Studies in Computational Intelligence 1051

Anis Koubaa Editor

Robot
Operating
System (ROS)
The Complete Reference (Volume 7)
Studies in Computational Intelligence

Volume 1051

Series Editor
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
ments and advances in the various areas of computational intelligence—quickly and
with a high quality. The intent is to cover the theory, applications, and design
methods of computational intelligence, as embedded in the fields of engineering,
computer science, physics and life sciences, as well as the methodologies behind
them. The series contains monographs, lecture notes and edited volumes in
computational intelligence spanning the areas of neural networks, connectionist
systems, genetic algorithms, evolutionary computation, artificial intelligence,
cellular automata, self-organizing systems, soft computing, fuzzy systems, and
hybrid intelligent systems. Of particular value to both the contributors and the
readership are the short publication timeframe and the world-wide distribution,
which enable both wide and rapid dissemination of research output.
Indexed by SCOPUS, DBLP, WTI Frankfurt eG, zbMATH, SCImago.
All books published in the series are submitted for consideration in Web of Science.
Anis Koubaa
Editor

Robot Operating System

(ROS)
The Complete Reference (Volume 7)
Editor
Anis Koubaa
College of Computer Science
and Information Systems
Prince Sultan University
Riyadh, Saudi Arabia

ISSN 1860-949X ISSN 1860-9503 (electronic)

Studies in Computational Intelligence
ISBN 978-3-031-09061-5 ISBN 978-3-031-09062-2 (eBook)
https://doi.org/10.1007/978-3-031-09062-2

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Switzerland AG 2023
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse
of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Contents

ROS2
Micro-ROS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Kaiwalya Belsare, Antonio Cuadros Rodriguez,
Pablo Garrido Sánchez, Juanjo Hierro, Tomasz Kołcon,
Ralph Lange, Ingo Lütkebohle, Alexandre Malki, Jaime Martin Losa,
Francisco Melendez, Maria Merlan Rodriguez, Arne Nordmann,
Jan Staschulat, and Julian von Mendel
A Tutorial to Use the MediaPipe Framework with ROS2 . . . . . . . . . . . . . . 57
Dieisson Martinelli, Jonathan Cerbaro,
Marco Antonio Simoes Teixeira, João Alberto Fabro,
and André Schneider de Oliveira

ROS Frameworks and Applications

Autonomous 3D Thermal Mapping of Disaster Environments
for Victims Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Christyan Cruz Ulloa, Guido Torres Llerena, Antonio Barrientos,
and Jaime del Cerro
Lab-Scale Smart Factory Implementation Using ROS . . . . . . . . . . . . . . . . . 119
Marwan Abdelatti and Manbir Sodhi

Cooperation, Sensing and Control

RosDrive: An Open-Source ROS-Based Vehicular Simulator
for STEM Control Systems Classes Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Enio Vasconcelos Filho, Jones Yudi, Mohamed Abdelkader,
Anis Koubaa, and Eduardo Tovar

v
vi Contents

Autonomous Laser-Induced Breakdown Spectroscopy System

for Chemical and Mineralogical Designation of Interplanetary
Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
George Stavrinos, Elias Chatzitheodoridis, and Olga Sykioti
High Fidelity IMU and Wheel Encoder Models for ROS Based
AGV Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
Emre Ozdemir, Hakan Gokoglu, M. Koray Yilmaz, Umut Dumandag,
Ismail Hakki Savci, and Haluk Bayram
ROS2
Micro-ROS

Kaiwalya Belsare, Antonio Cuadros Rodriguez, Pablo Garrido Sánchez,

Juanjo Hierro, Tomasz Kołcon, Ralph Lange, Ingo Lütkebohle,
Alexandre Malki, Jaime Martin Losa, Francisco Melendez,
Maria Merlan Rodriguez, Arne Nordmann, Jan Staschulat,
and Julian von Mendel

Abstract The micro-ROS stack (micro.ros.org) integrates microcontrollers seam-

lessly with standard ROS 2 and brings all major ROS concepts such as nodes,
publisher, subscriptions, parameters, and lifecycle onto deeply embedded systems.
This enables accessing all software using the same ROS tools and APIs, regard-
less of the underlying computing hardware and operating system. Micro-ROS
supports a broad spectrum of microcontroller families and the main open-source
real-time operating systems like FreeRTOS, Zepyhr, or NuttX. It supports vari-
ous microcontroller- or RTOS-specific build systems and provides ROS-CLI-based
build tooling. Micro-ROS is an open-source project that has been under development
at github.com/micro-ROS since 2018. It was initiated by the EU-funded innovation
activity OFERA. During the the last two years, micro-ROS has been adopted by a
relevant group of professional users inside the ROS community. In this chapter, we
give a technical introduction to the micro-ROS stack including APIs and architecture,

Because of the large number of authors, the names have been sorted alphabetically by surname.

K. Belsare · R. Lange (B) · I. Lütkebohle · A. Nordmann · J. Staschulat

Robert Bosch GmbH, Robert-Bosch-Campus 1, 71272 Renningen, Germany
e-mail: [email protected]
K. Belsare
e-mail: [email protected]
I. Lütkebohle
e-mail: [email protected]
A. Nordmann
e-mail: [email protected]
J. Staschulat
e-mail: [email protected]
A. C. Rodriguez · P. G. Sánchez · J. M. Losa · M. M. Rodriguez
eProsima, Plaza de la Encina 10-11, Nucleo 4, 2a Planta, 28760 Madrid, Tres Cantos, Spain
e-mail: [email protected]
P. G. Sánchez
e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 3

A. Koubaa (ed.), Robot Operating System (ROS),
Studies in Computational Intelligence 1051,
https://doi.org/10.1007/978-3-031-09062-2_2
4 K. Belsare et al.

as well as the corresponding middleware Micro XRCE-DDS. Furthermore, tutorials

for a simple application with an ESP32 microcontroller are provided together with
a report on three use-cases from industrial and research applications.

Keywords ROS 2 · Microcontroller · Micro-ROS · Embedded · Robotics

1 Introduction

The system architecture of a robot generally includes a network of one or more

stronger microprocessors and multiple microcontrollers. The microcontrollers are
used for accessing sensors and actuators, for low-latency control functions (e.g., force
or torque control at  1 kHz), for power saving (in particular in consumer applica-
tions), and for safety functions. Microcontrollers typically feature about 100 kB to
10 MB of RAM only and use highly resource-optimized real-time operating systems
(RTOS).
Due to their relevance in robotic applications, the integration of microcontrollers
has always been an important topic in ROS [1]. The most prominent integration
framework for classical ROS is rosserial [2]. It can be characterized as a lightweight
protocol for exchanging ROS messages between a Linux-based microprocessor and a
microcontroller, but offers only very few other ROS concepts on the microcontroller
side beyond communication. Rosserial supports Arduino-compatible boards, Mbed-
compatible platforms, Embedded Linux, and various other platforms.

J. M. Losa
e-mail: [email protected]
M. M. Rodriguez
e-mail: [email protected]
T. Kołcon · A. Malki
Łukasiewicz Research Network, Industrial Research Institute for Automation and Measurements
PIAP, Al. Jerozolimskie 202, 02-486 Warsaw, Poland
e-mail: [email protected]
A. Malki
e-mail: [email protected]
J. Hierro · F. Melendez
FIWARE Foundation e.V., Franklinstraße 13A, 10587 Berlin, Germany
e-mail: [email protected]
F. Melendez
e-mail: [email protected]
J. von Mendel
Bosch Rexroth AG, Robert-Bosch-Straße 2, 71701 Schwieberdingen, Germany
e-mail: [email protected]
Micro-ROS 5

Another solution for the integration of microcontrollers with classical ROS is

mROS. It provides a basic implementation of the core ROS concepts directly on
the embedded device [3]. This allows implementing multiple ROS nodes on the
same microcontroller. For message exchange between those nodes, mROS features
an efficient shared memory communication mechanism. For communication with
remote ROS nodes, mROS uses the lightweight TCP/IP stack lwIP [4]. The mROS
stack is based on the TOPPERS RTOS [5].

1.1 New Use-Case for ROS 2

For the design of ROS 2, six new use-cases were identified compared to classical
ROS [6]. The second use-case is the support of “small embedded platforms”, includ-
ing bare-metal microcontrollers, as “first-class participants in the ROS environment,
instead of being segregated from ROS by a device driver”. Our understanding of
this use-case implies that robotics software should be developed on microcontrollers
using the same ROS concepts as on strong microprocessors. This includes the struc-
turing by nodes, communication by topics, actions, and services, and many more
concepts. Similarly, the software on the microcontroller should be accessible with
the same developer tools for introspection, monitoring, runtime configuration, etc.
Device driver nodes that merely implement a ROS interface for the software
on a microcontroller and that communicate with the microcontroller via their own
protocol or communication technology are far from this idea.
The first undertaking to bring ROS 2 onto microcontrollers was the RIOT-ROS2
project [7, 8], in which the ROS 2 Client Support Library (rcl) and a tiny C client
library were ported to the RIOT operating system [9]. For communication, RIOT-
ROS2 used Named Data Networking (NDN) [10], unlike standard ROS 2, which
uses the Data Distribution Service (DDS) [11].
Recently, mROS 2 was published [12]. Analogously to mROS, it provides a basic
implementation of the core ROS 2 concepts directly on the embedded device. Embed-
dedRTPS [13, 14] is used for direct communication with the Data Distribution Ser-
vice (DDS) [11] used in ROS 2. In addition to the TOPPERS RTOS, mROS 2 also
supports Mbed OS [15].

1.2 Goals of Micro-ROS

The RIOT-ROS2 project and the preparation of a new standard DDS For Extremely
Resource Constrained Environments (DDS-XRCE) [16] led to the foundation of
the micro-ROS project, backed by the EU-funded research and innovation action
OFERA [17]. The micro-ROS project was launched in 2018 with the following three
goals:
6 K. Belsare et al.

ApplicaƟon
component
… ApplicaƟon
component
ApplicaƟon
component

library
Client
ROS API rcl + rclc: Convenience funcƟons, ROS 2

Build tools and integraƟons

executor, node graph, …
in C language stack
Benchmarking

ROS Middleware Interface (rmw)

Middle-
Micro XRCE-DDS Adapter

ware Micro XRCE-DDS Client

ROS 2
Ethernet, Agent
Bluetooth,
Serial
POSIX on a
RTOS

AddiƟonal drivers micro-ROS

normal
Zephyr / FreeRTOS /NuƩX arduino
OS

Microcontroller uP

Fig. 1 Overview of micro-ROS architecture and tooling. Reused ROS 2 packages and other third-
party open-source software are depicted in light blue. Dark blue colored components have been
developed in the project

1. Seamless integration of microcontrollers with ROS 2: The robotic software on the

microcontroller shall be accessible by standard ROS 2 interfaces and tools, which
comprises not only topics and services, but also concepts like parameters and the
new ROS 2 node lifecycle.
2. Ease portability of ROS 2 code to microcontrollers: The micro-ROS stack shall
provide all core ROS concepts with compatible APIs to minimize the efforts for
porting software from standard ROS 2 to micro-ROS. This specifically targets
the gap observed with classic ROS between robotics research and product devel-
opment: while research is typically conducted on Linux-based computers using
ROS, product development for small embedded hardware platforms does not use
ROS because these platforms are not adequately supported.
3. Ensure long term maintenance of micro-ROS stack: The micro-ROS stack and
project shall be setup and developed in a way that the maintenance is ensured
after the end of the EU-funded research and innovation action.
To achieve those goals, the founding partners of micro-ROS have designed a software
stack that utilizes the layered architecture of the standard ROS 2 stack and integrates
seamlessly with DDS. The micro-ROS stack reuses as many packages as possible
of the standard ROS 2 stack, which is a key difference to the other approaches
mentioned above. Relevant changes are contributed back to minimize the long-term
maintenance costs of micro-ROS. Figure 1 shows the high-level architecture of the
stack together with additional tooling.
The micro-ROS stack can be used with various open-source RTOS, including
FreeRTOS [18], Zephyr [19], and NuttX [20], but also comes with support for bare-
metal use-cases. On the middleware layer, it uses an open-source implementation
of the DDS-XRCE standard by eProsima, named Micro XRCE-DDS [21]. It comes
Micro-ROS 7

with an agent that bridges between standard DDS and DDS-XRCE. On the client
library layer, micro-ROS extends the rcl, which is implemented in the C programming
language, by the rclc packages to form a feature-complete client library in C.
A substantial portion of the development effort has gone into build tooling. Micro-
ROS provides tooling for the ROS 2 command line interface (CLI) as well as inte-
grations with various microcontroller- and RTOS-specific build systems. Finally,
micro-ROS is complemented by a generic benchmarking suite.
In addition to these technical measures, the partners successfully formed a global
subcommunity to distribute the work on more shoulders, particularly by initiating a
ROS Working Group named Embedded Systems WG, cf. [22].

1.3 Goals and Structure of This Chapter

The goals of this chapter are two-fold: First, to give a hands-on introduction and
technical overview to micro-ROS and the corresponding tooling. Second, to inspire
for practical use the micro-ROS stack by reporting on advanced use-cases from
industrial applications and research.
This chapter is structured along those two goals. In Sect. 2, requirements and
architecture are explained in detail as well as design decisions, followed by a tutorial
for a first micro-ROS application with an ESP32 microcontroller in Sect. 3. In Sect. 4
an overview to the DDS-XRCE standard and the implementation by Micro XRCE-
DDS is given. We also explain details on the use of Micro XRCE-DDS in micro-
ROS and on its customization. Next, in Sect. 5, we present the client library with
its implementation of common ROS concepts. Special focus is put on execution
management. In Sect. 6, we present the integration of micro-ROS with different
RTOS- or microcontroller-specific build tools as well as an own benchmarking tool.
Thereafter, in Sect. 7, we report on practical use-cases of micro-ROS from industrial
applications and research. The chapter is concluded in Sect. 8 with final remarks and
a brief outlook.

2 Architecture Overview and Design Rationales

In this section, we first discuss fundamental requirements to micro-ROS. Then, we

explain the micro-ROS stack layer by layer and compare it to the standard ROS 2
stack. Thereafter, we present major design decisions taken during the development.
Finally, we list supported RTOS and microcontrollers and explain the relation to
build tooling.
8 K. Belsare et al.

Robot Robot Robot Robot

MPU MPU MPU MCU
MCU
MCU MCU MCU MCU MCU
MCU
MCU
(1.) Fixed, wired (2.) Pluggable MCUs (3.) MCUs in the (4.) Networks of
MCUs environment MCUs without MPU

Fig. 2 Four typical computing and network system architectures used in robots

2.1 Requirements

In the first phase of the micro-ROS project, we analyzed a number use-cases and
derived fundamental requirements. A key observation was the importance of the
computing and network system architecture inside a robot. As indicated in the previ-
ous section, we distinguish between microprocessors with external RAM (typically
 0.5 GB), which are capable of running the standard ROS 2 stack on a desktop
operating system like Linux, and systems-on-a-chip microcontrollers with integrated
RAM, typically between 100 kB and 10 MB.
Throughout the rest of this chapter, we sometimes refer to a microprocessor as an
MPU and a microcontroller as an MCU.
From the use-cases, we deduced four typical system architecture scenarios
depicted in Fig. 2:

1. Fixed, wired MCUs: This is the most common scenario. The main software of
the robot runs on one (or more) MPUs connected to a number of MCUs that
interact with sensors and actuators, implement safety functions, or run low-latency
real-time-critical control loops. Advanced sensor and actuators often come with
integrated MCUs that have to be connected by UART, Ethernet, CAN, or similar
standards.
2. Pluggable MCUs: In this scenario, the system architecture can be extended by
pluggable devices. An example is an optional camera for a robot lawn mower that
enhances the perception capabilities. Such devices can be either connected by a
cable coupling or wirelessly.
3. MCUs in the environment: If sensors or actuators are deployed in the environment,
they are not explicitly plugged and configured, but the robot has to discover them
autonomously. An example is the wireless controller for a rolling shutter gate that
a self-driving vehicle in a warehouse can open if it comes close to the gate.
4. Networks of MCUs without MPU: The last scenario comprises all robot system
architectures without an MPU.
Micro-ROS 9

Table 1 Important requirements to the micro-ROS stack

Category Requirement
Communication 1. Support for 6LoWPAN or ZigBee, for Ethernet, and for serial protocols
2. Independent message serialization library
3. Dynamic discovery of participants without previous knowledge
4. Support for reliable communication
5. Efficient communication inside a microcontroller
Platform 1. Support for Olimex LTD STM32-E407 and an STM-L1 board
2. Support for the RTOS NuttX
3. Portability to other RTOS, in particular if they follow the POSIX standard
4. Fit on microcontrollers with 192 kB SRAM
Runtime 1. Provisioning of ROS 2 node lifecycle
configuration 2. Mechanisms for deterministic start-up across microcontrollers and microprocessors.
3. Support of dynamic node parameters
Data model 1. Usable with standard ROS message types for sensors, navigation, control, etc.
2. Support for custom message types
Diagnostics 1. Tailored diagnostics mechanisms
2. Compatibility with ROS diagnostics
Time 1. Quick boot
2. Time synchronization mechanism with <1 ms precision
Non-functional 1. Compliance with ROS 2
2. Intuitive use for experienced ROS developers
3. Extensibility by modules/packages
4. Simple upgrade to new versions
5. Easy transfer of a standard ROS 2 node to micro-ROS and vice-versa

With the price of microprocessors dropping and algorithms becoming more sophis-
ticated, we are expecting that the last scenario declines increasingly in importance.
Therefore, it was ruled out for the time being.
In addition to providing support for the Scenarios 1 to 3, we derived a number
of further requirements, which are explained in detail in a technical report [23].
The most important ones are given in Table 1. The current micro-ROS stack largely
meets these requirements. In terms of supported microcontroller families, RTOS, and
communication protocols, micro-ROS goes far beyond the original requirements as
explained in the next subsections.

2.2 Layer-by-Layer Walkthrough

As mentioned in the previous section, micro-ROS utilizes and follows the layered
architecture of ROS 2. This duality is depicted in Fig. 3.
10 K. Belsare et al.

ApplicaƟon
ApplicaƟon
component
… ApplicaƟon
component
ApplicaƟon
component
… ApplicaƟon
component

rclcpp rclpy
Client Library C API rcl + rclc: Convenience funcƟons,
executor, node graph, … rcl – Client Support Library

rmw – middleware interface rmw – middleware interface

Adapter Adapter

DDS-XRCE
Middleware micro-ROS

DDS
DDS ImplementaƟon
Micro XRCE-DDS Client UART, Agent
IP, Bluetooth, … (FastDDS, CycloneDDS, …)

POSIX
OperaƟng AddiƟonal drivers micro-ROS
Linux, macOS, Windows
System FreeRTOS, Zephyr, NuƩX arduino

Hardware microcontroller microprocessor

Fig. 3 Side-by-side comparison of micro-ROS stack (left) with standard ROS 2 (right)

2.2.1 The rcl and rmw Layers

The ROS 2 stack has been designed for high modularity and flexibility. The most
important layers in terms of the software architecture are the ROS Middleware Inter-
face (rmw) and the ROS Client Support Library (rcl).
The rmw defines a generic and slim interface to abstract from the underlying mid-
dleware being used for communication. It assumes three common concepts only: (1)
naming of endpoints, (2) publish-subscribe communication, and (3) client-service
(request-response) communication. These concepts are provided not only by imple-
mentations of the DDS standard but also by many other middlewares. To use a specific
middleware, the interface defined by rmw has to be implemented in an adapter.
The rcl implements the most fundamental ROS concepts like node, publisher,
subscription, client, service, and clock in the C programming language so that the
high-level language client libraries (for C++, Python, etc.) can rely on the same imple-
mentation by binding rcl via native or foreign-function interfaces. This is a crucial
difference to the architecture of classical ROS, where a custom middleware imple-
mentation is being used and the fundamental ROS concepts have been implemented
independently for every programming language.
Both layers, rmw and rcl, are used as they are in the micro-ROS stack, with the
exception that few features from rcl that are not available on a microcontroller are
disabled (e.g., command line arguments). For this purpose, the micro-ROS project
maintains a fork of the rcl repository1 in which those features are disabled by one
changeset applied after new releases of ROS 2. In addition, micro-ROS provides the
rclc packages,2 which extend rcl to make the combination rcl+rclc a feature-complete
client library for the C programming language.

1 https://github.com/micro-ROS/rcl.
2 https://github.com/ros2/rclc.
Micro-ROS 11

2.2.2 Micro XRCE-DDS Middleware

The micro-ROS stack uses an implementation of the new DDS-XRCE standard

provided by eProsima as open-source software under the name Micro XRCE-DDS
[21]. The corresponding rmw adapter can be found in the micro-ROS organisation
on GitHub.3 Micro XRCE-DDS is made up of two parts: The Client is a highly
resource-optimized library in the C programming language for MCUs. The Agent is
a standalone executable implemented in C++ for use on a microprocessor running
Linux or Windows. The Agent bridges between DDS-XRCE and normal DDS. In
particular, it performs the computationally intensive discovery tasks and quality-of-
service mechanisms for the microcontroller. Serialization happens directly on the
microcontroller in the Common Data Representation (CDR) format, which is also
used by normal DDS. Hence, the Micro XRCE-DDS Agent can simply relay the
binary messages in both directions.
The client-agent model of the XRCE-DDS standard fits very well to the typical
hardware setup in robots as explained above in the scenarios 1 to 3 of computing and
network system architectures.
Micro XRCE-DDS supports several transports for the connection between the
microcontroller and the main microprocessor running the Agent. These transports
include UDPv4, UDPv6, TCPv4, TCPv6, and serial. In addition, Micro XRCE-DDS
comes with APIs for custom transports as explained in Sect. 4.
For efficient communication between micro-ROS nodes on the microcontroller,
the middleware provides an optional Shared Memory Transport (SHM).

2.2.3 Real-Time Operating System

The ROS 2 stack has been developed for operating systems according to the Portable
Operating System Interface (POSIX) standards [24]. In fact, ROS 2 only depends
on a small subset of POSIX for process identification, clock, time, and filesystem
operations. The use of POSIX is hidden by a thin abstraction layer implemented
by the rcutils package and repository. This package also implements support for
Windows and streamlines tiny differences between different Unix-like operating
systems. Furthermore, rcutils implements an abstraction for atomic operations.
Micro-ROS adopts the assumption of POSIX but makes smaller changes to rcutils,
using the same changeset mechanism as explained above for rcl: filesystem operations
are disabled, dynamic memory allocations (in the context of error reporting) are
prevented, and a user-level implementation of 64-bit atomic operations is provided.

3 https://github.com/micro-ROS/rmw_microxrcedds.
12 K. Belsare et al.

2.2.4 License

All middleware- and client-library-related packages created in the micro-ROS project

are provided under the Apache 2.0 license, like the standard ROS 2 stack. Also, Micro
XRCE-DDS is provided under Apache 2.0.
Things get interesting at the RTOS level: The micro-ROS build tool and the
modules for external build systems are provided under Apache 2.0 but use (or are
combined with) very differently licensed RTOS and board support components. The
fact that typical embedded toolchains build the whole software (RTOS, micro-ROS,
and application) into one binary image makes the situation more complex compared
to desktop operating systems with clear separation between executables and kernel.
More information is provided in the overview section of the micro-ROS website.4

2.3 Major Feature Differences to ROS 2

Although micro-ROS aims at implementing all core features of ROS 2 for microcon-
trollers, there are a several important differences to take into account when creating
own applications:

– Fixed-size message only: To avoid dynamic memory allocations, micro-ROS sup-

ports fixed-size message types only, i.e., the definition of a message type has to
specify a size or upper bound for all arrays/vectors. If necessary, messages have
to be converted on the agent side to dynamically-sized types.
– Flexible, callback-level executor: To cope with the very different scheduling APIs
and mechanisms of the underlying RTOS, the rclc layer provides a flexible Execu-
tor on the level of individual callbacks.
– No launch system: Again, due to the very different underlying RTOS mechanisms,
micro-ROS does not provide a launch system like standard ROS 2, but leaves the
start of the micro-ROS nodes up to the developer.
– Build-time node composition only: The composition mechanism of the ROS 2 C++
Client Library allows deciding at launch time about the distribution of nodes to
processes. It even allows for dynamic loading of nodes into running processes.
Such mechanisms are not provided by micro-ROS. If multiple micro-ROS nodes
should be executed in the same process, the composition has to be performed at
compile-time by creating the nodes in the same main function of an executable.5
– No DDS-XRCE security: While there is a dedicated security standard for normal
DDS [25], the DDS-XRCE standard does not define explicit security mechanisms
but relies on such mechanisms of the underlying transport. If the underlying trans-
port does not provide such mechanisms (e.g., plain UART) the connection is not

4 https://micro.ros.org/docs/overview/license.
5 Note that the concepts of executables and processes are not available in every RTOS.
Micro-ROS 13

encrypted, which is typically acceptable for wired connections inside a robot. Of

course, the agent participates in the DDS and ROS 2 security mechanisms.
A continuously updated feature comparison between micro-ROS and ROS 2 can be
found in the overview section of the micro-ROS website.6

2.4 Design Rationales

During the development of micro-ROS, we have made several important design

decisions—starting with the use of DDS-XRCE, which has already been discussed
above. In the following, we present and give reasons for four other major such
decisions.

2.4.1 Reuse of Existing ROS 2 Layers

In the first months, we discussed two development strategies: First, developing a

dedicated stack from scratch, similar to the rosserial client library of classical ROS.
Second, developing micro-ROS based on existing ROS 2 packages and layers.
An important advantage of the first strategy would have been that the implemen-
tation could be optimized for microcontrollers from the very beginning, for example
in terms of memory usage and execution management. However, due to our goal
to bring all core ROS concepts on the microcontroller, we decided for the second
strategy. The reuse of rmw and rcl ensures maximum compatibility and minimizes
long-term maintenance costs.

2.4.2 Focus on POSIX-Based RTOS

Conceptual differences between RTOSes are much higher than between the domi-
nating three desktop operating systems Windows, macOS, and Linux. Given the fact
that ROS 2 was developed for POSIX-based OS and that there exist several open-
source POSIX-based RTOS, we decided to focus on them first. In the other side,
we avoided optimizing the micro-ROS stack for a specific RTOS. In the meanwhile,
micro-ROS also supports bare-metal use. An example of a port for a non-POSIX
RTOS is presented in Sect. 7.2.

6 https://micro.ros.org/docs/overview/ROS_2_feature_comparison.
14 K. Belsare et al.

2.4.3 Client Library by rcl+rclc

For the client library, we have discussed three options: An new implementation in C,
a new implementation in C++ not requiring a full-fledged libstdc++, and a rework of
the existing rclcpp package for use on microcontrollers. The third option was ruled
out early, as rclcpp makes intensive use of dynamic memory allocations. In early
2019, we finally decided for the first option based on rcl and against the second
option, to maximize compatibility with MCU-specific build tool chains. A detailed
design decision paper7 from 2019 gives more details.
As of 2021, the second option has been reconsidered in the Embedded Systems
WG as an additional client library for micro-ROS.

2.4.4 Tooling for ROS and Deeply Embedded Developers

During the requirements analysis, we identified two very different user groups for
micro-ROS: First, ROS developers, who are used to Linux and the ROS concepts but
have little experience with MCU or RTOS specific build tools, and, second, deeply
embedded developers, who are used to specific MCU families and RTOS but likely
new to ROS. To meet the needs of both groups, we decided to provide components
or modules for different microcontroller and RTOS build tools as well as a generic
build system named micro_ros_setup as a ROS 2 package and based on the ROS 2
CLI.

2.5 Supported RTOS and Microcontrollers

Micro-ROS comes with support for the three RTOS FreeRTOS [18], Zephyr [19],
and NuttX [20] and a number of microcontrollers boards including Espressif ESP32,
Renesas EK RA6M5, a couple of Arduino boards, Raspberry Pi Pico, and several
boards by STMicroelectronics or with STM32 microcontrollers. There is also exper-
imental support for bare metal use, the Arduino IDE, Mbed OS, and Azure RTOS.
The term “support” implies two aspects: First, microcontroller-specific exten-
sions such as hardware drivers for the RTOS or micro-ROS stack. Second, pro-
visioning of RTOS- and/or MCU-specific build tooling. At the same time both
aspects are closely intertwined. Therefore, some of the extensions can be found
in the generic [NameOfRTOS]_apps repositories of the micro-ROS organization
at GitHub and some in microcontroller-specific repositories such as
micro_ros_espidf_component.
Section 6.1 gives details on the generic micro_ros_setup package and the pro-
vided integrations in RTOS- and microcontroller-specific build toolchains in form of
components/modules.

7 https://micro.ros.org/download/client_library_decision_paper_2019.pdf.
 Micro-ROS 15

Up-to-date lists of supported RTOS and microcontrollers can be found in the

overview section of the micro-ROS website.8,9

3 First Application Tutorial

In this section, we will guide you through all steps to build your own first micro-ROS
application, which will implement a simple message ping-pong between a micro-
ROS node and standard command line tools (CLI) under ROS 2 on a Linux host.
All the build commands will be exemplified for the Espressif ESP32 microcontroller
family, the Foxy distribution of ROS 2 and micro-ROS, and the real-time operating
system FreeRTOS [18]. At very important commands, we will hint to other microcon-
trollers, distributions, and RTOSes. Complete tutorials with other microcontrollers
and RTOSes can be found in the the tutorials section of the micro-ROS website.10
This section is organized in two parts: In the first subsection, we demonstrate the
programming and build steps with the tooling from the micro_ros_setup package,
which is based on the ROS 2 CLI tools. In the second subsection, we explain how
to build the same application with micro-ROS component for the Espressif IoT
Development Framework (ESP-IDF). More about build and development tools can
be found in Sect. 6.1.

3.1 Ping-Pong Application with micro_ros_setup

First of all, install ROS 2 Foxy Fitzroy on your Ubuntu 20.04 LTS computer. Do so
from binaries, via Debian packages. Alternatively, you can use a Docker container
with a fresh ROS 2 Foxy installation. The one that serves the purpose is the container
run by the command:
1 docker run -it --net=host -v /dev:/dev --privileged ros:foxy
Once you have a ROS 2 installation in the computer, follow these steps to download
and build the micro_ros_setup package:
1 source /opt/ros/$ROS_DISTRO/setup.bash
2 mkdir microros_ws
3 cd microros_ws
4 git clone -b $ROS_DISTRO
→ https://github.com/micro-ROS/micro_ros_setup.git
→ src/micro_ros_setup
5 sudo apt update && rosdep update
6 rosdep install --from-path src --ignore-src -y

8 https://micro.ros.org/docs/overview/rtos.
9 https://micro.ros.org/docs/overview/hardware.
10 https://micro.ros.org/docs/tutorials.
 16 K. Belsare et al.

7 sudo apt-get install python3-pip

8 colcon build
9 source install/local_setup.bash

These instructions will setup a workspace with a ready-to-use micro-ROS build

system. The build system is in charge of downloading the required cross-compilation
tools and building the apps for the required platforms. The build system’s workflow
is a four-step procedure:
1. Create step: This step is in charge of downloading all the required code reposito-
ries and cross-compilation toolchains for the specific hardware platform. Among
these repositories, it will also download a collection of ready to use micro-ROS
apps.
2. Configure step: In this step, the user can select which app is going to be cross-
compiled by the toolchain. Some other options, such as transport, agent’s IP
address/port (for UDP transport) or device ID (for serial connections) will be also
selected in this step.
3. Build step: Here is where the cross-compilation takes place and the platform-
specific binaries are generated.
4. Flash step: The binaries generated in the previous step are flashed onto the hard-
ware platform memory, in order to allow the execution of the micro-ROS app.

3.1.1 Creating a New Firmware Workspace

Once the build system is installed, create a firmware workspace that targets all the
required code and tools for the ESP32 by:
1 ros2 run micro_ros_setup create_firmware_ws.sh freertos esp32

Once the command is executed, a folder named firmware must be present in your
workspace.
This step is in charge, among other things, of downloading a set of micro-ROS
apps for the specific platform you are addressing. In the case of FreeRTOS, these
are located at firmware/freertos_apps/apps. Each app is represented by a folder
containing the following files:
– app.c: This C source file contains the logic of the application.
– app-colcon.meta: This file contains the micro-ROS app specific colcon config-
uration. Detailed info on how to configure the rmw via this file can be found in a
dedicated tutorial in the micro-ROS website.11
For creating a custom application, a folder <my_app> needs to be registered in this
location, containing the two files just described.

11 https://micro.ros.org/docs/tutorials/advanced/microxrcedds_rmw_configuration.
 Micro-ROS 17

3.1.2 Configuring the Firmware

The configuration step will set up the main micro-ROS options and select the desired
application. It can be executed with the following command:
1 ros2 run micro_ros_setup configure_firmware.sh [APP] [OPTIONS]
The options available for this configuration step are:
1 . --transport or -t: udp, serial or any hardware-specific transport label
2 . --dev or -d: agent string descriptor in a serial-like transport
3 . --ip or -i: agent IP in a network-like transport
4 . --port or -p: agent port in a network-like transport
In this tutorial, we will use a serial transport (labeled as serial) and focus on the
out-of-the-box ping_pong application located at firmware/freertos_apps/
apps/ping_pong. To execute this application with the chosen transport, run the
configuration command above by specifying the [APP] and [OPTIONS] parameters
as below:
1 ros2 run micro_ros_setup configure_firmware.sh ping_pong -t udp
→ -i [LOCAL MACHINE ADDRESS] -p 8888
Then use the following command to set your wifi credentials for ESP32 to connect
to your network:
1 ros2 run micro_ros_setup build_firmware.sh menuconfig
In the menuconfig window go to micro-ROS Transport Settings and then WiFi Con-
figuration menu and fill your WiFi SSID and password. Save your changes, exit the
window.
You can check the complete content of the ping_pong app in apps/
ping_pong/app.c in the freertos_apps repository.12
This example showcases a micro-ROS node with two publisher-subscriber pairs
associated with a topic named ping and a topic named pong, respectively. The node
sends a message on ping with a unique identifier every two seconds. The ping
subscriber receives all those messages, but sends a response on the pong topic only
if the ping message comes from an external node.
To show that this micro-ROS node is functioning correctly, we implement com-
munication with ROS 2 CLI tools that listen to the topics ping and pong published by
the micro-ROS node and that publish a fake_ping message on ping that is received
by the micro-ROS node. As a consequence, the pong publisher on the micro-ROS
node will respond with a pong message to signal that it received the fake_ping
message correctly.
The diagram shown in Fig. 4 clarifies the communication flow within the micro-
ROS node and between this node and the ROS 2 CLI commands given below.
The contents of the application files can be found in main.c and
CMakeLists.txt in the micro-ROS-demos repository.13

12 https://github.com/micro-ROS/freertos_apps/tree/foxy/apps/ping_pong.
13 https://github.com/micro-ROS/micro-ROS-demos/blob/foxy/rclc/ping_pong.
 18 K. Belsare et al.

ROS 2 CLI tooling micro-ROS node

ros2 topic pub --once /microROS/ping Publisher Publisher (0.2 Hz)

[...] '{frame_id: "fake_ping"}' microROS/ping microROS/ping

Subscription Subscription
ros2 topic echo /microROS/ping
microROS/ping microROS/ping

Trigger pong only if ping was

received from external node

Subscription Publisher Subscription

ros2 topic echo /microROS/pong
microROS/pong microROS/pong microROS/pong

Fig. 4 Ping pong communication flow between micro-ROS application node and ROS 2 CLI tools

3.1.3 Building and Flashing the Application

When the configuring step ends, build the firmware by

1 ros2 run micro_ros_setup build_firmware.sh

Then, connect your ESP32 to the computer with a micro-USB cable and run:
1 ros2 run micro_ros_setup flash_firmware.sh

3.1.4 Creating the Agent

The micro-ROS app is now ready to be connected to a micro-ROS agent to start

talking with the rest of the ROS 2 world. To do that, let’s first of all create a micro-
ROS agent:
1 ros2 run micro_ros_setup create_agent_ws.sh

Now, let’s build the agent packages and, when this is done, source the installation by
1 ros2 run micro_ros_setup build_agent.sh
2 source install/local_setup.bash

A docker image is also available.

3.1.5 Running the Agent

At this point, you have both the client and the agent correctly installed in the ESP32
and in your host machine, respectively. To give micro-ROS access to the ROS 2
dataspace run the agent by
1 ros2 run micro_ros_agent micro_ros_agent udp4 --port 8888

or using docker by
 Micro-ROS 19

1 docker run -it --rm --net=host microros/micro-ros-agent udp4

→ --port 8888

3.1.6 Testing the Application

We now want to check that everything is working. Open a new command line. We
are going to listen to the ping topic with ROS 2 to check whether the micro-ROS
node is correctly publishing the expected pings:
1 source /opt/ros/$ROS_DISTRO/setup.bash
2 ros2 topic echo /microROS/ping

You should see the topic messages published by the Ping Pong node every five
seconds:
1 user@user:~$ ros2 topic echo /microROS/ping
2 stamp:
3 sec: 20
4 nanosec: 867000000
5 frame_id: ’1344887256_1085377743’
6 ---
7 stamp:
8 sec: 25
9 nanosec: 942000000
10 frame_id: ’730417256_1085377743’
11 ---

At this point, we know that our app is publishing pings. Let’s check if it also answers
to someone else’s pings. If this works, it’ll publish a pong.
First, we subscribe with ROS 2 to the pong topic from a new shell (notice that
initially we don’t expect to receive any pong since none has been sent yet):
1 source /opt/ros/$ROS_DISTRO/setup.bash
2 ros2 topic echo /microROS/pong

And now, let’s publish a fake_ping message on the ping topic with from yet another
command line:
1 source /opt/ros/$ROS_DISTRO/setup.bash
2 ros2 topic pub --once /microROS/ping std_msgs/msg/Header
→ ’{frame_id: "fake_ping"}’

Now, we should see this fake_ping in the ping subscriber console, along with the
micro-ROS pings:
1 user@user:~$ ros2 topic echo /microROS/ping
2 stamp:
3 sec: 0
4 nanosec: 0
5 frame_id: fake_ping
6 ---
7 stamp:
 20 K. Belsare et al.

8 sec: 305
9 nanosec: 973000000
10 frame_id: ’451230256_1085377743’

Also, we expect that, because of having received the fake_ping, the micro-ROS
node on the ESP32 will answer with a pong:
1 user@user:~$ ros2 topic echo /microROS/pong
2 stamp:
3 sec: 0
4 nanosec: 0
5 frame_id: fake_ping
6 ---

3.2 Ping-Pong Application with Component for ESP-IDF

In this section, we explain how to build the above ping-pong with the micro-ROS
Component for ESP-IDF. This has been tested in ESP-IDF v4.1, v4.2, and v4.3 with
ESP32, ESP32-S2, and ESP32-C3.

3.2.1 Dependencies

This component needs colcon and other Python 3 packages inside the IDF virtual
environment in order to build micro-ROS packages:
1 $IDF_PATH/install.sh
2 . $IDF_PATH/export.sh
3 pip3 install catkin_pkg lark-parser empy colcon-common-extensions

3.2.2 Usage

Clone the micro_ros_espidf_component repository directly in the components folder

of your project:
1 cd components
2 git clone
→ https://github.com/micro-ROS/micro_ros_espidf_component.git

If you encounter issues during the build process, ensure that you are running in a
clean shell environment without the ROS 2 setup script sourced.

3.2.3 Example

In order to test a ping_pong example:

 Micro-ROS 21

1 . $IDF_PATH/export.sh
2 cd examples/ping_pong
3 # Set target board [esp32|esp32s2|esp32c3]
4 idf.py set-target esp32
5 idf.py menuconfig
6 # Set configuration and WiFi credentials under micro-ROS settings
7 idf.py build
8 idf.py flash
9 idf.py monitor

To clean and rebuild all the micro-ROS library:

1 make -f libmicroros.mk clean

It is possible to use a micro-ROS Agent just with this Docker command:

1 docker run -it --rm --net=host microros/micro-ros-agent:galactic
→ udp4 --port 8888 -v6

The other steps to test the application are the same is in Sect. 3.1.

3.3 Build with Docker Container

It is also possible to build the ping-pong application using a preconfigured Docker

container. Docker is a good entry point in the case the tools are not available on the
development machine by default. Execute this line to build an example app using
Docker container:
1 docker run -it --rm --user espidf
→ --volume="/etc/timezone:/etc/timezone:ro" -v
→ $(pwd):/micro_ros_espidf_component -v /dev:/dev
→ --privileged --workdir /micro_ros_espidf_component
→ microros/esp-idf-microros:latest /bin/bash -c "cd
→ examples/ping_pong;idf.py menuconfig build flash monitor"

The Dockerfile for this container is provided in the ./docker directory and available
in Dockerhub.

3.3.1 Using Serial Transport

By default, micro-ROS component uses UDP transport, but is possible to enable

UART transport or any other custom transport by setting the colcon.meta like:
1 ...
2 "rmw_microxrcedds": {
3 "cmake-args": [
4 ...
5 "-DRMW_UXRCE_TRANSPORT=custom",
6 ...
7 ]
 22 K. Belsare et al.

8 },
9 ...

An example on how to implement such a transport is provided by the example of

a custom serial transport in examples/ping_pong_custom_transport. Available
ports are 0, 1, and 2 corresponding to UART_NUM_0, UART_NUM_1, and UART_NUM_2.
Again, it is possible to start the micro-ROS agent just with this Docker command:
1 docker run -it --rm -v /dev:/dev --privileged --net=host
→ microros/micro-ros-agent:galactic serial --dev [YOUR
→ BOARD PORT] -v6

4 Middleware Micro XRCE-DDS

In this section, we explain the DDS-XRCE standard, the implementation by Micro

XRCE-DDS, and the integration in the micro-ROS stack. We also report memory
profiling of Micro XRCE-DDS in connection with micro-ROS.

4.1 The DDS-XRCE Standard

The DDS standards are created and managed by the Object Management Group
(OMG), where many distributed computing standards have been created by industry
consortium members since 1989, being DDS-XRCE [16] one of them.
XRCE stands for eXtremely Resource Constrained Environments and refers to
generally small devices with limited memory capacities. The DDS-XRCE protocol
offers a way to communicate these constrained devices with full featured DDS sys-
tems. The DDS-XRCE protocol has been developed for devices and networks with
the following characteristics:
– Operability over low-bandwidth networks (40–100 Kbps)
– Devices with sleep/wake cycles
– Support for specialized languages or frameworks
– Plug-and-play characteristics due to dynamic discovery
– Publish or subscribe to any Interface Definition Language (IDL) data types
– Support for data updates to multiple on the same or multiple DDS Topics
– Support for both reliably and best-effort communication streams
– Support for secure communication at the transport level
– Provides full read/write access
– Full implementation with less than 100 kB of code
The DDS-XRCE protocol allows the DDS-XRCE devices, called clients, to commu-
nicate with a server, called agent. This Micro XRCE-DDS Agent allows the devices
to publish or subscribe to topics in the DDS domain, called Global Data Space. In
Micro-ROS 23

Fig. 5 Overview to architecture of DDS-XRCE standard

general, the DDS-XRCE standard describes a wire protocol that enables the XRCE
clients to create and interact with DDS entities in the Micro XRCE-DDS Agent.
Those DDS entities in the agent will act on behalf of the clients (Fig. 5).
As the DDS-XRCE protocol is applied on low-powered and resource-limited
devices, the goal is not to embed the entire functionality of DDS into such a restricted
environment but rather to allow them to act as participants in a larger DDS ecosystem.

4.2 Micro XRCE-DDS Architecture and Implementation

This section describes the most relevant features of Micro XRCE-DDS,14,15 the
default middleware implementation for micro-ROS. Micro XRCE-DDS is the DDS-
XRCE implementation of eProsima. It is fully open source and comes with an
Apache 2.0 license.
As explained in Sect. 4.1, the XRCE standard follows a client-server architecture,
which are the two main elements of Micro XRCE-DDS:
– Micro XRCE-DDS Agent: a C++11 out-of-the-box application which implements
the agent functionality16
– Micro XRCE-DDS Client: a C99 library which implements the Micro XRCE-DDS
Client side functionality17
In addition, the Micro XRCE-DDS suite offers two more components:
– Micro CDR: a CDR (Common Data Representation) compliant de-/serialization
engine used in the Micro XRCE-DDS suite.18

14 https://www.eprosima.com/index.php/products-all/eprosima-micro-xrce-dds.
15 https://micro-xrce-dds.docs.eprosima.com/en/latest.
16 https://github.com/eProsima/Micro-XRCE-DDS-Agent.
17 https://github.com/eProsima/Micro-XRCE-DDS-Client.
18 https://github.com/eProsima/Micro-CDR.
24 K. Belsare et al.

– Micro XRCE-DDS Gen: a code generator tool used for generating Micro CDR
de/serialization functions from IDL sources.19

4.2.1 Low Resource Consumption

Micro XRCE-DDS is focused on microcontroller applications. Therefore, the design

and implementation of this middleware have been carried out taking into account the
memory constraints of these kind of devices.
Given the limited memory resources offered by the systems targeted by the Micro
XRCE-DDS Client library, the memory size manipulation is highly relevant and
takes place at two different levels:

– At configuration time: To fix the size of the executable code size, the library can
be compiled enabling or disabling several profiles. As described later, the client
library follows a profile concept that enables to choose, add, or remove some
features at configuration time, allowing the user to customize its size.
– At runtime: The client library is both dynamic and static memory free, imply-
ing that all memory footprint depends only on how the stack grows during the
execution. The stack size is determined by the library functions, along with
user configured parameters as the Maximum Transmission Unit (MTU) and the
communication streams used:
• The MTU is transport-dependent and can be configured by the user. The
selected value represents the maximum message size that can be sent or
received without fragmenting the message. The transport uses the MTU
value to create an internal buffer, which is the memory block where the
messages will be written to and stored in when interchanged.
• As for the streams, the user can define a maximum of 127 best-effort streams
and 128 reliable streams, but for the majority of purposes only one stream
in either best-effort or reliable mode is needed. Moreover, reliable streams
have a message history associated, whose size can be tailored to fit the
specific requirements of the application. In this case, the size of the stream
equals to the size of the configured MTU times the associated history. In the
best-effort case, no history is stored and the memory reserved for the stream
equals the size of the MTU.
Detailed information about the memory consumption as a function of message size,
entity number, and internal memory management of the middleware library is pro-
vided in Sect. 4.4.

19 https://github.com/eProsima/Micro-XRCE-DDS-Gen.
Micro-ROS 25

4.2.2 Transport Agnosticity

The user can choose between several transport layers for the communication between
the clients and the agent. In contrast to other IoT middlewares which support only
one particular transport layer, XRCE supports multiple transport protocols natively.
The latest version of Micro XRCE-DDS supports: UDPv4, UDPv6, TCPv4, TCPv6,
and serial.
Apart from this, eProsima’s Micro XRCE-DDS provides a custom transport API
that allows the users to implement their own transport protocol in both the client
and agent libraries. Thanks to this, the Micro XRCE-DDS wire protocol can be used
over virtually any protocol, network, or communication mechanism. In order to do
so, two general communication modes are provided:
– Stream-oriented mode: the communication mechanism implemented does not have
the concept of a packet. Instead, High-Level Data Link Control (HDLC) framing20
will be used.
– Packet-oriented mode: the communication mechanism implemented is able to send
a whole packet that includes an XRCE-DDS message.
These two modes can be selected by activating and deactivating the framing param-
eter in both the micro-ROS client and the micro-ROS agent transport configuration.

4.2.3 Efficient Communication Inside a Microcontroller

Additionally, the Micro XRCE-DDS Client provides a Shared Memory Transport

(SHM) for efficient communication between micro-ROS nodes within the same
microcontroller memory space without the need of interacting with the micro-ROS
agent. SHM provides clients with self-introspection capabilities to detect and com-
municate with compatible publishers, subscribers or any other DDS entities with-
out the need to recur to the agent. The agent acknowledges the existence of those
compatible entities within a client and avoids sending redundant information to the
client. SHM can be activated by the flag UCLIENT_PROFILE_SHARED_MEMORY in the
colcon.meta file of a micro-ROS application project.

4.2.4 Multi-Platform Support

Micro XRCE-DDS is portable to all low-mid range microcontrollers, to any RTOS,

and bare metal. The only system requirements are timing functions that are usually
provided by the RTOS or can be implemented easily in bare-metal systems. Up to
date, the Micro XRCE-DDS Client supports FreeRTOS, Zephyr, NuttX, and Mbed
OS as embedded RTOS. Moreover, it also runs on Windows and Linux. On the other
hand, the Micro XRCE-DDS Agent supports Windows and Linux.

20 See https://micro-xrce-dds.docs.eprosima.com/en/latest/transport.html for details.

26 K. Belsare et al.

4.2.5 QoS Support

The Micro XRCE-DDS Client library allows the user to use different approaches for
creating DDS entities in the Micro XRCE-DDS Agent. This entity creation allows
to define at the client side the required Quality of Services parameters of the DDS
entities. The Micro XRCE-DDS suite allows to define the DDS entities using three
modes: XML, references and binary encoded format.

4.2.6 Best-Effort Versus Reliable Operation

Micro XRCE-DDS messages flow between client and agent through streams. A
stream represents an independently ordered flow of information, like a messaging
queue. There are two kinds of streams: best-effort and reliable. Both streams have a
raw buffer (uint8_t array) associated with them, but the layout is different. Best-
effort streams handle a single message at a time, while reliable streams handle a
sequence of messages using a message history with reception acknowledgment.

4.3 Micro XRCE-DDS in the Micro-ROS Stack

On the microcontroller, the micro-ROS stack uses the Micro XRCE-DDS Client
by its default rmw adapter layer rmw_microxrcedds. In this layer, micro-ROS
manages the memory for ROS 2 entities with a static pool approach. In general,
rmw_microxrcedds has a defined struct for handling ROS 2 memory and one extra
data structure for handling topics and services buffers.
When rmw_microxrcedds is initialized, it uses one static memory slot for han-
dling a Micro XRCE-DDS session. For example, this session structure has handles
for four XRCE-DDS streams: input reliable, input best-effort, output reliable, and
output best-effort. The best-effort buffers will be MTU long and the reliable ones will
have RMW_UXRCE_STREAM_HISTORY slots of MTU size each.
Similarly, the node, publisher, subscriber, and service entities will have their asso-
ciated static memory items with all the required members for maintaining their states.
The size of the static memory pools associated to the ROS 2 entities can be configured
at build time with CMake variables, e.g., RMW_UXRCE_MAX_NODES for nodes.
There is one special static memory pool in rmw_microxrcedds named Topic
Handling Pool. This static memory pool is used for storing the data received by
the middleware when a “read data” operation is performed and before the “take
data from middleware” operation retrieves the data. Its size is determined by
RMW_UXRCE_MAX_HISTORY and each slot has MTU × RMW_UXRCE_STREAM_HISTORY
because it is the biggest message size that the middleware can receive. The Topic
Handling Pool is shared by all entities that receive data in a micro-ROS client.
Also, there are some build-time parameters related to client-to-agent connec-
tion (e.g., CONFIG_RMW_DEFAULT_UDP_PORT, CONFIG_RMW_DEFAULT_UDP_IP, and
Micro-ROS 27

CONFIG_RMW_DEFAULT_SERIAL_DEVICE) that can be configured either at build-time

or at runtime. This means that they can be defined in the client-host-colcon.meta
configuration file. Furthermore, some rmw and middleware parameters can be mod-
ified at run-time via a user configuration API.
Notice, that it is also possible to set the Micro XRCE-DDS client_key, which
identifies a XRCE-DDS client and is set randomly by default. This feature is useful
for reusing DDS entities already created on the agent side. Further information can
be found in the rmw-microxrcedds repository.21

4.4 Memory Profiling of Micro XRCE-DDS in Micro-ROS

Micro-ROS and its middleware target low- and mid-range devices that do not have
enough memory for running a full ROS 2 stack. Therefore, it is important to evaluate
the memory usage of the different entities in the micro-ROS stack. A full memory
profiling of the micro-ROS stack in terms of static, dynamic, and code memory
is presented in the concepts section of the micro-ROS website.22 This webpage is
updated with every major improvement on the micro-ROS memory management.
At the moment of writing this chapter, the memory usage of each entity in the
micro-ROS stack is shown in Fig. 6. For example, a micro-ROS application that has
only one publisher will use about 20 kB of memory. As the application increases
the number of used entities, the memory usage of the application increases slightly.
More complex entities, such as service servers or clients will require more memory
than basic publishers or subscribers.
Note that most of the used memory is static memory assigned at build time.
The dynamic memory usage in micro-ROS is guaranteed to be allocated during the
entity creation stage and never during the entity operation. The stack usage in the
benchmarking process has been measured as the peak stack used in a FreeRTOS-
based system when creating the micro-ROS entity and using it for several minutes.

5 Client Library

This section describes the client library of micro-ROS, which implements the most
fundamental ROS concepts in the C programming language by extending the ROS 2
Client support library rcl with the rclc packages.23 The first subsection gives code
snippets on how to use those concepts in user code. The second subsection explains
execution management in micro-ROS, which is more customizable than in rclcpp or
rclpy of standard ROS 2. The last three subsections give introductions to advanced

21 https://github.com/micro-ROS/rmw-microxrcedds#rmw-micro-xrce-dds-implementation.
22 https://micro.ros.org/docs/concepts/benchmarking/memo_prof.
23 https://github.com/ros2/rclc.
 28 K. Belsare et al.

Fig. 6 Memory usage of micro-ROS entities

functionalities for querying the ROS graph, runtime lifecycle management, and diag-
nostics, which go beyond the core ROS concepts.

5.1 Programming with rcl and rclc

The rclc packages provide convenience functions to easily program a ROS application
in the C programming language. The programming interface is demonstrated by a
simple example: A publisher shall publish every second a message and a subscriber
shall print out this message. First, a node can be created by specifying its name and
namespace. Further more, memory allocation is organized by an allocator and the
ROS context is initialized by using a support object:
1 rcl_node_t my_node;
2 rclc_support_t support;
3 rcl_allocator_t allocator = rcl_get_default_allocator();
4 rclc_support_init(&support, argc, argv, &allocator);
5 rclc_node_init_default(&my_node, "name", "namespace", &support);

A publisher is created with three parameters: node object, message type, and
topic name. In this example, a string publisher with topic name ping is cre-
 Micro-ROS 29

ated. A macro is available to convert message types familiar from ROS 2 (e.g.,
std_msgs::msg::String).
1 rcl_publisher_t my_pub;
2 const char * topic = "ping";
3 const rosidl_message_type_support_t * msg_type =
→ ROSIDL_GET_MSG_TYPE_SUPPORT(std_msgs, msg, String);
4 rc = rclc_publisher_init_default(&my_pub, &my_node, msg_type,
→ topic);
The memory of the string message that shall be published has to be allocated stati-
cally:
1 std_msgs__msg__String pub_msg;
2 std_msgs__msg__String__init(&pub_msg);
3 const unsigned int PUB_MSG_SIZE = 20;
4 char pub_string[PUB_MSG_SIZE];
5 snprintf(pub_string, 13, "%s", "Hello World!");
6 rosidl_generator_c__String__assignn(&pub_msg, pub_string,
→ PUB_MSG_SIZE);
To publish this message, a timer is setup to call rcl_publish every second:
1 rcl_timer_t my_timer;
2 rclc_timer_init_default(&my_timer,&support,RCL_MS_TO_NS(1000),timer_cb);
3
4 void timer_cb(rcl_timer_t * timer, int64_t last_call_time)
5 {
6 if (timer != NULL) {
7 rcl_publish(&my_pub, &pub_msg, NULL);
8 }
9 }

Analogously, we create a subscriber that listens to the same string type and topic
name:
1 rcl_subscription_t my_sub;
2 std_msgs__msg__String sub_msg;
3 std_msgs__msg__String__init(&sub_msg);
4 rclc_subscription_init_default(&my_sub, &my_node, msg_type,
→ topic);
5
6 void sub_cb(const void * msgin)
7 {
8 const std_msgs__msg__String * msg =
→ (const std_msgs__msg__String *)msgin;
9 if (msg != NULL) {
10 printf("I heard: %s\n", msg->data.data);
11 }
12 }
ROS 2 services is another communication mechanism between nodes. Services
implement a client-server paradigm based on ROS 2 messages and types. A code
example how to use services in micro-ROS can be found in the rclc_examples
package.24

24 https://github.com/ros2/rclc/tree/master/rclc_examples.
 30 K. Belsare et al.

A parameter is a configuration value of a ROS 2 node which can be dynamically

reconfigured. Parameters can also be defined and used in rclc. A code example is
again available in the rclc_examples package.
Guard conditions are objects that can be used to notify the Executor to wake up
based on some external event similar to the concept of condition variables in multi-
threaded C++ programs. Guard conditions are also available in rclc. They can also
be used to react on hardware events.
After defining all objects, the spin method needs to be called, like in rclcpp or rclpy.
As the ROS 2 Executors are implemented in the language-specific client libraries and
not in rcl, the main rclc package provides its own Executor. This concept is explained
in more detail in the next section.
The following code snippet completes the example by adding the timer and the
subscription to an Executor instance. To avoid dynamic memory allocation at run-
time, the user has to specify how many objects (handles) the Executor shall manage.
Subscriptions and other objects can be added only until the maximum number of
handles is reached. Handles can also be removed at runtime.
1 rclc_executor_t executor;
2 unsigned int num_handles = 2;
3 rclc_executor_init(&executor, &support.context, num_handles,
→ &allocator);
4 rclc_executor_add_timer(&executor, &my_timer);
5 rclc_executor_add_subscription(&executor, &my_sub, &sub_msg,
→ &sub_cb, ON_NEW_DATA);
6 rclc_executor_spin(&executor);

5.2 Execution Management with the rclc Executor

The development of an execution management mechanism for micro-ROS is based

on an analysis of processing patterns commonly used in robotic applications.
The most basic pattern is the sense-plan-act control loop as depicted in Fig. 7.
All callbacks of the sense phase, which usually comprise of sensor data acquisition,
shall be processed before the plan phase to ensure that planning algorithms use the
most recent sensor data information.
A second pattern is the synchronization of sensor data with different update rates.
For example, an IMU has usually a very high update rate, while a laser sensor a low
one. Synchronizing sensor data with different rates deterministically is difficult due
to latency jitter and clock drift. An alternative pattern is to wait for a new laser input
and then request available IMU data as shown in Fig. 8.
A third common pattern is the prioritization of certain processing paths, e.g., for
safe obstacle avoidance.
The need for these patterns are often non-functional requirements that a robotic
application must meet: bounded end-to-end latencies, low jitter of response times
Micro-ROS 31

Fig. 7 Typical sense-plan-act control loop in robotics

Fig. 8 Sensor fusion of

sensor data with different
update rates

of cause-effect chains, deterministic processing, and quick response times even in

overload situations.
Embedded real-time applications can be scheduled with different schemes. The
most common is the periodic scheduling scheme, in which all functions are mapped to
predefined periods, e.g., 1, 10, and 100 ms. This scheduling scheme is well supported
by all real-time operating systems and formal performance analysis approaches are
well known for decades.
In contrast, ROS 2 follows an event-driven approach. Messages are communicated
between nodes using the publish and subscribe paradigm. It is the responsibility of
an Executor to react upon new messages. In detail, this Executor coordinates the
execution of callbacks issued by the participating nodes by checking the incoming
messages from the DDS queue and dispatching them to the underlying threads for
execution. Currently, the dispatching mechanism is very basic: the Executor looks
up wait queues, which notifies it of any pending messages in the DDS queue.
If there are pending messages, the Executor executes the corresponding callbacks
one after the other, which is also called round-robin to completion. In addition, the
Executor also checks for events from application-level timers, which are always pro-
cessed before messages. There is no further notion of prioritization or categorization
of these callback calls. Moreover, the ROS 2 Executor from rclcpp in its current
form does not leverage the real-time capabilities of the underlying operating system
scheduler to have finer control on the execution order. The overall implication of
this behavior is that time-critical callbacks could suffer possible deadline misses and
degraded performance since they are serviced later than non-critical callbacks. Addi-
tionally, due to the round-robin mechanism and the resulting dependencies it causes
on the execution time, it is difficult to determine usable bounds on the worst-case
latency that each callback execution may incur [26].
To address the aforementioned common processing patterns found in robotic
applications with the event-driven ROS 2 execution scheme, we have developed a
very flexible approach in micro-ROS named rclc Executor [27].
It supports three main features: First, a trigger condition allows to define when
the processing of a callback shall start. This is useful to implement sense-plan-act
control loops or more complex processing structures with directed acyclic graphs.
Second, a user can specify the processing order in which these callbacks will be
32 K. Belsare et al.

Fig. 9 Budget-based real-time Executor design

executed. With this feature, the pattern of sensor fusion with multiple rates, in which
data is requested from a sensor based on the arrival of some other sensor, can be
easily implemented. Third, the rclc Executor allows to set scheduling parameters
(e.g., priorities) of the underlying operating system. With this feature, prioritized
processing can be implemented.
By the example of NuttX, we even demonstrated how to use budget-based schedul-
ing in micro-ROS [28]. Figure 9 shows the Executor design for this purpose: One
Executor thread is responsible for checking for new data from the DDS queue. For
every callback, a thread is spawned with the desired scheduling parameters of the
operating system. The Executor thread dispatches new data to the corresponding call-
back functions, which are then executed based on the operating system scheduling
parameters.
Last, but not least, since memory is very limited on microcontrollers, dynamic
memory is only allocated at the configuration stage of the rclc Executor. This allows
a safe execution on the microcontroller.

5.3 ROS Graph

A micro-ROS system is by definition a system where is not possible to run a full

featured DDS middleware. From ROS 2 Foxy, the ROS 2 graph discovery and han-
dling is based on a hidden DDS topic named ros_discovery_info. This topic and
its associated packages (such as rmw_dds_common) heavily rely on DDS identifiers
such as GUIDs. This way, the node concept in ROS 2 is separated from the DDS
participant.
The micro-ROS stack is not able to handle the ROS 2 graph in a fully compliant
way because it does not have direct access to the DDS middleware but handles DDS
Micro-ROS 33

entities through DDS-XRCE (cf. Sect. 4). In order to have full support for ROS 2
graph in micro-ROS, the micro-ROS Agent implements a graph manager. This graph
manager is in charge of two main functionalities related to the ROS 2 graph:
1. It announces all the micro-ROS participants, nodes, publisher, and subscribers in
the ros_discovery_info topic, allowing ROS 2 nodes to discover the entities
created by a micro-ROS application.
2. It retrieves the information from the ros_discovery_info topic from the ROS 2
nodes and prepares a memory-aware topic ready to be subscribed by micro-ROS
nodes. This ros_to_microros_graph topic summarizes the ROS 2 graph infor-
mation in a way that the micro-ROS stack can handle it while having the full
information about the nodes names and associated topic names and types.
In the micro-ROS client part, the ROS 2 graph management tools are disabled by
default. Users can enable them by setting a flag in the micro-ROS colcon.meta
file. When the micro-ROS client graph manager is enabled, a subscription to
ros_to_mic- roros_graph will be configured and static and dynamic memory
will be used in order to handle the ROS 2 graph data.

5.4 Advanced Runtime Lifecycle Management

The C++ ROS 2 client library rclcpp does not only provide abstractions for standard
ROS nodes but also so-called managed lifecycle nodes, allowing greater control
over the state of a ROS 2 system.25 It allows to ensure that all components have been
instantiated correctly before beginning executing its behavior, restarting, or replacing
nodes. For this purpose, a standard lifecycle has been defined for all such nodes, see
the gray and blue boxes in Fig. 10.
We took this concept as basis for advanced lifecycle management to enable sleep
modes, power saving, etc., often required on resource-constrained devices. First,
we ported managed lifecycle nodes to rclc in the rclc_lifecycle package for
use on microcontrollers. Second, we extended the standard lifecycle by additional
modes—termed system modes—that provide additional abstractions for easy (re-
)configuration of nodes and entire (sub-)systems.26
Based on a specification of a ROS 2 system in terms of its ROS 2 nodes, a hierarchy
of nodes (sub-systems), and a definition of their system modes, a mode inference
mechanism and a mode manager node enable easy (re-)configuration of entire ROS 2
systems. In detail, based on the current state and parameterization of the nodes, the
system mode of the entire system can be inferred and switching an entire system
to a specified system mode results in according state and parameterization of all
contained ROS 2 nodes.

25 See the ROS 2 design document at https://design.ros2.org/articles/node_lifecycle.html for details

on the purpose of this concept.
26 https://github.com/micro-ROS/system_modes.
34 K. Belsare et al.

As the mode inference mechanism and mode management node are quite complex,
these are implemented on top of rclcpp and intended to run on a microprocessor.
They manage standard ROS 2 rclcpp lifecycle nodes as well as rclc lifecycle nodes
running on a microcontroller, see Fig. 11. The implementation of lifecycle nodes is
not influenced the system modes concept as the interface between the mode manager
and lifecycle nodes purely relies on standard lifecycle messages and parameter events.
Nordmann et al. provide more information on the system modes concept and its
application in [29].

5.5 Diagnostics

Core requirements for micro-ROS diagnostics were to (1) be slim and avoid dynamic
memory allocation and (2) integrate with the ROS 2 diagnostics stack.
Requirement 1 cannot be met with standard ROS 2 diagnostics messages27
because of the main message type DiagnosticArray: It is an array of arbitrary

INACTIVE

activating deactivating error processing

x x
Modes
ACTIVE
MODE A
x
MODE B shutting down
... x

FINALIZED

Fig. 10 Default node lifecycle (blue boxes for primary states, gray boxes for transition states) of
ROS 2 and the extension by system modes (green boxes) developed with micro-ROS. Red dots
indicate potential transitions to the error processing state

Lifecycle Mode Lifecycle Lifecycle

Node … Manager lifecycle &
parameter
Node … Node

rclcpp, rclcpp_lifecycle events rclc, rclc_lifecycle

ROS 2 micro-ROS
uP Microcontroller

Fig. 11 System modes architecture: The mode inference and management runs in the mode manager
on a microprocessor. It manages (re-)configuration of lifecycle nodes (and hierarchies of them) on
the microprocessor itself as well as on connected microcontrollers

27Package diagnostic_msgs is part of ROS 2 common interfaces, cf. https://github.com/ros2/

common_interfaces.
Micro-ROS 35

ApplicaƟon
component
ApplicaƟon
component
… ApplicaƟon
component
common
diagnosƟcs

Aggregator Updater Updater Updater

ROS 2 DiagnosƟcs micro-ROS DiagnosƟcs
micro-ROS
micro-ROS
ROS 2 diagnosƟc bridge
diagnosƟc
msgs micro-ROS

uP Microcontroller

Fig. 12 micro-ROS diagnostics architecture, showing the diagnostics stack running on the micro-
controller (right), sending optimized diagnostics messages to a ROS 2 bridge component that trans-
lates these messages to standard ROS 2 diagnostic messages

size, the array element DiagnosticStatus makes heavy use of strings, and it con-
tains another array of arbitrary size of KeyValue messages. This constraint and
Requirement 2 led to the micro-ROS diagnostics architecture28 depicted in Fig. 12.
The micro-ROS diagnostics packages do not provide any diagnostic aggregator as
we assume that such aggregation takes place on a microprocessor running standard
ROS 2.
First, micro-ROS diagnostics provides a specific MicroROSDiagnosticStatus
message type that neither contains arrays nor contains strings. Instead, key and value
of the diagnostic status as well as IDs of the hardware and updater are encoded with
numeric values, referencing entries in a lookup table. Based on the lookup table,
the diagnostic bridge, a standard ROS 2 component implemented with rclcpp, can
translate MicroROSDiagnosticStatus into standard ROS 2 DiagnosticStatus.
Another part of micro-ROS diagnostics is an rclc-based library for diagnostic
updaters. Each updater references a set of diagnostic tasks, i.e., c-functions that
return a diagnostic value such as a boolean value, a numeric value (integer or double
precision), or an ID that references a value entry in the lookup table. A diagnostic
updater additionally holds IDs referencing a diagnostic key, an updater ID, and a
hardware ID from the lookup table. Upon each execution of the updater, the diagnostic
task is executed and a MicroROSDiagnosticStatus message is created and sent to
a reserved topic /diagnostics_uros.
The diagnostic bridge, running on a microprocessor and instantiated with the
lookup table, subscribes to /diagnostics_uros, translates MicroROS-
DiagnosticStatus messages into DiagnosticStatus messages, and publishes
them to the standard /diagnostics topic. These messages are fully transparent,
i.e., a standard ROS 2 diagnostic aggregator is able to treat the translated micro-ROS
diagnostic messages in the same way it is treating diagnostic messages from standard
ROS 2 components, hence fulfilling Requirement 2 stated above.

28 https://github.com/micro-ROS/micro_ros_diagnostics.
36 K. Belsare et al.

6 Build and Development Tools

This section gives and overview to development tools for micro-ROS. The first part
is focused on build tools, while the second part presents a generic benchmarking
tool.

6.1 Build Tools

Micro-ROS provides multiple ways of building a library containing the whole micro-
ROS stack. Usually those build systems generate a static library and a set of header
files to be included and linked against the user application. Also, some of them
integrate this library directly in a platform specific SDK.

6.1.1 micro_ros_setup29

The principal package for creating micro-ROS apps is micro_ros_setup. It is pro-

vided as a ROS 2 package that can be included in a ROS 2 workspace and offers a
set of utilities for creating, building, and flashing micro-ROS in an embedded target.
Using micro_ros_setup, a ROS 2 developer can use the same tooling to build
and flash a micro-ROS applications without worrying about the underlying platform
specific details such as: RTOS SDK, compiler version, additional platform configu-
ration, etc.
It includes most of the micro-ROS supported platforms (e.g., Olimex STM32-
E407, ST Nucleo boards, and ESP32) and RTOSes (including FreeRTOS, Zephyr,
NuttX, and Mbed). Instructions about how to use it can be found in the tutorial section
of the micro-ROS website.30
The micro_ros_setup package also provides a tool for generating a standalone
precompiled static library that can be copied and used in external projects or plat-
forms. It is called custom library generator 31 and works just providing the following
files:
– CMake toolchain for defining the building target, flags and options
– colcon.meta file for configuring each layer of the micro-ROS stack
The micro_ros_setup package also allows building micro-ROS in a Linux envi-
ronment for debugging and testing purposes.

29 https://github.com/micro-ROS/micro_ros_setup.
30 https://micro.ros.org/docs/tutorials.
31 https://micro.ros.org/docs/tutorials/advanced/create_custom_static_library.
Micro-ROS 37

6.1.2 Micro-ROS Component for ESP-IDF32

This is a component that integrates micro-ROS in the Espressif ESP32 family: ESP32,
ESP32-S2, and ESP32-C3. It builds and link a micro-ROS static library against an
ESP-IDF application and provides serial, WiFi, and Ethernet transports.

6.1.3 Micro-ROS Module for Zephyr33

This module integrates micro-ROS in the Zephyr RTOS build system. It builds and
link a micro-ROS static library against a Zephyr application and provides Serial and
UDP transports. It can be used to integrate micro-ROS in any of the Zephyr RTOS
supported boards.34
For debugging and testing purposes, this package allows building micro-ROS in
a Linux environment using the Zephyr Native POSIX port.

6.1.4 Micro-ROS for Arduino35

This support package is a special port of micro-ROS provided as a set of precompiled

libraries for specific Arduino-enabled platforms. The main reason for this approach
is that Arduino does not allow the build of a complex library such as micro-ROS, so
by using this approach a ready-to-use solution is provided to the Arduino users.
Along with this package, there are detailed instructions for building the micro-
ROS for Arduino library manually, for users that need to tune the default configura-
tion.

6.1.5 Micro-ROS for STM32CubeMX and STM32CubeIDE36

This package is a set of utilities which enables the seamless configuration, setup,
and integration of micro-ROS into an STM32 controller based project. As such,
it allows micro-ROS to be virtually supported by the full set of boards offered by
STMicroelectronics.
Its usage is based on Dockers, via a prepared Dockerfile which eases micro-ROS
library generation outside of a ROS 2 environment.

32 https://github.com/micro-ROS/micro_ros_espidf_component.
33 https://github.com/micro-ROS/micro_ros_zephyr_module.
34 https://docs.zephyrproject.org/latest/boards.
35 https://github.com/micro-ROS/micro_ros_arduino.
36 https://github.com/micro-ROS/micro_ros_stm32cubemx_utils.
38 K. Belsare et al.

6.1.6 Micro-ROS Module for Raspberry Pi Pico SDK37

This module integrates micro-ROS in the Raspberry Pi Pico build system. It builds
and links a micro-ROS static library ready to be used with serial transport in the
Raspberry Pi Pico microcontroller.

6.1.7 Micro-ROS App for NuttX38

This module integrates micro-ROS in the NuttX 10 build system. It builds and links
a micro-ROS static library ready to be used with serial and networking transports
virtually in any NuttX supported platform.39

6.1.8 Micro-ROS for Mbed OS40

This is a module that integrates micro-ROS in the ARM Mbed build system. It builds
and link a micro-ROS static library ready to be used with serial transport.

6.1.9 Micro-ROS for Azure RTOS41

This repository provides a demo application that integrates micro-ROS in the

Microsoft Azure RTOS build system. It builds and links a micro-ROS static library
ready to be used with wireless transport.

6.2 Benchmarking

Developing a working and stable application from the scribbles to the final executing
binary is a long and hard task. During this process developers may come across sta-
bility issues and performance issues. In addition to these issues, some specified QoS
might be difficult to quantify. Solving those problems without the proper tools might
be frustrating, tedious tasks leading to reduced developer efficiency. An adapted
benchmarking tool can overcome important obstacles. There are different key per-
formance indicators (KPIs) that developers might be interested. In the proposed
benchmarking framework, the KPIs can be chosen freely by the developer. This way,

37 https://github.com/micro-ROS/micro_ros_raspberrypi_pico_sdk.
38 https://github.com/micro-ROS/micro_ros_nuttx_app.
39 https://nuttx.apache.org/docs/latest/introduction/supported_platforms.html.
40 https://github.com/micro-ROS/micro_ros_mbed.
41 https://github.com/micro-ROS/micro_ros_azure_rtos_app.
Micro-ROS 39

the benchmarking tool will remain flexible and allow the community to constantly
add some support for a lot of different KPIs. The problems we want to tackle are:
– Out there, many benchmarking tools exist, each of targeting different KPIs
– Different platforms (Linux, RTOSes, bare metal)
– Too few time/resources to code benchmarking tool for each
– Avoid code overhead, i.e., keep code clarity
– Avoid execution overhead, i.e., do not make execution slower when benchmarking

6.2.1 Overview

The benchmarking tool under development is providing a framework to allow devel-

opers to create their own benchmarking tool. Each part a developer wants to bench-
mark can be added as a plugin using the provided framework. This way, plugins
can be shared and this improves reusability as much as possible. The source code
of the tool is provided in the micro-ROS organization on GitHub.42 The framework
includes the features explained in the following paragraphs (Fig. 13).

6.2.2 Trace Framework Abstraction

The Shadow Builder alone only parses comments from the application and passes
them along to the Trace Framework Abstraction (TFA) core. The TFA core is aware
of the plugins that are available, all capabilities of the plugins, and of the platform
target. The process goes as explained below:

1. The line containing the functionality Benchmarking::XX::YY will be checked

against all the available plugins.
2. Plugins that are capable of handling functionality will respond with a piece of
code that will be replaced with a piece of code.
3. Then the output file will be added in a folder corresponding to the platform type
and benchmarking type.

Being generic is the key for this benchmarking tool. The plugins, in contrast, will
bring the specific implementation needed to benchmark a specific platform. Every
plugin will provide information as requested by the parser:
– List of supported platforms
– List of functions that are handled
– Code snippets that will be added for benchmarking
– List of patches and/or patch code
– Optional end script to run and execute the benchmarks.

42 https://github.com/micro-ROS/benchmarking.
40 K. Belsare et al.

Fig. 13 Architecture of the

micro-ROS benchmarking
tool

6.2.3 Shadow Builder

The Shadow builder is a tool that will transparently instrument the code to benchmark
and, if the benchmarking is not activated, it should be bypassed. The tools will be
able to output an “instrumented code” that will be later be compiled as a normal
code. The following steps describe what the shadow builder process flow:

1. Get Benchmarking Configuration File file from the user

2. Get appropriate sources
3. Execute Trace Framework Abstraction Configuration file
4. Parse the sources file needed Injecting code
5. Compile the targeted binary for different platform
6. If needed, depending on the type of benchmark being undertaken, compile another
target binary.

6.2.4 Binary Generation for Instrumented Code

The binary generation is the process of compiling the source code. In order to bench-
mark, previously to compile the source code, it is necessary to instrument the code.
The code will be instrumented in a transparent way for the developer/user. Therefore,
a configuration file provided by the developer will be parsed and code injected as
described in a configuration file. The two main steps in this process are:
1. Receiving inputs: The binary generation’s pipeline receives two inputs to work
with the Configuration Benchmarking File and the source code to benchmark. In
short, the configuration describes:
– What is benchmarked (sources)
– Where to benchmark
– What type of benchmark
– Optionally against what base line to compare (base line source)
Micro-ROS 41

2. Parse and check: Once the input received the Shadow Builder parses the config-
uration file. From the configuration file, the Shadow Builder obtains:
– The different benchmarking to be achieved
– The targeted platforms
In addition to parsing, the Shadow Builder is in charge of checking capabilities and
consistency within the configuration file and the different TFA plugins registered in
the TFA module.

6.2.5 TFA Execution

Once parsed and checked against the TFA module capabilities, the Shadow Builder
will be in charge of translating configuration into source code. The translated sources
will also be achieved in cooperation with the TFA module. At the end of this step, the
TFA will generate the new forged source code ready for compilation. In addition to
patched source code, the TFA will generate scripts that will complete the benchmarks.

6.2.6 Compilation

The compilation will happen for every kind of benchmark and target platform.
Depending on the kind of benchmark that is being executed, there will be one or
more binaries per benchmark session. The number of binary generated also depends
on what plugins are provided by the user to the shadow builder. The shadow builder
will retrieve capabilities of the plugins requested by the developer, match them, and
generate software according to the matches.

6.2.7 Steps to Start Benchmarking

The Shadow Builder will be executed as follows:

1. Software sources are passed to the Shadow Builder.
2. The source are passed and upon comments containing /Benchmarking::XX::YY/
(a tag) the code line is passed to the Trace Framework Abstraction module. Using
comments is preferable, i.e., no includes are needed.
3. For all plugins that are registered with the TFA, the Benchmarking::XX::YY
functionality will return a piece of code that will be added to the source.
4. Once all parsed, the Shadow Builder will compile for all the different platforms
requested either by plugins or by user configuration.
42 K. Belsare et al.

6.2.8 Results

The list of benchmarking targets are:

– Communication performance
– Real-time performances
– Execution performances
– Function call count
– Static memory consumption
– Dynamic memory allocation consumption
– Power consumption
We used the benchmarking tool to measure the performance of the micro-ROS stack
on a typical microcontroller with different communication technologies:

– Serial at 115200 baud 8 bits/No parity/1 stop

– Ethernet 100/10 Mbps link,
– 6LoWPAN over 802.15 link.

All the results, along with interpretations, are available on the micro-ROS website.43

7 Use-Cases

In this section, we report on three use-cases of micro-ROS in industrial applications

and research. The use-cases illustrate different usage scenarios: First, a pure sensing
use-case. Second, the use for microcontrollers in an automated/robotic vehicle. Third,
a highly automated environment with multiple autonomous robots and numerous
additional fixed sensors.

7.1 Micro-ROS-Enabled GNSS Module by Hydrasystem

Hydrasystem is young and rapidly expanding enterprise created by experienced engi-

neers on electronics and automation. The company provides state of the art position-
ing solutions for agricultural machines with work automation in mind. The main
product is a system for precise localization based on real-time kinematic corrections,
which enable field navigation with 1–2 cm precision. Starting with localization this
system can be expanded with further modules, providing navigation and automated
work functions.
The company offers several ROS-compatible modules, which are shown in Fig. 14.
In this use-case the Hydra Box DGPS module is used (Fig. 15).

43 https://micro.ros.org/docs/concepts/benchmarking/results.
 Micro-ROS 43

Fig. 14 ROS-compatible GNSS products by Hydrasystem

Fig. 15 GNSS family parameters

7.1.1 Requirements

There are many applications in robotic systems where GNSS receivers are used
for positioning. Usually, to connect a GNSS receiver to a ROS-based system, it is
needed to connect it to the computer on which ROS is installed and create a dedicated
node to support it. Hydrasystem has adapted a simple GNSS receiver to work with
micro-ROS. In the case of Hydra Box DGPS, it is only needed to configure the
network connection and the module itself starts the publishing standard NavSatFix44
messages.

7.1.2 Technical Approach

The Hydra Box DGPS module (Fig. 16) is based on the ESP32. The module offers
an absolute accuracy of approx. 1 m. The principle of operation is shown in Fig. 17.
The GPS receiver transmits data in the NMEA format to the ESP32 module via
UART. The data is then parsed and converted into NavSatFix messages. This message
is published by micro-ROS. In this case, WiFi has been selected for communication
with the computer with ROS 2, but it can also be, for example, Bluetooth. The micro-
ROS agent is running on the computer with ROS 2. The correct operation can be
checked as follows:
1 ros2 topic echo /freertos_navsatfix_publisher
2 ---
3 header:

44 See http://docs.ros.org/en/api/sensor_msgs/html/msg/NavSatFix.html for the message type def-

inition.
 44 K. Belsare et al.

Fig. 16 Hydra Box DGPS module

Fig. 17 Hydra Box DGPS diagram

4 stamp:
5 sec: -1515870811
6 nsecs: 2779096485
7 frame_id: ’’
8 status:
9 status: -91
10 service: 42405
11 latitude: 52.19599914550781
12 longitude: 20.926616668701172
13 altitude: 0.0
14 position_covariance:
15 - 0.0
16 - 3.6720809936523438
17 - 5.30345388e-315
18 - 0.0
19 ...
20 position_covariance_type: 0
 Micro-ROS 45

Fig. 18 The BODAS Controller RC40/5-6 (in the center) with an angle sensor and a pressure sensor
(left side) as well as a speed sensor and a display (right side) from the BODAS hardware portfolio.
(Photos by Bosch Rexroth.)

21 ---

A video showing how Hydra Box DGPS works can be seen at YouTube.45
The most important advantage of this solution is very simple integration with the
ROS 2 system. Just plug in the power supply and configure the network connection.

7.1.3 Discussion and Outlook

The use-case shows that micro-ROS can be successfully used on resource-constrained

devices where ROS/ROS 2 cannot be run. Thanks to the use of micro-ROS, it is
possible to easily integrate various types of sensors or other types of IoT devices.

7.2 Off-Road Vehicle Control with Micro-ROS Interface by

Bosch Rexroth

Bosch Rexroth is a company of the Bosch Group and one of the world’s leading
suppliers of drive and control technology for mobile applications, machinery appli-
cations, and factory automation. Bosch Rexroth has about 31,000 associates in over
80 countries.
BODAS is an open and scalable mobile electronics platform by Bosch Rexroth
and intended in particular for safety-critical applications in mobile machinery, e.g.,
excavators, wheel loaders, and tractors. The BODAS hardware portfolio comprises
sensors, joysticks, pedals, connectivity units, and displays, as illustrated in excerpts
in Fig. 18. The heart of the platform are the BODAS Controllers with safety-rated
microcontrollers in robust housings for off-road use and with a broad spectrum of
interfaces including CAN, CAN FD, ISOBUS, and GPIOs.

45 https://youtu.be/hTEnTju1Qkg.
46 K. Belsare et al.

The BODAS base software is built on the AUTOSAR Classic Platform [30] and
can be tailored by a comprehensive cloud service named easyConfig. Application
software can be implemented in C or IEC 61131-3 languages against a BODAS-
specific API.

7.2.1 Requirements

In the mobile machinery market, we observe an increasing interest in ROS (and

particularly ROS 2) for the development of advanced automation functions for con-
struction and agriculture. While the main, computationally intensive algorithms are
typically implemented on larger microprocessors, microcontroller platforms like
BODAS Controllers are used to interact with the hardware, to realize low-latency
control functions, and to implement safety mechanisms. To bridge the technological
gap between those two classes of platforms, Bosch Rexroth and Bosch Research are
porting micro-ROS to BODAS Controllers. The two top-level requirements for this
endeavor are:
1. Seamless integration with the BODAS API and AUTOSAR: The micro-ROS stack
shall be integrated in such a way that it does not interfere with non-ROS software.
First of all, this means that micro-ROS has to fit into the scheduling model and
memory management of AUTOSAR Classic, which again is based on the OSEK
RTOS standard. Second, in the long term, freedom from interference in accordance
with ISO 26262 shall be guaranteed.
2. Transport over CAN: The Controller Area Network (CAN) is the dominating vehi-
cle bus system. For the communication between micro-ROS nodes on a BODAS
Controller with the micro-ROS agent on a larger microprocessor, it is essential
that CAN is supported as a transport of DDS-XRCE.
These requirements raise several technical challenges, three of which we will discuss
next.

7.2.2 CAN Transport for DDS-XRCE

The DDS-XRCE standard [16] defines three transports—i.e., mappings to underlying

protocols—namely to UDP, TCP, and serial (e.g., SPI, I2C, and RS-232). However,
it is not limited to those ones. For this purpose, Section 11.1 of the standard defines
a transport model, which consists of five requirements to an underlying transport.
CAN does not meet exactly one of the requirements, namely the capability to
deliver messages of at least 64 bytes. The base frame format of CAN has a payload
of 8 bytes only. In fact, Micro XRCE-DDS assumes 42 bytes only, but that is still
too much. Therefore, we added mechanisms to packetize DDS-XRCE messages
including checksum into 7 byte fragments and a sequence number on the sender side
and to unpacketize them accordingly on the receiver side.
Micro-ROS 47

The transport model of DDS-XRCE also defines three functionalities that are
explicitly not required: The transport does not need to provide reliability and ordering.
It even does not need to provide notification of dropped messages. This simplifies the
packetizing mechanism. If the unpacketizing routine detects a broken or incomplete
DDS-XRCE message, it may simply drop it and rely on the error handling of DDS-
XRCE.

7.2.3 Execution Management on Periodic Tasks

AUTOSAR Classic and the underlying OSEK RTOS use a scheduling model which
is very different from POSIX. It consists of fixed periodic tasks (e.g., 50, 10, and
2 ms), whose priority is inverse to the period. Hence, the tasks may preempt each
other in a strict ordering only. There may be multiple functions inside a task. In
each iteration, these functions are executed in a fixed sequential order and run to
completion. All data across iterations is stored in the global memory. Therefore, the
whole system can be implemented by a single stack—unlike POSIX, where each
thread has its own stack.
On the one hand, the periodic execution model does not fit at all to the event-based
execution model of ROS and micro-ROS. On the other hand, we expect that many
functionalities on the application level are in fact periodic. Many sensors provide their
readings in a periodic fashion. Similarly, many actuators are controlled periodically.
Therefore, we decided to embed the micro-ROS execution model directly in this
task scheme as depicted in Fig. 19. CAN frames are received by an interrupt service
routine (ISR). A middleware interface function running in a high-frequency task
(e.g., 1 ms) unpacketizes the CAN frames into DDS-XRCE messages. The Executor
runs in the same or a slightly slower task and takes incoming DDS-XRCE message
into ROS messages. Such a take does not imply that the corresponding application-
level callback is invoked immediately but the callbacks are executed in worker tasks,
often at much lower frequency. The assignment of callbacks to worker tasks has to

CAN Interrupt Middleware Interface Executor Task User Callback Tasks

Service Routine Task (1 ms) (10 ms) Task A Task B
cond_check(dataByte.available)
(10 ms) (100 ms)
Fill waitSet
cond_check(msgA.available) cond_check(msgB.available)
Populate CAN data buffer rcl_wait

can_data_buffer

Create XRCE-DDS msg

xrce_dds_data rcl_take B.msg

execute subsB_cb(&B.msg)

Fig. 19 Mapping of micro-ROS execution model to AUTOSAR Classic and OSEK task scheme
48 K. Belsare et al.

be implemented by the developer. The developer has to observe that the callback can
run to completion in the period of the worker task.

7.2.4 Memory Management Without Explicit Heap

In the current version, micro-ROS avoids dynamic memory operations to a large

extent, but cannot do without them entirely. Applications can be designed that allo-
cations occur during some initialization phase only. In this case, micro-ROS performs
some dynamic memory allocations but memory is only freed at shutdown. Concep-
tually, such use can be considered global/static variables, but technically malloc is
called.
AUTOSAR Classic does not provide OS-level services for dynamic memory man-
agement at all, cf. Section 4.6.3 of [30], as such operations are generally considered
harmful in safety-critical systems due to memory fragmentation. Therefore, we are
currently designing a micro-ROS-specific allocation function backed by a static mem-
ory block whose size has to be determined by the developer.

7.2.5 Discussion and Outlook

The port of micro-ROS to the BODAS Controllers is still on-going. The CAN trans-
port for Micro XRCE-DDS has been already implemented and tested successfully
using the interface for custom transports. We have also implemented a first prototype
of an Executor supporting worker tasks/threads, cf. Sect. 5.2. This implementation
is currently integrated with AUTOSAR and the BODAS API.
The first release will not provide any freedom-from-interference mechanism but
is mainly intended for prototyping.

7.3 Smart Warehouse Use-Case by Łukasiewcz-PIAP

Łukasiewicz-PIAP (PIAP) is a research institution with strong commercial branch,

established in 1965. Currently it has approximately 300 employees, out of which
more than 2/3 are scientists and/or engineers. PIAP is a part of Łukasiewicz Research
Network. Activity topics include: Mobile robotics, Cyber-physical systems develop-
ment, Industrial automation, Measurement devices, Rapid prototyping centre. PIAP
is mainly Polish and one of the biggest European producers of IEDD/EOD/CBRN
mobile robots and robotic accessories. PIAP is also a member of the EU-funded
project OFERA [17], which has initiated the micro-ROS project.
The Smart warehouse is a complex use-case that intends to demonstrate that
micro-ROS works well in a dynamic, distributed system of heterogeneous devices,
including the low-power, microcontroller-based hardware and regular robots.
Micro-ROS 49

7.3.1 Requirements

In robotic or other automation systems there are many sensors or actuators that are
based on microcontrollers where ROS/ROS 2 cannot be executed. Currently, it is
required to connect such devices to a computer with ROS/ROS 2, containing custom
implementation of their support and only then were they available in a ROS/ROS 2
system. Micro-ROS, which works with ROS 2, changes this dramatically as it can run
directly on microcontrollers and other low resource devices. This allows for direct,
convenient, and quick integration into the ROS 2 based system and also into classical
ROS using the ROS bridge. In this use-case, a mobile platform roams around smart
warehouse, doing scheduled tasks. These tasks will consist of reaching the sensors
or actuators and interacting with these radio-controlled, micro-ROS enabled devices.

7.3.2 Technical Approach

A specially prepared laboratory simulates a smart warehouse. It contains all the

devices needed to run the planned scenario, such as locating devices and several
micro-ROS enabled devices. The layout of the warehouse is shown in Fig. 20. The
warehouse is equipped with:
– Humidity and temperature sensor
– Laser scanner (distance measurements), shown in Fig. 22
– Door opener
– Final effector
The mobile robot platform, depicted in Fig. 21, is built on the basis of the standard
Scout robot produced by Łukasiewicz-PIAP. The mobile robot platform has been
equipped with
– an autonomy module (PC computer/lidar/indoor positioning/outdoor positioning),
– communication interfaces (6LoWPAN/WiFi), and
– other components, e.g., a wireless emergency stop.
The whole use-case is using a mixture of classical ROS, ROS 2, and micro-ROS
environments. The autonomy of the mobile platform is based on ROS. The mobile

Fig. 20 Smart warehouse

mission plan
50 K. Belsare et al.

Fig. 21 Smart warehouse mobile platform

Fig. 22 Smart warehouse sensor example: distance sensor

platform’s communication with sensors is based on ROS 2 and micro-ROS. The

sensors and actuators use micro-ROS. A map of all those environments is shown on
the Fig. 23.
The robot mobile platform is controlled by the autonomy module, but it is also
possible to control it remotely using a standard joystick. The laptop (robot control
unit) can provide information about the robot’s states and its sensors’ values. The
Micro-ROS 51

Fig. 23 Architecture of the smart warehouse use-case

ROS visualization tool rviz,46 is used to give humans readable information. SMACH
is used to design states machines to make missions steps.
The mobile platform will perform specific missions according to the developed
scenario. The task for the robot is set in the mission planner. Missions are focused
on presentation of micro-ROS features. One selected mission is described below.
The mission is performed according to the simplest scenario, taking into account
communication with all sensors and actuators used in the use-case:
– approach the humidity sensor and obtain the measurement,
– go towards the warehouse gate and after communication with the door opener and
their opening, the platform moves towards the road,
– the distance sensor next to the road provide information to the platform whether
the road is empty and we can go,
– after crossing the road and reaching the lamp, the platform sends command to the
lamp to switch it on,
– stop and wait for the next task.

This mission is focused on evaluating the communication and the correct operation
of sensors and actuators. In the first task (approaching the humidity sensor), the
robot mobile base moves near to the humidity sensor and tries to connect to it.
After obtaining the connection, the measurement results are sent from the sensor to
robot. Then robot mobile base goes towards the door. Like before, a connection is
established with the door opener. After opening the door, we have to ensure that the
exit is clear and there are no obstacles. This is checked by a distance sensor placed
outside the warehouse. Now robot mobile base can go to the lamp, i.e., the final

46 http://wiki.ros.org/rviz.
52 K. Belsare et al.

effector. After approaching final effector, it remotely lights the lamp. The video with
the recorded mission is available on YouTube.47

7.3.3 Discussion and Outlook

The warehouse use-case is a good demonstration of what is achievable in an indus-

trial environment. This smart warehouse use-case also demonstrates that micro-ROS
works well in a dynamic, distributed system of heterogeneous devices. In this case
the focus was on the radio communication over 6LoWPAN. Showing that micro-
ROS works in such conditions confirms the fulfillment of the micro-ROS project
assumptions.

8 Final Remarks

After more than three years of active development and maintenance, micro-ROS
is the first choice for integrating microcontrollers into ROS-2-based systems and
for bringing ROS 2 itself onto microcontrollers. The micro-ROS stack turns ROS 2
into a great superset to cover an extended range of applications for new industrial
deployments in Industry 4.0, IoT, and many other domains. The maturity of micro-
ROS has been proven in several use-cases and projects from research to product
development.
An important key factor to long-term success of the micro-ROS project is building
a subcommunity inside the ROS community which takes over the maintenance of the
micro-ROS stack. For this purpose we have initiated the ROS Embedded Systems
Working Group (WG), held on a monthly basis [22], and we are happy about the
large number of active participants.
Encouragingly, also major microcontroller vendors who are seeking good posi-
tioning in the robotics and autonomous systems market recognize the importance
of micro-ROS. For example, in September 2021, Renesas Electronics Corporation
integrated micro-ROS into the EK RA6M5 family and e2 studio.48 The objective of
this hardware and software combination is to provide a first to market comprehensive
tool based on an Eclipse environment that covers the full development cycle. This
will strongly impact on reducing the time to market of robotic solutions targeting a
wide range of applications in logistics, security, agriculture, or healthcare.
Besides the support of further microcontroller families and RTOS, future tech-
nical topics are of course also on the micro-ROS roadmap. These topics include an
embedded C++ client library and framework mechanisms for the use in wireless
sensors (or even actuators) in the environment.

47https://youtu.be/XsUw1w_f8ks.
48See press release at https://www.renesas.com/eu/en/about/press-room/renesas-and-eprosima-
simplify-development-professional-robotics-applications-ra-mcus-micro-ros.
Micro-ROS 53

Acknowledgements The development of micro-ROS was supported by the European Union’s

Horizon 2020 research and innovation programme under grant agreement No. 780785 in the project
Open Framework for Embedded Robot Applications (OFERA) [17].

References

1. Bouchier, P.: Embedded ROS. IEEE Robot. Autom. Mag. 20(2), 17–19 (2013)
2. Ferguson, M., Bouchier, P., Purvis, M.: rosserial Package Summary. http://wiki.ros.org/
rosserial. Accessed 12 June 2021
3. Takase, H., Mori, T., Takagi, K., Takagi, N.: mROS: a lightweight runtime environment for robot
software components onto embedded devices. In: Proceedings of the 10th International Sym-
posium on Highly-Efficient Accelerators and Reconfigurable Technologies (HEART 2019),
Nagasaki, Japan, ACM (2019)
4. Goldschmidt, S., Ziegelmeier, D.: lwIP - A Lightweight TCP/IP stack - Summary. https://
savannah.nongnu.org/projects/lwip/. Accessed 12 June 2021
5. TOPPERS Project, Inc.: TOPPERS: Toyohashi OPen Platform for Embedded Real-time Sys-
tems. https://www.toppers.jp/en/asp-kernel.html. Accessed 12 June 2021
6. Gerkey, B.: Why ROS 2. http://design.ros2.org/articles/why_ros2.html. Accessed 12 June 2021
Accessed 12 June 2021
7. Dauphin, L., Baccelli, E., Adjih, C.: Riot-ros2: Low-cost robots in iot controlled via
information-centric networking. In: 2018 IFIP/IEEE International Conference on Performance
Evaluation and Modeling in Wired and Wireless Networks (PEMWN) (2018)
8. Dauphin, L., Vilches, V.M.: RIOT-ROS2 (Git Repository). https://github.com/astralien3000/
riot-ros2. Accessed 12 June 2021
9. RIOT developers: RIOT Project Website. https://www.riot-os.org/. Accessed 12 June 2021
10. NDN Consortium: Named Data Networking (NDN) Project Website. https://named-data.net/.
Accessed 12 June 2021
11. OMG and contributors: Data Distribution Service (DDS) 1.4. Standard, Object Management
Group (OMG), Milford, MA, USA (2015)
12. Takase, H., et al.: mROS 2. https://github.com/mROS-base/mros2. Accessed 25 Nov 2021
13. Kampmann, A., Wüstenberg, A., Alrifaee, B., Kowalewski, S.: A portable implementation of
the real-time publish-subscribe protocol for microcontrollers in distributed robotic applications.
In: IEEE Intelligent Transportation Systems Conference (ITSC ’19), pp. 443–448 (2019)
14. Kampmann, A., et al.: embeddedRTPS. https://github.com/embedded-software-laboratory/
embeddedRTPS. Accessed 25 Nov 2021
15. ARM Limited: Mbed OS. https://os.mbed.com/. Accessed 25 Nov 2021
16. OMG and contributors: DDS For Extremely Resource Constrained Environments (DDS-
XRCE) 1.0. Standard, Object Management Group (OMG), Milford, MA, USA (2020)
17. eProsima (project coordinator): OFERA Project: Open Framework for Embedded Robot Appli-
cations. http://www.ofera.eu/. Accessed 12 June 2021
18. Amazon Web Services, Inc.: FreeRTOS Project Website. https://freertos.org/. Accessed 12 June
2021
19. Linux Foundation: Zephyr Project Website. https://www.zephyrproject.org/. Accessed 12 June
2021
20. The Apache Software Foundation: Apache NuttX Project Website. https://nuttx.apache.org/.
Accessed 12 June 2021
21. eProsima: eProsima Micro XRCE-DDS. http://micro-xrce-dds.readthedocs.io/. Accessed
12 June 2021
22. Open Robotics et al.: ROS 2 – Project Governance. https://docs.ros.org/en/rolling/Governance.
html. Accessed 25 Nov 2021
54 K. Belsare et al.

23. Dabrowski, A., Lütkebohle, I., Lange, R., Losa, J.M., Gamarra, B.O., Goenaga, I.M., Vilches,
V.M.: Reference Scenarios and Technical System Requirements Definition. H2020 project
deliverable, OFERA Project (2018)
24. IEEE-SA Standards Board: IEEE/Open Group 1003.1-2017 - IEEE Standard for Information
Technology–Portable Operating System Interface (POSIX(TM)) Base Specifications, Issue 7.
Technical report, IEEE and Open Group (2017)
25. OMG and contributors: DDS Security (DDS-SECURITY) 1.1. Standard, Object Management
Group (OMG), Milford, MA, USA (2018)
26. Casini, D., Blaß, T., Lütkebohle, I., Brandenburg, B.B.: Response-time analysis of ROS 2
processing chains under reservation-based scheduling (Artifact). Dagstuhl Artifacts Ser. 5(1),
5:1–5:2 (2019)
27. Staschulat, J., Lütkebohle, I., Lange, R.: The rclc executor: Domain-specific deterministic
scheduling mechanisms for ros applications on microcontrollers (work-in-progress). In: Pro-
ceedings of the International Conference on Embedded Software (EMSOFT), Virtual event,
IEEE, pp. 18–19 (2020)
28. Staschulat, J., Lange, R., Dasari, D.N.: Budget-based real-time executor for micro-ros (2021)
29. Nordmann, A., Lange, R., Rico, F.M.: System modes - digestible system (re-)configuration
for robotics. In: Proceedings of 3rd International Workshop on Robotics Software Engineering
(RoSE), Virtual event (2021)
30. AUTOSAR Partners: AUTOSAR Classic Platform 4.4.0 – Specification of Operating System.
Standard, AUTOSAR GbR, Munich, Germany (2018)

Kaiwalya Belsare holds a master’s degree in information technology with specialization in

embedded systems and microelectronics from the University of Stuttgart, Germany. During his
tenure as a master thesis student at Bosch Research, he worked on integrating the micro-ROS
stack on an AUTOSAR-based electronic control unit, especially regarding freedom from interfer-
ence with existing safety-critical software functions. Currently, he works as an embedded software
developer at Robert Bosch Power Tools GmbH, Germany.

Antonio Cuadros Rodriguez is a B.Eng. in Electronics, Robotics, and Mechatronics specialized

in embedded systems. He is part of the micro-ROS team in eProsima, responsible for maintaining
and evolving this framework to facilitate and enchance ROS 2 functionality in mid-range MCUs
and embedded platforms with RTOS such as FreeRTOS, Zephyr or in bare metal.

Pablo Garrido-Sanchez is a M.Sc.Eng. Telecommunication Engineer specialized in embed-

ded systems, digital hardware design, and low-level programming. He is the Technical Leader of
micro-ROS in eProsima, responsible for maintaining and evolving this framework to facilitate and
enhance ROS 2 functionality in mid-range MCUs and embedded platforms with RTOS such as
FreeRTOS, Zephyr or in bare metal.

Juanjo Hierro is the CTO of the FIWARE Foundation. He obtained a degree in Computer Sci-
ence in 1990 from the Universidad Politecnica de Madrid (UPM) and owns a certificate of research
proficiency. Juanjo Hierro also plays the role of Chairman of the Technical Steering Committee
of the FIWARE Open Source Community and is the Chief architect of the FIWARE platform, a
growing open de-facto standard for the development of Smart Applications in multiple vertical
sectors.

Tomasz Kołcon is a M.Sc.Eng. Electronics Engineer specialized in embedded systems, digital

and analog hardware design, and low-level programming. He is the Project Manager of micro-
ROS in Łukasiewicz-PIAP, responsible for managing the team dealing with the preparation of
micro-ROS benchmark tools and one of the most complex use-cases in the project.
Micro-ROS 55

Ralph Lange joined Bosch Research in 2013 and is Chief Expert for Robotic Systems and Soft-
ware Engineering. He was principal investigator in the EU-funded project OFERA, which has ini-
tiated the open-source project micro-ROS. Ralph actively follows and contributes to the develop-
ment of ROS 2. Before joining Bosch, he worked as a software engineer for TRUMPF Machine
Tools. He obtained his Ph.D. in computer science from the University of Stuttgart, Germany in
2010.

Ingo Lütkebohle (computer science Ph.D., Bielefeld University) is a software architect at Bosch
Research, working on consumer robotics since 2014. He was co-PI for the EU-funded OFERA
project, working on micro-ROS until early 2020, an active member of the ROS community since
2010 and worked on robot software integration and quality since 1995, in several publicly funded
projects.

Alexandre Malki coming from a M.Sc.Eng. of Mechatronic specialized in microsystems. And

working as software embedded developer engineer at Łukasiewicz-PIAP, in charge of developing
micro-ROS benchmark tools and developing one of the most complex use-cases in the project.

Jaime Martin Losa is the eProsima CEO, with 21 years of experience in project management
and software development focused on networking middleware and integration services. He is the
Architect for all the company products, including eProsima Fast DDS and eProsima RPC over
DDS. Jaime leads the eProsima OMG membership, and is the responsible of eProsima contribu-
tion to the DDS standards, as the Web-enabled DDS, DDS Security, RPC over DDS, OPC-UA
DDS Bridge and DDS-XRCE.

Francisco Melendez is the FIWARE’s Technical Expert in charge of the Robotics Roadmap. He
is also one of the representatives of FIWARE in the Digital Twin Consortium and some interna-
tional projects where FIWARE is developing its technical robotics roadmap. As a mobile robotics
engineer and researcher with more than seven years of experience, his work focuses on the devel-
opment of convenient interfaces between physical robots and digital twins in smart robotics appli-
cations.

Maria Merlan Rodriguez is Ph.D. Telecom Engineer by the University Pierre et Marie Curie.
Solid scientific-technological background, specialized in the field of wireless communications.
Her main field of research is documented in scientific publications focused on the field of Radio
Frequency contactless techniques and machine learning techniques to monitor the hydro-
mechanical properties of new materials. Maria holds 10 years of experience in Technical Projects
and International Business Development.

Arne Nordmann joined Bosch Research in 2015 after obtaining his Ph.D. in model-driven engi-
neering in robotics from the Bielefeld University, Germany. He is now focusing on dependable
cyber-physical systems, such as robotics and highly-automated driving applications, and actively
following and contributing to the development of ROS 2.

Jan Staschulat is Ph.D. Computer Scientist by Braunschweig Technical University in the field
of performance analysis of real-time embedded systems. He joined Bosch Research in 2014 with
main research focus on deterministic and real-time computing in robotics and
autonomous driving. He activly contributes to the ROS 2 Real-Time Working Group.

Julian von Mendel joined Bosch Rexroth Mobile Electronics in 2018 and is project manager
for safety-critical robotics controller software as well as for Rexroth’s AUTOSAR cloud platform.
Before joining Bosch Rexroth, he was CEO at an embedded electronics startup and worked as a
software engineer at cibite AG, developing Linux and microcontroller solutions for optical inspec-
tion systems. He graduated 2014 in computer science (B.Sc).
A Tutorial to Use the MediaPipe
Framework with ROS2

Dieisson Martinelli, Jonathan Cerbaro, Marco Antonio Simoes Teixeira,

João Alberto Fabro, and André Schneider de Oliveira

Abstract This chapter presents a tutorial on the use of the MediaPipe package for
ROS2, developed by the authors. Through the MediaPipe framework it is possible to
collect data from the positions of the hand, face and body of an operator, using a web-
cam, and publish them in ROS2 topics, which can be used for various applications
in robotics. A complete example application using this technology is also covered
in this chapter. MediaPipe is used to collect data from the image of an operator’s
pose (hands, fingers, face and body positions), and this information is then used to
control the navigation of a robot inside a simulated environment. The programming
language used is Python and the only hardware needed for execution is a computer
with Linux Ubuntu 20.04, ROS2 Galactic and a webcam. At the end of the chapter,
the reader will be able to develop control strategies or other functionalities that use
information about the movements of the operator, captured directly by the developed
ROS2 package, through a webcam. All files used in this chapter can be accessed
from the link https://github.com/dmartinelli1997/media_pipe_ros2, as well as the
project videos. All code used in this project can be found in the following reposi-
tory: https://github.com/dmartinelli1997/Script_ROSBOOK2021.git. (This project
was partially funded by National Counsil of Technological and Scientific Develop-
ment of Brazil (CNPq), and by Coordination for the Improvement of Higher Level

D. Martinelli (B) · J. Cerbaro · M. A. S. Teixeira · J. A. Fabro · A. Schneider de Oliveira

Graduate Program in Electrical and Computer Engineering (CPGEI), Federal University of
Technology, Paraná (UTFPR), Av. Sete de Setembro, Curitiba, PR 3165, Brazil
e-mail: [email protected]
J. Cerbaro
e-mail: [email protected]
J. A. Fabro
e-mail: [email protected]
A. Schneider de Oliveira
e-mail: [email protected]
URL: http://www.utfpr.edu.br/

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 57

A. Koubaa (ed.), Robot Operating System (ROS),
Studies in Computational Intelligence 1051,
https://doi.org/10.1007/978-3-031-09062-2_1
58 D. Martinelli et al.

People (CAPES).) (This study was financed by the Human Resource Program of
The Brazilian National Agency for Petroleum, Natural Gas, and Biofuels—PRH-
ANP—FINEP Management, contractual instrument n◦ 0.1.19.0240.00, Ref 0431/19
- PRH21-UTFPR.)

Keywords MediaPipe · ROS2 · Teleoperation · Human-Robot interface

1 Introduction

The purpose of this tutorial is to present a tutorial on the use of a package developed
to use the features of the MediaPipe framework for ROS2. This tutorial includes the
installation and an example of application that uses the developed ROS2 package, by
controlling the navigation of a mobile robot using gestures. The entire tutorial was
developed in Ubuntu version 20.04.2 LTS, using Python 3, MediaPipe version 0.8.9
and the Galactic version of ROS2. Such computer must have a webcam installed,
turned toward the operator. A computer without an accelerated video card was used
to demonstrate that, even without great graphics processing capacity, MediaPipe
can perform tasks that require a robust hardware in other technologies. This tutorial
can serve as a base for creating other human-robot interaction techniques or other
functionalities in robots equipped with ROS2.
For a case study, this tutorial will cover the installation of the MediaPipe frame-
work, an example of the use of the without framework ROS2, installation and expla-
nation of the developed MediaPipe ROS2 package, installation of the simulation
environment of the mobile robot used in the example application, and the develop-
ment of this application itself.
The simulated environment used for testing is composed of the Gazebo simulator
version 11.5.1 with a TurtleBot3 robot.
This chapter is split in 5 sections: Introduction, MediaPipe Framework, Medi-
aPipe for ROS2, Example Application and Conclusion. After this brief introduction
(Sect. 1), Sect. 2 presents background information about the MediaPipe framework
from Google, and the OpenPose library. In Sect. 3, the MediaPipe for ROS2 package
is presented, covering installation of the package, and its integration with Python.
Section 4 presents the complete development of an example application for ROS2
using the proposed package. Section 5 presents the conclusions and the final obser-
vations of the obtained results.

2 The MediaPipe Framework

This chapter is intended to present a package that allows for the use of the Medi-
aPipe framework [1] in ROS2. MediaPipe is an open-source framework developed
by Google for building applied, multimodal and cross-platform machine learning
A Tutorial to Use the MediaPipe Framework with ROS2 59

Table 1 Table of solutions offered by MediaPipe for each platform

Solutions Android iOS C++ Python JS
Face detection x x x x x
Face Mesh x x x x x
Iris x x x
Hands x x x x x
Pose x x x x x
Holistic x x x x x
Selfie segmentation x x x x x
Hair segmentation x x
Object detection x x x
Box tracking x x x
Instant motion tracking x
Objectron x x
KNIFT x
AutoFlip x
MediaSequence x
YouTube 8M x

pipelines that supports IOS, Android, Python, Javascript and C++. MediaPipe offers
several features, such as Face Mesh, Hands, Pose, and Holistic detection, which
were the features used in the developed ROS2 package. The complete list of features
provided by MediaPipe can be seen in Table 1.
In some works, techniques such as detection of hands or pose, connected to a ROS
network, are used to control mobile robots [2–4]. For this end it is common to use
a library called OpenPose, that is the first real-time multi-person system to detect
points of interest in the human body, hands, face and feet, providing a total of 135
points for each image processed [5].
The MediaPipe framework has as its main advantage over OpenPose the pos-
sibility of a faster processing time, spending less processing resources. The main
objective of this chapter is present the MediaPipe ROS2 package, that was developed
specifically to allow the development of applications using some of the MediaPipe
framework features directly in ROS2.

3 MediaPipe ROS2 Package

This section presents the features that were implemented in the MediaPipe for ROS2
package. This section will provide installation instructions for this package, as well
as a detailed description of each of the features implemented, that were selected by
being the most relevant for robotics applications:
60 D. Martinelli et al.

– Hands Detector: Responsible for detecting both hands of the operator informing
the respective points for each one.
– Pose Detector: Detects the operator’s body points such as torso, arms, legs and
feet.
– Face Mesh: Maps all respective points on the operator’s face.
– Holistic: A combination of all the above functionalities at the same time, having
a complete profile of the operator points.

3.1 MediaPipe ROS2 Installation

To use the ROS package, first it is necessary to install MediaPipe. In this work it is
used the Python language version 3.8.10 in Ubuntu 20.04.2 LTS. MediaPipe offers
built-in and customizable Python solutions and is available on PyPI for Linux, macOS
and Windows.
To install MediaPipe, first you need to install OpenCV, as it is a dependency. To
do this, run the following command:
$ pip i n s t a l l opencv - p y t h o n

OpenCV (Open Source Computer Vision Library) is an open-source, cross-

platform computer vision library. It was developed in the mid 2000s by Intel [6].
It brings together all the resources necessary run the most varied computer vision
applications, such as MediaPipe.
After installing OpenCV, run the following command to install MediaPipe:
$ pip i n s t a l l m e d i a p i p e

Now the package is ready to use. You can quickly validate the installation using
one of the examples provided by the documentation itself, such as the hands detector.
Create a folder called MediaPipe and a file called hands.py inside it:
$ mkdir m e d i a p i p e && cd m e d i a p i p e
$ gedit hands . py

Inside this file, add the following code, which can be found in the hands.py
file in the following repository: https://github.com/dmartinelli1997/
Script_ROSBOOK2021.
A Tutorial to Use the MediaPipe Framework with ROS2 61

import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_hands = mp.solutions.hands
drawing_styles = mp.solutions.drawing_styles

cap = cv2.VideoCapture(0)
with mp_hands.Hands(
min_detection_confidence=0.5,
min_tracking_confidence=0.5) as hands:
while cap.isOpened():
success, image = cap.read()
if not success:
print("Ignoring empty camera frame.")
# If loading a video, use 'break' instead of 'continue'.
continue

# Flip the image horizontally for a later selfie-view display,

# and convert the BGR image to RGB.
image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
# To improve performance, optionally mark the image as not writeable to
# pass by reference.
image.flags.writeable = False
results = hands.process(image)

# Draw the hand annotations on the image.

image.flags.writeable = True
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
if results.multi_hand_landmarks:
for hand_landmarks in results.multi_hand_landmarks:
mp_drawing.draw_landmarks(
image, hand_landmarks, mp_hands.HAND_CONNECTIONS,
drawing_styles.get_default_hand_landmark_style(),
drawing_styles.get_default_hand_connection_style())
cv2.imshow('MediaPipe Hands', image)
if cv2.waitKey(5) & 0xFF == 27:
break
cap.release()

Make sure your webcam is connected and its reference name is correctly indicated
in the above code (in this case, the camera was connected to
dev
video0):
cap = cv2 . V i d e o C a p t u r e ( " n u m b e r of y o u r w e b c a m in \ dev \ v i d e o " )

After that, a window with the camera image will open and it will be possible to
observe the MediaPipe working (as can be seen in Fig. 1) when you show your hands,
validating the installation of MediaPipe on your machine. Other examples of imple-
mentations such as pose, face and object detection can be followed in MediaPipe’s
webpage [7].
At this point, MediaPipe is working but it is not communicating with ROS yet, so
now you need to install the package developed for this chapter. The package is found
in the media_pipe_ros2 repository on Github [7]. Just clone the repository into your
ROS2 workspace /src directory and then run the build command from your ROS2
workspace:
62 D. Martinelli et al.

Fig. 1 Image representing the MediaPipe Hands output

$cd ros2_workspace/src
$git clone \url{https://github.com/dmartinelli1997/media_pipe_ros2}
$cd ..
$colcon build

The MediaPipe package for ROS2 is now installed and ready to use. It is currently
possible to use four functionalities: Pose detection, Hands Detection, Face Mesh and
Holistic. Each of these features will be covered in the next subsections.
Development of the MediaPipe ROS2 package was all done in Python3. Medi-
aPipe’s own framework offers the position information of each point on the human
body through an RGB image. This information was captured and sent to an outline
for use through ROS2.
In order for all this to work, some ROS messages were created. Messages are the
data read and written. When posting or subscribing to a topic, you must specify which
type of message will be used, that is, which type of data will be written or read. For
our development it was only necessary the information of x, y and z of the position
of the points and an identification name of the point. Then the messages, HandPoint,
A Tutorial to Use the MediaPipe Framework with ROS2 63

Table 2 Example of message created for writing and reading in MediaPipe package for ROS2
HandPoint.msg
String Name
Float64 x
Float64 y
Float64 z

Table 3 Using example from previous message as a list in another message

MediaPipeHumanHand.msg
HandPoint [21] right_hand_key_points
HandPoint [21] left_hand_key_points

FaceMeshPoint and PosePoint were created, representing each of the points of its
segment. An example of one of these messages can be seen in the Table 2.
Other messages like MediaPipeHumanHandList.msg were created to have a com-
plete list of all the points in one of the segments, for example, from the operator’s
hand, facilitating its use in a ROS2 node. Its structure can be seen in Table 3.
Then, MediaPipe’s ROS2 package uses the position data of a segment’s points
(operator’s hands), creates a topic with the message type referring to that segment
(MediaPipeHumanHandList.msg), and publishes the information of the points in it.
Now, we can create a node in ROS2 capable of reading this topic and using these
points for some application.

3.2 MediaPipe ROS2 Hands Detector

Hands Detector is the node responsible for collecting data from the operator’s hand
and sending it to a ROS topic.
MediaPipe Hands is a high fidelity hand and finger tracking solution. It employs
machine learning to infer 21 3D points of each hand from just a single frame, as seen
in Fig. 2.
To execute this node, run the following commands:
$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 run m e d i a _ p i p e _ r o s 2 h a n d s _ d e t e c t o r

A window will open showing the image of your webcam with the MediaPipe
Hands already working, as in Fig. 3.
You can view the topic responsible for the messages sent from the hands_detector
node. To do this open another bash and run:

$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 topic list
64 D. Martinelli et al.

Fig. 2 Representation of 21 points of hands landmark. Source MediaPipe [1]

Fig. 3 Image representing the ROS2 MediaPipe Hands image output

A Tutorial to Use the MediaPipe Framework with ROS2 65

The topic that should appear is “mediapipe/human_hand_list”. In this message

there is a field responsible for informing the number of people on the image, which
is manually defined by default as always being one person, and a hand list for right
and left hands.
Each hand descriptor has 21 points in total, containing the point name and values
for x, y, and z positions. The values for x and y are normalized to the interval
[0.0, 1.0] with respect to the width and height of the camera image. The value for z
represents the depth of the reference point at the wrist, and the smaller the value, the
closer the reference point is to the camera. The magnitude of z uses approximately
the same scale as x. If hands are not identified, the fields will have the value 0.

3.3 MediaPipe ROS2 Pose Detector

Pose Detector is the node responsible for collecting data from the operator’s body
and sending it to a ROS topic. MediaPipe Pose is an ML solution for high-fidelity
full-body pose tracking, inferring 33 3D reference points across the entire body, as
seen in Fig. 4.

Fig. 4 The 33 pose landmarks

66 D. Martinelli et al.

Fig. 5 Image representing the MediaPipe Pose output

The execution of this node is similar to the previous one, just run the following
commands:
$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 run m e d i a _ p i p e _ r o s 2 p o s e _ d e t e c t o r

A window will open showing the image of your webcam with MediaPipe Pose
already running, as in Fig. 5.
It is possible to view the topic responsible for the messages of the node pose_
detector. To do this, open another bash and run:
$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 topic list

The topic that will appear now is “mediapipe/human_pose_list”. The message

structure of this topic is composed of a field responsible for informing the number
of people on the image and a list of body points. It has 33 points in total, containing
the name of the point and the x, y and z values, following the same pattern as the
previous topic.
A Tutorial to Use the MediaPipe Framework with ROS2 67

Fig. 6 Image representing the MediaPipe Face Mesh output

3.4 MediaPipe ROS2 Face Mesh

Face Mesh is the node responsible for collecting data from the operator’s face and
sending it to a ROS topic. MediaPipe Face Mesh is a facial geometry solution that
estimates 468 3D reference points. It employs machine learning to infer 3D surface
geometry, not requiring a depth sensor.
The execution is done through the following commands:
$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 run m e d i a _ p i p e _ r o s 2 f a c e _ m e s h _ d e t e c t o r

As before, a window will open showing the image of your webcam with the
MediaPipe Face Mesh already working, as in Fig. 6.
It is possible to view the topic responsible for the messages of the node face_mesh_
detector. To do this, open another bash terminal and run:
$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 topic list

The topic that will appear this time is “mediapipe/human_face_mesh_list”. The

message structure of this topic is composed of a field responsible for informing the
number of people on the image and a list of face points. It has 468 points in total,
containing values of x, y and z.
68 D. Martinelli et al.

Fig. 7 Image representing the MediaPipe Holistic output

3.5 MediaPipe ROS2 Holistic

Holistic is the node responsible for collecting the operator’s hand, face and pose
data and sending them to a ROS topic. MediaPipe Holistic is the combination of
the three features previously explained. It has a total of 543 points. MediaPipe’s
holistic pipeline integrates separate models for pose, face and hand components,
each optimized for its specific domain.
The execution is done through the following commands:
$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 run m e d i a _ p i p e _ r o s 2 h o l i s t i c _ d e t e c t o r

A window will open this time showing the image of your webcam with MediaPipe
Holistc already running, as shown in Fig. 7.
At this point, all functionalities are running simultaneously and working with-
out problems. It is possible to view the topic responsible for the messages of the
holistic_detector node. To do this open another bash terminal and run:
$ s o u r c e r o s 2 _ w o r k s p a c e / i n s t a l l / setup . bash
$ ros2 topic list

The topic that will appear this time is “mediapipe/human_holistic_list”. The mes-
sage structure of this topic is composed of a field responsible for informing the
number of people on the image and a list of points on the face, body and a list of
A Tutorial to Use the MediaPipe Framework with ROS2 69

hands containing right and left hands, containing values of x, y and z, and name of
the points for each of the cases.

4 Example Application

With the MediaPipe ROS package it is possible to run several applications, from
autonomous robot control to security applications, such as solutions for human
behaviour understanding [8]. For validation, the functionality of Hands Detector
will be used to control a mobile robot. All development will be explained in this
section. An application that uses OpenPose to control a mobile robot is used as a
reference for the development, which in the end should work similarly.

4.1 Mobile Robot

In order to perform this task, you will first need a mobile robot. For this, a simulation
in Gazebo can be used. In order to test the application, the TurtleBot 3 simulated
robot was chosen, since it has what it takes to develop this challenge. Gazebo [9]
is a powerful robotics simulator that calculates physics, generates sensor data and
provides convenient interfaces to ROS2.
To install and run the gazebo simulator, just run the following commands and the
simulator window should appear, as shown in Fig. 8:
$ sudo apt i n s t a l l ros - < distro > - gazebo - ros - pkgs
$ gazebo

In our installation, we replaced < distr o > by galactic.

Now let’s install TurtleBot 3. TurtleBot is a standard ROS platform robot. It is
a small, affordable, programmable mobile robot for use in education and research
[10].
For its installation we will have to download the source files, compile and install
the software:
$ sudo apt i n s t a l l ros -( d i s t r o ) - turtlebot3 -*
$ mkdir -p ~ / t u r t l e b o t 3 _ w s / src
$ cd ~ / t u r t l e b o t 3 _ w s / src
$ git clone -b galactic - devel https : // g i t h u b . com / ROBOTIS - GIT /
t u r t l e b o t 3 _ s i m u l a t i o n s . git

Now that the source code has been downloaded from the repository, compile it
and provide the ROS configuration file to add the newly installed executables to the
environment:
$cd
$cd turtlebot3_ws/
$colcon build
$echo 'source ~/turtlebot3_ws/install/setup.bash' >> ~/.bashrc
70 D. Martinelli et al.

Fig. 8 Gazebo simulator graphical interface

Two environment variables must also be set. GAZEBO_MODEL_PATH variable

to allow Gazebo to find TurtleBot model definition files and TURTLEBOT3_MODEL
as waffle_pi to simulate TurtleBot3 Waffle Pi. After adding these lines to the .bashrc
file, create the source file.
$echo 'export GAZEBO_MODEL_PATH=$GAZEBO_MODEL_PATH:~/turtlebot3_ws/src/
turtlebot3/turtlebot3_simulations/turtlebot3_gazebo/models' >> ~/.
bashrc

$echo 'export TURTLEBOT3_MODEL=waffle_pi' >> ~/.bashrc

$source ~/.bashrc

Now, all the software is installed and configured and everything is ready to run
the simulation. Just open a terminal and run:
$ r o s 2 l a u n c h t u r t l e b o t 3 _ g a z e b o e m p t y _ w o r l d . l a u n c h . py

The Gazebo simulator should open with the TurtleBot3 simulated model, as shown
in Fig. 9, and we will have access to all the mobile robot control topics.
For our application, the ROS topic of interest is “/cmd_vel”. This topic is respon-
sible for sending angular and linear velocities to the robot. You can test the operation
using the following command:
$ ros2 topic pub -- once / c m d _ v e l g e o m e t r y _ m s g s / msg / Twist " {
l i n e a r : { x : 2.0 , y : 0.0 , z : 0.0} , a n g u l a r : { x : 0.0 , y :
0.0 , z : 1.8}} "

In Gazebo, you can see TurtleBot moving in a circle. Now with everything set up
and running it is possible to develop the control strategy.
A Tutorial to Use the MediaPipe Framework with ROS2 71

Fig. 9 TurtleBot3 simulated on Gazebo

4.2 Control Strategy

The strategy will use the hand detector functionality for this example. The control
idea is based on the work of Martinelli [4], in which the authors use OpenPose to
obtain data from the operator’s hand, as can be seen in Fig. 10.
The strategy addressed by the authors is to use the right hand to control linear
velocity and the left hand to control angular velocity. For this, it is calculated the angle
difference between the points six and eight of the index finger on both left and right
hand. Whenever the finger is moved, an angle between the two points is generated

Fig. 10 Image resulting

from applying OpenPose to
an RGB image. Source
Martinelli [4]
72 D. Martinelli et al.

Fig. 11 Left hand, used for linear velocity control. The velocity is proportional to the angle, the
closer to 180 degrees, the closer to 1 m/s the velocity will be. The closer to 0 degrees, the closer to
−1 m/s the velocity will be. Source Martinelli [4]

and a linear and/or angular velocity is sent. The angular difference is calculated by
the function atan2. Equation 1 represents the use of the function to obtain the angular
difference between points six and eight.

α = atan2((P[6]i − p[8]i ), (P[6] j − p[8] j )) (1)

As can be seen in Fig. 11, when the difference in the left hand is greater than 90
degrees the linear velocity will be positive, if it is smaller, it will be negative. The
same is true for the right hand, but the angular velocity is controlled.
It was also implemented a control system to proportionally change the velocity as
the angle difference changes. Equation 2 represents the decision taken by this control
system: ⎧
⎨ 1.5 ∗ α /90, i f α ≥ 95
vel = −1.5 ∗ α /90, i f α ≤ 85 (2)
⎩
0, else

Figure 12 represents the control strategy developed by the authors, in a flowchart

format.
Following these criteria, the operator’s finger movement will move the robot as
in the work mentioned above, but using the MediaPipe for ROS2 package as a way
to capture points in the operator’s hands.
A Tutorial to Use the MediaPipe Framework with ROS2 73

Fig. 12 Flowchart representing the control strategy adopted. Source Martinelli [4]
74 D. Martinelli et al.

4.3 Execution and Results

Following the definitions discussed in the previous subsection, the code developed is
as follows, and can be found in the control_logic.py file in the following repository:
https://github.com/dmartinelli1997/Script_ROSBOOK2021.
import rclpy
from rclpy.node import Node
import math
from media_pipe_ros2_msg.msg import MediaPipeHumanHandList
from geometry_msgs.msg import Twist

global handLeft, handRigth

handLeft = []
handRigth = []

class CmdSub(Node):
global handLeft, handRigth
def __init__(self):
super().__init__('sends_command')
self.publisher_ = self.create_publisher(Twist, '/cmd_vel', 10)
self.subscription = self.create_subscription(
MediaPipeHumanHandList,
'/mediapipe/human_hand_list',
self.listener_callback,
10)
self.subscription

def listener_callback(self, msg):

global linear
global angular
handLeft = []
handRigth = []
cmd_vel = Twist()
if (msg.human_hand_list):
if (msg.human_hand_list.left_hand_key_points):
for i in range(0,len(msg.human_hand_list.left_hand_key_points)
):
handLeft.append(msg.human_hand_list.left_hand_key_points[i
])

if (msg.human_hand_list.right_hand_key_points):
for i in range(0,len(msg.human_hand_list.right_hand_key_points
)):
handRigth.append(msg.human_hand_list.right_hand_key_points
[i])

if (len(handLeft)>0):
if (handLeft[5]):
angle = CalcAngularDif([[handLeft[5].x, handLeft[5].y],[
handLeft[8].x, handLeft[8].y]])
if (angle>=95):
linear = (angle/90)*0.5
elif(angle < 85 and angle != 0.0):
linear = (angle/90)*-1*1.5
elif(angle == 0.0):
linear = 0.0
else:
linear = 0.0

if (len(handRigth)>0):
if (handRigth[5]):
angle = CalcAngularDif([[handRigth[5].x, handRigth[5].y],[
handRigth[8].x, handRigth[8].y]])
A Tutorial to Use the MediaPipe Framework with ROS2 75

if (angle>=95):
angular = (angle/90)*0.5
elif(angle < 85 and angle != 0.0):
angular = (angle/90)*-1*1.5
elif(angle == 0.0):
angular = 0.0
else:
angular = 0.0

cmd_vel.linear.x = linear
cmd_vel.angular.z = angular
self.publisher_.publish(cmd_vel)

def CalcAngularDif(data):
difAngular = math.atan2(data[1][0]-data[0][0], data[1][1]-data[0][1])
difAngular = abs(math.degrees(difAngular))
return (difAngular)

def main(args=None):
rclpy.init(args=args)

cmd_sub = CmdSub()

rclpy.spin(cmd_sub)

cmd_sub.destroy_node()
rclpy.shutdown()

if __name__ == '__main__':
main()

For better understanding, each part of the code will be explained separately. The
first few lines of code import rclpy so its Node class can be used.
import rclpy
from rclpy . n o d e i m p o r t Node

The next line imports the math library that provides the a tan2 function.
import math

Then, the structure of the node that will read the hand points is imported. The
Twist message, where the velocity commands for the robot will be published, is also
imported.
from m e d i a _ p i p e _ r o s 2 _ m s g . msg i m p o r t MediaPipeHumanHandList
from g e o m e t r y _ m s g s . msg i m p o r t Twist

Next, the CmdSub class is created, which inherits from (or is a subclass of) Node.

class CmdSub ( Node ) :

Inside the CmdSub class, a publisher (cmd_vel) and a subscriber

(/mediapipe/human_hand_list) are created. The subscriber has as a callback func-
tion listener_callback where all the control strategy is developed, and the resulting
command to the robot is published in the cmd_vel topic.
76 D. Martinelli et al.

def __init__(self):
super().__init__('sends_command')
self.publisher_ = self.create_publisher(Twist, '/cmd_vel', 10)
self.subscription = self.create_subscription(
MediaPipeHumanHandList,
'/mediapipe/human_hand_list',
self.listener_callback,
10)
self.subscription

def listener_callback(self, msg):

global linear
global angular
handLeft = []
handRigth = []
cmd_vel = Twist()
if (msg.human_hand_list):
if (msg.human_hand_list.left_hand_key_points):
for i in range(0,len(msg.human_hand_list.left_hand_key_points)):
handLeft.append(msg.human_hand_list.left_hand_key_points[i])

if (msg.human_hand_list.right_hand_key_points):
for i in range(0,len(msg.human_hand_list.right_hand_key_points)):
handRigth.append(msg.human_hand_list.right_hand_key_points[i])

if (len(handLeft)>0):
if (handLeft[5]):
angle = CalcAngularDif([[handLeft[5].x, handLeft[5].y],[handLeft
[8].x, handLeft[8].y]])
if (angle>=95):
linear = (angle/90)*0.5
elif(angle < 85 and angle != 0.0):
linear = (angle/90)*-1*1.5
elif(angle == 0.0):
linear = 0.0
else:
linear = 0.0

if (len(handRigth)>0):
if (handRigth[5]):
angle = CalcAngularDif([[handRigth[5].x, handRigth[5].y],[
handRigth[8].x, handRigth[8].y]])
if (angle>=95):
angular = (angle/90)*0.5
elif(angle < 85 and angle != 0.0):
angular = (angle/90)*-1*1.5
elif(angle == 0.0):
angular = 0.0
else:
angular = 0.0

cmd_vel.linear.x = linear
cmd_vel.angular.z = angular
self.publisher_.publish(cmd_vel)
A Tutorial to Use the MediaPipe Framework with ROS2 77

The angular difference calculation uses the a tan 2 function to be computed and
uses the abs function to return the absolute result.
def CalcAngularDif(dados):
difAngular = math.atan2(dados[1][0]-dados[0][0], dados[1][1]-dados
[0][1])
difAngular = abs(math.degrees(difAngular))
return (difAngular)

Just add this code in a file called control_hands_strategy.py, for example, and as
soon as you run it will be possible to move the simulated robot.
With the logic developed, simulator installed and MediaPipe ROS2 package con-
figured, it is possible to run all together and get the final result.
bash1 :
$ r o s 2 l a u n c h t u r t l e b o t 3 _ g a z e b o e m p t y _ w o r l d . l a u n c h . py

bash2 :
$ ros2 run m e d i a _ p i p e _ r o s 2 h a n d s _ d e t e c t o r

bash3 :
$ p y t h o n 3 c o n t r o l _ h a n d s _ s t r a t e g y . py

A workspace similar to Fig. 13 will appear. When moving the fingers, the simulated
robot will receive linear and angular velocity commands, causing it to move through
the scenario.

Fig. 13 Final workspace with MediaPipe and Gazebo in operation

78 D. Martinelli et al.

5 Conclusion

The objective of this chapter is to present the MediaPipe for ROS2 package, that is
developed in order to allow the use of the features offered by MediaPipe framework
on ROS2. By presenting a brief tutorial, the use of this package is detailed, with a
complete example application for a simulated mobile robot in Gazebo.
By developing the example application, it is possible to verify that any operator
would be able to control the movements of any robot by simply moving the hand-
s/fingers. In the developed application, it is shown how to control the movement of
a mobile robot in an simulated scenario using the MediaPipe package, using only
body recognition and no need for additional hardware, such as accelerated graphics
cards.
Compared to the similar technology of OpenPose, MediaPipe shows an advantage
due to its processing speed. The entire process was performed on a Core i5 notebook
without an accelerated graphics card. In order to get the same results with OpenPose
and its holistic detector, it is necessary an accelerated graphics card with at least
1.5 GB of RAM [11].
This tutorial presented the MediaPipe for ROS2 package, that makes it easy
to create applications for interfacing with robots by using hand, body or face
movements. All open-source code for this work is available in https://github.com/
dmartinelli1997/media_pipe_ros2. The example application in https://github.com/
dmartinelli1997/Script_ROSBOOK2021.git.

References

1. LLC, G.: Mediapipe (2020). https://google.github.io/mediapipe/

2. Yam-Viramontes, B.A., Mercado-Ravell, D.: Implementation of a natural user interface to
command a drone. In: 2020 International Conference on Unmanned Aircraft Systems (ICUAS),
pp. 1139–1144. IEEE (2020)
3. Vighneshwara, C., Kini, M.S., Shenoy, K.P.: Robot motion control using openpose. In: Smart
Sensors Measurements and Instrumentation, pp. 49–57. Springer (2021)
4. Martinelli, D., Sousa, A.L., Augusto, M.E., Kalempa, V.C., de Oliveira, A.S., Rohrich, R.F.,
Teixeira, M.A.: Remote control for mobile robots using gestures captured by the rgb camera
and recognized by deep learning techniques. In: Latin American Robotics Symposium (LARS),
2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education
(WRE), pp. 98–103. IEEE (2019)
5. Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., Sheikh, Y.: Openpose: realtime multi-person 2d
pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43(1), 172–
186 (2019)
6. Team, O.: Opencv (2021). https://opencv.org/
7. LLC, G.: Mediapipe (2020). https://google.github.io/mediapipe/getting_started/getting_
started.html
8. da Silva Neto, J.G., Teixeira, J.M.X.N., Teichrieb, V.: Analyzing embedded pose estimation
solutions for human behaviour understanding. In: Anais Estendidos do XXII Simpósio de
Realidade Virtual e Aumentada, pp. 30–34. SBC (2020)
9. Foundation, O.S.R.: Gazebosim (2014). http://gazebosim.org/
A Tutorial to Use the MediaPipe Framework with ROS2 79

10. Foundation, O.S.R: Gazebosim (2014). https://www.turtlebot.com/

11. Team, O.: Openposedoc (2021). https://github.com/tramper2/openpose/blob/master/doc/
installation.md

Dieisson Martinelli Received the M.Sc. degree in electrical engineering, concentrate on teleoper-
ation techniques through gesture recognition, from the Federal University of Technology - Paraná
(UTFPR) in 2022, where is today a Ph.D. degree candidate. Bachelor’s degree in Information Sys-
tems from the State University of Santa Catarina (UDESC). His research focuses on using deep
learning techniques for teleoperated controls and mobile robotics.

Jonathan Cerbaro Received the M.Sc. degree in electrical engineering, concentrate on manip-
ulator kinematics techniques, from the Federal University of Technology - Paraná (UTFPR) in
2022, where is today a Ph.D. degree candidate. Electronic Engineer graduated from University of
Caxias do Sul (UCS) in 2020. His area of expertise encompasses control and automation, mobile
robotics and industrial manipulators, with a focus on kinematics and Clifford’s Algebra.

Marco Antonio Simoes received the M.Sc. and D.Sc. degree in Electrical and Computer Engi-
neering from the Federal Universityof Technology—Parana, Brazil. His researchinterests com-
prise mobile robots in the areas of perception and intelligent systems, computervision, navigation,
mapping, and applied AI.

João Alberto Fabro holds a degree in Informatics (bachelor’s degree) from the Federal Univer-
sity of Paraná (UFPR-1994), a master’s degree in Electrical Engineering from the State Univer-
sity of Campinas (UNICAMP-1996), a doctorate in Electrical Engineering and Industrial Infor-
matics from the Federal Technological University of Paraná (UTFPR-2003). He was a professor
at UNIOESTE (State University of Western Paraná) from 1998 to 2007. Since 2008, he is a pro-
fessor at the Federal University of Technology - Paraná, based in the Department of Informatics -
Curitiba. He is an Associate Professor. From 2012 to 2013, he was the Head of the M.Sc. course
on Applied Computing of UTFPR. In 2014, Prof. Dr. Fabro was visiting fellow at the Faculty of
Engineering of the University of Porto, Portugal. Has experience in Computer Science, focusing
on Artificial Intelligence, researching the following subjects: Autonomous Mobile Robotics, Com-
putational Intelligence (Neural Networks, Fuzzy Systems, Evolutionary Computing) and Educa-
tional Robotics. Is the coordinator of Research Groups on Computational Intelligence and the
LASER-Advanced Laboratory of Embedded Systems and Robotics (http://laser.dainf.ct.utfpr.edu.
br), at UTFPR. Is a collaborator of LIACC-Laboratory of Artificial Intelligence and Computer
Science, University of Porto, Portugal (https://liacc.fe.up.pt). Since 2009, he has participated in
several national, Latin American and international robotics championships (Robocup), in robot
soccer and service robot categories (Robocup@Home), both as competitor and organizer.

André Schneider de Oliveira (Member, IEEE) received the M.Sc. degree in mechanical engi-
neering, concentrate in force control of rigid ma-nipulators from the Federal University of Santa-
Catarina (UFSC) in 2007 and the Ph.D. de-gree in engineering of automation and systemsin 2011.
His Ph.D. thesis was on differentialkinematics through dual-quaternions for vehicle-manipulator
systems. He is an Associate Pro-fessor with the Federal University of Technol-ogy-Parana
(UTFPR) and a member of Ad-vanced Laboratory of Robotics and Embedded Systems (LASER)
andLaboratory of Automation and Advanced Control Systems (LASCA). Hisresearch interests lie
in the areas of robotics, mechatronics and automa-tion with particular focus in navigation and
localization of mobile robots,autonomous and intelligent systems, perception and environment
iden-tification, cognition, and deliberative decisions, human-interaction, andnavigation control.
ROS Frameworks and Applications
Autonomous 3D Thermal Mapping
of Disaster Environments for Victims
Detection

Christyan Cruz Ulloa, Guido Torres Llerena, Antonio Barrientos,

and Jaime del Cerro

Abstract Early detection of victims in post-disaster environments is an essential

task for rescue teams, and effective victim identification will save the greatest num-
ber of lives. One of the most significant challenges to carry out these tasks lies mainly
in the harsh conditions typical of post-disaster environments such as low light, vic-
tims trapped both in rubble and by some materials that make their location difficult
(blankets, wood, glass, Etc.) by simple visual inspection. To face this problem, this
Chapter describes a robust method for identifying victims and their location within
a heat map of the analyzed environment, using specific equipment (Thermal Camera
Optris PI640, Microsoft Kinect, and Robot Dr. Jaguar). For this task development, a
novel method developed in ROS is proposed based on the processing of thermal and
depth images, in such a way that the first ones are used to detect victims through a
Convolutional Neural Network and to generate Thermal Local Point Clouds of the
environments in combination with depth images. Iterative Closest Point algorithms
were used for a subsequent complete thermal map concatenation of the 3D environ-
ment using Point Clouds (Thermal Global Point Clouds). The tests to validate this
method were carried out indoors (CAR Arena at Centre for Automation and Robotics)
and outdoors (Robotics and Cybernetic Laboratory), recreating environments with
post-disaster conditions. The main results show an average of 89% in concatenation
efficiency to generate Thermal Global Point Clouds. For different scenarios. On the
other hand, the detection of victims was carried out with an efficiency greater than
90% for the different proposed scenarios.

C. C. Ulloa (B) · G. T. Llerena · A. Barrientos · J. del Cerro

Centro de Automatica y Robotica (UPM-CSIC), Universidad Politecnica de Madrid,
28006 Madrid, Spain
e-mail: [email protected]
G. T. Llerena
e-mail: [email protected]
A. Barrientos
e-mail: [email protected]
J. del Cerro
e-mail: [email protected]
URL: https://www.car.upm-csic.es/

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 83

A. Koubaa (ed.), Robot Operating System (ROS),
Studies in Computational Intelligence 1051,
https://doi.org/10.1007/978-3-031-09062-2_3
84 C. C. Ulloa et al.

Keywords ROS · Thermal image · Point Clouds · Victim identification · 3D

thermal mapping

1 Introduction

Search and rescue robots have had great relevance in recent years because a robot
can enter where a rescuer or a dog cannot and gather valuable information from
its environment to help in rescue tasks. Post-disaster scenarios result from natural
events such as earthquakes, hurricanes, or attacks, completely devastated areas or
cities with totally or partially collapsed buildings, rubble, Etc. They also produce
high economic losses and, above all, the death of people trapped by the structures and
debris generated by these disasters. The confined access spaces generated as tunnels
and structures at risk of collapse pose a very high risk to rescue teams and front-line
equipment during search and rescue missions in post-disaster environments.
Figure 1 shows the increase in publications in the Web Of Science (WOS) with
topics related to search and rescue robotics, from 1999 to 2020, having an increase
of 36% between each year.
Being a topic with great relevance, several competitions seek to reward the best
search and rescue methods through collaborative or individual robots, through a
series of cameras and sensors, to have extensive information on the environment to be
searched. One of the main competitions is the DARPA (Defense Advanced Research
Projects Agency) Subterranean Challenge, Rescue Robot Competitions at the AAAI
(American Association for Artificial Intelligence) and the IJCAI (International Joint
Conference on Artificial Intelligence). The competition scenarios are regulated by
the NIST (National Institute of Standards and Technology) [1]. Where a group of
robots or robots goes into a cave to locate victims in the environment to be scanned.

Fig. 1 Number of publications per year on the web of science WOS. Source Authors
Autonomous 3D Thermal Mapping of Disaster Environments … 85

The mapping of catastrophic environments is a great advantage for the different

rescue groups to locate the different victims, and with this, it is possible to adapt
different types of sensors or cameras, such as RGB, Thermal, Laser cameras, Etc.
The coupling of different sensors and cameras gives better results to know the state of
people. The mapping with thermal cameras has an advantage over the others because
they do not depend on artificial and natural light, and they have different types of
pallets to use in different applications. Different works within the state-of-the-art
highlight the advantages of working with thermal images in search and rescue tasks
[2–5].
This chapter describes the method used to locate victims in disaster areas and its
3D thermal mapping. For this purpose, an infrared Kinect sensor has been used, it
is capable of projecting infrared structured light and capturing environmental dis-
tance information in the form of depth images, which, based on a normalization of
parameters, have been transformed into Point Clouds. Each point within the cloud
corresponds to the point distance of the object concerning the position of the sensor
[6–9].
This sensor has been used mainly in indoor areas for extensive mapping areas. A
3D reconstruction of the environment was carried out using Point Clouds through
thermal images (Th-Im) and depth images (Dp-Im).
A Convolutional Neural Network (CNN) was trained for victims location, carried
out with a datasheet of several disaster areas such as exteriors and interiors to obtain
a better result.
Thermal imaging is performed using an OPTRIS PI690 thermal camera and a
Kinect V1 sensor, mounted on a Dr. JAGUAR V4 mobile robot, said an operator
remotely manipulated robot.
The treatment and generation of the Thermal Local Point Clouds (T-LPC) are done
offline; after a complete scan of the area, the images are processed to generate the
Thermal Global Point Clouds (T-GPC) using Iterative Closest Point (ICP) algorithms.
Once the 3D map has been prepared, and with the data acquired from the neural
network training using thermal images, we obtain the location of the victim and her
representation on the map.
The development of this method involves integrating different fields described
within the state-of-the-art, which involve: 3D mapping, digital processing of thermal
images, victim detection, neural networks, and search and rescue robotics. The main
developments respectively within these fields are highlighted below.
Among the main developments in the processing of neural networks and thermal
images, the approaches in detection of intruders in night environments stand out
[10], comparison of the detection of people indoors and outdoors [11], the influence
of atmospheric weather in the detection of people [12]. On the other hand, other
methods focus on the detection of people using neural networks with images of low
results both thermal and RGB [13–16], together with some developments that they
use drones for this purpose [17].
On the other hand, the mapping techniques commonly used are based on repre-
senting Point Clouds in false color using grayscales or RGB images [18–20, 20, 21].
By processing the captured data, applying surface smoothing techniques [22], ICP
86 C. C. Ulloa et al.

[23], and visual odometry [24]. However, developments within the state-of-the-art
related directly to the proposed method are limited to equipment calibration [25, 26]
and building thermal maps [27–30], but not in generating an integral system that
generates support. For rescue teams in disaster areas, as is done in this chapter.
This chapter is structured as follows. In the first instance, there is the system
overview and the background of the proposed method, to later describe the configu-
ration of the ROS environment, to finally show the results, discussion, conclusions,
and future work.

2 System Overview

One of the main contributions of this chapter stands out in the capability to generate
thermal maps of a disaster environment, detect and locate (locate them in the T-GPC)
victims in a disaster environment. Through thermal images and a previously trained
network, through of the environmental inspection using a robot in the field capable
of transmitting images to a remote station (ERO—Remote Operator Station), the
Robot in this phase of experimentation of the system is teleoperated from a remote
station by an operator or rescuer previously capable.
The transmission of images and processing are carried out in real-time in the
ERO, so that the operator or rescuer can know a priori if there is a victim (which
could be an unconscious person or trapped in rubble) who needs primary assistance)
Furthermore, its location within the environment is based on the position of the
Robot.
This implementation has been completely developed in ROS, which has been used
for its great versatility and ability to integrate different sensory systems, robots and
carry out processes on multiple computers through its node and topic communications
architecture.
The Robot used for exploration in the disaster environment has been the Jaguar
V4, equipped with a thermal camera and a Kinect sensor, the following components
are mounted on the Robot and whose Hardware structure is shown in the Fig. 2:

– Computer Lenovo Flex 5.

– Thermal Camera Optris 640
– Sensor kinect V1

The robot’s movement control in the environment is executed by publishing linear

and angular velocity commands, through the topic /cmd_vel, for its part, the robot
transmits its position within the environment through the topic /position. The flow
of these data is shown in the Fig. 3. The structure of the published topics is:
Autonomous 3D Thermal Mapping of Disaster Environments … 87

Fig. 2 Thermal camera and kinect assembly in Robot Jaguar V4

Fig. 3 Diagram of interconnections of the systems using ROS, used for the detection of victims
and automatic generation of 3D thermal maps
88 C. C. Ulloa et al.

– Speed command transmission to the robot:

/cmd_vel (geometry_msgs/Twist)
linear:
x: 0.1 (vel_lin)
y: 0.0
z: 0.0

angular:
x: 0.0
y: 0.0
z: 0.1 (vel_ang)

– Robot position reception:

/pose (geometry_msgs/Twist)
geometry_msgs/Vector3 linear
float64 x (Position x)
float64 y (Position y)
float64 z

geometry_msgs/Vector3 angular float64 x

float64 y
float64 z (Orientation)

For the capture and subsequent transmission of thermal and depth images from the
Kinect camera and sensor integrated into the robot, drivers for ROS (in the Lenovo
Flex computer) have been installed, developed by the manufacturer, and available at
http://wiki.ros.org/optris_drivers.
The image is transmitted through the topic /image_compressed. The compressed
image has been used, because it increases the fps transmission rate, and the loss of
information is not significant for the development of the process.
The structure of the thermal image topic is:

– Thermal image reception:

/image_compressed (sensor_msgs/Image Message)

uint32 height uint32 width
string encoding (Contains the pixels and channels of the image)
uint8 is_bigendian
uint32 step
uint8[ ] data

For the real-time processing of the image received in the ERO, ROS packets have
been previously installed in the anaconda environment in such a way that with the
Autonomous 3D Thermal Mapping of Disaster Environments … 89

network already trained, inference and detection of people within the received image
can be carried out. as shown in Fig. 3.
An algorithm in python has been developed for artificial vision processing. It
receives as input the image transmitted by the robot through the subscription to the
related topic of ROS. It is processed using OpenCV libraries to apply the respective
corrections to the image, erosion operations, dilation, and noise elimination, to pass
as data to the neural network next stage.
Figure 3 shows in the lower part the CNN architecture, taking the image as input,
features extraction phase is applied through convolutions and pooling phases. At the
end, in the classification phase, the result of the fully connected layers is the victim
location(s), the bounding box for each victim, and the accuracy percentage for each
recognition.
The first convolution stage finds characteristics in the thermal image using detec-
tors. The pooling layer reduces the characteristic maps size, in this phase more
relevant characteristics are preserved.
After a sets of convolution-pooling phases, a flattening operation is applied, con-
verting into a 1D vector the pooling result per thermal image. This new vector serves
as final fully connected layers input, where internal weights are established through
back propagation.
Finally, the inference is made with the trained network, and it is established
whether or not there is a victim or a rescuer in the inspected area. This process runs
cyclically throughout the environment inspection.
The detection algorithm for thermal images victim detection has been imple-
mented in Python, a bounding box mark was established to detected victims and
rescuers on the original image.
Simultaneously the thermal images received are processed together with the depth
images captured by the Kinect, forming several T-LPCs. As the robot moves in the
environment, the T-LPCs are concatenated to form the T-GPCs and locate victims on
the global map, allowing rescuers to know the position of a possible victim within
the environment.

3 Background, Sensors and ROS Configuration

3.1 ROS Packages, Configuration and Data Acquisition

3.1.1 Packages for Thermal Images—Compressed Image

The thermal camera used for this development was the Optris PI640 http://wiki.ros.
org/optris_drivers. The files used for the configuration of the thermal camera used can
be downloaded from the GitHub repository (https://github.com/ChristyanCruz11/
Thermal-3DMapping.git). The same ones that have the structure shown in the Fig. 4
90 C. C. Ulloa et al.

Fig. 4 Calibration and configuration files for the thermal chamber

The following steps must be followed for the installation and launch of the thermal
chamber nodes:
1. sudo apt-get install cmake freeglut3-dev libusb-1.0-0-dev
2. In /src folder: git clone https://github.com/evocortex/optris_drivers
3. catkin_make
4. sudo ir_download_calibration
5. ir_download_calibration -t /tmp -d 18072067.xml
Subsequently its execution is carried out by means of the commands:
1. sudo rmmod uvcvideo
2. sudo modprobe uvcvideo nodrop=1
3. rosrun optris_drivers optris_imager_node_xmlConfig:=/home/explorador/
catkin_ws/src/optris_drivers/config/generic.xml
4. rosrun optris_drivers optris_colorconvert_node _palette:=6 _paletteScaling:=1
_temperatureMin:=20 _temperatureMax:=40
The palette scaling should be considered because it will adapt to the maximum
temperature existing in the environment to maximize the tones and adapt them to it.
The Fig. 5 shows the difference between a uniform pallet through temperature values
(Param: _paletteScaling:=1), with respect to a non-uniform pallet that adapts to the
maximum temperature value (Param: _paletteScaling:=2).
The topics generated after launching these nodes are shown in the Fig. 6, of which
we will work mainly with the compressed image to optimize processing times.
– /thermal_image_view/compressed.

3.1.2 Packages for Kinect—Depth Images

To work with depth images of the Kinect, the Freenect libraries have been used, the
installation and execution of which are detailed below:
1. In /src: git clone https://github.com/OpenKinect/libfreenect.git
2. cd libfreenect
Autonomous 3D Thermal Mapping of Disaster Environments … 91

(a) No uniform palette (b) Uniform palette

(c) No uniform palette (d) Uniform palette

Fig. 5 Differences between uniform and non-uniform temperature pallet

3. mkdir build
4. cd build
5. cmake -L ..
6. make
7. sudo make install
8. sudo apt-get install ros-melodic-freenect-launch
9. roslaunch freenect_launch freenect.launch

3.1.3 Packages for Robot Dr. Jaguar

The files used to launch the .launch file to command the robot are located in the “Robot
Dr. Robot Jaguar Files” folder at: https://github.com/ChristyanCruz11/Thermal-
3DMapping.git. To launch the nodes, execute the commands from the robot’s com-
puter:

1. roscore
2. roslaunch jaguar jaguar

The Fig. 7 shows the topics obtained when launching the .launch file; these topics
will be used to command the robot based on linear and angular speeds, as well as to
move the front and rear fliers through a joystick in a remote station.
92 C. C. Ulloa et al.

Fig. 6 Topics for thermal imaging

Fig. 7 Thermal camera

calibration
Autonomous 3D Thermal Mapping of Disaster Environments … 93

Fig. 8 Master configuration

Fig. 9 Slave configuration

3.1.4 Master—Slave Communication

Master-Slave does the configuration of the system for information management. In

such a way that the most powerful computer that receives all the information and
sends speed commands is configured as a master. By the other hand the robot is
configured as a slave. Figures 8 and 9 show respectively the IP addresses of the
master and slave.

3.2 Equipment Calibration

3.2.1 Thermal Camera

Some cameras introduce significant distortion to images. Two main types of radial
distortion and tangential distortion, Eqs. 1 and 2.
Radial distortion makes straight lines appear curved. Radial distortion increases
as points move away from the center of the image.

xdistor tion = x(1 + k1r 2 + k2 r 4 + k3r 6 ) (1)

ydistor tion = x(1 + k1r 2 + k2 r 4 + k3r 6 ) (2)

Similarly, tangential distortion occurs because the imaging lens is not aligned
perfectly parallel to the image plane. Therefore, some areas of the image may appear
closer than expected. The amount of tangential distortion can be represented by the
Eqs. 3 and 4:
xdistor tion = x + [2 p1 x y + p2 (r 2 + 2x 2 )] (3)

ydistor tion = y + [ p1 (r 2 + 2x 2 ) + 2 p2 x y] (4)

In summary, we need to find five parameters, known as distortion coefficients,

given by the Eq. 5:

Distor tioncoe f f icients = (k1, k2, p1, p2, k3) (5)

94 C. C. Ulloa et al.

Besides this, we need some other information, such as the intrinsic and extrinsic
parameters of the camera (Eq. 6). Intrinsic parameters are specific to a camera. They
include focal length (fx, fy) and optical centers (cx, cy). The focal length and optical
centers can be used to create a camera array, which can be used to eliminate distortion
due to the lenses of a specific camera. The camera matrix is unique to a specific
camera, so once calculated, it can be reused in other images taken by the same
camera. It is expressed as a 3×3 matrix:
⎡ ⎤
f x 0 cx
⎣ 0 f y cy ⎦ (6)
0 0 1

The calibration of the Optris thermal camera uses the [31] method. It consists of
registering the corners of a chessboard and using code, obtaining the fundamen-
tal matrix variables (Eq. 6), vectors of translation and rotation Eq. 5, previously
explained. For this, some sheets of paper with checkerboard-shaped perforations
were heated, and the thermal camera was adjusted on the White-Hot pallet to record
the characteristic points from each corner of the chessboard. Once the calibration
code has been developed, we obtain the desired values, the Fig. 10 shows the edges
of the object to be calibrated (checkerboard), the fundamental matrix of the thermal
chamber (7), distortion vector (8), rotation matrix (9), and the translation matrix (10)
said calibration, a distortion error of 0.38487294716991416.
⎡ ⎤
f x = 458.41626958 0 cx = 327.15150492
⎣ 0 f y = 457.27416629 c y = 213.86162691⎦ (7)
0 0 1
 
−0.63118568 0.89263329 0.02207553 − 0.00361449 − 1.11144398
(8)

Fig. 10 Thermal camera

calibration
Autonomous 3D Thermal Mapping of Disaster Environments … 95
⎡⎡ ⎤⎡ ⎤⎤
−0.01884591 −0.14199782
⎣⎣ 0.05737353 ⎦ ⎣−0.02233268⎦⎦ (9)
−0.40815257 0.02366859
⎡⎡ ⎤⎡ ⎤⎤
−2.68330769 −0.92708789
⎣⎣−1.10260326⎦ ⎣−1.36752316⎦⎦ (10)
12.13103857 9.63877082

3.2.2 Image Rectification

After obtaining the fundamental matrix, distortion vector, translation, and correction,
we obtain the new correction matrix with the cv2.getOptimalNewCameraMatrix
library. From this process, we extract the distortion map for rectification of the image
with cv2.initUndistortRectifyMap with the new rectification matrix and fundamental
matrix of calibration. Once the distortion map is obtained, we project said map with
a test image with cv2.remap (Fig. 11).

3.2.3 Image Preprocessing

Erosion and dilation operations have been applied to pre-process the image trans-
mitted by the robot in the field. These morphological operations have been applied
sequentially. In the first instance, erosion has been applied to erode the limits of the
object in the foreground, eliminating tiny objects that are not of interest and elim-
inating white noise; later, a dilation is applied, which Brings together the broken
parts of an object and accentuates the characteristics of the objects in the image. The
selection of the parameters used was: Erosion (kernel = 5; iterations = 1), Dilation
(kernel = 3; iterations = 1).

3.3 Thermal Image Processing and Transmission Using

OpenCV and ROS

– Installing ROS and optris drivers and plugins

– Generation of initial parameters generiz.xml
– Enable usb port for sudo rmmod uvcvideo thermal camera; sudo modprobe
uvcvideo nodrop = 1
– Launch node with generic parameters rosrun optris drivers optris imager node
xmlConfig:=/home/guido/catkin_ws/src/optris_drivers/config/generic.xml
– Obtaining thermal camera video with Palette Hot heat, without temperature self-
regulation, minimum temperature of 20 and maximum of 40 (rosrun optris drivers
96 C. C. Ulloa et al.

Fig. 11 Calibration process

optris colorconvert node palette:=6 paletteScaling:=1 temperatureMin:=20 tem-

peratureMax:=40)

Figure 12 shows the acquisition of thermal images.

3.4 Integration of Thermal and Depth Camera with ROS

The acquisition of images for the creation of Point Clouds was executed using ROS.
The process to develop this task is detailed below:
Autonomous 3D Thermal Mapping of Disaster Environments … 97

Fig. 12 Thermal image

acquisition

Launch of Nodes for Thermal Camera

1. sudo rmmod uvcvideo; sudo modprobe uvcvideo nodrop=1
Subscribe the thermal camera serial port to COM 1
2. rosrun optris.drivers optris.imager.node .xmlConfig:=
/home/guido/catkin.ws/src/optris.drivers/config/generic.xml
Execute the generic calibration parameters of the thermal chamber generated in
the installation generic.xml
3. rosrun optris.drivers optris.colorconvert.node .palette:=6 .paletteScaling:=1 .tem-
peratureMin:=20 .temperatureMax:=40 Execute the thermal chamber node with
palette 6 (Iron heat), deactivating the temperature auto scaling (paletteSaling: = 1),
minimum temperature of 20 (.temperatureMin: = 20) and maximum temperature
of 40 (.temperatureMax: = 40)
Kinect Sensor Releases
– roslaunch freenect.launch freenect.launch depth.registration:=true
We launch the freenect node of the Kinect sensor and activate the different registers
for depth image.
Image Acquisition
1. Node subscription:
rospy.wait.for.message (‘/thermal.image.view’, Image) thermal camera node
rospy.wait.for.message(‘/camera/depth.registered/hw.registered/image.rect.
raw’, Image) rectified depth chamber node.

2. CV.BRIDGE:
For the bridge between ROS and image processing we use cv.bridge, for ther-
mal images we use RGB format and for depth images passthrough (No image
conversion).
98 C. C. Ulloa et al.

Fig. 13 Thermal and depth image

3. Depth Image Processing:

– Image declaration as floating
We declare the acquired images as floating for post processing (np.array (Ima-
gen.profundidad, dtype=np.float)).
– Depth image normalization
We normalize the images from 0 to 255 for the generation of Point Clouds
cv2.normalize(image.profundidad, image.profundidad, 0,255, cv2.NORM
.MINMAX).
4. Thermal Image Processing:
– New Camera Matrix
Obtain a new optimal thermal camera matrix, using the fundamental matrix
and rotation of the thermal camera, obtained these values in the calibra-
tion of the camera and the dimensions of the image (width and height),
cv2.getOptimalNewCameraMatrix.
– Rectified map without distortion
The new map without distortion is obtained through cv2.initUndistort_ Rec-
tifyMap, through the old fundamental matrix and the new matrix obtained
previously. The same process is carried out with the rotation vector.
– Remapping the image
Projection of the rectified image using the rectified map without distortion
(cv2.remap). The interpolation technique used has been of type Linear through
the python module cv2.INTER_LINEAR.
5. Save Images:
We save the processed images from a thermal camera and a depth camera, using
cv2.imwrite (location where to save the images, the type of image to save) (Fig. 13)

3.5 Neural Network for Victims Identification

A Convolutional Neural Network has been trained to detect victims within the
explored environment, using as input the thermal images transmitted by the robot.
This subsection details the process carried out for automatic victim detection.
Autonomous 3D Thermal Mapping of Disaster Environments … 99

3.5.1 Datasets Generation

The dataset generation has been carried out by extracting video frames captured
from the environment with the robot in a previous stage, choosing the images where
there are victims and rescuers. The robot moves in a teleoperated way, in different
scenarios, capturing the videos. The result of the extracted images is 1280 × 720
px.
The training database is 1747 images, with 7 labels:
1. Rescuer body
2. Rescuer legs
3. Victim body
4. Victim arm
5. Victim torso
6. Victim legs
7. Victim head
The validation of the network was given using a series of images taken after the
network’s training, with 2000 images.

3.5.2 Data Augmentation to Increase the Neural Network Robustness

To increase the neural network robustness, a series of treatments were applied to

these images such as Increase in brightness, Horizontal Reflection and Increase in
contrast, Gaussian Noise, as shown in the Figs. 14 and 15. Specific parameters used
for Data Augmentation in Gaussian Noise are:

– gauss = np.random.nor mal(mean, sigma, (r ow_img, col_img, ch_img)

– gauss = np.random.nor mal(0, 0.1, (720, 1280, 3))

Thus we increased our database, and the Recognition of victims can be in envi-
ronments with different environments. This image was taken at the Robotics and
Cybernetics (RobCib) Laboratory.
Figure 14 shows a couple of rescuers, for which the same image treatment pro-
cedure is performed to increase the database and thus improve the validation of the
work in different environments.

3.5.3 Neural Network Training and Hyper-Parameters

The neural network architecture consists of 24 Convolutional layers and two fully
connected layers, receives the image (1280 × 720 px) transmitted by the robot
remotely and has as output one of the raised labels, the detection percentage and the
location bounding box. The training has been carried out for 130 epochs, each epoch
with 290 batches with a batch size of 6 images, resulting in 37700 iterations.
100 C. C. Ulloa et al.

(a) Original Image (b) Gaussian Noise

(c) Brightness increase (d) Horizontal reflection

(e) Vertical reflection (f) Contrast increase

Fig. 14 Sample of the application of data augmentation to a thermal image of a rescuer

At the end of each training season, the adjustment weights of the network model
are stored, so based on the analysis of the most optimal season, the final model of
the network will be adjusted.
The neural network training has been carried out on a PC, MSI model, 16GB
in RAM, 1660Ti GPU. The approximate time for training the network was 6 h.
TensorFlow libraries have been used for this section development.
The model’s effectiveness has been analyzed in terms of Accuracy and Loss. The
results of the training and validation of the neural network model are shown below.
Autonomous 3D Thermal Mapping of Disaster Environments … 101

(a) Original Image (b) Gaussian Noise

(c) Brightness increase (d) Horizontal reflection

(e) Vertical reflection (f) Contrast increase

Fig. 15 Application of data augmentation to a victim thermal image

Figure 16 represents the Loss curve for training. This allows us to know how
far apart are the predictions of a model and its labels. Each time it decreases, its
performance improves. On the other hand, Fig. 17 shows the behaviour of the model
during the validation stage. It is observed that the error decreases uniformly until
time 107; from then on, there is no significant improvement, therefore which time
has been considered optimal to adopt the adjustment weights of the CNN in the
model.
102 C. C. Ulloa et al.

Fig. 16 Loss Training for 130 epochs (38 k Iterations). Source Authors

Fig. 17 Loss Validation Curve for 130 epochs. From epoch 107 on, the Loss value does not show a
downward trend, therefore, at this time of training, the values for the model have been taken. Source
Authors

Figure 18 shows the precision curve of our model, which increases as the training
time continues. Figure 19 shows the validation of the network model after the end of
each epoch. The curve shows that around epoch 107, there is no longer a significant
improvement in the training result, so that It has been considered the best option to
adjust the weights of the network based on those obtained at that time.
Autonomous 3D Thermal Mapping of Disaster Environments … 103

Fig. 18 Accuracy Training for 130 epochs (38 k Iterations). Source Authors

Fig. 19 Accuracy Validation Curve for 130 epochs. From epoch 107 on, the precision value does
not show a growth trend, therefore, at this time of training, the values for the model have been taken.
Source Authors

3.6 System Requirements for Execution

The following parameters are required To carry out the automatic mapping in order
to be established in such a way that the error is minimized when generating local
maps, and the global map, the requirements established based on experimentation
are shown below:
104 C. C. Ulloa et al.

3.6.1 Robot Mobility

The Table 1 describes the linear and angular velocity, and the angle of rotation in
which the robot was moving
The minimum rotation angle of the robot is 30 degrees. In the Fig. 20 shown the
acquisition of images from the thermal camera while the robot rotates.

Table 1 Robot mobility

Turning angle Angular velocity Linear velocity
(rad/seg) (ms)
30 0.15 0.2

Fig. 20 Robot rotation to capture depth images and thermal images (CAR Arena)
Autonomous 3D Thermal Mapping of Disaster Environments … 105

3.6.2 Image Acquisition

The acquisition of depth images have certain limitations:

– Shooting speed
– Image loss
– Variation of images in different positions
The images were taken every second and it was normalized from 0 to 255, but due
to the speed of image processing we have data loss in each image. The solution was
to choose the best image for each position and process it to do the job.

4 Autonomous 3D Thermal Mapping and Victims

Identification

4.1 Point Clouds Acquisition and Processing

The generation of Point Clouds is given through the open3d [32] library, with the
intrinsic parameters obtained from the calibration of the thermal camera, the depth
image of the Kinect v1 and the rectified image of the thermal camera.
Figure 21 shows the process of creation and processing of the Point Cloud. The
basis is the rectified thermal image, the depth image, and the intrinsic parameters of
the thermal camera.
Below we detail the steps in the Fig. 21:
1. Create from RGBD image:
The library to use is open3d.geometry.RGBDImage.create.from.color.and.depth,
whose inputs are (depth image and thermal image)
2. Create from Point Cloud
The library to use is o3d.camera.PinholeCameraIntrinsic, whose inputs are
(RGBD image and the internal parameters of the camera (fundamental matrix
obtained from stereo calibration))

Fig. 21 Point cloud creation and processing

106 C. C. Ulloa et al.

The steps above refer to the creation of the Point Cloud.

Next, we are going to detail the processing of these clouds and the process for ICP.
1. Remove points:
This process is to remove the points outside the Point Cloud using the point-
cloud.remove.statical.outliner library, whose entries are (selection of close neigh-
bors and a minimum radius)
2. Point Cloud reduction:
We reduce the Point Cloud to handle clouds of the same size, with point-
cloud.voxel.down.sample with the input (the value to reduce the Point Cloud)
3. Estimation of normals:
We need to estimate the normals to obtain the characteristic points of each Point
Cloud, for this, we use pointcloud.estimate.normals with the following inputs (the
method to use in our case was
open3d.geometry.KDTreeSearchParamthhibrid (with which we name the search
radius and the maximum number of neighbors to search))
4. Characteristic points:
We use the characteristic points to match with the RANSAC method and we
occupy the library open3d.registration.compute.fpfh.feature with the inputs (Point
Cloud, search radius and maximum neighbors)
5. Characteristic point matching:
For the matching of the characteristic points of two Point Clouds we use the
RANSAC method for this we use the library
open3d.registration.registration.ransac.based.on.feature.matching with the fol-
lowing entries:
– Cloudbase.voxel.down reduced base cloud
– Cloudbase.fpfh base cloud feature points
– Nubetarget.voxel.down reduced target cloud or second cloud
– Cloudtarget.fpfh characteristic points of target cloud
– Threshold maximum distance between clouds
– Estimation method
– Fit.ransac
– Matching criteria (maximum.iteration and maximum.validation)
6. ICP:
We do the ICP with the open3d.registration.registration.ICP library with the fol-
lowing input parameters
– Base cloud
– Cloud to which we attach
– Threshold range of motion threshold
– Matrix resulting from pairing of characteristic points
– Type of estimate (PointtoPoint, PointtoPlane)
Autonomous 3D Thermal Mapping of Disaster Environments … 107

Fig. 22 Point Clouds

concatenation

Point Clouds Concatenation

Figure 22 shows the Point Cloud concatenation process, which we describe below.
1. Once the ICP matrix is obtained, we apply transformation to the target cloud
(target.transformation (ipc.transformation))
2. Store the new target points with ICP and the base cloud
3. Apply the previous steps for the processing of the new Comb Point Cloud and
apply ICP between the Comb cloud and the new target
4. Apply the transformation matrix to the new target
5. Store the Comb cloud and transformed new.target in a new matrix, the process is
repeated until the last Point Cloud to be obtained.
Figure 23 shows the integration of a depth image with a thermal image, by means
of the open3d libraries.

4.2 Victims Detection

Figure 24 shows the detection of victims in exteriors is observed, with a precision of

90%.
Figure 25 shows a series of thermal images on the left and RGB images on the
right, thus seeing the difference between RGB and thermal images. Thermal images
have implemented the neural network for victim identification.
108 C. C. Ulloa et al.

Fig. 23 Integration of thermal and depth image, to generate a Local 3d thermal map (Rob-Cib Lab)

(a) External Validation (b) External Validation

Fig. 24 External Validation

5 Results and Discussion

The proposed method has proven to be highly robust for the reconstruction of post-
disaster environments and the provision of relevant information to rescuers during
the execution of missions, thanks to cutting-edge techniques such as thermal image
processing with neural networks. The main contributions lie above all in the provision
of thermal information and location of the victim in the generated map, concerning
conventional methods, which are limited to using RGB cameras for mapping or 3D
reconstruction by laser.
Autonomous 3D Thermal Mapping of Disaster Environments … 109

(a) Thermal and RGB image

(b) Thermal and RGB image

(c) Thermal and RGB image

Fig. 25 Thermal and RGB image at different rotation angles of the robot

5.1 Victims Identification in Post-disaster Environments

Figure 26 shows the process carried out for generating a 3D map in the area of a
corridor. Where there is a victim and a rescuer, the different views of the thermal
map are shown. The subjects of interest are shown with a celestial mark; This infor-
mation is very relevant to support the different brigades during a search and rescue
110 C. C. Ulloa et al.

Fig. 26 Point cloud generation process in corridor

Fig. 27 Point Clouds generation process in CAR Arena

intervention. The approximate dimensions of the created point cloud are (2.01, 7.51,
2.17)[m].
Figure 27 shows the general and detailed process of the generation of the thermal
map and the location of victims using the Convolutional Neural Network in the
thermal images, in such a way that a marker can be placed in the map position. This
information will serve the rescue teams to approach the different victims or areas
directly and prioritize the time of each rescue.
Autonomous 3D Thermal Mapping of Disaster Environments … 111

Table 2 Ship total cloud data of CAR ARENA

Point Images Points % ICP X, Y, Z (min, max)
cloud
X ∗10 [m] Y∗10 [m] Z ∗10 [m]
1 2 432516 88.5 (−0.11567910, (−0.12189946, (0.04323747,
0.31308051) 0.04621907) 0.12711484)
2 3 609894 89.1 (−0.11567910, (−0.12189946, (0.02801179,
0.31308051) 0.04920207) 0.12711484)
3 4 846777 90.1 (−0.11567910, (−0.12769378, (0.01406533,
0.31308051) 0.04920207) 0.12910332)
4 5 1072094 91.3 (−0.11567910, (−0.12769378, (0.01174412,
0.31308051) 0.05085112) 0.12950451)
5 6 1307255 88.6 (−0.11567910, (−0.12769378, (0.01174412,
0.31308051) 0.05085112) 0.12950451)
6 7 1511905 87.8 (−0.11567910, (−0.12769378, (0.00853401,
0.31308051) 0.05085112) 0.12950451)
7 8 1683523 90.5 (−0.11567910, (−0.12769378, (0.00853401,
0.31308051) 0.05085112) 0.129504513)
8 9 1924684 92.7 (−0.11567910, (−0.12769378, (0.00853401,
0.31308051) 0.05085112) 0.12950451)
9 10 2161782 93.5 (−0.11567910, (−0.12769378, (0.00853401,
0.31308051) 0.05085112) 0.12950451)
10 11 2401940 94.8 (−0.11567910, (−0.12769378, (0.00853401,
0.40739381) 0.05085112) 0.12950451)
11 12 2645056 90.1 (−0.11567910, (−0.12769378, (0.00853401,
0.40739381) 0.05085112) 0.21950451)
12 13 2865852 87.9 (−0.11567910, (−0.14489141, (0.00853401,
0.40739381) 0.05085112) 0.21950451)
13 14 3095092 86.6 (−0.11567910, (−0.14489141, (0.00853401,
0.40739381) 0.05085112) 0.21112295)
14 15 3338212 87.5 (−0.11567910, (−0.14489141, (0.00853401,
0.40739381) 0.05085112) 0.21421589)
15 16 3581818 89.6 (−0.11567910, (−0.14489141, (0.00853401,
0.43033820) 0.05085112) 0.215866660)
16 17 3747882 90.3 (−0.11567910, (−0.14489141, (0.00853401,
0.45019200) 0.05085112) 0.213586666)
17 18 3982299 87.3 (−0.11567910, (−0.14489141, (0.00853401,
0.50247442) 0.05085112) 0.21661679)
18 19 4226592 90.4 (−0.11567910, (−0.14489141, (0.00853401,
0.53304231) 0.05085112) 0.21661679)
19 20 4441642 91.5 (−0.11567910, (−0.14489141, (−0.01253458,
0.59267616) 0.05085112) 0.21661679)
20 21 4642145 93.4 (−0.11567910, (−0.15411624, (−0.01867867,
0.62411952) 0.05085112) 0.21661679)
21 22 4813992 89.7 (−0.11567910, (−0.15411624, (−0.02061485,
0.66031938) 0.05085112) 0.21661679)
112 C. C. Ulloa et al.

Fig. 28 Rob-Cib laboratory cloud generation process

The Fig. 27 shows the location of 2 victims in different positions and a rescuer.
The map varies in terms of obtaining depth images since said Kinect sensor does not
work properly outdoors.
The Table 2 describes the results in each T-LPC captured to form the T-GPC of
Fig. 27, the percentage of ICP between each Point Cloud (Fig. 27) and the minimum
and maximum distances of the X, Y, Z coordinates [meters] of each concatenation
of Point Clouds. The first Point Cloud is the first ICP of two initial points.
Figure 28 shows the Rob-Cib laboratory on the first floor in which a victim and
a rescuer are located in it, such an environment similar to the yellow sand of the
Robo-Cup competition both due to its limited space and location.
Table 3 describes the results of the different T-LPCs captured to form the T-GPC
of the Fig. 28, the percentage of ICP between each Point Cloud and the minimum
and maximum of the X, Y, Z coordinates [meters] of each concatenation of Point
Clouds, the first Point Cloud is the first ICP of two new initial points.
Autonomous 3D Thermal Mapping of Disaster Environments … 113

Table 3 Rob-Cib laboratory total cloud data

Point Images Points % ICP X, Y, Z (min, max)
cloud
X ∗10 [m] Y∗10 [m] Z ∗10 [m]
1 2 385868 85.74 (−0.22578875, (−0.10453029, (0.03945444,
0.04596534) 0.052261363) 0.21503266)
2 3 586889 92.1 (−0.2826026082, (−0.1122396439, (0.02035032,
0.04596534743) 0.05226136371) 0.21503266)
3 4 794060 90.2 (−0.2826026082, (−0.11223964, (−0.043887581,
0.0459653474) 0.05437704) 0.215032690)
4 5 1015257 93.5 (−0.282602608, (−0.112239643, (−0.010153032,
0.0459653474) 0.054377041) 0.215032690)
5 6 1218043 92.6 (−0.282602608, (−0.112239643, (−0.012963907,
0.045965347) 0.054377041) 0.21503269)
6 7 1431371 94.3 (−0.2826026082, (−0.1122396439, (−0.0129639074,
0.04596534743) 0.0543770417) 0.215032690)
7 8 1638043 95.4 (−0.282602608, (−0.112239643, (−0.01351100,
0.0459653474) 0.054377041) 0.21503269)
8 9 1846780 90.1 (−0.282602608, (−0.112239643, (−0.01351100,
0.0459653474) 0.054377041) 0.21503269)
9 10 2069730 94.6 (−0.282602608, (−0.112239643, (−0.01351100,
0.0527163445) 0.054377041) 0.21503269)
10 11 2264878 89.6 (−0.282602608, (−0.112239643, (−0.01351100,
0.0527163445) 0.054377041) 0.21503269)
11 12 2250620 87.4 (−0.282602608, (−0.112239643, (−0.01351100,
0.069837734) 0.054377041) 0.21503269)
12 13 2416492 93.2 (−0.282602608, (−0.112239643, (−0.01351100,
0.081097111) 0.054377041) 0.21503269)
13 14 2624830 95.4 (−0.282602608, (−0.112239643, (−0.01351100,
0.084412157) 0.054377041) 0.21503269)

6 Conclusion and Future Work

– An algorithm for automatic generation of 3D thermal maps and victim location was
developed and implemented after analysis with Convolutional Neural Networks.
Data was collected tele-operating a robot in the field, from a remote station in a
post-disaster environment.
– The main aspects for the automatic 3D Point Cloud generation consider were the
data collection (images), the normalization of depth images, the rectification of
thermal images, and the thermal camera’s intrinsic parameters obtained from the
calibration of said camera.
– The Point Cloud processing started from the refinement of the same and the elim-
ination of points outside a specific range (80 neighbors in a radius of 4), from
we find the characteristic points of each new cloud and perform matching by the
114 C. C. Ulloa et al.

RANSAC method. The previous process is called the registry of global variables,
the result of this process is the rotation and translation matrix of each Point Cloud,
which we apply to the ICP, and we obtain the integration of the Point Clouds.
– The training of the CNN neural network was given from creating an environment
similar to the yellow and red zone of the Robo-Cup competition, which the NIST
standardizes. In these environments, we acquired the images to create the data-set
for the Point Cloud training. The accuracy of the network training was 98%, while
in the validation, accuracies higher than 90% were obtained.
– The project’s main objective was the location of victims in 3d disaster environ-
ments, for which the objective was fulfilled, but the limitations of the Kinect sensor,
as it does not work in outdoor environments, give rise to proposing another type
of sensor for the acquisition of images. Deep. Another alternative to generating
Point Clouds would be to use 3d sensors such as lidar, Velodyne, Etc.
– Future works to continue the project’s development are the localization of victims
in disaster environments through deep learning with 3D laser sensors such as Lidar
Velodyne, to contrast the proposed method.
– A next stage is proposed where the exploration of the robot in the unknown envi-
ronment is developed autonomously. In this first approach, the problem has been
addressed in an assisted way, where an operator in a remote area guides the robot
through the environment, collecting information.

Acknowledgements This research has been possible thanks to the financing of Rob◦ City2030-
DIH-CM, Madrid Robotics Digital Innovation Hub, S2018/NMT-4331, funded by “Programas de
Actividades I+D en la Comunidad Madrid” and cofunded by Structural Funds of the EU and TASAR
(Team of Advanced Search And Rescue Robots), funded by “Proyectos de I+D+i del Ministerio de
Ciencia, Innovacion y Universidades”

References

1. Jaco, A., Messina, E., Weiss, B.A., Tadokoro, S., Nakagawa, Y.: Test arenas and performance
metrics for urban search and rescue robots. In: Proceedings 2003 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No. 03CH37453), vol. 4, pp.
3396–3403 (2003)
2. Doulamis, N., Agrafiotis, P., Athanasiou, G., Amditis, A.: Human object detection using very
low resolution thermal cameras for urban search and rescue. In: Proceedings of the 10th
International Conference on PErvasive Technologies Related to Assistive Environments, pp.
311–318. Association for Computing Machinery, Island of Rhodes, Greece (2017). ISBN:
9781450352277. https://doi.org/10.1145/3056540.3076201
3. Portmann, J., Lynen, S., Chli, M., Siegwart, R.: People detection and tracking from aerial
thermal views. In: 2014 IEEE International Conference on Robotics and Automation (ICRA),
pp. 1794–1800 (2014)
4. Rodin, C.D., et al.: Object classification in thermal images using convolutional neural networks
for search and rescue missions with unmanned aerial systems. In: 2018 International Joint
Conference on Neural Networks (IJCNN), pp. 1–8 (2018)
5. Cruz Ulloa, C., Prieto Sánchez, G., Barrientos, A., Del Cerro, J.: Autonomous thermal vision
robotic system for victims recognition in search and rescue missions. Sensors 21, 7346 (2021)
Autonomous 3D Thermal Mapping of Disaster Environments … 115

6. Turgut, K., Kaleci, B.: A PointNet application for semantic classification of ramps in search
and rescue arenas. Int. J. Intell. Syst. Appl. Eng. 7, 159–165 (2019)
7. Colas, F., Mahesh, S., Pomerleau, F., Liu, M., Siegwart, R.: 3D path planning and execution for
search and rescue ground robots. In: 2013 IEEE/RSJ International Conference on Intelligent
Robots and Systems, pp. 722–727 (2013)
8. Lauterbach, H.A., et al.: The Eins3D project — instantaneous UAV-based 3D mapping for
search and rescue applications. In: 2019 IEEE International Symposium on Safety, Security,
and Rescue Robotics (SSRR), pp. 1–6 (2019)
9. Cruz Ulloa, C., Krus, A., Barrientos, A., Del Cerro, J., Valero, C.: Robotic fertilisation using
localisation systems based on point clouds in strip-cropping fields. Agronomy 11, 11 (2021)
10. Park, J., Chen, J., Cho, Y.K., Kang, D.Y., Son, B.J.: CNN-based person detection using infrared
images for night-time intrusion warning systems. Sensors 20, 34 (2020)
11. Königs, A., Schulz, D.: Evaluation of thermal imaging for people detection in outdoor scenarios.
In: 2012 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pp.
1–6 (2012)
12. Krišto, M., Ivasic-Kos, M., Pobar, M.: Thermal object detection in difficult weather conditions
using YOLO. IEEE Access (2020)
13. Cerutti, G., Prasad, R., Farella, E.: Convolutional neural network on embedded platform for
people presence detection in low resolution thermal images. In: ICASSP 2019-2019 IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2019)
14. Gomez, A., Conti, F., Benini, L.: Thermal image-based CNN’s for ultralow power people recog-
nition. In: Proceedings of the 15th ACM International Conference on Computing Frontiers,
pp. 326–331 (2018)
15. Cerutti, G., Milosevic, B., Farella, E.: Outdoor people detection in low resolution ther-
mal images. In: 2018 3rd International Conference on Smart and Sustainable Technologies
(SpliTech) (2018)
16. Jiménez-Bravo, D.M., Mutombo, P.M., Braem, B., Marquez-Barja, J.M.: Applying faster R-
CNN in extremely low-resolution thermal images for people detection. In: 2020 IEEE/ACM
24th International Symposium on Distributed Simulation and Real Time Applications (DS-RT),
pp. 1–4 (2020)
17. Perdana, M.I., Risnumawan, A., Sulistijono, I.A.: Automatic aerial victim detection on low-
cost thermal camera using convolutional neural network. In: 2020 International Symposium
on Community-centric Systems (CcS), pp. 1–5 (2020)
18. Chen, L., Yao, X., Xu, P., Moon, S.K., Bi, G.: Rapid surface defect identification for additive
manufacturing with in-situ point cloud processing and machine learning. Virtual Phys. Prototyp.
16, 50–67 (2021)
19. Guo, M.-H., et al.: PCT: point cloud transformer. Comput. Vis. Media 187–199 (2021)
20. Rusu, R.B.: Semantic 3D Object Maps for Everyday Robot Manipulation. Springer, Berlin
(2013)
21. Rase, W.-D.: In: True-3D in Cartography, pp. 119–134. Springer, Berlin (2011)
22. Mineo, C., Pierce, S.G., Summan, R.: Novel algorithms for 3D surface point cloud boundary
detection and edge reconstruction. J. Comput. Design Eng. 6, 81–91 (2019)
23. Li, P., Wang, R., Wang, Y., Tao, W.: Evaluation of the ICP algorithm in 3D point cloud regis-
tration. IEEE Access 8, 68030–68048 (2020)
24. Scaramuzza, D., Zhang, Z.: Visual-inertial odometry of aerial robots (2019). arXiv:1906.03289
25. Borrmann, D., Afzal, H., Elseberg, J., Nüchter, A.: Mutual calibration for 3D thermal
mapping. In: IFAC Proceedings Volumes 45. 10th IFAC Symposium on Robot Control,
pp. 605–610 (2012). ISSN: 1474-6670. https://www.sciencedirect.com/science/article/pii/
S147466701633676X
26. Adamopoulos, E., et al.: 3D thermal mapping of architectural heritage in digital heritage.
In: Ioannides, M., Fink, E., Cantoni, L., Champion, E. (eds.) Progress in Cultural Heritage:
Documentation, Preservation, and Protection, pp. 26–37. Springer International Publishing,
Cham (2021). ISBN: 978-3-030-73043-7
116 C. C. Ulloa et al.

27. Adamopoulos, E., Volinia, M., Girotto, M., Rinaudo, F.: Three-dimensional thermal mapping
from IRT images for rapid architectural heritage NDT. Buildings 10 (2020). ISSN: 2075-5309.
https://www.mdpi.com/2075-5309/10/10/187
28. Vidas, S., Moghadam, P., Bosse, M.: 3D thermal mapping of building interiors using an RGB-D
and thermal camera. In: 2013 IEEE International Conference on Robotics and Automation, pp.
2311–2318 (2013)
29. Schönauer, C., Vonach, E., Gerstweiler, G., Kaufmann, H.: 3D building reconstruction and
thermal mapping in fire brigade operations. In: Proceedings of the 4th Augmented Human
International Conference. Association for Computing Machinery, Stuttgart, Germany (2013),
pp. 202–205. ISBN: 9781450319041. https://doi.org/10.1145/2459236.2459271
30. Borrmann, D., Elseberg, J., Nüchter, A.: In: Lee, S., Cho, H., Yoon, K.-J., Lee, J. (eds.) Intel-
ligent Autonomous Systems 12: Volume 1 Proceedings of the 12th International Conference
IAS- 12, held June 26–29, 2012, Jeju Island, Korea, pp. 173–182. Springer, Berlin, Heidelberg
(2013). ISBN: 978-3-642-33926-4. https://doi.org/10.1007/978-3-642-33926-4_16
31. Zhang, S., et al.: Optimized calibration method for ultra-field dual bands cameras based on
thermal radiation checkerboard. Infrared Phys. & Technol. 108, 103346 (2020)
32. Yang, J., Cao, Z., Zhang, Q.: A fast and robust local descriptor for 3D point cloud registration.
Inf. Sci. 346, 163–179 (2016)

Christyan Cruz Ulloa received the M.Sc degree Robotics and Automation from the Polytech-
nic University of Madrid in 2019, currently is Ph.D. candidate in Robotics and Automation in the
same University from 2019. In 2018–2019 he was Professor of the Army Forces University form
Ecuador. Currently, he is researcher of the Robotics and Cybernetics research group of the Cen-
tre for Automatic and Robotic in the Technical University of Madrid–Spanish National Research
Council. His main research activities are in the field of ground robotics and specific sensors for
artificial vision systems, focused on Search and Rescue Robotics (TASAR National Project) and
Robotics in Precision Agriculture (SUVEREV European Project). He has participated in inter-
national conferences as expositor is also autor and co-author of different international scientific
papers in journals and conferences (https://orcid.org/0000-0003-2824-6611). He has an active
participation as a reviewer for different journals as Sensors, Drones and Revista Iberoamericana
de Automática e Informática Industrial.

Guido Torres Llerena received the title of M.Sc Robotics and Automation at the Polytechnic
University of Madrid in 2021. In 2020–2021 Research Assistant at the Center for Automation and
Robotics of the Polytechnic University of Madrid. His research activities have been developed in
the handling of robots and specifically, in the handling of artificial vision systems with ROS, his
publications are related to the design of machines and automation.

Antonio Barrientos received the M.Sc Engineer degree by Automatic and Electronic from the
Polytechnic University of Madrid in 1982, and the Ph.D. in Robotics from the same University in
1986. In 2002 he obtained de M.Sc Degree in Biomedical Engineering by Universidad Nacional de
Educación a Distancia. Since 1988 he is Professor of robotics, computers and control engineering
at the Polytechnic University of Madrid. He has worked for more than 30 years in robotics, devel-
oping industrial and service robots for different areas. Currently, he is the head of the Robotics and
Cybernetics research group of the Centre for Automatic and Robotic in UPM-CSIC. The main
interests of his research group are in the field robotics (ground and aerial), seeking its application
in two areas: agriculture and security, search and rescue scenarios. He is author of several text-
books in Robotics and Manufacturing automation and has participated as coauthor in the white
books about the state of robotics and manufacturing in Spain. He is also co-author of about 200
scientific papers in journals and conferences (https://orcid.org/0000-0003-1691-3907). He has
been a reviewer for several journals and is an Editorial Board Member of the International Jour-
nal of Advanced Robotic Systems and of the Revista Iberoamericana de Automática e Informática
Autonomous 3D Thermal Mapping of Disaster Environments … 117

Industrial. Also, he is member of the Board of directors of the Comite Español de Automática a
senior member of the IEEE.

Jaime Del Cerro received his M.S. degree in Industrial Engineering and his Ph.D. degree in
Robotics and Automation from UPM (Spain) 2007. He currently teaches Control Basics, Robotics,
System Programming and Guidance Navigation and Control of Autonomous Vehicles at the ETSII-
UPM. He collaborated with the Universidad Internacional de la Rioja in the posgrade program of
Designing and Management of Technological projects and Universidad Europea de Madrid for
several years. His main research activities are related to robotics (ground and aerial) and multi-
robot systems applied to security and agriculture. He has worked on several national and European
R&D projects as Researcher.
Lab-Scale Smart Factory
Implementation Using ROS

Marwan Abdelatti and Manbir Sodhi

Abstract The central concepts underlying Industry 4.0 are the inter-connectivity
between smart machines, production equipment, storage systems, algorithms with
autonomous data exchange, action triggers, and, autonomous process controls. The
goal of Industry 4.0 is to improve monitoring and control of manufacturing processes
and systems, leading to high-quality parts, lower inventory, and reduced through-
put times. However, the limitations and incompatibilities of legacy systems present
significant challenges for the introduction of Industry 4.0. This use case chapter
describes the implementation of a fully functioning, lab-scale, Industry 4.0 smart
factory to demonstrate how legacy systems can be upgraded without major infras-
tructure changes. Using existing and proven frameworks and protocols such as robot
operating system (ROS), and telemetry transport protocol (MQTT), Industry 4.0 fea-
tures can be introduced to legacy systems by adding a single layer of hardware. Addi-
tionally, a supervisory control and data acquisition (SCADA) system is integrated
with the IIoT to monitor and control the entire system processes and to provide ana-
lytical reports from the collected data. By utilizing ROS, the system has standardized
functionalities including unified high-speed communication, hardware interfacing,
and multi-thread handling, to name a few. We present the ft_industry4_raspi ROS
Melodic package that handles all the distributed nodes over the multiple controllers
in the factory. The reconfigured system is upgraded to demonstrate Smart Manufac-
turing, Cybersecurity, and Cloud-based Manufacturing. This system can function as
a testbed for evaluating Cybersecurity, Cyber-Physical Systems concepts, and new
IIoT hardware and SCADA software, as well as for embedding a variety of operations
control approaches.

Keywords Industry 4.0 · Internet of Things · ROS · SCADA · Smart factory ·

Embedded systems

M. Abdelatti (B) · M. Sodhi

Department of Industrial Engineering, University of Rhode Island, Kingston, RI 02881, USA
e-mail: [email protected]
M. Sodhi
e-mail: [email protected]

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 119
A. Koubaa (ed.), Robot Operating System (ROS),
Studies in Computational Intelligence 1051,
https://doi.org/10.1007/978-3-031-09062-2_4
120 M. Abdelatti and M. Sodhi

1 Introduction

The factory of the future, as conceptualized in Industry 4.0, is an integrated sys-

tem where production machines, material handling systems, part input and output
processes, and humans interact within a framework of distributed and centralized
controls [1, 2]. Machines exchange parts and data autonomously and algorithms act
on this data to control processes and part movements. The data also gives insight
into the condition of machines, and maintenance is invoked in a predictive manner.
Much of this data is generated by components of the Internet of Things (IoT) [3,
4]. One of the challenges in implementing an Industry 4.0 system is assembling a
collection of heterogeneous devices in a common framework. Using the architecture
of Robot Operating System (ROS), it is possible to integrate the communications of
a variety of production devices. In this application, ROS has the advantage that many
of the material handling devices such as robots and other material transfer devices
such as AGVs may already have been set up to operate with ROS, facilitating their
integration expeditiously.
Small-scale factories or simplified physical models have been commonly used
in research involving operations research and maintenance. The main purpose of
using such systems is to generate representative data for modeling and testing since
it is difficult to get this data from industrial settings by disrupting production, and
furthermore, when available, it is not publicly accessible [5]. This cyber-physical
approach has many benefits over using a virtual model of a factory or machine since
it allows training in real environments and allows replication of details related to
industrial practice that can be difficult to simulate virtually [6]. Another advantage
of using lab-scale implementations is that these can be built using low cost, non-
industrial components. There are several platforms available nowadays including
the Smart-LEGO Factory, the Fischertechnik plant model for Industry 4.0 [7] a
Fischertechnik punching workstation [8], the Festo Learning Factory [9] to name a
few.
There is an increasing interest in the use of small-scale factories, especially for
exploring Industry 4.0 concepts. Andersen et al. [10] show how a learning factory
can be used in an engineering course on reconfigurable manufacturing in a blended
and problem-based learning approach. Nardello et al. [11] present an in-house devel-
oped learning factory to meet various manufacturing requirements and proving their
value in a production environment. A mechanism of how a generic client can access
data generated from a workstation has been demonstrated in [8]. Three application
cases from business process management (BPM) have been used in [12] to address
challenges in Industry 4.0 using DFKI Smart Lego Factory. A physical Fischertech-
nik factory model equipped with several sensors has been introduced in [5] as an
approach for the generation of predictive maintenance data. The development of an
ontology to represent a Fischertechnik manufacturing simulation model by reusing
existing ontologies has been investigated in [13]. A fully automated factory proto-
type built out of LEGO bricks has been introduced in [14] to demonstrate artificial
Lab-Scale Smart Factory Implementation Using ROS 121

intelligence applications in Industry 4.0. However, these all use proprietary software
that makes the process of expanding the factory challenging.
In this chapter, we present a laboratory-scale smart factory that can be assembled
with off-the-shelf components and IoT devices. The system has been created to
give students a facility to learn about Industry 4.0 concepts without the expense,
complexity and risk of working in life-scale installations. Another intended use of
this system is as a sand-box for testing different frameworks, such as ROS, with a
variety of Manufacturing Execution Systems (MESs) to detect performance issues
at at low expense. A video showing the system in action can be viewed at: https://
youtu.be/mH5jlzI3Shw. Our contributions in this work can be listed in the following
aspects:
– We introduce the “University of Rhode Island Lab-scale Factory for Industry
4.0 Kit” (i.e., URILF4.0 for short) as an open-source system for manufacturing
research. All proprietary interconnections including hardware, software, and pro-
tocols have been replaced by open-source components.
– We develop the ft_industry4_raspi ROS package to handle the industrial
processes of the factory. This will give the learners the opportunity to get practical
training on ROS for industrial applications.
– We efficiently developed a SCADA system to monitor and control industrial pro-
cesses by mapping between MQTT and ROS topics.
This chapter is organized as follows, a detailed description of the factory kit
is provided in Sect. 2 where various functioning modes are explained with dia-
grams and flowcharts. Section 4 discusses the proposed ROS package and details
its key files and environment configurations, and other software architecture. The
ft_industry4_raspi system is tested in Sect. 6. Section 5 discusses the SCADA
system concept and the use of the bridge package to map between ROS and MQTT
topics. Section 7 concludes this work with some remarks on possible future devel-
opments.

2 The Smart Factory Kit

The URILF4.0 is a physical simulation of a “smart factory” designed to familiarize

students with Industry 4.0 concepts and to serve as a test-bed for exploration and
experimentation. The current configuration uses the basic modules purchased from
Fischertechnik as part of the Industry 4.0 Learning Factory, but all proprietary inter-
connections including hardware, software and protocols have been replaced by open
components. URILF4.0 simulates the storing of raw materials, as well as product
manufacturing and delivery according to the orders placed by customers. Figure 1
shows the different components of the kit, it consists of factory modules like auto-
matic storage and retrieval station (ASRS), vacuum gripper robot (VGR), high-bay
warehouse (HBW), multi-processing station with kiln (MPS), and a sorting line with
color detection (SLD).
122 M. Abdelatti and M. Sodhi

Fig. 1 The smart factory kit

Fig. 2 New material process

flow

URILF4.0 supports two main processes: (i) Storing new (incoming) materials.
When a part arrives at the pickup location, the VGR picks it and transports it to the
ASRS. The ASRS checks for the first free location in the HBW and places the material
accordingly. This is shown in Fig. 2. (ii) Processing new orders, as documented by
Fig. 3. This process is activated by receiving an order from the SCADA system
representing a customer request. The customer may request a part with one of three
colors: red, white, or blue within a unique processing sequence. The ASRS checks for
the first available material location in the HBW and moves to it then places the part
where the VGR can pick it and transport it to the MPS. The material moves through
the MPS and passes along a conveyor belt to the sorting line where it is processed
according to its color. Finally, the VGR picks up the finished product and places it
into the delivery outlet where it is ready for customer pick up. The discussion below
details the functionalities of each unit component in the kit:
High-Bay Warehouse As shown in the flowchart in Fig. 4, there are two different
processes associated with the HBW, namely, new material reception and new order
processing. With regards to new material reception, an inventory piece is moved from
Lab-Scale Smart Factory Implementation Using ROS 123

Fig. 3 New order process flow

Fig. 4 The HBW flowchart

the vacuum gripper robot (VGR) into storage, whereas in the new order process an
inventory piece is moved out of storage and back to the custody of the vacuum gripper
robot. It is worth mentioning that the high-bay warehouse (HBW) communicates both
location and inventory color information for access by both the SCADA and the VGR
as will be detailed in the ROS section of this chapter.
Vacuum Gripper Robot The vacuum gripper robot (VGR) is equipped with a suction
end to pick and move materials. The role of the VGR is to transport the materials
between the different stations and can reach any station in the system. Figure 5 shows
the flowchart for the actions performed by the VGR for the two main processes in
the system. When a new part arrives at the pickup inlet, the photosensor sends a
124 M. Abdelatti and M. Sodhi

Fig. 5 The VGR flowchart

triggering flag to the VGR to initiate a pickup and subsequent placement at the color
detection unit where the color data is created and published along with a triggering
flag to the ASRS. The ASRS in turn reads HBW status then moves and picks up
an empty box which is transported to the HBW conveyor belt to the VGR dropping
side. Meanwhile, the VGR moves the material towards the HBW conveyor belt and
waits for the arrival of the empty box where the VGR drops it.
The second process, which defines how parts are processed in the system, starts
by placing a new order through the SCADA system. The customer can indicate the
part color and the number of parts desired. The VGR moves to the HBW conveyor
while the ASRS moves to the appropriate location at the HBW to pick a material
and place it on the conveyor belt. The VGR then picks the material and transports it
to the multi-processing station (MPS). The VGR waits at its home location until the
material processing is completed and the sorting line (SLD) moves the material to
its pickup location based on its color. Finally, the VGR picks the material from SLD
to its delivery outlet.
Multi-Processing Station The Multi-Processing Station (MPS) includes multiple
processing units like klin, and a milling machine. These units communicate through
I2C protocol to simulate a legacy system. Various conveying techniques are used
such as a conveyor belt, a turntable, and a vacuum suction gripper.
Sorting Line Station The sorting line (SLD) is responsible for sorting the completed
parts by color. Its function lies in the new order process only where it is the last
station before the finished product is picked up by the (VGR) and marked as ready
Lab-Scale Smart Factory Implementation Using ROS 125

Fig. 6 The SLD flowchart

for delivery. The flowchart in Fig. 6 illustrates the SLD process which is activated
once a finished material is dropped from the MPS station. A conveyor belt moves
the material to a color detecting unit then one of three pneumatic valves is activated,
according to the detected color, moving the processed part onto the corresponding
chute. A trigger signal accompanied by the color is published to activate the VGR to
transport the finished piece to the delivery outlet.

3 Legacy System Upgrade

The original Industry 4.0 kit is available from Fischertechnik company with mul-
tiple options such as basic running voltages of 9v or 24v, proprietary TXT control
units or PLCs etc. Technical details about their product and the different purchasing
options are available on their website: https://www.fischertechnik.de/en/products/
simulating/training-models. The base kit used in this project is the 9v kit with TXT
controllers. It was delivered pre-assembled and was equipped with proprietary con-
trollers, and software accessible through the company’s website. To increase the
flexibility and to support expansion of the system using hardware from different
suppliers without compatibility issues, the controllers were replaced by open-source
models. We chose Arduinos and Raspberry Pis because of the large established user
community, relatively low cost components, and free and broad ranges of libraries of
codes. Furthermore, there is increasing interest in using ruggedized versions of these
controllers in factory settings. Since the original system utilizes separate controllers
126 M. Abdelatti and M. Sodhi

Fig. 7 Rosserial between

Arduino and Raspberry Pi

for each processing unit, we follow the same principle in our upgrade and expand
it into a distributed system that can: (1) provide the system with reliability and high
fault tolerance so that a crash on one unit does not affect the others; (2) enable a
scalability level so that more machines can be added in the future with a flexibility
to install, implement and debug new services; and (3) reduce the computational load
if central PCs are used.
To further support the open-source idea and facilitate reproducibility of the pro-
posed system, ROS is adopted as the basic integration tool. Along with being gen-
eral and new parts can be added easily in the system, ROS can be implemented
with sparse storage and computational resources which makes it suitable for use on
embedded computers. Additionally, there is a large library of community developed
packages. For example, instead of writing an exclusive master-slave program to inter-
face between the Raspberry Pi’s and Arduino boards in our system, the rosserial
package is used for the task, which reduces development and maintenance effort.
Moreover, ROS is language agnostic which means that the system can run with parts
of codes written in different languages, increasing the scalability of the system.
In order to support the idea of adding a single layer of software and hardware to the
pre-built legacy system, the SCADA software interface was retained in its original
condition without altering the communication protocol supporting it. This reduces
the modifications needed to the original system. Since the SCADA software being
used in the system uses the MQTT protocol and does not support ROS messaging,
the ROS bridge package is used to map between the two protocols as explained in
the sequel. MQTT itself was not utilized to interface to the embedded computers or
in the sensor/actuator layer for the same reasons as discussed above. As illustrated
by Fig. 7, the Arduino controller is used to perform low-level control such as read-
ing inputs from sensors and sending outputs to motors and pneumatic valves, and
handling I2C communication to other machine controllers. Whereas the Raspberry
Pi takes care of the WiFi communication and integrates Arduino controllers in the
network loop through USB connectivity and the rosserial package.

rosserial is a communication protocol to transfer data through a serial inter-

face or network socket [15]. It’s based on a client-server model where a rosserial
service runs on a computer running ROS (the Raspberry Pi in our implementation)
and a rosserial client on the Arduino side. Data is transferred in the form of
ROS messages between them. rosserial_client package is available for sev-
Lab-Scale Smart Factory Implementation Using ROS 127

eral microcontroller types including Arduino, STM32, Raspberry Pi to name a few.

Rosserial packages are available for both Python or C++.

3.1 ROS Components and Network

Since our system mimics a real factory with multiple machines involved, the proposed
ROS framework is based on a distributed package concept. ROS nodes are run on
different machines and communicate over a network. Every machine is responsible
for specific tasks, as explained earlier and, therefore, they respond to different ROS
topics. The system has one PC functioning as a Human Machine Interface (HMI)
with the SCADA system running on it.
Raspberry Pi 3 B+ boards were selected to handle the ROS communication
between the local Arduino board as well as with the ROS core over WiFi. The 1.4
GHz processor, 1 GB RAM, and the installed Linux operating system (Ubuntu 18.04
server) were sufficient for this project. Four processing stations were interfaced to
the Raspberry Pi’s: three of them, called “unit Pis”, take care of the ROS nodes of the
factory units and necessary packages, and the fourth, called “main Pi”, hosts the ROS
core, MQTT server, and other configuration files necessary to the system. To be more
specific, each unit Pi runs rosserial_server package to communicate with its
Arduino board, and the proposed ft_industry4_raspi package to handle the
factory tasks. In addition to these packages, the main Pi runs the ROS core, and the
mqtt_bridge package to map between the ROS topics and the MQTT topics as
detailed in the following sections as well as the Mosquitto MQTT server. The HMI
PC has a SCADA software installed. The PC does not directly interact with ROS
since the SCADA software supports MQTT rather than ROS, the mapping performed
by the mqtt_bridge package installed on the main Pi is responsible for this PC
interface.
The remainder of basic control operations that require direct physical link to sen-
sors or actuator drivers like color detection, robot manipulation, and servo motor
and solenoid control are performed on the Arduino boards. The board used is an
Arduino-Leonardo compatible board called ftDuino. It has built in motor drivers,
eight universal inputs that can be utilized for analog or digital readings, four counter
inputs, and eight outputs in addition to I2C, and USB connections. More informa-
tion about the board can be found in this link: https://harbaum.github.io/ftduino/
www/de/. It fits perfectly into the Arduino ecosystem for handy development, as
discussed in Sect. 3.2, but the lack of network connectivity in ftDuino was one of the
motivators behind using Raspberry Pi as another board to grant network access. The
communication network uses a router and all the devices are connected via WiFi. The
formed network consists of one PC, four Raspberry Pis 3 B+, and six ftDuinos. Every
ftDuino is connected to one Raspberry Pi for ROS connectivity, the two additional
ftDuinos are for the local Multi-processing station that rely on sensor based detec-
tion and I2C master-slave communication for operation. The network components
are detailed in the architecture given in Fig. 8 where the Raspberry Pis are connected
128 M. Abdelatti and M. Sodhi

Fig. 8 Physical and logic layers in the system

to a WiFi router gateway with unique static IP addresses, the PC is remotely con-
nected to the main Pi via internet. The main Pi functions as the ROS master that
handles the communications between the factory components on one hand, and on
the other hand acts like an MQTT broker that talks to the HMI PC to present data
on the SCADA. The mqtt_bridge package on the main Pi simultaneously maps
between the ROS messages and the MQTT messages in a bidirectional form. The
MQTT communication is done through port 1883 since encrypted messages are not
handled in this version of the system.

3.2 ftDuino Integration into the Arduino IDE

In order to access the ftDuino through the Arduino IDE, the following steps need to be
followed: after connecting the ftDuino to the PC where the Arduino IDE is installed,
the ftDuino board must be installed in the IDE. At the File -> Preferences
menu, the following URL is added in the “Additional Boards Manager URLs:” field:
https://harbaum.github.io/ftduino/package_ftduino_index.json. Then press “OK” as
shown in the screenshot in Fig. 9.
Once the link is provided, the board package can be installed from the Boards Man-
ager at Tools -> Board: ... -> Boards Manager as shown in Fig. 10.
The ftDuino can now be selected by choosing ftDuino from
Tools -> Board: ... -> ftDuino Boards as illustrated in Fig. 11. If
the board is already connected to the PC, it can now be selected under
Tools -> Port as in Fig. 12.
Lab-Scale Smart Factory Implementation Using ROS 129

Fig. 9 Arduino IDE preferences

Fig. 10 Arduino IDE boards manager

4 ROS Environment Configuration

The ROS version used for this project is ROS Melodic on Ubuntu 18.04 server con-
nected to four ROS machines that are running on a network with the same ROS
master. All necessary files including the configuration, and the nodes as well as the
launch files are available in the package repository in the following link:
130 M. Abdelatti and M. Sodhi

Fig. 11 Arduino IDE board selection

Fig. 12 Arduino IDE port

selection

https://github.com/MarwanAbdelatti/ft_industry4_raspi

The package should be cloned inside the src folder in the catkin_ws workspace.
The src folder inside the package includes five folders for the four system machines
as well as the “main Pi”. Some of the key files inside this package are discussed below:
– Every unit Pi has a shell file to configure its launch processes and remote envi-
ronment, namely, env_hbw.sh, env_vgr.sh, and env_sld.sh. The code
below shows an example of these files:

1 #!/bin/bash
2

3 export ROS_MASTER_URI=http://192.168.0.10:11311
4 export ROS_HOSTNAME=192.168.0.12
5

6 source /opt/ros/melodic/setup.bash
7 source /home/ubuntu/catkin_ws/devel/setup.bash
8

9 exec "$@"

where the ROS master remote IP is defined in line 3 followed by the local host IP
definition. The environment is configured in lines 6 and 7.
Lab-Scale Smart Factory Implementation Using ROS 131

– src/main/industry4.launch: is the main launch file that initiates the

nodes and sub-launch files for the system. The file can be divided into two parts:
(1) the machine declaration, where the parameters of the “main Pi” and every “unit
Pi” are defined; and (2) the node declaration, where every node required to run on
the system is defined. The following code shows part of the file:

1 <launch>
2 <!-- Declaring Machines -->
3 <group>
4 <machine
5 name="main"
6 address="192.168.0.10"
7 env-loader="/home/ubuntu/catkin_ws/devel/setup.bash"
8 default="true"
9 user="ubuntu"
10 password="RandomPassword">
11 </machine>
12

13 <!-- Include Files -->

14 <include file="$(find ft_industry4_raspi)/src/main/
ft_industry4_params.launch" />
15 </group>
16

17 <machine
18 name="hbw"
19 address="192.168.0.12"
20 env-loader="/home/ubuntu/env_hbw.sh"
21 .....
22 </machine>
23

24 <!-- Declaring Nodes -->

25

26 <node machine="main" pkg="ft_industry4_raspi" name="

status_update" type="status_update.py">
27 <param name="port" value="/dev/ttyACM0"/>
28 <param name="baud" value="57600"/>
29 </node>
30

31 <node machine="hbw" pkg="rosserial_python" name="hbw_node"

type="serial_node.py">
32 <param name="port" value="/dev/ttyACM0"/>
33 <param name="baud" value="57600"/>
34 </node>
35 .....
36 </launch>

In this file, we start with a <group> tag since we are using multiple launch
files in the “main” machine. Then, we define the machine parameters using the
<machine> tag where we provide the machine name, its local IP, the bash
environment loader file, and the username and password for the ssh remote
connection. Although explicitly providing passwords in a readable file is a
 132 M. Abdelatti and M. Sodhi

security risk, we provide it here for clarification and simplicity. A secure ver-
sion of the launch file utilizing public key authentications is detailed the text
file https://github.com/MarwanAbdelatti/ft_industry4_raspi/blob/main/Secured-
Approach-for-Launch-Files.md in the package repository link on Github. At
line 14 in the file we currently discuss, another launch file is included. This file
is responsible for launching the mqtt_bridge nodes that interface between
MQTT and ROS messaging. The file includes node configurations, and the con-
version map between ROS and MQTT topics which is available in the data file
ft_industry4_params.yaml. Similarly, the other machines are declared
such as the HBW in line 17 but there’s no <group> tag used here since there are
no sub-launch files to include, the other parameters are similar like the IP address,
the environment loader file, and so on.
The second part of this file is the node declaration where we launch the
rosserial_python node on each machine using the <node machine>
tag to enable the communication between each Raspberry Pi and the ftDuino con-
nected to it. The link is enabled on the USB port with ID ttyACM0 and with
communication speed 57600 bps. In addition to this, the status_update node
is declared on the main Pi which is responsible for updating the HBW occupancy
status of the shelves as well as the material color available. This data is saved in
the data.json status file. The dots in lines 21 and 35 are to indicate that the file
is clipped for the sake of space and that the clipped parts are similar.

– The rest of files contain Arduino codes (.ino files) that are meant to run on the
ftDuino’s of the units and contain the remainder of nodes that subscribe or publish
to the different ROS topics in the system. The following is a sample of the VGR
Arduino code:
1 // Use the following line if you have a Leonardo, FtDuino or MKR1000
2 #define USE_USBCON
3
4 #include <ros.h>
5 #include <std_msgs/String.h>
6 #include <std_msgs/Bool.h>
7 #include <Ftduino.h>
8
9 ros::NodeHandle nh;
10 std_msgs::String str_msg;
11 std_msgs::String str_loc_msg;
12
13 ros::Publisher vgr_new_material("vgr_new_material", &str_msg);
14 ros::Publisher provide_empty_loc("provide_empty_loc", &str_loc_msg);
15
16 char empty_loc_content[3] = "Z4";
17 char detected_color[6] = "NA";
18 bool move_cmd = true;
19 bool in_processing_object = false;
20
21 void messageCb_red(const std_msgs::Bool &payload_msg){
22
23 if (payload_msg.data && move_cmd){
24 nh.loginfo("VGR Received an Order at the RED Topic");
25 move_cmd = false;
26 //move to Sorting Line Drop-off area (red)
 Lab-Scale Smart Factory Implementation Using ROS 133

27 ftduino.motor_counter(Ftduino::M1, Ftduino::LEFT, Ftduino::MAX, 390);

28 while(ftduino.motor_counter_active(Ftduino::M1)); /*no-op*/
29
30 //pick up material (red)
31 ftduino.motor_counter(Ftduino::M3, Ftduino::LEFT, Ftduino::MAX, 450);
32 while(ftduino.motor_counter_active(Ftduino::M3)); /*no-op*/
33
34 ftduino.motor_counter(Ftduino::M2, Ftduino::LEFT, Ftduino::MAX, 850);
35 while(ftduino.motor_counter_active(Ftduino::M2)); /*no-op*/
36
37 ftduino.output_set(Ftduino::O7, Ftduino::HI, Ftduino::MAX);
38 ftduino.output_set(Ftduino::O8, Ftduino::HI, Ftduino::MAX);
39 delay(2000);
40
41 //move to end point
42
43 ....
44
45 //drop off material at end point
46 digitalWrite(LED_BUILTIN, HIGH);
47 dropMaterial();
48 delay(2000);
49 digitalWrite(LED_BUILTIN, LOW);
50
51 //move back home
52 goHome();
53 }
54 }
55
56 ros::Subscriber<std_msgs::Bool> vgr_blue_sub("sld_blue_pub",
57 &messageCb_blue);
58 ros::Subscriber<std_msgs::Bool> vgr_red_sub("sld_red_pub",
59 &messageCb_red);
60 ros::Subscriber<std_msgs::Bool> vgr_white_sub("sld_white_pub",
61 &messageCb_white);
62 ros::Subscriber<std_msgs::String> new_order_sub("new_order",
63 &material_processing);
64 ros::Subscriber<std_msgs::String> provide_empty_loc_sub("provide_empty_loc_back",
&empty_locCb);
65
66 void setup()
67 {
68 nh.initNode();
69 nh.subscribe(vgr_blue_sub);
70 nh.subscribe(vgr_red_sub);
71 nh.subscribe(vgr_white_sub);
72 nh.subscribe(new_order_sub);
73 nh.subscribe(provide_empty_loc_sub);
74
75 nh.advertise(provide_empty_loc);
76 nh.advertise(vgr_new_material);
77
78 pinMode(LED_BUILTIN, OUTPUT);
79
80 ftduino.init();
81 //Vertical motor inputs
82 ftduino.input_set_mode(Ftduino::I2, Ftduino::SWITCH);
83 ftduino.counter_set_mode(Ftduino::C2, Ftduino::C_EDGE_RISING);
84 //Horizontal motor inputs
85 ftduino.input_set_mode(Ftduino::I3, Ftduino::SWITCH);
86 ftduino.counter_set_mode(Ftduino::C3, Ftduino::C_EDGE_RISING);
87 //turn encoder
 134 M. Abdelatti and M. Sodhi

88 ftduino.input_set_mode(Ftduino::I1, Ftduino::SWITCH);
89 ftduino.counter_set_mode(Ftduino::C1, Ftduino::C_EDGE_RISING);
90 //in-processing sensor
91 ftduino.input_set_mode(Ftduino::I7, Ftduino::SWITCH);
92 //color sensor
93 ftduino.input_set_mode(Ftduino::I8, Ftduino::VOLTAGE);
94 //ending sensor
95 ftduino.input_set_mode(Ftduino::C4, Ftduino::SWITCH);
96 }
97
98 void loop()
99 {
100 while(move_cmd && ftduino.input_get(Ftduino::I7)){
101 nh.spinOnce();
102 delay(250);
103 }
104
105 if (!ftduino.input_get(Ftduino::I7)){
106 in_processing();
107 }
108 }

The first six lines are related to the rosserial_arduino package and enable
the ftDuino to connect to ROS topics. Line 7 imports the ftDuino library to commu-
nicate with the IO ports to read sensors and control devices connected to it. Lines
9–14 declare the ROS node handle, message objects used in the code, and pub-
lisher objects to different topics. Some global variables are defined in lines 16–19.
The messageCb_red() callback function is defined in lines 21–54 to be invoked
when a red part is detected at the SLD chute, the dots in line 43 are to cut some
text to save space. ROS subscriber objects are defined in lines 56–60 where the
corresponding callback functions are tied to them. The node is initialized and ROS
objects are connected to the topics by the setup() function declared in lines 62–92
where board-related I/O definitions are also defined. The main loop is defined in lines
94–104.

4.1 Prerequisite Packages

In order for the ft_industry4_raspi package to initiate and run successfully,

some packages are necessary. The communication between each Raspberry Pi and
the ftDuino board connected with requires the rosserial package to be available
on each Raspberry Pi and ftDuino. To handle the communication between the main
Pi and the SCADA HMI, an MQTT server and the mqtt_bridge package are
required on the main Pi to exchange the messages between ROS and MQTT back
and forth. Steps to properly install and configure these packages are summarized
below:
The rosserial Package is a metapackage that references multiple packages required
to handle the communication over the USB port in our system. This package
includes rosserial_arduino, rosserial_python, rosserial_msgs,
Lab-Scale Smart Factory Implementation Using ROS 135

and rosserial_client. More information can be found in ROS Wiki: http://

wiki.ros.org/rosserial. To install the package on ROS Melodic for Arduino-compatible
boards, the following steps should be followed:

These two commands install the packages on each Raspberry Pi:

$ sudo apt-get install ros-melodic-rosserial-arduino
$ sudo apt-get install ros-melodic-rosserial

Now that the necessary libraries are installed on the Raspberry Pi, packages are built
using catkin_make command. Header files and libraries necessary for Arduino
programs to interact with ROS are built using the following commands:
$ cd ~/Arduino/libraries
$ rm -rf ros_lib
$ rosrun rosserial_arduino make_libraries.py .

The goal of the first two commands is to remove any old ROS libraries installed
within the libraries folder, which exists inside the installation folder of the Arduino
IDE. Then, ROS libraries are built with the third command which requires a target
location for the libraries, and is specified by the last (dot) in the line.
The mqtt_ bridge Package is a package that maps between ROS and MQTT mes-
sages in a bidirectional form. Messages from ROS are serialized by a messagepack
json file for MQTT, and messages from MQTT are deserialized for ROS topic.
More information is available in the ROS Wiki: http://wiki.ros.org/mqtt_bridge. The
mqtt_bridge and its components are required on the main Pi only. First the MQTT
broker (i.e., server) is installed:
$ sudo apt install mosquitto mosquitto-clients

Then the following commands install the rest of the prerequisites before installing
the package:
$ sudo apt install python-rospkg
$ pip install rospkg
$ sudo apt install ros-melodic-rosbridge-library

The mqtt_bridge package is installed by the following command:

$ sudo apt install ros-melodic-mqtt-bridge

5 SCADA System

Supervisory Control and Data Acquisition (SCADA) system, as defined in [16], is:
A sub-class of industrial control systems (ICSs) in which control is performed over multiple,
distributed individual lower-level control systems (hence the word “supervisory”).
136 M. Abdelatti and M. Sodhi

A typical SCADA control center monitors and manages automation processes across
remote areas through smart controllers such as remote terminal units (RTUs) and
intelligent electronic devices (IEDs). The system status is presented on human-
machine interface (HMI) that represents the system graphically and displays events
and alarms lists. In addition to status monitoring, human operators can send control
commands via the HMI software to different ICS components. Such control com-
mands include but are not limited to changing a set point of a motor speed, start/stop
a compressor, or open/close valves. In order for a SCADA software to communicate
with the remote site, appropriate communication drivers must be available in the
software. The SCADA software in our proposed system supports MQTT data trans-
mission and functions as an MQTT client connecting to the MQTT broker located
in the main Pi.

5.1 MQTT Protocol

According to the official MQTT specifications in the OASIS standard [17], MQTT
is defined as:
MQTT is a Client Server publish/subscribe messaging transport protocol. It is light weight,
open, simple, and designed so as to be easy to implement. These characteristics make it ideal
for use in many situations, including constrained environments such as for communication
in Machine to Machine (M2M) and Internet of Things (IoT) contexts where a small code
footprint is required and/or network bandwidth is at a premium.

MQTT was developed in 1999 by Andy Stanford-Clark (IBM) and Arlen Nipper
(Arcom, now Cirrus Link) to connect to oil pipelines via satellite with minimal bat-
tery loss and minimal bandwidth. In contrast to client/server architecture where a
client communicates directly with an endpoint (server) and the server responds back,
MQTT adopts publish/subscribe architecture where there are three key components
illustrated in Fig. 13 forming the communication process, i.e. the publisher, the sub-
scriber and the broker. The broker decouples the client (the publisher) that sends a
message from the client or clients (the subscribers) that receive the messages. The
publishers and subscribers never have a direct contact to each other. The broker filters
all incoming messages and ensures they are correctly distributed to subscribers [18].
MQTT has been gaining popularity along with the proliferation of the Internet of
Things (IoT) that needs the capability to work with low-powered devices. Accord-
ing to ISO/IEC 20922, MQTT is an ideal communication protocol in Machine to
Machine (M2M) communications and within the IoT. The publish/subscribe archi-
tecture showed more efficiency than the conventional client/server model for industry
4.0 systems. Therefore, many systems are adopting MQTT. The publish-subscribe
architecture along with the quality of service (QoS), scalability, message filtering, as
well as the last will and testament (LWT) features give more flexibility to the connec-
tion of low-bandwidth devices with limited CPUs and with very little overhead [19].
 Lab-Scale Smart Factory Implementation Using ROS 137

Fig. 13 MQTT main components

MQTT has already been successfully implemented in many smart applications like
home automation, surveillance, healthcare, transportation, industry, and logistics.

5.2 ROS-MQTT Bridge Configuration

As discussed in Sect. 4, the file ft_industry4_params.yaml includes the

MQTT configurations, and the conversion map between ROS and MQTT topics.
The contents of the file are shown in the listing below which includes the mapping
between different ROS and MQTT topics in both directions.
1 mqtt:
2 client:
3 protocol: 4 # MQTTv311
4 connection:
5 host: localhost
6 port: 1883
7 keepalive: 60
8 private_path: device/001
9 #serializer: json:dumps
10 #deserializer: json:loads
11 bridge:
12 # new_order
13 - factory: mqtt_bridge.bridge:MqttToRosBridge
14 msg_type: std_msgs.msg:String
15 topic_from: mq_new_order
16 topic_to: /new_order
17 # /vgr_new_material
18 - factory: mqtt_bridge.bridge:RosToMqttBridge
19 msg_type: std_msgs.msg:String
20 topic_from: /vgr_new_material
 138 M. Abdelatti and M. Sodhi

21 topic_to: mq_vgr_new_material
22 # /sld_blue_pub
23 - factory: mqtt_bridge.bridge:RosToMqttBridge
24 msg_type: std_msgs.msg:Bool
25 topic_from: /sld_blue_pub
26 topic_to: mq_sld_blue_pub
27 # /sld_red_pub
28 - factory: mqtt_bridge.bridge:RosToMqttBridge
29 msg_type: std_msgs.msg:Bool
30 topic_from: /sld_red_pub
31 topic_to: mq_sld_red_pub
32 # /sld_white_pub
33 - factory: mqtt_bridge.bridge:RosToMqttBridge
34 msg_type: std_msgs.msg:Bool
35 topic_from: /sld_white_pub
36 topic_to: mq_sld_white_pub

The lines 1–8 contain the MQTT connection configuration for the bridge including
the broker name (here is localhost since the bridge and the broker are on the same
machine), and the port number. The mapping bridge configurations start at line 11,
the first block of configuration serves the order placing feature from the HMI which
sends the order to an MQTT topic called mq_new_order then the bridge forwards
it to the ROS topic /new_order. Line 13 tells the bridge to map from MQTT to
ROS. Line 14 specifies the message type based on ROS message types. Lines 15 and
16 specifies the source and destination topics respectively. The same applies to the
blocks at lines 17–36 but the difference is that the mapping is in the other direction
(i.e., from ROS to MQTT), the source and destination topics, as well as the message
type for the last three topics (lines 22–36).

6 Test of the System

In this section, steps for properly running the system are provided. A graph with the
system nodes and topics is given for better understanding how the system works and
the node running sequence. Also, an application video is linked at the end of this
section.
The system is initiated by running the nodes in related packages through a single
.launch file that also contains other .launch files for easy operation. The launch
file is run by the following command:
$ roslaunch ft_industry4_raspi industry4.launch

where ft_industry4_raspi is the package name and industry4.launch

is the launch file. By running this command on the main Pi, the ROS master connects
to the different unit Pi machines through SSH connections and uses the environment
configuration .sh files as shown in the first part of Fig. 14. The different roslaunch
components including the nodes, machines, as well as the connection parameters of
both rosserial and mqtt_bridge are listed in the summary part in the figure.
Lab-Scale Smart Factory Implementation Using ROS 139

Fig. 14 Screenshot of the ROS master initiation

The last part in the figure shows the successful initiation of all the nodes in the system
indicating that the system is up and ready to work.
Although part of the system we present works using the client/server mechanism
in the rosserial package, the system mainly depends on the pub/sub mechanism
of ROS. Most the node communications are based on topics that show up on the
$ rostopic list command once the nodes are run. These topics are:

/back
/diagnostics
/echo
140 M. Abdelatti and M. Sodhi

/new_order
/private/back
/private/echo
/provide_empty_loc
/provide_empty_loc_back
/sld_blue_pub
/sld_red_pub
/sld_white_pub
/vgr_new_material
/what_is_empty
/what_is_occupied

While some topics are feedback topics that return some information to nodes for
processing, others are command topics that are intended to directly initiate an action
in the system (e.g., moving the robot). The ROS graph in Fig. 15 depicts the sys-
tem nodes (as ellipses) and their corresponding connections to the topics that are
shown as rectangles. Now that the system is up and ready, we have two modes
of operations as discussed earlier in Sect. 2, namely, (i) storing new part, and (ii)
processing materials for new orders. Storing new material mode is automatically
initiated once the system detects a new material at its pickup inlet by the photosen-
sor connected to the ftDuino which runs the vgr_node. The vgr_node, as the
graph shows, publishes an inquiring message to the /provide_empty_loc topic
where the status_update node subscribes and checks the HBW shelf status i.e.,
data.json file for empty locations then returns this location back to the VGR
through the /provide_empty_loc_back.
The VGR moves the material to the color detection station then publishes to the
/vgr_new_material topic its color and the intended location to be placed in.
The hbw_node which subscribes to this topic moves its ASRS robot to pick the
material from the VGR and places it accordingly. Once the new material is placed
successfully, the node publishes to the /what_is_occupied topic the updated
location and color for the status_update node to update the shelf status file.
The typical data.json file looks like the following, where A1, A2,…,C3 are
pre-defined locations at the HBW:

Fig. 15 ROS graph, nodes and topics connections

 Lab-Scale Smart Factory Implementation Using ROS 141

1 {
2 "A1":"white",
3 "A2":"red",
4 "A3":"blue",
5 "B1":"white",
6 "B2":"red",
7 "B3":"blue",
8 "C1":"white",
9 "C2":"red",
10 "C3":"blue"
11 }

The second operation mode, on the other hand, responds to new order request from
customers. The mode is initiated by the following command from terminal or by the
SCADA HMI:
$ rosrun ft_industry4_ftduino place_order.py red

The rosrun command runs the place_order node which takes the requested
color as an argument, searches for an HBW location having a material with the
corresponding color, then publishes this location to the topic /new_order as
shown in the graph of Fig. 15. Both the hbw_node and vgr_node subscribe
to this topic and respond to it at the same time such that the HBW ASRS robot
moves to the target location, picks the material, and places it where the VGR robot
can pick. The ASRS brings the empty box back to the HBW shelf and the the
hbw_node node publishes to the /what_is_empty topic the empty location for
the status_update node to update the shelf status file. Meanwhile, the VGR
moves the material to the MPS station for processing and returns back home listen-
ing to the topics: /sld_blue_pub, /sld_red_pub, and /sld_white_pub.
Once the material processing is done, it’s kept at one of the three SLD locations based
on its color where the sld_node publishes to one of the three above-mentioned
topics so that the VGR can pick and place it at the delivery outlet.
In order for the SCADA to monitor the processes properly, the ROS node of name
mqtt_bridge subscribes to relevant topics namely, /vgr_new_material,
/sld_blue_pub, /sld_red_pub, and /sld_white_pub and publishes to
/new_order as illustrated with green connections in Fig. 15. Future work will
include other topics that provide manual override to VGR and ASRS robots to force-
fully send them to their home positions in case of emergencies.

7 Conclusion

URILF4.0 is designed as a laboratory-scale manufacturing test-bed for exploring and

experimenting with Industry 4.0 concepts. It is built with low cost, open-source con-
trollers and software. The sensors interface to Arduinos, connected with Raspberry
Pi 3 B+’s, which provide network communications and serve as ROS nodes. An addi-
tional Pi is used as the ROS master node. The ROS network is connected to a SCADA
142 M. Abdelatti and M. Sodhi

system and a widely used protocol, MQTT, is used to link SCADA and ROS. The
SCADA gives full access to the low level hardware and enables user inputs to initiate
(and process) orders through the manufacturing test-bed and to receive and store
materials in an ASRS. Work on expanding the manufacturing test-bed is ongoing.
A digital twin of this system is also under development. The source code has been
shared at https://github.com/MarwanAbdelatti/ft_industry4_raspi, and the authors
can also be contacted directly for any additional information.

References

1. Kagermann, H., Lukas, W.-D., Wahlster, W.: Industrie 4.0: Mit dem internet der dinge auf dem
weg zur 4. industriellen revolution. VDI nachrichten 13(1), 2–3 (2011)
2. Lasi, H., Fettke, P., Kemper, H.-G., Feld, T., Hoffmann, M.: Industry 4.0. Bus. & Inf. Syst.
Eng. 6(4), 239–242 (2014)
3. Lee, J., Kao, H.-A., Yang, S.: Service innovation and smart analytics for industry 4.0 and big
data environment. Procedia Cirp 16, 3–8 (2014)
4. Thames, L., Schaefer, D.: Cybersecurity for Industry 4.0. Springer, Berlin (2017)
5. Klein, Patrick, Bergmann, Ralph: Generation of complex data for ai-based predictive mainte-
nance research with a physical factory model. ICINCO 1, 40–50 (2019)
6. Abele, Eberhard, Metternich, Joachim, Tisch, Michael, Chryssolouris, George, Sihn, Wilfried,
ElMaraghy, Hoda, Hummel, Vera, Ranz, Fabian: Learning factories for research, education,
and training. Procedia CiRp 32, 1–6 (2015)
7. Lang, S., Reggelin, T., Jobran, M., Hofmann, W.: Towards a modular, decentralized and digital
industry 4.0 learning factory. In: 2018 Sixth International Conference on Enterprise Systems
(ES), pp. 123–128. IEEE (2018)
8. Angione, G., Barbosa, J., Gosewehr, F., Leitão, P., Massa, D., Matos, J., Peres, R.S., Rocha,
A.D., Wermann, J.: Integration and deployment of a distributed and pluggable industrial archi-
tecture for the perform project. Procedia Manuf. 11, 896–904 (2017)
9. Tisch, M., Abele, E., Metternich, J.: Overview on potentials and limitations of existing learning
factory concept variations. In: Learning Factories, pp. 289–321. Springer (2019)
10. Andersen, A.-L., Brunoe, T.D., Nielsen, K.: Engineering education in changeable and recon-
figurable manufacturing: using problem-based learning in a learning factory environment. Pro-
cedia Cirp 81, 7–12 (2019)
11. Nardello, M., Madsen, O., Møller, C.: The smart production laboratory: a learning factory
for industry 4.0 concepts. In: CEUR Workshop Proceedings, vol. 1898. CEUR Workshop
Proceedings (2017)
12. Rehse, J.-R., Dadashnia, S., Fettke, P.: Business process management for industry 4.0–three
application cases in the dfki-smart-lego-factory. IT-Inf. Technol. 60(3), 133–141 (2018)
13. Klein, P., Malburg, L., Bergmann, R.: Ftonto: a domain ontology for a fischertechnik simulation
production factory by reusing existing ontologies. In: LWDA, pp. 253–264 (2019)
14. Rehse, J.-R., Mehdiyev, N., Fettke, P.: Towards explainable process predictions for industry
4.0 in the dfki-smart-lego-factory. KI-Künstliche Intelligenz 33(2), 181–187 (2019)
15. Michael Ferguson. rosserial package summary (2018)
16. Colbert, E.J.M., Kott, A.: Cyber-security of SCADA and other industrial control systems, vol.
66. Springer (2016)
17. OASIS Standard Incorporating Approved Errata. Mqtt version 3.1. 1 plus errata 01. (2015)
18. Hillar, G.C.: MQTT Essentials-A lightweight IoT protocol. Packt Publishing Ltd (2017)
19. The HiveMQ Team. Publish & subscribe - mqtt essentials: Part 2 (2015)
Lab-Scale Smart Factory Implementation Using ROS 143

Marwan F. Abdelatti received the M.Sc. degree in Control Engineering from Cairo University,
Egypt in 2010. He is currently a Ph.D candidate in the department of Industrial Engineering at the
University of Rhode Island, USA. His current research focuses on smart manufacturing system,
intelligent agents, and parallel GPU computing. Marwan is involved in research projects including
GPU implementations of optimization algorithms for route-planning problems, and autonomous
robots, specifically, ground vehicles.

Manbir S. Sodhi obtained his undergraduate degree from Jadavpur University, Kolkata, and
graduate degrees from the University of Arizona. He is the Professor of Industrial and Man-
ufacturing Engineering at the University of Rhode Island. He has published many engineering
research articles involving manufacturing, sustainability, driving distractions and autonomy. For
over a decade, he has hosted the Global Product Sustainability Systems Meeting in Rhode Island.
He has also worked as a visiting scientist with NATO in Italy, and a visiting Professor at the Tech-
nical University of Braunschweig, Germany.
Cooperation, Sensing and Control
RosDrive: An Open-Source ROS-Based
Vehicular Simulator for STEM Control
Systems Classes Tutorial

Enio Vasconcelos Filho, Jones Yudi, Mohamed Abdelkader, Anis Koubaa,

and Eduardo Tovar

Abstract The study of control systems in the engineering courses is quite complex,
given the difficulty of some teachers in exemplifying and allowing the student to
understand how such systems affect the environment. In this context, the STEM
methodologies aim to fill this gap between the traditional classes and the student
comprehension of the topic through the active learning process. Realistic open-source
simulators can be interpreted as one solution for this STEM implementation, allowing
students to test, modify and create different configurations and sensors with a low-cost
environment. This work presents a flexible open-source 3D simulation framework,
based on ROS, of a line follower vehicle, using an embedded PID controller, a
camera for processing and detecting lines, and sonars for detecting and avoiding
obstacles. This simulator integrates several controller systems, allowing the student
to build consistent skills in control and related areas, analyze the impacts of models
configurations, and extends its knowledge to new techniques.

Keywords Control systems · Education · Simulator · STEM · Open-Source ·

ROS

E. Vasconcelos Filho (B) · A. Koubaa · E. Tovar

ISEP, CISTER Research Centre, Rua Alfredo Allen 535, 4200-135 Porto, Portugal
e-mail: [email protected]
A. Koubaa
e-mail: [email protected]
E. Tovar
e-mail: [email protected]
J. Yudi
Automation and Control Group, University of Brasilia, Brasilia, Brazil
e-mail: [email protected]
M. Abdelkader · A. Koubaa
Prince Sultan University, Riyadh, Saudi Arabia
e-mail: [email protected]; [email protected]
M. Abdelkader
Systemtrio Electronics L.L.C, Abu Dhabi, United Arab Emirates

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 147
A. Koubaa (ed.), Robot Operating System (ROS),
Studies in Computational Intelligence 1051,
https://doi.org/10.1007/978-3-031-09062-2_5
148 E. Vasconcelos Filho et al.

1 Introduction

Control system techniques are one of the most significant challenges in several engi-
neering courses. Since it requires extensive mathematical background, a theoretical
load is quite extensive, requiring effort to learn by students and teachers. Moreover,
there is still an inherent difficulty in transporting the studied theory to practice, mak-
ing it challenging to retain learning [1]. Thus, alternative teaching techniques [2]
can facilitate knowledge production and construction of the skills expected by the
agents involved. In addition, [3] concludes in their work that the student’s perception
of applicability and the ability to construct different solutions is a motivator for the
search for more knowledge.
This line of education development puts the student as a producer of dynamic
and practical knowledge. It should be encouraged to take an active and autonomous
attitude and not necessarily follow pre-established models [4]. So, the student can
go further and propose new solutions to existing problems and even create different
issues. Active student engagement in the learning process also helps to keep the moti-
vation to research and learn [5], using Active Learning techniques. Thus, integrating
different areas of knowledge, experimentation, and implementation allows the stu-
dent to retain more excellent expertise and develop new skills. This integration of
knowledge is called STEM—Science, Technology, Engineering, and Mathematics
[6].
A standard solution in many universities is using pre-defined laboratory sessions,
using commercial kits such as [7–9]. Although such solutions are attractive, efficient,
and robust, they are often expensive and not flexible for experimenting and devel-
oping different solutions. Nevertheless, using Arduino development kits has shown
promising results as a learning tool [10]. This study suggests the development of
kits that can be used throughout the semesters, gradually increasing the project’s
difficulty [11] and even in specific dynamics and control systems projects [12].
In the same line of knowledge integration, other low-cost projects have been devel-
oped and implemented, giving students greater flexibility in experimenting with tech-
niques and knowledge. For example, in [13], the authors proposed an educational
line-following robot based on Arduino, allowing the implementation of low-level
control techniques. A similar application is presented in [14], proposing an even
lower cost robot with less flexibility. The increased complexity of possible control
algorithms is achieved on other platforms, such as those seen in [15, 16]. How-
ever, such applications imply a significant increase in project costs. Thus, although
Arduino-based solutions integrate the theoretical model and practice regarding con-
trol aspects, they present a limitation regarding the complexity of the algorithms,
given the restriction of processing capacity and design flexibility, due to the need to
purchase different sensors.
A solution that combines a low-cost implementation with flexibility and allows
knowledge retention through experimentation and active learning is based on realistic
open-source simulators. In STEM, a simulator represents a crucial stage of devel-
opment and education, reducing the time to produce prototypes. Thus, emulating
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 149

a real scenario with physical interactions allows the development of safety tests in
different environments and situations. As a result, it is possible to experiment with
techniques, analyze results and propose solutions flexibly, with great speed and less
cost.
The work done in [17] presents some of these tools, comparing simulators such
as Webots [18], Gazebo, and ROS [19], using criteria such as supported operating
systems, programming languages, documentation, tutorials, among others. In addi-
tion to these tools, others have been developed over time, such as the one presented
in [20], where a virtual laboratory is designed so that students can experiment with
models of line-following robots for competitions. However, such a simulator does
not allow the 3D visualization of the models, allowing only the testing of the pro-
posed algorithms. Another interesting simulator is proposed in [21], which presents
CARLA, an open-source simulator aimed at autonomous-driving research in this
work. It is a very realistic simulator with many items, with several physical inter-
actions between the components. However, despite being an extensible platform for
new developments, its vehicle control methods are limited to artificial intelligence
learning models without control models.
The authors of [22] present a simulator that uses a competition model to teach
robotics based on ROS. An autonomous robot capable of traveling a path is used in
this simulator, following directions on the track. Such a simulator showed promising
results when crossing the designated paths but presented the limitation of not using
a realistic vehicle model or even different control models. It has also been used
in competition simulation, which increases students’ comprehension and stimulates
self-learning [23, 24].
Seeking to use the advantages of a simulator capable of emulating realistic vehi-
cles, RosDrive is presented. A flexible platform based on ROS and the 3D simulator,
Gazebo, for studying different models of vehicle control. RosDrive uses an electric
vehicle model [25], with several sensors capable of covering different routes and
avoiding obstacles. Thus, the student will be able to implement additional control
strategies, analyze the system’s responses, and visualize the impacts of theoretical
models and their variables on the simulated scenarios. In addition, the tool allows
the use of different strategies in different vehicles, allowing the comparison between
the adopted models. For instance, a line follower controller mode with obstacle
avoidance will be implemented to exhibit the simulator results. The tool’s flexibility
allowed its extension for the study of communication models [26], the development
of hardware in the loop (HIL) simulation [27], and the implementation of the same
control model on a testbed platform [28]. As it is an open-source tool, the full code
access is provided in Sect. 5, for general use, with all the necessary steps for its
installation. The general simulator environment is illustrated in Fig. 1. The concrete
contributions of this paper are presented below:

– Present RosDrive, a realistic open-source 3D vehicle simulator that allows the stu-
dents to apply different control models techniques, improve their practical knowl-
edge in the control area, and create new strategies to perform vehicle movements;
150 E. Vasconcelos Filho et al.

Fig. 1 General simulator architecture

– Introduce a flexible simulator architecture, that allows modules exchange, param-

eters configurations, and system response analysis and visualization;
– Demonstrate the simulator flexibility due to the use of a camera and sonar sensors
to perform a Line Follower algorithm, and an Obstacle Detection and Avoidance
strategy integrated with a PID Cruise Controller model;

This paper is organized as follows: Sect. 2 shows the Simulator Architecture, the
leading technologies, and the minimal system requirements. The control and percep-
tion modules including the speed control algorithm, the image processing module
with the heading controller, and the collision avoidance module are introduced in
Sect. 3. The results of designed scenarios are presented in Sect. 4, while the main
conclusions are drawn in Sect. 5. Finally, the software installation and execution
instructions are shown in Sect. 5.

2 Simulator Architecture

This section will introduce the simulator tools and their general architecture, provid-
ing details about the vehicle model and data analysis.
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 151

Fig. 2 Publish/subscribe
model

2.1 Robot Operating System (ROS)

ROS is an open software developed by Open Source Robotics Foundation. It is a

robotic middleware with many software frameworks for robot software development.
It provides hardware abstraction, enabling users to avoid low-level problems, with
profound device control, communication between nodes and processes, and packet
management. ROS-based functions are realized in nodes that may post, receive and
reproduce control features, sensor data, state of the node, or general messages. ROS
is not a real-time framework or a Real-time Operating System (RTOS). This project
will be used in ROS Melodic distribution.
The basic concepts of ROS are nodes, Master, messages, and topics. The Master
node works as a central node of the system, storing data and information regarding
the ROS Nodes. Nodes inform their registration information to the Master and then
can receive data from other nodes. The Master is also responsible for reporting the
nodes, using Callbacks, if new information or connections are made. The nodes
exchanges messages using the publish/subscribe method, as described in Fig. 2.
Due to its flexibility, ROS has been used in several vehicular applications, such as
ground [29], aerial [30], and water [31] and many other robotic platforms. As a con-
solidated open-source community, several new libraries are available and supported,
at the same time that it is highly portable between platforms, including embedded
platforms [32]. The extensive material allows a quick learning curve for the student,
enabling a simple familiarization with the commands and interfaces and quickly
creating new modules.

2.2 Gazebo

One of the critical aspects of a learning-oriented system is its ability to present the
results of user interactions intuitively. Thus, the high capacity of ROS to integrate with
other platforms shows itself to be a competitive advantage since its functionalities
can be extended, expanding the experimentation horizon. For example, integration
with a robotic simulation tool helps to visualize the iterations between objects simply,
152 E. Vasconcelos Filho et al.

aiding in learning [33]. One of the most used tools for robotic simulation in ROS is
Gazebo. The Gazebo is an open-source 3D robotics simulator, for indoor and outdoor
environments, with multi-robot support that allows a complete implementation of
dynamic and kinematic physics and a pluggable physics engine. Furthermore, it pro-
vides a realistic rendering of backgrounds, including high-quality lighting, shadows,
and textures. In addition, it can model sensors that “see” the simulated environment,
such as laser range finders, cameras (including wide-angle), Kinect style sensors,
among others.
The Gazebo present the same message interface as the rest of the ROS ecosystem.
So, the development of ROS nodes is compatible with simulation, logged data, and
hardware. Many projects integrate ROS with Gazebo, such as the QuadRotor pre-
sented in [34], the Humanoid implementation in [35], and the Ground Vehicle in [29].
As a powerful and very visual tool, Gazebo has also been used as the simulation envi-
ronment for several technology challenges and competitions, such as NASA Space
Robotics Challenge (SRC) [36], Agile Robotics for Industrial Automation Compe-
tition (ARIAC) [37], and Toyota Prius Challenge [38].
Gazebo is responsible for realistically mimicking the system’s fundamental
dynamics, representing physical issues such as mass, inertia moment, friction, and
even collisions. To ensure better representation, Gazebo supports four engines: Sim-
body [39], Bullet Physics [40], ODE [41], and DART [42]. Such engines guarantee
a wide range of representations, bringing simulations closer to reality, offering the
student a greater possibility of representing theoretical concepts practically.
Although some Gazebo components show some lag with new technologies, its
overview still has more advantages than the alternatives presented. For example,
Unity [43] has similarities in the implemented physics, but its integration with ROS
is still complex. Furthermore, the Webbots recently developed a ROS integration
but still do not have the same flexibility in implementing different physical models.
Finally, the Coppelia [44] does not have the same rendering quality [45] as Gazebo,
although it has similar flexibility and quality in physical representation.

2.3 Scenario and 3D Vehicle Model

The Gazebo allows the construction of several different scenarios, including as many
objects as desired. Those objects can be static or dynamic and controlled in the
simulation. For illustration, this work introduces the track presented in Fig. 3 with
and without obstacles. Those obstacles can be removed or added by the user.
ROS applications have a launch file that allows the easy start of several applications
with previously saved scenarios and desired configurations. In this project, the file
car_demo.launch is responsible for starting the track, and the cars.launch defines the
vehicle’s initial coordinates and model. The simulator flexibility allows different car
models, including or removing sensors, modifying their positions and configurations.
The sensors data can be real-time observed though the RViz software.
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 153

Fig. 3 Track model

Simulated 3D Prius front view Prius Sensors Position

Fig. 4 Prius Gazebo model [25]

The 3D car model used in this work was presented in [25]. Figure 4 illustrates
the Hybrid Prius 3D model’s main details. Its fundamental dynamics are contained
in the node PriusHybridPlugin.cpp, and the model’s characteristics can be edited in
prius.urdf.
The primary vehicle controllers, such as throttle, brake, steering, and gear, can
be actuated by publishing to a ROS topic. Thus, the vehicle Powertrain will control
154 E. Vasconcelos Filho et al.

Fig. 5 Prius information structure

the gear control in this simulation. The simulated vehicle also has multiple sensors:
16-beam LIDAR on the roof, eight ultrasonic sensors, four cameras, and two planar
LIDARs. However, adding or removing sensors is a simple task that allows adjust-
ments, as necessary. Furthermore, implementing the vehicle with all the kinematics
and basic controls enables the study of other project aspects, such as motion control,
platooning, stability, and detection and avoidance models.
The vehicle information flowchart is illustrated in Fig. 5. All the vehicles have
the same model, and the simulation is composed of n ∈ N vehicles. The full set of
cars can be defined as carn = {i ∈ N|0 ≤ i ≤ n}. The information provided by each
module/node is:

– cari /initial Pose: defines the vehicle’s initial position

– cari / prius: new vehicle settings—throttle, break, steering
– cari /jointstates: conditions of each vehicle component—wheels and steering
– cari /car I N F O: vehicle’s current state—throttle, brake, speed, latitude, longi-
tude, steering, heading, etc.
– cari /camera: vehicle’s onboard cameras info
– cari /sonar : vehicle’s onboard sonars info
– cari /lidar : vehicle’s onboard LIDARs info
– PriusHybridPlugin: dynamics and vehicle model
– Environment Interaction: Gazebo calculation about interactions
– Camera (1 . . . n), Sonar (1 . . . m) and Lidar (1 . . . p): sensor nodes

All the sensors can be added, removed, or modified in the file prius.urdf. The
vehicle control is managed through the data sent to cari / prius topic, which works
as the vehicle’s input center, receiving throttle, brake, and steering. The throttle and
brake have a limit from 0 to 1, and the steering has a range from −30◦ to +30◦ . Its
format is defined in the “Control.msg”. To better understand the text, the rest of this
text will refer to a generic simulated vehicle identified by the “i” index, unless in
cases where some differentiation is necessary.
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 155

2.4 System’s Outputs

As a simulator for learning purposes, the system’s outputs are essential. Moreover,
as a STEM application, with many details, several analysis must be performed using
a mathematical approach. The output data will allow the study and comparison of
each simulation, allowing the student to evaluate the impact of slight differences
in the system’s response in each experiment. The system’s outputs are provided in
.csv files generated during the simulation. The module listener.py is responsible for
collecting the desired vehicle’s data in the related topics and exporting that to a .csv
file.
During the simulation, the topic cari /car I N F O can be used to perform a Real-
Time system evaluation, showing the vehicle’s most important information, like
coordinates, heading, speed, throttle, and brake conditions. The listener.py collects
this data and adds some information to the simulation’s output file, triggered by the
car’s movement or spent time. The output file contains the timestamp, coordinates,
speed, speed error, throttle and brake percentage, heading, heading error, and sonar
information, in this version.

3 Control Algorithms

This section will introduce the controller models used in this simulator. Then, it
will discuss the Cruise Controller (CC), the Line Follower characteristics, and the
Obstacle Avoidance Strategy. The Prius model simulates sensors that publish to the
cari /car I N F O topic. This topic contains the main data about the vehicle, like
latitude, longitude, altitude, heading, speed, direction, steering angle, acceleration
pedal percentage, and brake pedal percentage. All the data is updated every 0.01 s.

3.1 Vehicle Model

The vehicle model used in this work is based on the two-degree-of-freedom bicycle
system, as shown in Fig. 6. This model considers the car’s rotation around the z-
axis (θ) and its lateral velocity. Assuming x and y as the vehicle’s frame coordinates,
respectively, and θ its rotation in the z-axis, X, Y, and  are their absolute equivalents
in the global frame. Thus, the vehicle frame can be expressed using the rotation angle
θ in the global frame ( = θ). Finally, The steering angle, expressed in the vehicle’s
frame, is defined as δ and admits that both wheels turn the same value. By applying
Euler–Newton equations [46], it is possible to simplify the vehicle’s dynamics in the
plane as:
m ẍ = m ẏ θ̇ + Fx F + Fx R , (1a)
156 E. Vasconcelos Filho et al.

Fig. 6 Vehicle 2D model and coordinates

m ÿ = −m ẋ θ̇ + FyF + Fy R , (1b)

I θ̈ = a FyF − bFy R + c(−Fx F,l + Fx F,r − Fx R,l + Fx R,r ), (1c)

where I is the inertia moment, m is the vehicle mass, and Fx and Fy are the forces
in x and y directions, and the subscriptions r and l indicates the force direction
compound. Finally, the kinematic model, translated to X and Y coordinates, can be
described as:
Ẋ = ẋ cos Θ − ẏ sin Θ, (2a)

Ẏ = ẋ sin Θ − ẏ cos Θ, (2b)

Θ̇ = θ̇. (2c)

3.2 Cruise Controller (CC)

Like an actual vehicle, the Prius model does not allow direct speed control but only
throttle and brake adjustments. So, this simulator adopts a Proportional Integral
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 157

Fig. 7 PID CC

Derivative (PID) strategy to the vehicle speed controller. Although this controller is
quite simple, it will help the student to develop basic control skills and move towards
other implementations, including several autopilot strategies and tuning models [47].
In this way, the Cruise Controller (CC) will be responsible for keeping the vehi-
cle constant speed during the vehicle’s movement and adjusting it when necessary,
changing the brake and the throttle pedals, through the cari / prius topic. The PID
equation is defined as follows:

Δεσ (t)
α(t) = K P ∗ εσ (t) + K I ∗ εσ (t)dt + K D ∗ , (3)
dt

where K P , K I , and K D denote the Proportional, Integral, and Derivative gain con-
stants respectively, εσ (t) is the speed error, measured by the difference between the
current speed value and the desired one and α is the desired system acceleration.
The α is then normalized to a value between −1 . . . 1, representing the Throttle and
Brake pedals usage. A positive value indicates that the Throttle pedal has been used
while the Brake is free. Conversely, the Brake is pressed for a α negative value, and
the Throttle pedal is free. The complete controller is illustrated in Fig. 7, where it is
assumed that the time constant of the actuator is much bigger than the motor one,
and the CC algorithm is summarized in Algorithm 1.

Algorithm 1 Cruise Controller Algorithm

Input: Speed Set Point, Current Speed
Output: Throttle and Brake percentage
1: εσ ← speed_set_ point − curr ent_speed
2: α ← P I D(εσ )
3: αcontr ol ← N or mali zed[−1 . . . 1](α)
4: if αcontr ol ≥ 0 then
5: throttle ← αcontr ol
6: brake ← 0
7: else
8: throttle ← 0
9: brake ← αcontr ol
10: end if
158 E. Vasconcelos Filho et al.

3.3 Line Follower

In this work, the vehicle will simulate a standard trajectory path following method,
using a road line [48]. The simulated car has several cameras, and one of them is used
to identify the road line and follow it with real-time detection. The Line Follower
(LF) algorithm processes the captured image and delivers information regarding
the line position to the controller. The vehicle’s controller will keep its center over
the line with a second PID controller. The implemented algorithm is similar to the
one proposed in [49]. Nevertheless, as Gazebo provides a realistic camera view, it is
possible to implement algorithms without a real one, changing the image coordinates,
frame rate, data size, among other image capture characteristics, and evaluate the
changes’ impact over the controller.
An OpenCV node was implemented to read the data from the onboard front
camera. This node subscribes to the topic cari / f r ont_camera and virtually receives
all the images from the camera in an 800 × 800 pixels frame. Then, the LF algorithm
filters the image to find a vertical line in the track, and the detection is performed using
the Progressive Probabilistic Hough Transform (HT) [50]. This method is commonly
used in image processing and can help detect any shape if it can be represented in
mathematical form.
The Line Detection (LD) algorithm is illustrated in three frames of Fig. 8. The first
one, in Fig. 8a shows the vehicle camera simulated view. The LD algorithm applies
a mask over this image to filter it, highlighting a particular color. This color can be
adjusted following the Red Green Blue (RGB) model. The filtered image is then
converted to a greyscale picture, as presented in Fig. 8b, allowing the edges detec-
tion using the Canny Edge detection [51], using vectors with Cartesian coordinates.
Finally, these edges are integrated with the HT, defining a most probably line to be
followed, as demonstrated in Fig. 8c.
The line coordinates are published in cari /line_data topic and can be used by
the LF controller module. This module is called controller and is responsible for the
vehicle’s motion controller. The vehicle heading error (εθ ) related to the line reference

Vector and position extraction

Vehicle front camera view Line Detection - Final View

Fig. 8 Line detection process

RosDrive: An Open-Source ROS-Based Vehicular Simulator … 159

Fig. 9 Vehicle heading error

(εθ )

is defined as a relative measurement, using the center of the image frame as illustrated
in Fig. 9. In this figure, the Detected Line is the output of the LD algorithm, with
(x1 , y1 ) and (x2 , y2 ) respectively the initial and the final coordinates. The X max and
Ymax represent the frame limits and X C is the frame center point in X axis. The εθ is
defined as the angular difference between the X C and the (x2 , y2 ) coordinates, given
by Eq. 4. Finally, the controller calculates the car’s Steering Wheel Angle, using the
PID control action presented in Eq. 5.

X C − x2
εθ (t) = arcsin (4)
y2 − YC

Fig. 10 Line detection and

driving controller
160 E. Vasconcelos Filho et al.

Δεθ (t)
θwheels (t) = K Pθ θ
∗ ε (t) + K Iθ ∗ εθ dt + K Dθ ∗ , (5)
dt

where K Pθ , K Iθ and K Dθ denote the Proportional, Integrator, and Derivative gain con-
stants, and θwheels is the Steering Wheel Angle to be applied to the vehicle. Figure 10
shows the general LF flowchart, including the controller action, while the complete
LF algorithm can be observed in Algorithm 2.

Algorithm 2 Line Follower Algorithm

Input: Image Frame
Output: Steering Angle
1: Mask image to find Vertical Lines
2: Filter image to obtain Data Vectors
3: Line_V ector s ← H ough_Line_T rans f or m
4: Line_Coor dinates ← M E RG E(Line_V ector s)
5: εθ ← Eq. 4
6: θwheels ← P I D(εθ )

3.4 Obstacle Detection and Avoidance

Obstacle detection and avoidance is one of the most common autonomous vehicular
application, given the demanded safety conditions. So, in this simulator demonstra-
tion, sonars are used to detect and avoid unpredicted obstacles and help the vehicles
to keep the LF algorithm. The cari will use six sonars: four in the car’s front and
one on each side of it, as seen in Fig. 11. The simulator allows the user to change
the sonar’s positions and ranges and add or remove them in prius.urdf file. This
simulator assumes that the obstacles are positioned near the reference line and have
the same lateral size as the vehicles.

Fig. 11 Sonar visualization

RosDrive: An Open-Source ROS-Based Vehicular Simulator … 161

Algorithm 3 Line Follower with Detection and Avoidance Algorithm

Input: Sonars Info, Image Frame
Output: Steering Angle
1: while Object_Detected do
2: if Right_Object_Detected then
3: θwheels ← Le f t_Deviation
4: else if Le f t_Object_Detected then
5: θwheels ← Right_Deviation
6: end if
7: end while
8: Line_Follower _Algorithm (Algorithm 2)

Fig. 12 General vehicle architecture, with line detection and detection and avoidance modules

The sonars are used together with the LF algorithm. However, as the Detection and
Avoidance (DA) algorithm has priority over the LF, it assumes the vehicle controller
until the obstacle is out of view and the LF is reactivated. So, the algorithm 3 is an
extension of the LF algorithm. When the sonars detect an obstacle, the DA controller
turns the vehicle in the opposite direction, within a fixed θwheels value. This heading
adjustment is continued until the four front sonars stop detecting the obstacle. Then,
the lateral sonars avoid the vehicle trying to return to the line before it overtakes the
obstacle. Finally, the LF algorithm uses the last information about the detected line
to return to the desired trajectory. The DA block diagram is presented in Fig. 12.

4 Experimental Validation

A control simulation environment should present several controller tools to the stu-
dent. This section will introduce three main tools developed in RosDrive that allow
the student to analyze the vehicle’s controller performance and elaborate on different
strategies to guarantee its safety. The vehicle’s controller performance can be defined
in several ways, including fuel consumption, final speed, acceleration, among others.
In this chapter, the performance is measured by the vehicle’s capacity to track the
setpoint, both in speed and heading adjustments.
162 E. Vasconcelos Filho et al.

Table 1 Cruise Controller PID Settings

CC KP KI KD
P I D1 10.8 0.0 0.0
P I D2 10.8 2.16 0.270
P I D3 10.8 4.32 0.135
P I D4 10.8 2.16 0.135

4.1 Cruise Controller Implementation

The CC was developed as an independent module. So, it works as a black box

implementation, where the inputs are the setpoint and current vehicle’s speed, and
the outputs are the throttle and brake percentage, while the controller parameters
are adjusted inside the module. This architecture choice increases the simulator’s
flexibility, allowing the user to replace the controller and adjust its parameters.
Taking into account Fig. 3, the straight line between the points 7 and 1, without
obstacles, was used to evaluate the CC and check how does the vehicle behaves with
several accelerations and decelerations. In this scenario, the vehicle speed setpoint
was changed from 20.0, to 14.0, 16.0, 12.0 m/s and finally 0.0 m/s. All the speed
settings are defined in the controller.py file in the parameters section. They are related
to the vehicle’s current position on the track.
The controller parameters K P , K I , and K D were defined with the Ziegler Nichols
(ZN) empirical method [52]. The vehicle was accelerated from a rest position until
it reached the first setpoint speed in the proposed scenario. Increasing K P until the
system oscillation limit, it was possible to determine the ultimate gain (K u ), at 18,
with a period of 0.1 ms. These values show how the vehicle’s actuator has a rapid
response since the oscillation period is fast. In this test, the Ziegler Nichols Tuning
parameters are K P = 10.8, K I = 2.16, and K D = 0.135. The system’s response is
presented in Fig. 13.
As described above, the RosDrive was designed so that the student can change the
system’s characteristics and observe the impact on the vehicle’s response. In addition
to changing the controller model, the change of control parameters already implies
different responses to be analyzed, providing the user with a practical study of the
characteristics of each one of them. Three variations of the parameters obtained with
ZN are proposed to exemplify their impacts on the vehicle’s control action. These
parameters are shown on the Table 1, where P I D1 is a proportional-only controller,
P I D2 increases the derivative component, P I D3 enforces the integrator component,
and finally P I D4 shows the parameters obtained by the method ZN.
Figure 13a and b show in detail the impact of controller changes on the system
response, and in Fig. 13b it is possible to observe that the control proportional-only
(P I D1) presents a more significant oscillation and that the increase of the derivative
component (P I D2) makes the response slower, but with a smaller overlap. On the
other hand, the increase in the integrator component (P I D3) makes the system faster
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 163

Speed Comparison Speed Comparison

25
PID1 - 10.8-0.0-0.0 20.15 PID1 - 10.8-0.0-0.0
PID2 - 10.8-2.16-0.270 PID2 - 10.8-2.16-0.270
PID3 - 10.8-4.32-0.135 PID3 - 10.8-4.32-0.135
20.1
20 PID4 - 10.8-2.16-0.135 PID4 - 10.8-2.16-0.135

20.05
Speed (m/s)

15

Speed (m/s)
20

19.95
10
19.9

19.85
5

19.8

0
0 5 10 15 20 25 30 5.9 6 6.1 6.2 6.3 6.4 6.5 6.6
Time (s) Time (s)

Vehicle’s Speed Response Vehicle’s Speed Detail Response

Fig. 13 Vehicle speed response to different PID parameters

but with a greater overshoot on the setpoint. In this scenario, the parameters obtained
by ZN show a better response as they are at an intermediate point of response time and
overshoot. In all models presented, the controller error was minimal with a maximum
steady-state value lower than 0.006 m/s.
The PID CC example shows the student the basic vehicle controller models. It
translates the conceptual controller view to a practical application, reducing the gap
between the theoretical aspects and the implementation one, allowing the develop-
ment of active skills and opening the doors to the student’s creativity. A shortly
CC RosDrive demonstrator is presented in https://youtu.be/QFVwgFyhaF4. All the
video demonstrators links are presented in Sect. 5.

4.2 Line Follower (LF) Controller

The LF controller is responsible for the vehicle’s heading adjustment, performed by

the Heading Controller (HC). This control ensures that the vehicle safely makes the
circuit curves, preventing accidents. As can be seen in Fig. 3, in this scenario, tighter
curves were chosen, allowing the student to analyze more complex situations, such
as car skidding. Under these conditions, the vehicle’s controller is adjusted in one
curve and then evaluated its performance on the whole circuit.
Initially, the circuit’s curves radius were analyzed to define the maximum speed
that would prevent the vehicle from √ going off at the curve’s tangent. The maximum
speed (vout ) is given by |vout | = μ · |g| · R, where μ is the friction’s coefficient,
|g| is the gravity acceleration and R is the curvature ray. In the proposed scenario, it
is defined that μ = 0.9, g = −9.8 m/s, and R = 18.38 m, which means that vout =
13.34 m/s.
164 E. Vasconcelos Filho et al.

Table 2 HC PID parameters

HC Speed K Pθ K Iθ θ
KD HC Speed K Pθ K Iθ θ
KD
(m/s) (m/s)
RE F 13 10.0 0.0 0.5 P I D8 15 10.0 1.0 1.0
P I D5 15 10.0 0.0 0.5 P I D9 15 10.0 0.5 0.0
P I D6 15 10.0 0.0 1.0 P I D10 15 10.0 0.5 0.5
P I D7 15 10.0 0.0 1.0 P I D11 15 10.0 1.0 0.0

The heading controller has a different evaluation in comparison with the CC.
In the CC, the setpoints are defined through a step function, while in the HC, the
setpoints function follows the curve design, with a long transition phase. It means
that the system’s response should be evaluated after the desired heading is constant.
Due to this condition, a more complex scenario is proposed to evaluate the system’s
response in adversarial conditions. Initially, the vehicle’s trajectory was fixed with
vout as the heading reference. Then, the objective was to find the most suitable HC
PID parameters for the system with v = 15 m/s. It means that the HC will suffer
from skidding. In this scenario, the student’s experience determining the best HC
PID parameters will be necessary since the ideal conditions presented in theory are
not present. Furthermore, it will increase the student’s perception of the problem and
stimulate creative new solutions since the vehicle’s speed increase will increase the
skidding, compromising the system’s stability. It is also important to highlight that
the user can set up any speed and check its response.
The HC PID parameters were obtained initially in curve 7 given the long straight
lines before and after. The obtained parameters are presented in Table 2. Figure 14a
presents the vehicle’s trajectory on curve 7, while Fig. 14b perform an in-depth view
of the same curve. Both figures illustrate how the reference HC has a smoother
trajectory, with no skidding. As expected, there is some skidding in all the HC PID
configurations, with a speed setpoint of 15 m/s. However, these figures analysis allow
the identification of the best controller performance, even on these conditions. So,
in the proposed scenario, the P I D7 presents a better response due to the derivative
action, avoiding extreme adjusts keeping the vehicle’s trajectory near to the R E F
trajectory. On the other hand, the integral action presented in P I D8 and P I D11
configurations produces more oscillation and increases the distance between the
R E F and the performed trajectory due to the skidding.
Figure 15 presents the vehicle’s heading error (εθ ), due to the different HC PID
parameters. As expected, while the LF reference adjusts the heading setpoint, the
vehicle’s heading suffers from much oscillation, trying to respond to the new con-
ditions. However, the system’s response can be better studied after the transition,
when the LF algorithm sets the new line. This situation can be observed in Fig. 14b.
This figure highlights the smooth response of P I D7, with little oscillation above the
R E F response. Nevertheless, the systems’ response with P I D8 and P I D11 have
a considerable overshoot and take much more time until the stabilization.
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 165

Trajectory Comparison - Curve 7 Trajectory Comparison - Curve 7

180
REF 65
REF
PID5 - 10-00-00
160 PID5 - 10-00-00
PID6 - 10-00-0.5
60 PID6 - 10-00-0.5
PID7 - 10-00-01 PID7 - 10-00-01
140 PID8 - 10-01-01 PID8 - 10-01-01
PID9 - 10-0.5-00 55 PID9 - 10-0.5-00
PID10 - 10-0.5-0.5 PID10 - 10-0.5-0.5
120 PID11 - 10-01-00 PID11 - 10-01-00
Latitude (m)

Latitude (m)
50
100
45
80
40
60
35
40
30
20
50 100 150 200 250 300 60 70 80 90 100 110
Longitude (m) Longitude (m)

Vehicle’s Trajectory - Curve 7 Vehicle’s Detail Trajectory - Curve 7

Fig. 14 Vehicle trajectory analysis (curve 7) under different HC PID settings

Heading Error (Degrees) Heading Error (Degrees)

0.8
REF
PID5 - 10-00-00
0
0.6 PID6 - 10-00-0.5
PID7 - 10-00-01
REF PID8 - 10-01-01
-1
PID5 - 10-00-00 0.4 PID9 - 10-0.5-00
PID6 - 10-00-0.5 PID10 - 10-0.5-0.5
-2 PID7 - 10-00-01 PID11 - 10-01-00
Error (Degree)

Error (Degree)

PID8 - 10-01-01 0.2

PID9 - 10-0.5-00
-3 PID10 - 10-0.5-0.5
PID11 - 10-01-00 0

-4
-0.2

-5
-0.4

-6
-0.6

100 150 200 250 90 95 100 105 110 115 120 125
Time (s) Time (s)

Vehicle’s Heading Error (εθ ) Vehicle’s Heading Error (εθ ) Detailed View

Fig. 15 Vehicle heading error(εθ ) at curve 7 under different HC PID settings

The HC PID response analysis can be extended to the entire circuit. Looking at the
heading error (εθ ) presented in Fig. 15, the best controllers response were performed
by the PD configurations, namely the P I D6 and P I D7. Furthermore, a full lap was
performed to evaluate the vehicle’s heading controller, comparing its trajectory and
the general εθ . Figure 16a present the vehicle’s trajectory comparison in the full lap.
It shows the vehicle’s skidding in all the curves and the most distinguished one in
curve 4. Thus, Fig. 16b highlight the vehicle’s trajectory in this curve, showing that
although all the HC PID configurations suffer from high skidding on this curve, the
P I D7 configuration provides a smaller skidding and is the faster one to stabilize
the system after the curve. Finally, the Fig. 16c shows a comparison between the
general εθ during the full lap. It demonstrates that the R E F configuration has the
smallest error variation during the circuit and that the P I D7 error response is the
most approximate to it.
166 E. Vasconcelos Filho et al.

Trajectory Comparison - Full Circuit Trajectory Comparison - Full Circuit

400 200
REF REF
PID5 - 10-00-00 PID5 - 10-00-00
350 PID6 - 10-00-0.5 190 PID6 - 10-00-0.5
PID7 - 10-00-01 PID7 - 10-00-01
300 180

250 170
Latitude (m)

Latitude (m)
200 160

150 150

100 140

50 130

50 100 150 200 250 300 350 400 450 130 140 150 160 170 180 190 200 210
Longitude (m) Longitude (m)

Vehicle’s Trajectory - Full Lap Vehicle’s Detail Trajectory - Curve 4

Heading Error - Full Circuit
8

4
Error (Degrees)

-2

-4

-6
EF

7
D

D
R

PI

PI

PI

Heading Controller (HC)

Vehicle’s Heading Error (εθ ) at a Full Lap

under best HC PID Settings

Fig. 16 Vehicle trajectory analysis (Full Lap) under different HC PID settings

This scenario was built to illustrate the simulator’s flexibility, merging the LF algo-
rithm with the HC method under an adversarial context. In this way, the student will
be able to extend its capabilities, changing the controller’s parameters and checking
the system’s response, proposing new situations, and evaluating them. Furthermore,
it will help students build and reinforce their capabilities and skills without damaging
any equipment by extrapolating the commonly encountered theoretical conditions.

4.3 Obstacle Detection and Avoidance

In addition to the analysis of the CC and HC controllers on the vehicle’s perfor-

mance in isolated scenarios, RosDrive allows the analysis of its interaction with
other vehicles, whether static or dynamic.
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 167

Trajectory Comparison - DA Static Obstacles Trajectory Comparison - DA Static Obstacles

60
REF REF
350 PID5 - 10-00-00 PID5 - 10-00-00
PID6 - 10-00-0.5 PID6 - 10-00-0.5
PID7 - 10-00-01 55 PID7 - 10-00-01
300

250 50
Latitude (m)

Latitude (m)
200

45
150

100
40

50

35
0
50 100 150 200 250 300 350 400 450 225 230 235 240 245 250
Longitude (m) Longitude (m)

Vehicle’s Trajectory Static Obstacle Vehicle’s Trajectory Detail Static Obstacle

Avoidance - Full Lap Avoidance
Heading Error - DA Static Obstacles
8

4
Error (Degrees)

-2

-4

-6
EF

7
D

D
R

PI

PI

PI

Heading Controller

Vehicle’s Heading Error (εθ ) with Static

Obstacle Avoidance

Fig. 17 Vehicle trajectory analysis (static obstacle avoidance) under different HC PID settings

4.3.1 Static Obstacle Detection and Avoidance

The vehicle’s ability to perform a trajectory avoiding several close obstacles was
initially analyzed, comparing the heading error (εθ ) given the HC parameters changes.
The 19 static obstacles are illustrated in Fig. 3 and are modeled as Pickup vehicles.
These obstacles are positioned on the straight circuit lines avoiding curves overtaking.
Again, the R E F vehicle running with a 13.0 m/s speed was presented against the
P I D5, P I D6, and P I D7 HC configuration, running at 15.0 m/s. The vehicle’s
trajectory is presented in Fig. 17a, illustrating that the skidding is still present, mostly
in curve 4.
Furthermore, the vehicles do not necessarily follow the same trajectory to avoid
obstacles. This situation is illustrated in the straight line between curves 4 and 5,
where the vehicle with the HC P I D6 avoids the last obstacle with a left turn and
the others perform a right curve. This obstacle avoidance action responds to the first
168 E. Vasconcelos Filho et al.

Trajectory Comparison - DA Dynamic Obstacles Trajectory Comparison - DA Dynamic Obstacles

450 47
REF - 12 m/s 14 m/s 16 m/s
13 m/s 15 m/s 20 m/s
400 46

350 45

300 44
Longitude (m)

Longitude (m)
250 43

200 42
REF - 12 m/s
150 13 m/s 41
14 m/s
15 m/s
100 16 m/s 40
20 m/s

50 39
0 5 10 15 20 25 30 50 100 150 200 250 300 350 400 450
Time (s) Time (s)

Vehicle’s Dynamic Obstacle Avoidance Vehicle’s Dynamic Obstacle Avoidance

Longitude trajectory

Fig. 18 Vehicle trajectory analysis (dynamic obstacle avoidance)

sonar activated in the vehicle. As the vehicle’s trajectories are slightly different due
to the HC response, the car’s angular position at that point is not the same for all the
configurations, providing different sonar activation. The same situation is observed
in the last obstacle between curve 7 with all the HC PID configurations compared
with the R E F.
To compare the HC PID’s performance in a static detection and avoidance con-
dition, Fig. 17b emphasize the vehicle’s movement over the second track obstacle. It
is possible to observe that the HC PID performance follows the model presented in
the LF algorithm, with P I D7 providing the best system’s response in comparison
with R E F. However, the εθ ’s variation between R E F, P I D6, and P I D7 is similar,
given the rapid heading transitions triggered by the ODA and the LF algorithms, as
presented in Fig. 17c. On the other hand, the boxplots of P I D7 εθ in both Figs. 16c
and 17c show that although the maximum variation is similar, these errors appear
more frequently, no longer presenting themselves as outliners but as values that are
repeatedly perceived. These errors happen due to several obstacles and the constant
need to adjust the vehicle’s position caused by the HC action.

4.3.2 Dynamic Obstacle Detection and Avoidance

RosDrive’s flexibility allows different control models, algorithms, and movement

strategies. Thus, it is possible to evaluate strategies for overtaking vehicles in motion,
observing how the fastest vehicle behaves and if it can perform the maneuver safely.
To conduct this demonstration, the car1 and car2 , with v1 (t) > v2 (t) were defined,
with x2 (0) = x1 (0) + 30 m. This way, car1 has time to reach its maximum speed
before the sonar detects the presence of car2 and only then starts the overdrive
process.
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 169

The vehicles.launch allows the setup and launches of the necessary vehicles, with
no additional development. In this file, the vehicle models, their initial positions, and
the algorithms to be used are instantiated. Initially car1 (R E F) and car2 are defined
with the same CC and HC parameters as P I D4 and P I D7. In the same file, the car2
sonars are deactivated, avoiding its reaction to car 1 presence. So, car 1 accelerates,
detects the presence of car2 and performs the overtake action. As the same ODA
strategy presented in Sect. 3.4 is applied, after the obstacle detection, car 1 will return
to the line only after overtaking the obstacle, when the lateral sonars indicate that
there are no more obstacles there.
In this scenario, v2 was defined as 13.0, 14.0, 15.0, 16.0, and 20.0 m/s and v1 =
12 m/s, while the front sonar’s ranges are set to 20 m and the lateral ones are 2 m.
When theoretically studying physical systems, it is common to analyze that vehi-
cles are points in space and that overtaking, for example, is just a matter of validating
the relationship between space traveled in time, having as reference the speed of the
two points. However, in a realistic simulator, vehicles cannot be treated as points in
space but as bodies that can collide and must avoid this to remain safe. Thus, the
overtaking process begins with detecting the body ahead, followed by a diversion
action and consequent movement.
This controller action of car1 is illustrated in Fig. 18. Figure 18a illustrates a
more simplified view of the system, indicating the longitudinal trajectory of car1
and car2 under all different velocity conditions of car1 . It is important to note that
the movement of car2 is the same in all scenarios, as its speed is constant, and its
movement is not affected by car1 . In this figure, it is possible to observe the crossing
point when the curve referring to car 1 crosses the curve of car2 . Thus, it is seen that
the speed of 13.0 m/s, car1 is not able to exceed car2 on the desired route, indicating
an unsafe maneuver.
However, extending this view to a 2D dimension, as presented in Fig. 18b, it
is possible to analyze how does the overtake movement is performed. Under the
proposed conditions, only the vehicle with a 20 m/s speed has overtaken car2 with
no oscillatory movement. With other speeds, car1 lateral sonar detects car2 presence,
and adjust car1 heading position, avoiding a collision. This figure illustrates how car1
with a 13.0 m/s speed cannot overtake car2 in the desired time and finally, how the
same movement with 14.0 m/s is dangerous since the overtaken process ends just at
the limit of the desired trajectory.
In this way, the student has more information to check how the movement was
performed and propose different safety strategies. In addition, this scenario allows
the student to create new sonars detection algorithms, different controller strategies,
and some intelligent systems to increase the system’s safety.
170 E. Vasconcelos Filho et al.

5 Conclusions and Future Works

This chapter presents RosDrive, an open-source simulator for developing the study of
vehicle control techniques in a realistic way. It presents the conditions that make this
simulator able to bring together the students’ theoretical lessons with their practical
implementation, reducing gaps in their training. By using ROS as a development
platform, RosDrive is compatible with the most diverse Linux systems and has con-
tinuous support. In addition, its wide adoption by the community allows for the rapid
introduction of new sensors and components and integration with other platforms.
Scenarios were developed to evaluate vehicle speed and heading control models
using systems similar to actual vehicles. These systems use acceleration and braking
variation and cameras to analyze the environment and take action. Furthermore, algo-
rithms were introduced to avoid collisions between controlled vehicles and static or
dynamic obstacles, providing different analyzes and conditions for different control
and security systems.
For this, RosDrive features a modular architecture based on a flexible set of easily
adjustable and interchangeable control tools to assist in the extrapolations imagined
by students and teachers who will use the tool. With this, we firmly believe that active
learning can be reinforced in classrooms, introducing low-cost dynamic models that
mimic reality, increasing the applicability of knowledge, and consolidating the skills
necessary for the formation of the control engineer.
Shortly, we hope to develop communication between vehicles, enabling the
exchange of information between them, simulating scenarios referring to the ITS
models. We also hope to extend the configuration modules for more intuitive plat-
forms using windows that facilitate the user’s vision. Finally, we plan to introduce
the use of RosDrive in a classroom context, visualizing the difficulties of students
and teachers and analyzing the impacts of its adoption in teaching control systems.

Supplementary Material: Simulator Installation and

Execution Instructions

This section presents the main requirements for installing and configuring RosDrive
Simulator. Furthermore, it summarizes the prerequisites and indicates the current
repository for downloading the necessary files.

5.1 Main Code Repository

– The complete code can be found at: https://github.com/enioprates/rosdrive

RosDrive: An Open-Source ROS-Based Vehicular Simulator … 171

5.2 Main Requirements

– Operating System: Ubuntu 18.04

– ROS Distribution: Melodic
– Gazebo: 9 or above
– Python: 2.7 or above
– GCC: 7.3 or above

5.3 Setup Project

1. Install ROS Melodic following the instructions of:

http://wiki.ros.org/melodic/Installation/Ubuntu.
2. Download files from Github RosDrive Simulator:
https://github.com/enioprates/rosdrive
3. RosDrive Setup
– RosDrive Simulator
(a) Open a new terminal inside .../CISTER_car_simulator
(b) Type: catkin_make
– RosDrive Simulator Controllers
(a) Open a new terminal inside .../CISTER_image_processing
(b) Type: catkin_make

5.4 How to Run RosDrive Simulator

1. Run the simulator:

(a) Open a new terminal inside .../CISTER_car_simulator
(b) Type: source devel/setup.launch
(c) Type: roslaunch car_demo demo_t.launch
The GAZEBO should open on your screen with the vehicle in the position defined
in cars_t_curve_2.launch. The vehicle will remain stopped until the Vehicle Con-
troller is turned on.
2. PAUSE the simulation and reset the time!
3. Starting the Vehicle Controllers:
172 E. Vasconcelos Filho et al.

(a) Open a new terminal inside .../CISTER_image_processing

(b) Type: source devel/setup.launch
(c) Type: roslaunch image_processing vehicle.launch
This launcher file will start the vehicle’s Line Follower Detection and the Vehicle’s
movement controller, respectively with the nodes lane_lines_detection.py and
simulation_connector.py.
4. Start the Gazebo simulation

5.5 Optional Configuration

1. Speed PID Configuration

(a) Open ....../CISTER_image_processing/src/scripts/simulation_connector.py
(b) Find #Speed PID
(c) Adjust K P → kp_vel, K I → ki_vel, and K D → ki_vel
2. Heading PID Configuration
(a) Open ....../CISTER_image_processing/src/scripts/simulation_connector.py
(b) Find #Steering PID
(c) Adjust K Pθ → kp_steer _l, K Iθ → ki_steer _l, and K Dθ → kd_steer _l

5.6 RosDrive Video Demonstrations

1. Cruise Controller demonstrator:

https://youtu.be/QFVwgFyhaF4.
2. Line Follower demonstrator:
https://youtu.be/sRlXk2K1IJc
3. Static Obstacle Detection and Avoidance demonstrator:
https://youtu.be/YEoO2CQUiKc.
4. Dynamic Obstacle Detection and Avoidance demonstrator:
https://youtu.be/5135G3aafLw

Acknowledgements This work was partially supported by National Funds through FCT/MCTES
(Portuguese Foundation for Science and Technology), within the CISTER Research Unit
(UIDP/UIDB/04234/2020); also by the FCT and the EU ECSEL JU under the H2020 Framework
Programme, within project(s) ECSEL/0010/2019, JU grant nr. 876019 (ADACORSA). Disclaimer:
This document reflects only the author’s view and the Commission is not responsible for any use
that may be made of the information it contains.
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 173

References

1. Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., Sugai, G.: Evidence-based practices in
classroom management: considerations for research to practice. Educ. Treat. Child. 31(1),
351–380 (2008)
2. Maciejewski, A.A., Chen, T.W., Byrne, Z.S., De Miranda, M.A., Mcmeeking, L.B.S., Notaros,
B.M., Pezeshki, A., Roy, S., Leland, A.M., Reese, M.D., Rosales, A.H., Siller, T.J., Toftness,
R.F., Notaros, O.: A holistic approach to transforming undergraduate electrical engineering
education. IEEE Access 5, 8148–8161 (2017)
3. Jamison, A., Kolmos, A., Holgaard, J.E.: Hybrid learning: an integrative approach to engineer-
ing education. J. Eng. Educ. 103, 253–273 (2014)
4. McKenna, A.F.: Educating engineers: designing for the future of the field. J. Higher Educ. 81,
717–719 (2010)
5. Shooter, S., Mcneill, M.: Interdisciplinary collaborative learning in mechatronics at Bucknell
University. J. Eng. Educ. 91, 339–344 (2002)
6. Xie, Y., Fang, M., Shauman, K.: STEM education. Ann. Rev. Sociol. 41, 331–357 (2015)
7. WEG: Learning workbenches for training. Teaching Equipment. Teaching Equipment. Electric
Panels. WEG - Products, May 2020. Library Catalog: https://www.weg.net/
8. Nulle, L.: Lucas Nülle - Lucas-Nuelle Training Systems for vocational training and didactic
(2020)
9. Feedback PLC: Welcome to Feedback plc (2020). Library Catalog: https://fbkmed.com/
10. Galadima, A.A.: Arduino as a learning tool. In: 2014 11th International Conference on Elec-
tronics, Computer and Computation (ICECCO), pp. 1–4 (2014)
11. Chancharoen, R., Sripakagorn, A., Maneeratana, K.: An Arduino kit for learning mechatronics
and its scalability in semester projects. In: 2014 IEEE International Conference on Teaching,
Assessment and Learning for Engineering (TALE), pp. 505–510 (2014)
12. Omar, H.M.: Enhancing automatic control learning through Arduino-based projects. Eur. J.
Eng. Educ. 43, 652–663 (2018)
13. Gonçalves, J., Pinto, V.H., Costa, P.: A line follower educational mobile robot performance
robustness increase using a competition as benchmark. In: 2019 6th International Conference
on Control, Decision and Information Technologies (CoDIT), pp. 934–939 (2019). ISSN:
2576-3555
14. Serrano Pérez, E., Juárez López, F.: An ultra-low cost line follower robot as educational tool
for teaching programming and circuit’s foundations. Comput. Appl. Eng. Educ. 27, 288–302
(2019)
15. Cruz-Martín, A., Fernández-Madrigal, J., Galindo, C., González-Jiménez, J., Stockmans-Daou,
C., Blanco-Claraco, J.: A LEGO mindstorms NXT approach for teaching at data acquisition,
control systems engineering and real-time systems undergraduate courses. Comput. & Educ.
59, 974–988 (2012)
16. Leonard, M., Morgan, J., Coffelt, J.P.: Digital systems teaching and research (dstr) robot: a
flexible platform for education and applied research. In: Proceedings of the 2018 ASEE Gulf-
Southwest Section Annual Conference. The University of Texas at Austin, p. 5, ASEE (2018)
17. Staranowicz, A., Mariottini, G.L.: A survey and comparison of commercial and open-source
robotic simulator software. In: Proceedings of the 4th International Conference on PErvasive
Technologies Related to Assistive Environments - PETRA ’11, Heraklion, Crete, Greece, p. 1.
ACM Press (2011)
18. Michel, O.: Webots: professional mobile robot simulation. Int. J. Adv. Rob. Syst. 1, 5 (2004)
19. Open Source Robotic Foundation: ROS/Introduction (2018)
20. Suwasono, S., Prihanto, D., Dwi Wahyono, I., Nafalski, A.: Virtual laboratory for line follower
robot competition. Int. J. Electr. Comput. Eng. (IJECE) 7, 2253 (2017)
21. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An Open Urban Driving
Simulator (2017). arXiv:1711.03938 [cs]
22. Costa, V., Rossetti, R., Sousa, A.: Simulator for teaching robotics, ROS and autonomous driving
in a competitive mindset. Int. J. Technol. Human Interact. 13, 19–32 (2017)
174 E. Vasconcelos Filho et al.

23. Grover, R., Krishnan, S., Shoup, T., Khanbaghi, M.: A competition-based approach for under-
graduate mechatronics education using the arduino platform. In: Fourth Interdisciplinary Engi-
neering Design Education Conference, pp. 78–83 (2014). ISSN: 2161-5330
24. Regueras, L.M., Verdu, E., Munoz, M.F., Perez, M.A., de Castro, J.P., Verdu, M.J.: Effects of
competitive E-learning tools on higher education students: a case study. IEEE Trans. Educ. 52,
279–285 (2009). Conference Name: IEEE Transactions on Education
25. Tully Foote: Demo of Prius in ROS/GAZEBO (2017). https://github.com/osrf/car_demo
26. Vieira, B., Severino, R., Filho, E.V., Koubaa, A., Tovar, E.: COPADRIVe - a realistic simulation
framework for cooperative autonomous driving applications. In: IEEE International Conference
on Connected Vehicles and Expo - ICCVE 2019, Graz, Austria, pp. 1–6. IEEE (2019)
27. Filho, E.V., Severino, R., Rodrigues, J., Gonçalves, B., Koubaa, A., Tovar, E.: CopaDrive: an
integrated ROS cooperative driving test and validation framework. In: Robot Operating System
(ROS), vol. 962, pp. 121–174. Springer International Publishing (2021). Series Title: Studies
in Computational Intelligence
28. Filho, E.V., Guedes, N., Vieira, B., Mestre, M., Severino, R., Gonçalves, B., Koubaa, A., Tovar,
E.: Towards a cooperative robotic platooning testbed. In: IEEE International Conference on
Autonomous Robot Systems and Competitions (ICARSC), Ponta Delgada, Portugal, pp. 332–
337. IEEE (2020)
29. Rivera, Z.B., De Simone, M.C., Guida, D.: Unmanned ground vehicle modelling in
gazebo/ROS-based environments. Machines 7, 42 (2019)
30. Furrer, F., Burri, M., Achtelik, M., Siegwart, R.: RotorS—A Modular Gazebo MAV Simulator
Framework, pp. 595–625. Springer International Publishing, Cham (2016)
31. Vaz, F.C., Portugal, D., Araújo, A., Couceiro, M.S., Rocha, R.P.: A localization approach for
autonomous underwater vehicles: a ros-gazebo framework (2018)
32. Kim, M.S., Delgado, R., Choi, B.-W.: Comparative study of ros on embedded system for a
mobile robot. J. Autom. Mob. Robotics Intell. Syst. 12, 61–67 (2018)
33. Kuosa, K., Distante, D., Tervakari, A., Cerulo, L., Fernández, A., Koro, J., Kailanto, M.:
Interactive visualization tools to improve learning and teaching in online learning environments.
Int. J. Distance Educ. Technol. 14, 1–21 (2016)
34. Meyer, J., Sendobry, A., Kohlbrecher, S., Klingauf, U., von Stryk, O.: Comprehensive simula-
tion of quadrotor UAVs using ROS and gazebo. In: Simulation, Modeling, and Programming
for Autonomous Robots, vol. 7628, pp. 400–411. Springer, Berlin, Heidelberg (2012)
35. Feng, H., Wong, C., Liu, C., Xiao, S.: ROS-based humanoid robot pose control system design.
In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki,
Japan, pp. 4089–4093 (2018)
36. Hambuchen, K.A., Roman, M.C., Sivak, A., Herblet, A., Koenig, N., Newmyer, D., Ambrose,
R.: NASA’s space robotics challenge: advancing robotics for future exploration missions. In:
AIAA SPACE and Astronautics Forum and Exposition, Orlando, FL. American Institute of
Aeronautics and Astronautics (2017)
37. NIST: Agile Robotics for Industrial Automation Competition (2020). https://www.nist.gov/el/
intelligent-systems-division-73500/agile-robotics-industrial-automation-competition. Last
Modified: 2020-07-09T09:19-04:00
38. Open Source Robotics Foundation: Prius Challenge (2017)
39. Sherman, M., Eastman, P.: Simtk - simbody: Multibody physics api
40. PyBullet: Bullet real-time physics simulation
41. Smith, R.: Ode - open dynamics engine
42. Lab, G., Lab, H.R.: Dynamic animation and robotics toolkit - dart
43. Hussein, A., García, F., Olaverri-Monreal, C.: Ros and unity based framework for intelligent
vehicles control and simulation. In: 2018 IEEE International Conference on Vehicular Elec-
tronics and Safety (ICVES), pp. 1–6 (2018)
44. Robotics, C.: Coppelia robotics
45. Rizzardo, C., Katyara, S., Fernandes, M., Chen, F.: The importance and the limitations of
sim2real for robotic manipulation in precision agriculture (2020)
RosDrive: An Open-Source ROS-Based Vehicular Simulator … 175

46. Attia, R., Orjuela, R., Basset, M.: Combined longitudinal and lateral control for automated
vehicle guidance. Veh. Syst. Dyn. 52, 261–279 (2014)
47. Vartika, V., Singh, S., Das, S., Mishra, S.K., Sahu, S.S.: A review on intelligent pid controllers in
autonomous vehicle. In: Reddy, M.J.B., Mohanta, D.K., Kumar, D., Ghosh, D. (eds.) Advances
in Smart Grid Automation and Industry 4.0 (Singapore), pp. 391–399. Springer Singapore
(2021)
48. Tesla: Transitioning to Tesla Vision (2021)
49. Kondakor, A., Torcsvari, Z., Nagy, A., Vajk, I.: A line tracking algorithm based on image pro-
cessing. In: 2018 IEEE 12th International Symposium on Applied Computational Intelligence
and Informatics (SACI), Timisoara, pp. 000039–000044. IEEE (2018)
50. OpenCV: Hough Line Transform (2019)
51. OpenCV: Canny Edge Detection (2022)
52. Kobatake, K., Okazaki, T., Arima, M.: Study on optimal tuning of pid autopilot for autonomous
surface vehicle. IFAC-PapersOnLine 52(21), 335–340 (2019). 12th IFAC Conference on Con-
trol Applications in Marine Systems, Robotics, and Vehicles CAMS 2019

Enio Vasconcelos Filho is currently Assistant Professor at the

Federal Institute of Goias (IFG). Since 2018, he has been carry-
ing out his Ph.D. in Electrical and Computer Engineering at the
University of Porto (FEUP), Portugal. He joined the CISTER
Research Unit in 2018, working with Cyber-Physical Cooper-
ative Systems, notably in vehicular communications. He has a
BSc degree in Automation and Control (2006) and an M.Sc.
degree in Mechatronics Systems (2012), both from Universidade
de Brasilia (UnB). His most significant interests are autonomous
vehicles, communication networks, and artificial intelligence in
real-time applications.

Jones Yudi is a Professor at the University of Brasília since

2010, in the area of Industrial Automation. Doctor in Electrical
Engineering & Information Technology from Ruhr-Universität
Bochum (2018), Master in Mechatronic Systems from the
University of Brasília (2010), with a degree in Control and
Automation Engineering (Mechatronics) from the same Univer-
sity (2007). Specialist in the design of hardware/software archi-
tectures for high-performance embedded systems, focusing on
image and signal processing, instrumentation and control. Expe-
rience in distributed systems, sensor networks, ubiquitous com-
puting, Industrial Internet of Things and Industry 4.0.
176 E. Vasconcelos Filho et al.

Mohamed Abdelkader is currently an Assistant Professor at

Prince Sultan University, Riyadh, KSA, where he leads mul-
tiple R&D projects related to autonomous delivery drone sys-
tem, and autonomous drone hunting system, in the Robotics
and Internet of Things lab. Abdelkader obtained his Ph.D. in
Mechanical Engineering from King Abdullah University of Sci-
ence Technology (KAUST) working with real-time distributed
autonomous multi-robot systems for pursuit-evasion applica-
tions. His main research works are in the development of AI-
powered swarm robotic systems, and safe autonomous naviga-
tion. He is a technical team lead at Systemtrio, Abu Dhabi,
UAE, developing autonomous robotics solutions including a
UAV swarm system for security and surveillance, affiliated with
the IEEE Robotics and Automation Society, and a reviewer
in several robotics conferences and journals including ICUAS,
IROS, and JINT.

Anis Koubaa is a Professor in Computer Science, Advisor to

the Rector of Research Governance, Director of the Research
and Initiatives Center, and Director of the Robotics and Inter-
net of Things Research Lab, in Prince Sultan University. He is
also a Senior Researcher in CISTER/INESC TEC and ISEP-
IPP, Porto, Portugal. He is also a Senior Fellow of the Higher
Education Academy (HEA) in the UK. He received several dis-
tinctions and awards including the Rector Research Award in
2010 at Al-Imam Mohamed bin Saud University and the Rec-
tor Teaching Award in 2016 at Prince Sultan University. He is
on the list of top 2.

Eduardo Tovar received the Licentiate, M.Sc. and Ph.D.

degrees in electrical and computer engineering from the Univer-
sity of Porto, Porto, Portugal, in 1990, 1995 and 1999, respec-
tively. Currently he his Professor in the Computer Engineering
Department at the School of Engineering (ISEP) of Polytechnic
Institute of Porto (P. Porto), where he is also engaged in research
on real-time distributed systems, wireless sensor networks, mul-
tiprocessor systems, cyber-physical systems and industrial com-
munication systems. He heads the CISTER Labs, an internation-
ally renowned research centre focusing on RTD in real-time and
embedded computing systems. He is currently the Vice-chair
of ACM SIGBED and is member of the Executive Committee
of the IEEE Technical Committee on Real-Time Systems (TC-
RTS). He is currently deeply involved in the core team setting-
up a Collaborative (industry-academic) Lab on Cyber-Physical
Systems and Cyber-Security Systems.
Autonomous Laser-Induced Breakdown
Spectroscopy System for Chemical and
Mineralogical Designation of
Interplanetary Materials

George Stavrinos, Elias Chatzitheodoridis, and Olga Sykioti

Abstract The invention of the laser has pushed the boundaries of technological
advancements with its variety of uses. This use case chapter analyses the imple-
mentation of an Autonomous Laser-Induced Breakdown Spectroscopy System for
Chemical and Mineralogical Designation of Interplanetary Materials. In more detail,
it highlights all the major components of the development of an autonomous system
that maps the material composition of a meteorite sample using the Laser-Induced
Breakdown Spectroscopy method. In the first chapter all key concepts are presented
while offering a short literature survey regarding (calibration-free) Laser-Induced
Breakdown Spectroscopy. It also highlights the use of the Robot Operating System
(ROS) as a tool for the implementation of hardware drivers and autonomy compo-
nents. Continuing with the details of the implementation, the manuscript focuses
on the reference and description of all major hardware components. In the third
chapter, the software components of the implementation are discussed. These com-
ponents include hardware drivers, autonomy capabilities, data visualisation and user
interfacing. For interested readers, chapter four offers a quick tutorial on compiling,
configuring and running the implemented software. Chapter five summarises the
experimental setup and presents results. Finally, this chapter concludes with a quick
summary and an overview of the limitations of the implementation along with future
work.

Keywords Laser-Induced Breakdown Spectroscopy · LIBS · Mineralogy

G. Stavrinos (B) · E. Chatzitheodoridis · O. Sykioti

National Observatory of Athens and National Technical University of Athens, Athens, Greece
e-mail: [email protected]
URL: https://gstavrinos.github.io/
E. Chatzitheodoridis
e-mail: [email protected]
URL: https://www.eliasch.gr/
O. Sykioti
e-mail: [email protected]
URL: https://members.noa.gr/sykioti/

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 177
A. Koubaa (ed.), Robot Operating System (ROS),
Studies in Computational Intelligence 1051,
https://doi.org/10.1007/978-3-031-09062-2_6
178 G. Stavrinos et al.

1 Introduction

From the invention of the laser in 1959 by Gordon Gould [17], there have been a lot of
studies on how to exploit its capabilities. One of the most significant observations was
that a concentrated pulsing laser beam could generate radiating plasma on specific
materials. This discovery was of major importance since it could potentially be used
in the field of spectrochemistry.

1.1 Laser-Induced Breakdown Spectroscopy

Laser-Induced Breakdown Spectroscopy (LIBS), sometimes mentioned as Laser-

Induced Plasma Spectroscopy (LIPS), is based on the high photon density laser
pulses, which when reacting with a material’s surface they create plasma. In the last
forty years, LIBS has been flourishing as a technique to qualitatively and quantita-
tively analyse sample materials.
The main principle of the LIBS technique is to extract the emission spectrum from
a spot of a material, in order to acquire its elemental composition. This seemingly
simple procedure can lead to extraordinary results and can be applied to a wide
variety of applications [21]. Some applications include analysis of cultural heritage
items [1] and archaeological conservation state [30], to meteorite samples [14] and
interplanetary soil [24].
A typical LIBS setup includes a pulsed laser system that is controlled by a pulse
generator. The pulsing laser beam is focused on a sample (in this case a meteorite),
on which the high density of photons induces radiation from the generated plasma.
This radiation is received by an optical fiber which is connected to a spectrograph.
The whole spectra acquisition procedure is driven by a computer system, which also
performs all data processing and storage.
Once the spectra data are acquired on the computer, a wide variety of elemental
analysis can be conducted on various materials, from molten steel [36] and glass [34]
to jewellery [16] and meteorites [12].
A typical setup of a LIBS system is shown in Fig. 1 in the form of a flow diagram.
The importance of a pulse generator lies on the fact that the spectrograph needs
to acquire the spectrum after a very strict time frame (of about 1 µs). This delay
is required because the pulsed laser system during plasma generator creates other
physical phenomena like fluorescence that have a strong effect on the acquisition
data. Thus, in order to minimise their effects, the pulse generator induces a precise
delay between the laser pulse and the spectrograph acquisition.
The above is demonstrated on Fig. 2 as presented in [11]. Initially, and immedi-
ately together with the pulse we have a strong signal from the plasma continuum.
When plasma cools, then this diminishes/reduces and then we can open the spec-
trograph (gate) for acquisition. Then the characteristic element excitation is clearly
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 179

Fig. 1 A typical Laser-Induced Breakdown Spectroscopy setup

visible compared to the reduced plasma continuum (see inserts left top without the
delayed gating, and top right the same material with delayed gating of the acquisition
spectrograph).

1.2 LIBS Advantages and Disadvantages

The LIBS technique has been extensively studied and already there is an extensive
number of publications [15, 19, 37] through which major advantages over other
techniques are reported as well as some limitations.
On of the most important advantages is that the LIBS experimental apparatus is
very simple to setup and use. Additionally, it can be highly miniaturised to allow
fieldwork or to be employed as payload of a mission to another planet [10, 27, 43].
Another important advantage of the LIBS technique is its ability to analyse any
type of material, being gas [40], liquid [32] or solid [26]. This makes it very versatile
and allows for a variety of experiments without the need to change the apparatus
setup or prepare the sample. Additionally, it can analyse the very light and heavy
elements.
180 G. Stavrinos et al.

Fig. 2 A schematic overview of the temporal history of a LIBS plasma. The gate delay and window
are shown. Inserts illustrate the kind of spectra one might observe at the different times

The results of the LIBS analytical technique are available instantly after each laser
pulse. This makes its application extremely useful for in-situ and real-time scenarios.
The limitations of the LIBS are the matrix interference, the requirement for post-
processing techniques and the bad repeatability of the measurement that does not
allow easy quantification.
The most important one could be interference (matrix) effects during the excitation
of the material and the generation of radiating plasma. The presence of the matrix
effects is due to the LIBS technique itself. With LIBS, in order to analyse a sample,
plasma needs to be created. This generation of plasma involves the ablation of the
prototype material, which leads to fluctuation of the expected result. This fluctuation
is mainly due to the fluorescence of the plasma generation procedure. In addition to
the ablation of the sample, the plasma generation also produces free electrons which
can falsely be read by the spectrograph as part of the sample. The plasma that is
generated by the pulsing laser beam can also absorb part of the beam itself while
cooling, leading, again, to inaccurate results. All of these constitute the matrix effects
that can greatly affect the quality of the acquisition results.
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 181

Table 1 LIBS advantages and disadvantages summarising table

Advantage Disadvantage
Simple apparatus Matrix noise effects
Works with any material Need for post-processing
Detects all elements even the light ones Fewer research resources
Instant results Setup-dependent precision
In-situ and real-time applications

The generated noise leads us to the next disadvantage of LIBS, which is the
need for post-processing filtering techniques. Post-processing techniques can vary
from simple averaging of multiple acquisitions to more complicated LIBS dedicated
methods [21] like calibration-free LIBS [8].
Another disadvantage that has been in the centre of research using the LIBS
technique is the poor precision values and the precision dependence on a variety of
experimental setup factors (i.e. sample composition) [5].
LIBS is still a technique under development and the even increasing bibliography
and its technical advancements are expected to bring it to the required maturity,
especially in performing quantitative measurements.
Table 1 summarises the advantages and disadvantages of the LIBS technique.

1.3 Calibration-Free LIBS

Studying the disadvantages of the LIBS method, a major observation arises; even
though the qualitative results of the technique are instant and easy to acquire, they
lack the accuracy found in other methods, such as the acquisition using a scanning
electron microscope (SEM), for quantitative analyses, and thus require corrections
during post-processing that is performed through the comparison with standards
of similar chemical matrix or calibration curves. This process can prove very time-
consuming, and this is why researchers have worked into implementing a new method
of LIBS called Calibration-Free Laser-Induced Breakdown Spectroscopy (CF-LIBS)
[8].
With traditional LIBS, as stated earlier, the matrix effects must be reported and
documented as accurately as possible in order to subtract them from the result of the
experiment. CF-LIBS avoids this last step, nullifying the need for intricate matrix
databases.
In short, the CF-LIBS technique tries to bypass the problem of matrix noise effects
by incorporating the matrix itself into the problem of analysing the sample materials.
These changes in the matrix itself could be studied in order to derive information
about the underlying material composition.
182 G. Stavrinos et al.

In order for this hypothesis to be valid and dependable, a series of assumptions

need to be adopted (as seen in [42]):
– The plasma composition is representative of the unperturbed target composition.
This means simply that there are no probabilistic models behind each plasma
reading.
– The generated plasma is in Local Thermodynamic Equilibrium (LTE) in the tem-
poral and spatial sampling window. This means that ionisation states are modelled
based on the Saha-Boltzmann equilibrium equation. Practically (and simplified)
this means that the plasma composition is considered uniform and stable for the
duration of acquisition.
– The plasma can be modelled as a spatially homogeneous source.
– The spectral lines included in the calculation are optically thin.
These assumptions are very difficult to satisfy in most experiments. This is the
reason why CF-LIBS often presents divergence from the expected/desired results.
The work of Tognoni et al. [42] on the state of the art of CF-LIBS allows for an
in-depth analysis of the method which are out of the scope of this study.

1.4 LIBS Applications

LIBS applications include tests on analysing various materials, but also LIBS
exploitation for other multi-disciplinary problems.
All applications discussed in this section, have been implemented using either the
base LIBS method or the calibration-free one.
As a start, the wide variety of materials analysed using the (CF-)LIBS method
should be highlighted. As stated earlier, the LIBS methods have been used on molten
steel [36], glass [34], jewellery [16], meteorites [12], gases [40], water and ice [2]
and bacteria [33].
The LIBS technique offers only one type of information: the spectrum of each
laser pulse. This fact has not discouraged researchers to use it for an immensely
wide variety of applications. Using the materials discussed earlier as an example,
researchers can characterise (and even appraise) jewellery [16]. Another example is
the identification of bacteria [33] under different condition that enhances biologists’
understanding on bacteria discrimination based on different experimental environ-
ments.
Another very interesting field of LIBS application is interplanetary materials.
Many studies have been conducted in various materials, with the ultimate goals
to either understand the composition of a meteor from far away [12] or analyse
the surface of a planet with instrument payloads on rovers, such as the Curiosity
rover on Mars [33]. These studies can surely enhance our knowledge of our outer-
Earth environment. They can also provide vital information on habitable worlds and
exploitation techniques for interplanetary resource acquisition.
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 183

1.5 The Focus of This Study

A topic that a lot of studies have been conducted on, is the process of mapping sample
materials [3, 20, 23, 41]. This, in short, means that researchers are not only interested
in getting a single spot measurement from a sample, but rather scan it for a series
of matrix-like measurements to produce a material composition map. This can be
extended to different modes of analyses, such as line and depth profiles, along the
surface of the sample and vertical to it inside the sample, or three-dimensional (3D)
maps, taking advantage of the destructive character of the technique (i.e., removal
of material with ablation and crater formation).
In general, mapping is achieved by moving the sample material (or, in cases, the
focus point of a beam) in a two-dimensional plane while reading data from the pulsing
laser beam in each individual position of the spatial array. This can be achieved by
an X-Y moving plate like those found in 3D printers and CNC machines. Figure 3
shows a typical LIBS mapping setup as found in [4].
The autonomous mapping process is the main goal of this work. In order to achieve
such a task, precise control of the XY plate is required, as well as constant knowledge
of the position of the plate. With a robust control method, elemental mapping can be
achieved by introducing the LIBS technique.
The final product of this manuscript aims to be a basis of a complete autonomous
LIBS mapping system that could be used in the future for quantitative analyses using
the calibration-free LIBS method.
The steps required to create such a system include the design, collection and
construction of the mechanical/hardware part i.e. the laser system, the spectrograph,
the kinematic stage etc. Apart from the hardware, complementary software needs to
be developed in order to drive (control) each hardware component and also integrate

Fig. 3 A typical Laser-Induced Breakdown Spectroscopy mapping setup as presented in [4]

184 G. Stavrinos et al.

it into a single, seamless pipeline. The software implementation is extremely vital

for this custom setup, since it is the basis of all the features and capabilities of the
system. In addition, it should be implemented in a way that offers plug-and-play
hardware functionality, for easier component upgrade and swapping.
It has to be noted that this study focuses on the analysis of interplanetary materials
and more specifically meteorite samples. With that in mind, the system is focused
on geological/geochemical and mineralogical analyses of such samples and aims to
offer the base tools for easier interpretation of the results.

1.6 The Robot Operating System

Mapping the LIBS technique requires interfacing, visualising and planning the map-
ping process as a different challenge. In order to effectively tackle the mobility,
positioning and generally the automation tasks, the Robot Operating System (ROS)
[31] middleware was selected to be used. Among a variety of robotics and automation
frameworks, ROS has the largest community of users and available (open source)
software packages.
ROS is a set of software packages that offer interfaces for hardware abstraction
and almost plug-and-play autonomy capabilities. It is built around the philosophy
that each process, called node, runs in a distributed environment, allowing for easy
scalability and transferability of software. Even though it is oriented around robotic
platforms being wheeled or legged, the overall value of its already implemented
packages remains for the autonomous LIBS mapping implementation. More specifi-
cally, this study uses the ros_control [7] package and implements the robot hardware
interface (RobotHW) that allows for hardware abstraction, use of multiple input
commands, integration with planners etc. Details on the implementation of the ROS
software and its integration with the rest of the system is discussed later in this
manuscript.

2 In-Depth LIBS Instrumentation

2.1 Concept

As discussed in the introductory chapter, an autonomous LIBS mapping and analysis

system consists of a group of interconnected devices. Those devices, as shown in
Figs. 1 and 3, are the following:
– A computer (mainly) for control automation and data processing.
– A pulse generator with at least two transistor-transistor logic (TTL) digital outputs
to trigger for both the laser beam and the spectrograph.
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 185

– A pulsed laser, such as the commonly used solid state Nd:YAG (neodymium-doped
yttrium aluminum garnet) laser.
– A kinematic stage capable of moving on an X-Y plane, and in cases vertically to
it (Z direction).
– A sample material (in the case of this study, a meteorite sample), which is preferably
cut to a flat polished surface.
– An optical fiber to collect the radiation from the cooling plasma. It can be adapted
with a focusing lens in the entry side of the fiber and this focus point should be
positioned in the volume of the plasma.
– A spectrograph to analyse the spectra collected by the optical fiber.
The rest of this chapter is dedicated to reference and analyse each of the hard-
ware components used in this study. It has to be noted that the optical fibre and the
meteorite samples are not discussed in this chapter due to their simplicity and variety
respectively. The software side of the implementation and its usage are introduced
in the next chapters.

2.2 Instrumentation

All of the hardware equipment used for this study along with the facilities of the
experiments are located in Prof. Elias Chatzitheodoridis’ lab [6] in the School of Min-
ing and Metallurgical Engineering of the National Technical University of Athens,
Greece.

2.2.1 Instrumentation Summary

This chapter focuses on the hardware side of the autonomous LIBS mapping system.
The information circle that starts and ends on the computer is analysed. This circle
includes apart from the processing unit, the pulse generator, the laser device, the XY
plate movement system and the spectrograph. Figure 4 shows the complete setup
used for the work of this chapter. In addition, Table 2 summarises all the hardware
components of the setup.
Complementary to the hardware components of the LIBS system, is the software.
Software includes hardware drivers, communication protocols, autonomy, visualisa-
tion and user interface. All of these software components are discussed in detail in
the next chapter.

2.2.2 The Processing Unit

It would be rightful to report on the processing unit of the autonomous LIBS system
first. In the LIBS system of this use case chapter, the computer is the easiest to acquire
186 G. Stavrinos et al.

Fig. 4 The top view of the autonomous LIBS mapping setup of this study

Table 2 Hardware setup summary of this study

Computer Lenovo B50-80
Pulse generator BNC 588-OEM-1ns
Laser Litron Nano LG 250-20
Movement system motors Wantai 57BYGH420
Spectrograph Ocean optics USB4000

Table 3 Laptop specifications

Manufacturer Lenovo
Model B50-80
CPU i7-5500U
RAM 8 GB (DDR3)
GPU AMD Radeon R5 M330
VRAM 2 GB
Storage 1 TB (HDD)

and integrate with the rest of the components. Thus, the processing takes place in a
consumer grade laptop. Table 3 summarises the specifications of the processing unit
that has been used for development. It has to be noted that the implemented LIBS
system did not introduce any hard constraints on the required computer hardware.
Information regarding the software that was run on this laptop is discussed in the
next chapter.
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 187

2.2.3 The Pulse Generator

After the processing unit, the flow of information of a LIBS system leads to the
pulse generator. As stated earlier, the pulse generator is responsible for generating
the required TTL 5 volt signals for triggering a laser pulse from the laser system in
order to irradiate a sample and generate plasma on the meteorite sample.
The pulse generator used in this study is the Berkeley Nucleonics Corp (BNC)
Model 588-OEM-1ns. This model is a board level, digital pulse generator. These
specifications are important in order for the device to generate accurate time gaps
in the range of several µs between triggering signals. Table 4 presents the specifica-
tions regarding the model used for this manuscript. The given specifications are not
exhaustive for the system; only the most important are included which were used for
the development and operation of the newly setup LIBS system. Figure 5 shows the
pulse generator as seen by the original website of the company [9].
The pulse generator is connected to the main computer over a USB serial link.
Through this link, a set of commands are uploaded to the system with which the two
pulse channels would produce two consequent pulses. The code used is included in
the first annex of the chapter.

Table 4 Pulse generator specifications

Manufacturer Berkeley nucleonics corp
Model 588-OEM-1ns
Channel modes Single shot, burst, normal, duty cycle
Delay 0–1000 s
Pulse width 10 ns–1000 s
Resolution 1 ns
Accuracy 1.5 ns + 0.0001 delay
Time base 50 MHz, 25 PPM crystal oscillator

Fig. 5 The BNC

588-OEM-1ns pulse
generator
188 G. Stavrinos et al.

2.2.4 The Laser

The signals generated by the pulse generator are relayed to the power, control and
cooling unit of the laser system. The laser device used in this study is the Litron Nano
LG 250-20. Table 5 highlights the basic specifications of this laser device. It should
be noted that not all details have been included, but rather only those of interest for
the current study. Figure 6 shows the laser device as seen by the original website of
the company [25].

Table 5 Laser specifications

Manufacturer Litron
Model Nano LG 250-20
Laser type Solid State Nd:YAG, multimode, Q-switched
Max repetition rate 20 Hz
Max output energy @ 1064 nm/20 Hz 260 mJ
Max output energy @ 532 nm/20 Hz 139 mJ
Max output energy @ 266 nm/20 Hz 34 mJ
Beam diameter About 5 mm
Beam divergence <0.8 mrad
Pulse length @ 1064 nm 7–11 ns
Resonator type Telescopic
Timing jitter <0.5 ns

Fig. 6 The Litron Nano LG 250-20

Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 189

2.2.5 The XY Plate Movement System

The XY plate is one of the moving parts of the LIBS analytical system of this
study, and this should carry the sample as well i.e. it is responsible for moving the
flat surface of the meteorite sample along the laser beam in a controlled way, in
order to generate plasma at certain points on the surface of the sample and acquire
spectra. It is a custom-made build, based on OpenBuilds [28] parts and Wantai [44]
Nema 23 stepper motors. The hardware and the overall setup are based on concepts
made for CNC machines that offer high accuracy in movement. Table 6 presents the
specification of the movement system of the LIBS setup. Figure 7 shows a photograph
of the XY plate when it was disconnected from the other part of the setup for testing
purposes.

2.2.6 The Spectrograph

The final component of any LIBS system is the spectrograph that performs the data
acquisition, i.e. this device is responsible for reading the emitted radiation from the
generated plasma and create spectrum graphs. The spectrograph used in this study
is the Ocean Optics (Ocean Insight) USB4000. As its name suggests, it is can be
interfaced via USB, allowing for easy connection to any computing machine. Table 7
presents the specifications of the spectrograph. Figure 8 shows the device itself as
seen on the official website of the company [29].
Apart from the USB port, the USB400 spectrograph has a 22-pin connect for
external to external hardware for control and automation. From the 22 pins of the
connector, only two will be used in our setup, which are:
– The pin Nr 6, which is the electrical ground
– The pin Nr 7, which is the external trigger pin which receives TTL signals to
trigger acquisition. All other acquisition parameters are set by the software on the
computer through the USB port.
Figure 9 shows the 22-pin connector.

Table 6 XY plate movement system specifications

Motors manufacturer Wantai
Motors model 57BYGH420
Motors form factor Nema 23
Motors driving modes Single, double, interleaved, microstepping
Motors step accuracy ±5%
Motors step angle 1.8 degrees
X max travel 12.9 cm
Y max travel 17.8 cm
Resolution 250 µm
190 G. Stavrinos et al.

Fig. 7 The XY plate

movement system

Table 7 Ocean optics Manufacturer Ocean optics

USB4000 spectrograph
specifications Model USB4000
Detector sensor Toshiba TCD1304AP linear
CCD array
No. of elements 3648 pixels
Pixel size 8 µm × 200 µm
Detector range 200–1100 nm
A/D resolution 16bit
Entrance aperture 100 µm

Fig. 8 The Ocean Optics

USB4000 spectrograph
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 191

Fig. 9 The 22-pin connector

of the Ocean Optics
USB4000 spectrograph

3 LIBS System Software

The software components of the implemented autonomous LIBS mapping system

include custom device drivers and libraries already developed by the hardware
provider and software that was developed for the current LIBS system. The soft-
ware can be classified in the following four main categories:
– Hardware drivers, which mainly includes the software that controls the motors and
also the communication between the various nodes of the system.
– Autonomy, which mainly includes the planning and mapping procedures.
– Visualisation, which mainly includes code that generates ROS-compatible mes-
sages that can be visualised using the main ROS visualisation utility called RViz
[22].
– User Interface, which mainly includes user-oriented software that aims to simplify
the parameter configuration process, by providing graphical interface methods to
interact with the whole system.
The implemented software is open source and available through a github repos-
itory [38]. It consists of a variety of ROS packages each with its own nodes and
configuration files. The implemented ROS packages are:
– auto_libs_msgs: This package contains the custom ROS Services used for the
autonomous LIBS mapping system.
– auto_libs_description: This package contains the physical description of the XY
moving platform for visualisation, control and planning purposes.
– auto_libs_control: This package contains hardware drivers, laser/spectrograph
control software and movement-related configuration files.
– auto_libs_navigation: This package includes the planning software along with the
navigation control graphical user interface application and related configuration
files.
This chapter analyses all of the above components and offers a detailed description
of all the capabilities of the autonomous LIBS mapping system for meteorite samples.
This chapter focuses solely on the details of the implementation and not the usage or
the configurability. The latter are topics that are discussed in the next tutorial chapter.
192 G. Stavrinos et al.

3.1 Hardware Drivers

The XY kinematic stage is the piece of hardware that was implemented from ground
up. Other hardware components already came with their own factory-implemented
drivers like the Litron Nano LG 250-20 laser module and the BNC 588-OEM-1ns
pulse generator. These devices have either hardware interfaces or communicate with
other systems over serial ports and implement an internal command language set. The
Ocean Optics USB4000 spectrograph came with both proprietary and open source
drivers.
The XY kinematic stage driver software had to be implemented from scratch, since
it consisted of general purpose devices (CNC and 3D printing oriented), such as the
Wantai Nema-23 type motors, and a series of linear movement hardware components.
The specifications of the XY movement system have already been provided in the
previous chapter.
The Wantai motors are driven using an Arduino Mega. The Arduino is responsible
to send to the computer and also receive from it messages that allow the integration
with the rest of the software components in a distributed manner. In short, distributed
computing relies on a variety of machines and/or processes that simultaneously work
towards the solution of a problem while updating the required components with their
status.
The Arduino code, uploaded to the Arduino over its serial port and permanently
running when it is on power, follows the ROS communication protocol. This commu-
nication protocol has the rationale of “talkers” and “listeners”. Talkers (publishers in
the ROS language) send data to other software components and listeners (subscribers
in the ROS language) receive data and process them. Table 8 summarises the input
and output data of the XY movement system node.
With the data exchange in mind, it should be clear that the motor controlling
software receives commands either in the form of velocity or position and acts based
on that. As an output, the drivers only report the status of the system and also provide
feedback for debugging purposes.
The motor controlling software itself consists of a small set of functions that
include:

– The setup function.

– The motor controlling function.
– The velocity commands receiver function.
– The goal position receiver function.

Table 8 XY movement system drivers inputs and outputs

Input Output
Velocity commands Movement system status
Goal position Debugging info
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 193

– The movement system status reporting function.

– The main loop.
In the setup function, all the objects and variables are initialised including position
loggers and communication objects (publishers and subscribers). Most importantly,
in this function the movement system is self-calibrated. This is achieved by driving
the movement plate to the maximum X and Y positions and then recentering the plate
while also resetting the recorded position variables. This self-calibration takes place
when the device (Arduino) is powered on, when a new serial device is connected to
it and also per user request.
The motor controlling function implements the main capabilities of the motor
driver (Adafruit hardware integrated on the Arduino) by sending the appropriate
current to the stepper motors. As discussed in the previous chapter, the motors used
in this setup are also capable of performing multiple stepping modes apart from
single stepping.
The ‘interleaved’ and ‘microstepping’ modes are of major importance for the
autonomous LIBS mapping system. They provide double and quadruple stepping
accuracy respectively compared to the single stepping mode. The motor controlling
function exploits all three stepping modes in order to provide accurate and fast
control. More specifically, the motor controller always tries to maximise the step
distance, based on the absolute distance between the desired and the current position.
Figure 10 shows the motor stepping mode selection process in the form of a decision
tree diagram.
The velocity commands receiver function is the next motor controlling driver
function that should be discussed. This function is a simple ROS subscriber callback.
This means that it is triggered only when a new velocity command is sent (published)
by another node. The callback simply receives the commanded velocity and translates
it to a desired position based on the Eqs. 1 and 2.
In Eqs. 1 and 2, and considering that i represents either the x or the y movement
direction, pi is the desired position, i is the current position, veli is the commanded
velocity, si are the motor steps per centimeter and ki is a constant multiplier.

px = x + vel x × sx × 100 × k x , vel x ∈ [−1.0, 1.0] (1)

p y = y + vel y × s y × 100 × k y , vel y ∈ [−1.0, 1.0] (2)

Following the same rationale, the goal position receiver function is triggered only
when a goal position message is published by another node. The callback receives a
two-dimensional position and translates it to motor steps, while respecting hardware
limitations. In the case of the autonomous LIBS mapping system, these limitations
refer to the X and Y movement limits of the platform. Equations 3 and 4 show the
calculations required for the conversion from real world coordinates to motor step
positions. In Eqs. 3 and 4, and again i is replaced by axes x and y, pi is the desired
position, gi is the input position, and ti is the maximum travel in cm.
194 G. Stavrinos et al.

Fig. 10 The motor controller decision tree diagram

   
tx −tx tx
px = gx × 100 + × sx , gx ∈ , (3)
2 2 2
   
ty −t y t y
p y = g y × 100 + × sy , gy ∈ , (4)
2 2 2

The movement system status reporting function is a message publishing routine.

It is constantly called to report the position of the XY movement platform. This
function does not make any calculations. It just generates the appropriate message
for the Arduino complementary code. This complementary code is discussed later
in this section.
Finally, the motor controlling driver utilises the main loop function. The main loop
is responsible for making the appropriate calls to other functions in order to ensure
the normal driver operation. This infinite loop calls the motor controlling function
that was discussed earlier first. After successfully returning from that function, it
calls the status reporting function to update the corresponding subscribed nodes.
Finally, it calls the spin() function included in the main ROS library, in order to poll
for incoming messages.
Now that all of the Arduino code has been covered, another part of the motor
controlling driver should be referenced. As mentioned in the description of the status
reporting function, there is also complementary code for the Arduino part of the
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 195

Fig. 11 The complete hardware driver flowchart diagram

driver. This code consumes the status messages generated from the Arduino code,
and updates the platform’s “joints”. In the case of the autonomous LIBS mapping
system, the joints refer to the X and Y positions of the moving plate. In more compli-
cated setups, the joints could refer to actual robot arm/legs joints, or other physically
moving parts. The complementary code was developed in order to offer full ROS
integration by implementing the ros_control [7] interface. The complete list of advan-
tages of implementing the ros_control interface is out of the scope of this chapter,
but parts of it will be apparent in the following sections.
Figure 11 shows a complete diagram of the software threads connected in the
hardware driver implementation.

3.2 Autonomy

The ability to effectively control the moving parts of the hardware, allows for a
wide variety of additional capabilities and features, including autonomy. In the
autonomous LIBS mapping system, autonomy refers to the ability of the system
to plan, move, generate plasma, acquisition and data storage without requiring exter-
nal (human) input.
196 G. Stavrinos et al.

With this definition of autonomy in mind, this section discusses the following
software components:
– Planning software.
– Laser control.
– Sampling/acquisition.
– Saving data.

The list of software components is not exhaustive. Other minor parts were devel-
oped which will be discussed later in this section.
In order to achieve robust autonomy, the architecture of the software relies on a
highly distributed architecture, following the ROS rationale. More specifically, all
of the different software components send and receive messages that work either
as triggers or as input data for calculations. Figure 12 shows the architecture of the
autonomy software components. Additionally, Table 9 summarises the inputs and
outputs of the autonomy software components. It should be noted that the rows of
the table are not directly related. Interested readers should continue reading for a
detailed analysis of each software component.
Starting with the planning software component of the autonomy package, the main
purpose of this node is to create a valid plan for the moving platform to follow. In
order to achieve such task, the planner needs a series of parameters that discretise the
search space. For the purpose of this chapter, the two-dimensional space is discretised
into an XY grid.
Apart from space discretisation, the implemented software allows for half stepping
and goal tolerance configuration. Goal tolerance refers to the maximum distance the
platform can be away from the goal in order to be considered as correctly positioned.
Half stepping enables double resolution by configuring the planner to generate paths
that pass twice from each grid cell. The planner has a series of other parameters
that can be set either on initialisation or dynamically during run time. Details on the
parameters are discussed in the next chapter.
With the LIBS meteorite sample mapping as a goal, the planning software gener-
ates paths that require the moving platform to pass through all of the configured grid
cells, based on a (user) defined resolution. The paths generated are optimal, meaning
that the required displacement of the platform from one position to the next is always
minimum. This optimality does not include the first position of the plan. It is always
the top left corner of the sample, regardless of the current position of the platform.
Figure 13 shows the base rationale for all planning procedures.
The planner keeps track of the moving platform’s position. This allows it to
discriminate between a goal in progress and an achieved goal position, based on a
(user) defined goal tolerance. When a goal position is reached, the planner pauses,
and send a fire trigger to the laser controller.
The laser controller, when a fire signal is received, fires the pulsed laser into the
sample, based on (user) defined parameters. At the same time, the laser controller
enables the spectrograph’s optical fibre and samples/acquires the radiating plasma
on the meteorite sample. When the sampling/acquisition process is over, the data
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 197

Fig. 12 The architecture of the autonomy software components

Table 9 Autonomy software inputs and outputs

Input Output
External goal position Next plan position
External start trigger Visualisation elements
External dynamic parameter update
Joint States

collected is saved into the hard disk and the laser controller sends a continue trigger
to the planner. At that point, the planner sends the next goal position to the motor
controller. This vicious circle continues until all planned positions have been reached
and spectra from the radiating meteorite plasma have been collected.
It should be highlighted that the spectra acquisition procedure includes a dark
current noise [18] (also referred to as instrument signature) removal based on a
typical baseline subtraction and the spectrograph manufacturer’s available dark cur-
198 G. Stavrinos et al.

Fig. 13 The grid planning

rationale

rent specifications that are available at runtime. More specifically, the dark current
removal procedure can be described with Eq. 5. In Eq. 5, Is is the new intensity, Is is
the intensity that includes the dark current noise, k is the number of spectra values
and dk is dark current baseline noise provided by the manufacturer.
 
1 
k

Is = Is − × dk , ∀s ∈ [0, k], k = N o.spectra (5)
k 0

In the autonomy software components architecture diagram, there is a node outside

the main autonomy package called ‘Goal Fix’. This is a simple ROS node that trans-
lates all input goal position to the correct reference frames for the moving platform.
This offers seamless integration between a variety of different input methods. Since
the autonomy code itself has reference frame awareness (contradictory to humans),
the goal fix node is only used for human interfacing.

3.3 Visualisation

The core functionality of the autonomous LIBS mapping system would not be com-
plete without the appropriate software components for visualisation/debugging tools.
This section discusses all the visualisation capabilities of the implemented software.
The visualisation software components are majorly based on RViz, the main Robot
Operating System visualisation tool. Visualisation capabilities include:

– Moving platform position visualisation. (in RViz)

Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 199

Fig. 14 The movement system real time visualisation in RViz

– Generated plan visualisation. (in RViz)

– Planning grid visualisation. (in RViz)
– Laser beam firing visualisation. (in RViz)
– Spectrograph sampling visualisation.

The above list of visualisation capabilities offers a complete reference for humans
to follow and understand the progress of the autonomous LIBS mapping process.
The moving platform position visualisation appears inside in a three-dimensional
window (in RViz) the current state of the moving plate. It follows the movement of
the plate in real time, based on the reported joint states of the ros_control interface of
the hardware driver. Figure 14 shows a screenshot with the three-dimensional state
of the movement system.
The visualisation of the generated plan (path) consists of a series of connected
straight lines. These lines connect the positions that constitute the whole navigation
plan. As stated earlier, the generated plan is based on a specified grid that needs to
be covered. The visualisation of the planning grid offers allows humans to quickly
evaluate the coverage of the generated plan. Figure 15 shows a plan on top of the
planning grid. The thin green line is the generated plan, and the blue layer below
it is the planning grid. Again, the visualisation is integrated into RViz, using the
appropriate message types, for real time updates. The real time aspect of the planning
phase, involves the removal of already visited positions of the plan.
Another part of the visual software components is the visualisation of the laser
beam. Enabling the capability for humans to see the laser beam in RViz, allows for
200 G. Stavrinos et al.

Fig. 15 The generated plan and grid real time visualisation in RViz

better understanding of the autonomous LIBS mapping process. This involves the
appearance and disappearance of a red line whenever the laser controller is pulsing a
beam or not respectively. Figure 16 shows a complete visualisation screenshot from
RViz, while the laser controller is firing a beam.
Finally, the spectrograph sampling visualisation tool is an external application
designed to read and visualise the log files generated during the acquisition process
from the laser controller. The interface consists of five components. The grid cell
selector, the spectrum visualiser and its visualisation toolbar as well as the elemental
map and its visualisation toolbar.
The grid cell selector shows a grid that users can click on to choose a cell for
which the spectrum will be visualised. The selected grid cell is highlighted to offer
an enhanced user experience. The spectrum visualiser is the plot itself, while the
toolbar consists of tool to configure a series of parameters of the plot visualisation.
The last feature of the spectrograph sampling visualisation tool is the elemental map
along with its toolbar. Located in the top left corner of the application, the elemental
map is populated based on the currently selected spectrum limits. This means that
users can alter the area of interest by zooming in an intensity of interest in the
spectrum visualiser, and thus generate and elemental map based on a spectrum of
choice. The corresponding elemental map toolbar allows users to zoom and pan on
the map for thorougher inspection. Finally, to offer easier spectrum investigation, a
‘(x)’ has been included that selects all grid cells for an overall sum of intensities.
Figure 17 shows a screenshot from the spectrograph’s sampling visualisation tool.
It should be noted that an extra random synthetic dataset generator has been script
implemented for instant log access. The random generation script populates a log
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 201

Fig. 16 Screenshot from RViz showing the complete visualisation capabilities of the system, includ-
ing position and laser beam states, generated plan and the planning grid

Fig. 17 Screenshot from the spectrograph sampling visualisation tool

file with random intensities in the range of zero (0) to one (1). The probability for
an intensity value to be higher than 0.2 is 5% or 0.05. This probability tries to (very)
roughly simulate the output of the spectrograph sampling procedure.
202 G. Stavrinos et al.

3.4 User Interface

In order to offer a complete autonomous LIBS mapping solution, a number of ways

are required for humans to interface to the system. For this purpose three ways
were integrated into the system, that are listed below from the simplest to the most
complicated:

– Sending a single goal position through RViz

– Teleoperating the moving platform with a joystick
– Configuring and controlling the planning process through a graphical user inter-
face.

Starting with the simplest, the goal position button (called “2D Nav Goal”) that
is part of the RViz tools was integrated. It is of vital importance to offer to the user
a robust and transparent to the user way to send navigation goals. For this purpose,
the goal fix node consumes (receives) the goal position before actually sending it to
the rest of the system. As discussed earlier, this procedure ensures reference frame
stability.
The process of sending a navigation goal using the RViz goal position button is
simple. The user first has to select the goal position tool from the RViz top tool panel.
Then the user needs to click anywhere inside the main RViz screen where the state of
the moving platform is visualised. As soon as the mouse click is released, a message
is sent to the goal fix node in order for it to translate the point to the correct reference
frame and relay it to the rest of the autonomy pipeline.
The second way for users to interact with the LIBS mapping system, is to use a
joystick (gamepad) to teleoperate the moving platform. This process is quite straight
forward, since users do not have to do anything other than use the analogue sticks of
an ordinary wired or wireless gamepad. As a reference, two gamepads were used as
a test, a Logitech F710 (wireless) and a Logitech F340 (wired). In order to achieve
this seamless integration of the gamepad, the ROS joystick driver [35] is used. The
joystick driver sends commands directly to the moving platform drivers where the
commanded velocity is handled accordingly.
The last user interaction method is through a graphical user interface (GUI). This
application allows users to configure and control the autonomous planning routine
as well as monitor the spectra acquisition process. Figure 18 shows a screenshot of
the navigation control application.
In order to make the application even more user friendly, a USB camera was added
to the hardware setup. The camera integration allows users to draw a rectangle in its
image. This rectangle specifies the planning grid limits.
The navigation control GUI allows users to configure a series of planning param-
eters:
– The planning grid cell size (equivalent to the laser spot size, depending on the
magnification of the objective lens). This parameter indirectly sets the number of
cells of the grid, by trying to fit the cells inside the user defined grid rectangle.
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 203

Fig. 18 Screenshot of the navigation control application

This is a simple division calculation after having set up the optical microscopic
configuration of the system and having measured the spot size.
– The goal tolerance. This parameter sets the minimum distance that the platform
needs to be away from a goal position in order to be considered as reached by the
planner.
– The camera height from the plate. This parameter is only used for visualisation
purposes. It sets the z-axis distance that the grid and the plan will be drawn with
the plate as the origin.
– The centre frame. This is a highly advanced parameter that allows users to change
the reference frame of the centre of the platform. It is included in the case of
application to a different setup.
– Half stepping toggle mode. This boolean parameters sets the planner’s half step-
ping mode. Half stepping was discussed in a previous chapter. In short, when
enabled, it doubles the control points of the planning function.
As stated earlier, apart from parameter configuration, the navigation control appli-
cation also allows for easy selection of a part of the sample to be analysed. This can
be achieved with two different methods.
The first method involves the camera image. It allows users to draw a rectangle
with the mouse, which specifies the limits of the analysis rectangle. This rectangle
pixels are translated into real world coordinates, based on the fact that the camera
height from the moving plate is known and constant.
204 G. Stavrinos et al.

The second method requires the “Manual Editing” option to be enabled. It allows
users to specify the exact limits of the analysis rectangle by typing in the appropriate
boxes.
It should be clear that even though the first method is far more intuitive for humans,
it is less accurate. The second method was implemented to offer a less user friendly
but precise implementation.
As highlighted earlier, the navigation control application also offers a live spectra
acquisition visualisation. It is located at the bottom of the window and offers a rough
estimate of the current sampling intensity.

4 Software Tutorial

Before moving on to the experimental setup and the final results of this chapter, a
proper tutorial should be included for interested readers. This chapter contains infor-
mation regarding on how to compile and run the implemented software. Additionally,
detailed description of all configuration parameters and files are provided.

4.1 Compiling the Software

The compilation section of this chapter considers ROS already installed. The process
of installing ROS is out of scope of this chapter. Minor familiarity with the ROS
ecosystem is also assumed.
The first step in order for users to be able to use the implemented software is to
download the required code. The whole implementation is open source and available
on github. It consists of the main autonomous LIBS mapping implementation [38]
and also a fork [39] of the unofficial open source Ocean Optics spectrograph device
driver [13].
Before describing all of the compilation details, it should be noted that the software
was tested under MX Linux 18.1 (Debian 9 based Linux distribution). The software
is not Debian or even Linux specific, so users of other operating systems should not
have any major problems compiling (Windows or Mac).
The seabreeze fork does not include any major changes over the official release,
apart of modernised CMake changes for instant compilation on more recent C++
compilers. The compilation process of the spectrograph device driver follows the
typical CMake based compilation procedure. The following commands allow users
to download, compile and install the seabreeze device driver as a system library. It
should be highlighted again, that the process was tested in a Debian based Linux
distribution.

– Download: git clone https://github.com/gstavrinos/seabreeze

– Compile:
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 205

Table 10 Software dependencies

Messages/services Functionality packages
nav_msgs tf
std_srvs roscpp
sensor_msgs rosserial
geometry_msgs rosserial_arduino
visualization_msgs controller_manager
Joy
usb_cam
hardware_interface

• mkdir seabreeze/SeaBreeze/build
• cd seabreeze/SeaBreeze/build
• cmake ..
• make
• cd ../..

– Install: sh install_seabreeze.sh

With the spectrograph device driver compiled and installed, users should be able
to compile and use the main software implementation.
Before compiling the ROS package, users should make sure that all ROS depen-
dencies are installed on their system. The ROS dependencies of the auto_libs package
are summarised in Table 10.
With all the dependencies in Table 10 satisfied, users can normally compile the
auto_libs related packages, using the catkin_make tool. The following commands
allow users to download and compile the implemented packages:
– Download:
• cd <catkin workspace location>/src
• git clone https://github.com/gstavrinos/auto_libs
• cd ..

– Compile: catkin_make
With all software compiled and installed, users should be ready to configure and
run the whole autonomous LIBS mapping implementation. These tasks are discussed
in the following sections of this chapter.
206 G. Stavrinos et al.

4.2 Configuring the Software

As discussed in the previous chapter, the configuration of the whole autonomous

LIBS pipeline, can be achieved in two ways. The first one involves editing config-
uration files. The second one, allows only a portion of parameters to be configured
and can be done via a graphical user interface.
All of the available parameters are presented in Table 11. For the sake of simplicity,
only the equivalent ROS package for each parameter has been included in the table.
Interested readers should refer to the referenced packages in order to note on which
node each one refers to.
Configuration files are an established way to experiment with parameters in the
ROS community. They are deterministic and precise. Even though configuration files
are an effective way to set parameters for the autonomous LIBS mapping system,
they also have a series of flaws. These include lack of simplicity, since they require
users to search through files and edit them, but also lack of dynamic configuration
capabilities.
The disadvantages of configuration files are tackled by the second configuration
method, that involves a graphical user interface. As discussed in the previous chapter,
the graphical user interface allows user to set a large subset of the parameters of sys-
tem while offering user friendly interaction and dynamic reconfiguration capabilities.
Figure 18 from the previous chapter, shows a screenshot of the navigation control
application.

4.3 Running the Software

Since the implemented software could rely on configuration files to initialise some
parameters, the need for a way to execute the system combined with the appropriate
configuration files is needed. Luckily, ROS offers an easy way to achieve this.
Batch execution (“launch”) of ROS nodes along with their configuration files can
be done using an XML file with the .launch extension. This XML file specifies exactly
which nodes should be run and where the appropriate parameter can be found.
For the implemented autonomous LIBS mapping system, a single launch file has
been included that executes everything needed to control and visualise the planning
and sampling autonomous procedure.
In order to execute the whole autonomous LIBS mapping pipeline, users have to
run:
roslaunch auto_libs_navigation auto_libs_complete.launch
The auto_libs_complete.launch file runs the following software:
– The XY moving platform hardware driver.
– The laser/spectrograph controller.
– The USB camera hardware driver.
– The goal fix node for reference frame abstraction.
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 207

Table 11 Software Configuration Parameters

Name Description Package
teleop_twist_joy This is a set of parameters that allows for sensitivity auto_libs_control
setting of the joystick commands
cell_size This parameter allows for planning grid cell size auto_libs_navigation
configuration (in meters)
half_cell_step This parameter enables or disables the half stepping auto_libs_navigation
mode of the planner that allows for double planning
resolution by visiting each cell on its edges instead
of its centre.
goal_tolerance This parameter configures the minimum distance auto_libs_navigation
between the moving platform and a goal point in
order for the later to be considered reached (in
meters)
boundaries This parameter configures the rectangle of interest auto_libs_navigation
for which the planner will generate a path for the
moving platform to follow
visualise_beam This parameter enables or disables the visualisation auto_libs_control
of the laser beam
center_link This parameter sets the name of the reference frame auto_libs_control
for the centre of the XY axes of movement
laser_link This parameter sets the name of the reference frame auto_libs_control
for the (vertical) source of the laser beam

– The autonomous LIBS mapping planner.

– An instance of RViz for visualisation.
– The navigation control graphical user interface application.

The only part of the implementation that is not run using the auto_libs_
complete.launch file, is the teleoperation node. The teleoperation software has been
isolated in a dedicated launch file, since it is not part of the standard autonomous
LIBS mapping procedure. Users that need to execute the teleoperation node, should
run:
roslaunch auto_libs_control teleop.launch
The teleop.launch executes the appropriate joystick hardware drivers along with
the required button mappings and configurations. It also run a velocity command
multiplexer that manages simultaneous inputs based on a predefined multiplexing
logic.
Figure 19 shows the complete graph of nodes when running both the auto_libs_
complete.launch and teleop.launch files. It should be noted that some nodes that
appear unconnected user ROS Services to communicate, which is not visible in the
graph due to its on-demand nature.
208 G. Stavrinos et al.

Fig. 19 The complete ROS node graph

5 Experiments and Results

The experimental and validation setup of the autonomous LIBS mapping imple-
mentation precedes a preliminary test phase. This test phase includes the execution
of the whole elemental mapping pipeline in synthetic samples that could easily offer
a first impression of the overall validity of the acquisition/mapping results.
For the testing phase two synthetic samples were prepared and used. Both of them
consisted of an aluminium (Al) plate as a substrate and smaller copper (Cu) details
on it, formed or having specific shapes, in order to be easily identifiable during LIBS
analysis.
Single spot analyses from these materials (Cu and Al) have been acquired, as well
as two spectra from atmospheric air (Fig. 20). Table 12 contains spectral lines from

Fig. 20 Reference spectra from pure metals of copper (first from the top) and of aluminium (second
from the top), as well as two analyses from atmospheric air (two lower spectra). Few unidentifiable
peaks belong to the air analysed with the solid materials
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 209

Table 12 Spectral lines of reference materials used in this study

Element Wavelength (nm) Element Wavelength (nm)
Al 358.7 N 375.54
Al 394.4 N 391.4
Al 396.2 N 400.5
Al 453 N 405.94
Al 466.3 N 417.6
Al 625 N 422.8
Al 685 N 423.5
Al 704 N 444.7
Ar 696.5 N 460
C 392.03 N 462.1
C 426.87 N 472.35
C 437.14 N 478.8
C 477.18 N 480.3
C 493.2 N 489.5
C 504.01 N 498.7
C 505 N 499.4
C 505.24 N 500.1
C 526.91 N 500.7
C 530.65 N 504.5
C 538.02 N 648
C 589.07 N 665
C 658 N 672
C 906 N 742.4
C 909 N 744.2
C 940 N 746.8
Cu 510 N 820
Cu 515 N 821
Cu 521 N 862
Cu 522 N 865
Cu 529 N 868
Cu 578 N 872
H 410 N 906
H 434 N 939
H 486.1 O 407
(continued)
210 G. Stavrinos et al.

Table 12 (continued)
Element Wavelength (nm) Element Wavelength (nm)
H 656.33 O 463
O 616
O 636
O 715
O 777.2
O 795
O 822
O 844.7
O 929.6

Fig. 21 Elemental maps performed during the first test of the experimentation process to discrimi-
nate between different materials, i.e., Cu strips (top map) on an Aluminum substrate (bottom map).
Each map is formed by extracting the intensity of a characteristic peak from each material

the materials in Fig. 20. These should be used as a reference for interpreting results
from the following studies.
The first synthetic sample had long parallel stripes of a Cu foil, each of about 1 mm
width and 1 mm spacing from each other. The result of this first test is shown in Fig. 21,
which show two elemental maps, extracted from the spectra from representative
spectral lines for Cu (top elemental map) and for Al (bottom elemental map).
The second test consisted of a copper (Cu) ring on top of an aluminium (Al)
plate/substrate. Such rings are specially fabricated by lithographic processes and are
used in microscopy to calibrate the instruments or as a scale reference since they are
very precise in shape and sizes. In our experiment, the ring’s outer diameter was 3 mm
and its inner hole diameter was 1 mm. The results of this second test are shown in
Fig. 22. The figure shows two elemental maps (top images), each of different selected
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 211

Fig. 22 The second test of the experimentation process that included Cu rings (spectral line at
510 nm) on a Al substrate (spectral lines at between 374 and 395 nm, and at 680 nm)-see top images.
On the bottom left is an optical microscope image showing the copper ring positioned on an already
irradiated area. On the bottom right the total spectrum is shown of the current experiment, contained
characteristic peaks from each element and from the surrounding environment (atmospheric air)

spectral line, that leads to the designation of the two materials, for copper (top left
image; peak at 510 nm) and aluminium (top right image; mapping range between 374
and 395 nm—the same image is acquired when the 680 nm line is used). Figure 23
shows spectra from points on the Cu ring and on the Al substrate. Furthermore, to
better demonstrate the geometrical characteristics of the laser stops, Fig. 24 shows this
synthetic sample in scale with magnification of x200 (optical microscope objective
lens of 20x magnification). The craters on the aluminium substrate created formed by
the pulsed laser pulses are now clearly visible and have an elliptical shape of 110 ×
200 µm (top image); one crater is further magnified using the microscope objective
of 50x (bottom image).
212 G. Stavrinos et al.

Fig. 23 Spectra from spots on the Cu ring (left) and on the Al substrate (right). The other peaks
are characteristic of the air composition

The visibility (under a microscope) of the craters created from the laser pulses
offers a chance to evaluate the precision of the kinematic plate (motors and mechan-
ical linear movement axes). While testing the autonomous LIBS mappings system,
during the second test a drift of the motors was noticed in the microscopic level.
Figure 25 shows a plain aluminium plate scanned by the implemented autonomous
LIBS system in a resolution of a quarter of a millimetre (0.25 mm). The image in the
figure is acquired with a microscope with 10x objective lens. It should noted that the
sampled area in not a rectangle as expected. The drift displacement is around half
a millimetre (0.5 mm) which is not acceptable for the use case of a LIBS elemental
mapper.
Experimentation with different motor step values for the implemented autonomous
LIBS mapping system led to the conclusion that the minimum robust value is around
a third of a millimetre (0.3 mm). This realisation made the experimentation process
more robust and the elemental maps generated during the following experiments of
better quality. Figure 26 shows the precision of the system magnified with 20x objec-
tive lens on the microscope, both on the Al substrate and on a meteorite sample, the
NWA 3118, a CV3 chondrite. The constant distance between laser pulse craters is
clearly demonstrated on the Al sample, but it is also discernible on the meteorite
sample.
It should be mentioned that further results on the NWA 3118 CV3 chondrite
(meteorite) sample have been to the current stage omitted, due to their polymorphic
nature of the sample, i.e., mineral phases smaller than the spot and step size of the
LIBS system, as well as the topography of the surface of the sample (not a polished
section) do not produce recognisable patterns. To achieve this, a high caliber kine-
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 213

Fig. 24 Optical microscopic images of the synthetic sample of the second test of the experimen-
tation process. It is composed of a 3 mm copper ring with a 1 mm hole, placed on an aluminium
substrate (top image). Scales show the copper ring opening and the laser crater size (110 × 200 µm)
which is magnified in the bottom image with an objective lens of x50
214 G. Stavrinos et al.

Fig. 25 Observation of motor/kinematic stage drifting during the second test phase in a microscopic
level

matic motor stage is required as well as a high magnification microscope objective

instead of the 1 inch objective used in our LIBS system.
The final experiment using the implemented autonomous LIBS mapping sys-
tem was conducted on an iron-nickel alloy (NeFi) meteorite, the Gebel Kamil atax-
ite meteorite containing a characteristic iron-nickel phosphate mineral phase, the
schreibersite mineral with formula (Fe, Ni)3P. This meteorite also contains traces of
Rare Earth Elements (REEs), observed as grain accumulations, often occurring close
to schreibersite. Figure 27 shows a backscatter electron image of the schreibersite
phase, which then we mapped with LIBS. This image shows the schreibersite core,
which is the brightest of all phases in BSE, then around it the kamacite phase, and
finally the plessite matrix in which it resides. Figure 28 shows X-ray Energy Disper-
sion (EDX) spectra from the two major phases, schreibersite and kamacite. Kamacite
is composed of iron and minor nickel, while schreibersite is also composed of iron,
together with nickel and phosphor (see mineral formula above). During LIBS map-
ping it is expected that Fe will show less intense signal in schreibersite than kamacite.
This is true if we look at the LIBS elemental map at the top right of Fig. 29. There
are some points at the rim of the mineral inclusion that show brighter colours, and
these are clearly located in the Kamacite zone surrounding the iron nickel phosphate
(schreibersite mineral phase). In the same Figure, the top right optical image demon-
strates that enough spots exist inside schreibersite to resolve it in the map. The optical
images of Fig. 31 have been acquired under a petrographic optical microscope just
before irradiation with the laser (Fig. 29).
Before we proceed to any interpretation, we have assembled a graph (Fig. 30)
of the major elements we expect to detect in Gebel Kamil meteorite, i.e., Fe, Ni,
and P. From the reference spectra it is clear that the major characteristic peaks of
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 215

Fig. 26 Precise laser pulse craters spacing on an aluminium substrate (top) and on NWA 3118-CV3
Chondrite (meteorite) sample (bottom). The chondrite laser ablation spot pattern has a horizontal
arrangement; it is discernible, although with some difficulty. Scale is the same for both images
216 G. Stavrinos et al.

Fig. 27 Backscatter electron image (BSE) of the LIBS mapped schreibersite phase in the Gebel
Kamil meteorite. Sch = schreibersite, Kam = Kamacite, and Plessite, which is a fine-grained
mixture between Kamacite and Taenite

these elements are not visible by our spectrometer. This spectrometer is tuned to
work mostly with light elements and a new spectrometer is required, tuned to view
wavelengths starting as low as 200 nm to detect metals. This is a current hardware
restriction, and consequently, we do not expect that we will acquire characteristic
maps for these elements. However, we will investigate if we can detect minor peaks
of the elements and extract some information. As from Fig. 30, minor peaks of the
elements occur within the range covered by our spectrometer.
The elemental maps of Fig. 30 (top row) are extracted from the spectra at lines
568.6 and 510.1 nm (left and right map, respectively). The map of the 568.6 nm line
should represent the Fe-Ni matrix, where the schreibersite inclusion appears dark due
to lower contents of both elements relative to the matrix. We also produced the graphs
in Fig. 32 to demonstrate the difference of the spectra between schreibersite (yellow
trace) and kamacite (green trace). The absolute value of the difference between these
spectra is shown with a blue trace. It is clear that the largest difference between the
two spectra is visible at the 568.6 nm peak, as well as the 582 nm peak. Similarly, a
large difference is observed at 451.7 nm, however it is not easily visible because the
peak is small.
The 510.1 nm element map is expected to demonstrate the P-rich area, revealing
the schreibersite inclusion. On the other hand, the 568.6 nm peak should demonstrate
the Fe peak, because it is dark in schreibersite, brighter in Kamacite, and intermediate
in plessite, as expected from their chemistries. The demonstrated contrast is realistic;
however, we could not identify from the literature and from the NIST database any
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 217

Fig. 28 EDX (X-ray electron dispersion) spectra of schreibersite and kamacite phases. Schreibersite
has a phosphor peak, while this element is totally missing from Kamacite

plasma emission lines of the three major elements to fit those peaks. At the same time,
these peaks are not identifiable in the spectra taken from atmospheric air (Fig. 20).
The bottom images of Fig. 30 are optical microscopy images of schreibersite,
before irradiation (bottom left image) and after irradiation (bottom right image)
demonstrating the laser spot sizes. A number of spots are discernible to occur inside
schreibersite, suggesting that the spectra from its matrix are representative. We
require hardware upgrades in order to pursue this work further.

6 Conclusion

In this use case chapter an Autonomous LIBS Mapping System for Chemical and
Mineralogical Designation of Interplanetary Materials was implemented. The imple-
mentation includes hardware design and setup, as well software for device drivers,
218 G. Stavrinos et al.

Fig. 29 Schreibersite, (Fe, Ni)3P, mineral inclusion (bright area) in the Gebel Kamil FeNi meteorite
(Egypt, 2009–2010). The surrounding matrix (dark area) is the kamacite FeNi alloy, common to
iron meteorites. Elemental maps (top images) show different images (opposite) if two different
wavelengths are selected

control and planning, in addition to graphical user interfacing applications. It is then

demonstrated on different synthetic samples, as well as two meteorites.
The implemented system offers a complete solution to the autonomous LIBS map-
ping problem, but not without its limitations. From the hardware point of view, and
as seen in the previous chapter regarding the experiments and results of the imple-
mented system, a major drawback was the lack of high precision movement under
resolution requirements of equal or lower than a quarter of a millimetre (0.25 mm).
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 219

Fig. 30 The simulated spectra of the most important elements of the sample, those of Fe, Ni, and
P, calculated from the NIST database. Insets represent the wavelength range that our spectrometer
can view
220 G. Stavrinos et al.

Fig. 31 The iron-nickel alloy (NeFi) meteorite sample under the microscope with a 2.5x zoom lens

This is not adequate also for the current laser spot size which is smaller (0.20 mm
in its larger axis of the ellipsis, improved to a circular spot of about 0.1 mm diam-
eter when a better focusing is performed and a smaller aperture is used). Since the
linear movement mechanism along with the power source for the motors and the
motors themselves were hobbyist-level products, it was to be expected. Luckily, the
hardware abstraction offered by the software should make a future upgrade of the
linear movement system a straight forward task. Another hardware limitation was the
spectrometer. Its spectral range is between 340 and 1040 nm, which is appropriate
for light elements (i.e., gases) but not for the heavy elements (i.e., metals) which emit
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 221

Fig. 32 Spectra difference between schreibersite and kamacite (top spectrum) and zoom in at
568.6 nm peak (bottom spectrum). On the right of the bottom spectrum, the total elemental distri-
bution map is shown (top) and the elemental map extracted from the 568.5 nm peak

their characteristic major lines in the UV region. An upgrade of the spectrometer with
a properly selected grating would solve this problem. Moreover, the spectrometer
should contain faster electronics to allow precise control for delayed gating during
acquisition in order, for example, to reduce background due to luminescence. With
the current spectrometer acquisition was delayed by a variable unknown time interval
after triggering, degrading the elemental maps.
222 G. Stavrinos et al.

Apart from hardware limitations, this implementation was planned to reach only
certain specifications which have to do with the automation and usability of the
technique. Acquisition currently includes only the subtraction of the dark current
noise, but additional post-processing procedures would enhance the functionality of
the system and the analytical results. For example, calibration-free LIBS integration
to the already implemented system is a short-term goal that would greatly enhance
acquisition performance in respect to interpretation. Future work could also ideally
include more robust and sophisticated sample processing methods using machine
learning techniques.
To conclude, this chapter focused on the development of an Autonomous LIBS
Mapping System for Chemical and Mineralogical Designation of Interplanetary
Materials. The implementation includes the hardware design and setup as well as
all software components required for complete autonomy and user interfacing. The
implementation did not come without a set of drawback though, which can be poten-
tially be the focus of future work.

Annex A: Pulse Generator Configuration Commands

*RST
:PULS0:MOD SING
:PULS1:POL NORM
:PULS2:POL NORM
:PULS1:OUTP:MOD ADJ
:PULS2:OUTP:MOD ADJ
:PULS2:OUTP:AMPL 5
:PULS1:OUTP:AMPL 5
:PULS1:DEL 0
:PULS2:DEL 0
:PULS1:WIDT 100E-3
:PULS2:WIDT 100E-3
The command “RST” resets the system. Command ‘:PULS0’ and subcommands
refer to general settings of the pulse generator. The commands ‘PULS1’ and ‘PULS2’
pass parameters to the two individual channels, such as time delay (‘DEL’) and pulse
duration (‘WIDT’).
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 223

References

1. Anglos, D.: Shedding light on the past: optical technologies applied to cultural heritage (2017)
2. Arp, Z.A., Cremers, D.A., Wiens, R.C., Wayne, D.M., Sallé, B., Maurice, S.: Analysis of water
ice and water ice/soil mixtures using laser-induced breakdown spectroscopy: application to
mars polar exploration. Appl. Spectrosc. 58(8), 897–909 (2004)
3. Bonta, M., Gonzalez, J.J., Quarles, C.D., Russo, R.E., Hegedus, B., Limbeck, A.: Elemental
mapping of biological samples by the combined use of libs and la-icp-ms. J. Anal. At. Spectrom.
31(1), 252–258 (2016)
4. Bush, L.: Biological mapping with libs. Spectroscopy 26–31 (2016)
5. Castle, B.C., Talabardon, K., Smith, B.W., Winefordner, J.D.: Variables influencing the preci-
sion of laser-induced breakdown spectroscopy measurements. Appl. Spectrosc. 52(5), 649–657
(1998)
6. Chatzitheodoridis, E.: Lab (2021). http://minpetlab.metal.ntua.gr/
7. Chitta, S., Marder-Eppstein, E., Meeussen, W., Pradeep, V., Rodríguez Tsouroukdissian,
A., Bohren, J., Coleman, D., Magyar, B., Raiola, G., Lüdtke, M., Fernández Perdomo,
E.: ros_control: a generic and simple control framework for ros. J. Open Source Softw.
(2017). https://doi.org/10.21105/joss.00456. http://www.theoj.org/joss-papers/joss.00456/10.
21105.joss.00456.pdf
8. Ciucci, A., Corsi, M., Palleschi, V., Rastelli, S., Salvetti, A., Tognoni, E.: New procedure for
quantitative elemental analysis by laser-induced plasma spectroscopy. Appl. Spectrosc. 53(8),
960–964 (1999)
9. Corporation, B.N.: Bnc website (2021). https://www.berkeleynucleonics.com/
10. Cousin, A., Meslin, P., Wiens, R., Rapin, W., Mangold, N., Fabre, C., Gasnault, O., Forni, O.,
Tokar, R., Ollila, A., et al.: Compositions of coarse and fine particles in martian soils at gale:
a window into the production of soils. Icarus 249, 22–42 (2015)
11. Cremers, D.A., Radziemski, L.J.: Handbook of Laser-Induced Breakdown Spectroscopy. Wiley,
New York (2013)
12. De Giacomo, A., Dell’Aglio, M., De Pascale, O., Longo, S., Capitelli, M.: Laser induced
breakdown spectroscopy on meteorites. Spectrochim. Acta Part B 62(12), 1606–1611 (2007)
13. K. C. et al.: Seabreeze (2021). https://sourceforge.net/projects/seabreeze/
14. Ferus, M., Koukal, J., Lenza, L., Srba, J., Kubelık, P., Laitl, V., Zanozina, E.M., Vana, P.,
Kaiserova, T., Knızek, A., Rimmer, P., Chatzitheodoridis, E., Civis, S.: Calibration-free quan-
titative elemental analysis of meteor plasma using reference laser-induced breakdown spec-
troscopy of meteorite samples. A&A 610, A73 (2018). https://doi.org/10.1051/0004-6361/
201629950. https://doi.org/10.1051/0004-6361/201629950
15. Fichet, P., Tabarant, M., Salle, B., Gautier, C.: Comparisons between libs and icp/oes. Anal.
Bioanal. Chem. 385(2), 338–344 (2006)
16. Garcıa-Ayuso, L., Amador-Hernández, J., Fernández-Romero, J., De Castro, M.L.: Charac-
terization of jewellery products by laser-induced breakdown spectroscopy. Anal. Chim. Acta
457(2), 247–256 (2002)
17. Gould, R.G. et al.: The laser, light amplification by stimulated emission of radiation. In: The
Ann Arbor Conference on Optical Pumping, vol. 15, p. 92. The University of Michigan (1959)
18. Hagins, W., Penn, R., Yoshikami, S.: Dark current and photocurrent in retinal rods. Biophys.
J . 10(5), 380–412 (1970)
19. Hou, X., Jones, B.T.: Field instrumentation in atomic spectroscopy. Microchem. J. 66(1–3),
115–145 (2000)
20. Ilhardt, P.D., Nuñez, J.R., Denis, E.H., Rosnow, J.J., Krogstad, E.J., Renslow, R.S., Moran, J.J.:
High-resolution elemental mapping of the root-rhizosphere-soil continuum using laser-induced
breakdown spectroscopy (libs). Soil Biol. Biochem. 131, 119–132 (2019)
21. Jolivet, L., Leprince, M., Moncayo, S., Sorbier, L., Lienemann, C.-P., Motto-Ros, V.: Review
of the recent advances and applications of libs-based imaging. Spectrochim. Acta Part B 151,
41–53 (2019)
224 G. Stavrinos et al.

22. Kam, H.R., Lee, S.-H., Park, T., Kim, C.-H.: Rviz: a toolkit for real domain data visualization.
Telecommun. Syst. 60(2), 337–345 (2015)
23. Kim, T., Lin, C., Yoon, Y.: Compositional mapping by laser-induced breakdown spectroscopy.
J. Phys. Chem. B 102(22), 4284–4287 (1998)
24. Knight, A.K., Scherbarth, N.L., Cremers, D.A., Ferris, M.J.: Characterization of laser-induced
breakdown spectroscopy (libs) for application to space exploration. Appl. Spectrosc. 54(3),
331–340 (2000)
25. Lasers, L.: Litron website (2021). https://litron.co.uk/
26. Lasheras, R., Bello-Galvez, C., Rodriguez-Celis, E., Anzano, J.: Discrimination of organic
solid materials by libs using methods of correlation and normalized coordinates. J. Hazard.
Mater. 192(2), 704–713 (2011)
27. Neuland, M.B., Meyer, S., Mezger, K., Riedo, A., Tulej, M., Wurz, P.: Probing the allende
meteorite with a miniature laser-ablation mass analyser for space application. Planet. Space
Sci. 101, 196–209 (2014)
28. OpenBuilds: Openbuilds website (2021). https://openbuilds.com/
29. Optics, O.: Ocean optics website (2021). https://www.oceaninsight.com/
30. Pagnotta, S., Lezzerini, M., Ripoll-Seguer, L., Hidalgo, M., Grifoni, E., Legnaioli, S., Loren-
zetti, G., Poggialini, F., Palleschi, V.: Micro-laser-induced breakdown spectroscopy (micro-libs)
study on ancient roman mortars. Appl. Spectrosc. 71(4), 721–727 (2017)
31. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros:
an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3,
p. 5. Kobe, Japan (2009)
32. Rai, N.K., Rai, A.: Libs-an efficient approach for the determination of cr in industrial wastew-
ater. J. Hazard. Mater. 150(3), 835–838 (2008)
33. Rehse, S.J., Diedrich, J., Palchaudhuri, S.: Identification and discrimination of pseudomonas
aeruginosa bacteria grown in blood and bile by laser-induced breakdown spectroscopy. Spec-
trochim. Acta Part B 62(10), 1169–1176 (2007)
34. Rodriguez-Celis, E., Gornushkin, I., Heitmann, U., Almirall, J., Smith, B., Winefordner, J.,
Omenetto, N.: Laser induced breakdown spectroscopy as a tool for discrimination of glass for
forensic applications. Anal. Bioanal. Chem. 391(5), 1961 (2008)
35. ROS: joystick_drivers (2021). https://wiki.ros.org/joystick_drivers
36. Sattmann, R., Sturm, V., Noll, R.: Laser-induced breakdown spectroscopy of steel samples
using multiple q-switch nd: Yag laser pulses. J. Phys. D Appl. Phys. 28(10), 2181 (1995)
37. Senesi, G.S.: Laser-induced breakdown spectroscopy (libs) applied to terrestrial and extraterres-
trial analogue geomaterials with emphasis to minerals and rocks. Earth Sci. Rev. 139, 231–267
(2014)
38. Stavrinos, G.: auto_libs (2021). https://github.com/gstavrinos/auto-libs
39. Stavrinos, G.: seabreeze fork (2021). https://github.com/gstavrinos/seabreeze
40. Sturm, V., Noll, R.: Laser-induced breakdown spectroscopy of gas mixtures of air, co 2, n 2,
and c 3 h 8 for simultaneous c, h, o, and n measurement. Appl. Opt. 42(30), 6221–6225 (2003)
41. Taschuk, M., Tsui, Y., Fedosejevs, R.: Detection and mapping of latent fingerprints by laser-
induced breakdown spectroscopy. Appl. Spectrosc. 60(11), 1322–1327 (2006)
42. Tognoni, E., Cristoforetti, G., Legnaioli, S., Palleschi, V.: Calibration-free laser-induced break-
down spectroscopy: state of the art. Spectrochim. Acta Part B 65(1), 1–14 (2010)
43. Wang, W., Li, S., Qi, H., Ayhan, B., Kwan, C., Vance, S.: Revisiting the preprocessing proce-
dures for elemental concentration estimation based on chemcam libs on mars rover. In: 2014
6th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing
(WHISPERS), pp. 1–4. IEEE (2014)
44. Wantai: Wantai website (2021). https://wantmotor.com/

George Stavrinos received a B.Sc. degree in Computer Science from the Athens University
of Economics and Business in 2014 and a M.Sc. in Intelligent Systems and Robotics from De
Montfort University in 2018. In 2020, he received his second M.Sc. in Space Science, Tech-
nologies and Applications from the National Observatory of Athens. He is currently working as
Autonomous Laser-Induced Breakdown Spectroscopy System for Chemical … 225

a researcher in RoboSKEL Lab in the National Centre of Scientific Research “Demokritos” in

Athens, Greece. His main research interests are robots under zero-gravity (microgravity) condi-
tions (rovers or humanoids), autonomous spacecraft, flying robots (autonomous UAVs), naviga-
tion autonomy, dynamics and kinematics analysis and motion planning.

Elias Chatzitheodoridis is Professor of Mineralogy-Petrology at the National Technical Univer-

sity of Athens (NTUA), and Director of the Laboratory of Mineralogy, Petrology, and Economic
Geology. He studied Geology at the National and Kapodistrian University of Athens and acquired
his M.Sc. and Ph.D. degrees from the University of Manchester, UK. He worked at the Foundation
for Research and Technology—Hellas (FORTH) in Heraklion, Crete, at the Technical University
of Vienna (TUWien), and at the high-tech private company Austria Technologie and Systemtech-
nik AG (AT&S AG) at Leoben-Hinterberg, Austria. His research interests are in astrochemistry,
astrobiology, and planetary sciences. He has also a strong applied research background on pho-
tonic technologies, micro- and nanotechnologies, lithography, microsystems (MEMS), and on the
development of analytical instruments. He is President of the ‘Stellar Discoveries’ network, CEO
of the ‘Network of Researchers on the Chemical Evolution of Life’ (NoRCEL), and Council Mem-
ber of the ‘European Astrobiology Network Association’ (EANA). He has published more than a
hundred peer reviewed scientific and technological papers, and he organised conferences in astro-
biology and planetary sciences.

Olga A. Sykioti is Senior Researcher at the Institute of Astronomy, Astrophysics, Space Applica-
tions and Remote Sensing of the National Observatory of Athens. She obtained her B.S. in Geol-
ogy from the National and Kapodistrian University of Athens, her M.S. degrees in Remote Sens-
ing and in Hydrology, Geostatistics and Geochemistry as well as her Ph.D. in Geosciences and
Remote Sensing from Pierre and Marie Curie University (Paris, France). Her research interests
include imaging spectroscopy and machine learning applications in Geosciences with emphasis
on mineral/chemical exploration and geological mapping of terrestrial surfaces. She is President
of the Remote Sensing and Space Applications Committee of the Geological Society of Greece. In
2009, she was awarded with a Fulbright Scholar fellowship. She has published more than fifty peer
reviewed papers in high-ranked scientific journals and international conference proceedings and
she has participated in more than thirty international, European, European Space Agency (ESA)
and national funded research projects.
High Fidelity IMU and Wheel Encoder
Models for ROS Based AGV Simulations

Emre Ozdemir, Hakan Gokoglu, M. Koray Yilmaz, Umut Dumandag,

Ismail Hakki Savci, and Haluk Bayram

Abstract In this research chapter, we study the problem of how automated guided
vehicles (AGVs) are simulated considering the realistic settings in their sensors.
Wheel encoders and inertial measurement units (IMUs) are two fundamental sen-
sors in AGVs. These sensors are prone to error due to internal or environmental
disturbances and noises. To have a better representation of a robot in simulation
environments, these disturbances/noises should also be modeled. However, since
simulators have the simplified or idealized version of sensors, the high fidelity models
of the sensors are usually ignored, which leads to simulation results inconsistent with
the real-life scenarios. In this chapter, we develop related ROS nodes and Gazebo
plugins to incorporate the disturbance/noise into the sensors and model the error
occurrences using Poisson distribution. We validate the approach through Gazebo
simulations using an AGV model. The source codes and installation details are pro-
vided via a public repository.

1 Introduction

Industry 4.0 requires smart factories with automated guided vehicles (AGVs) [7].
AGVs have a growing usage in factories, warehouses and other areas. These robots
can autonomously navigate around their environments by using magnetic tapes,
reflector systems, rails or natural navigation and can carry out their assigned tasks

E. Ozdemir · H. Gokoglu · I. H. Savci

Light Mobility Laboratory, Ford Otosan Kocaeli Plant, Kocaeli, Turkey
e-mail: [email protected]
H. Gokoglu
e-mail: [email protected]
I. H. Savci
e-mail: [email protected]
M. K. Yilmaz · U. Dumandag · H. Bayram (B)
Field Robotics Laboratory, BILTAM, Istanbul Medeniyet University, Istanbul, Turkey
e-mail: [email protected]
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 227
A. Koubaa (ed.), Robot Operating System (ROS),
Studies in Computational Intelligence 1051,
https://doi.org/10.1007/978-3-031-09062-2_7
228 E. Ozdemir et al.

[16]. Natural navigation is one of the preferred ways for AGV localization and nav-
igation as it provides better flexibility and requires almost no infrastructure change,
such as placing artificial landmarks to the environment.
To expedite the development process of mobile robots, various framework and
simulations, such as ROS and Gazebo, have been developed. ROS and Gazebo frame-
works allow users to develop algorithms and quickly validate the system performance
using simulations. High fidelity sensor models have a profound effect on this per-
formance. For instance, localization task is commonly handled by reading various
sensor inputs (LiDAR, wheel encoder and IMU) and fusing them in real time. One
drawback in performing this task in simulators is the lack of disturbance or noise in
the wheel encoder and IMU models as often these have an important effect on the
localization performance.
In the literature, there has been a large body of studies on the noise/disturbance/i-
naccuracies in wheel encoder and IMU sensors. In these studies, the sources of
inaccuracies and how to deal with them are examined. For wheel encoder sensors,
these sources are given as in [15]: (1) wheel slips, (2) uneven surfaces, (3) skidding,
(4) variations in wheel diameter due to environmental changes (temperature) and
physical factors (pressure, speed), (5) irregularities in wheel diameters and wheel
configuration, (6) limitations (resolution, sample rate) of the wheel encoders. In
addition, the work in [12] states that although the scale factor of wheel encoders is
considered noised constants, the error characteristics of the measurements taken from

Fig. 1 Odometry drift vs. localization (amcl package) from our field experiments in Ford Otosan
Yenikoy Plant, Istanbul, Turkey
High Fidelity IMU and Wheel Encoder Models for ROS Based … 229

wheel encoders are “unknown and largely unstable”. As to IMU sensors, the sources
of errors can be listed in [1, 2, 15, 17] as follows: (1) biases, (2) scale factors, (3)
nonlinearities, (4) environmental effects (varying temperature, humidity, vibrations,
non-rigid mounting). In the simulations of robotic systems, these errors are generally
either ignored or characterized by a known distribution with fixed parameters.
In this work, we developed an approach to provide a more realistic simulation
environment by considering real life events such as wheel encoder uncertainties (due
to wheel slip) and IMU noise/disturbance. In doing so, Poisson distribution is used to
model the undesirable events, such as wheel slips. One example from a real industrial
scenario is given in Fig. 1 where the red trajectory shows the robot localization
using the ROS package amcl whereas the blue trajectory shows the odometry drift
occurring mostly due to wheel slips and mechanical coupling problems.

2 High Fidelity Wheel Encoder and IMU Models in ROS

and Gazebo

In this section, we show how to improve the fidelity level of the existing wheel
encoder and IMU models by incorporating uncertainties/disturbances using Poisson
distribution. All the source codes are accessible via the public repository [3].

2.1 Run-Time Adjustable Noise in Gazebo Wheel Encoder

The differential drive models developed for Gazebo, such as DiffDrivePlugin

[10] and gazebo_ros_diff_drive [8], do not have noise support. For instance,
this can be seen in the following code fragment (lines 377–378) in the original source
codes from the plugin gazebo_ros_diff_drive:
double sl = vl∗(wheel_diameter_/2.0)∗seconds_since_last_update ;
double sr = vr∗(wheel_diameter_/2.0)∗seconds_since_last_update ;

Here, the displacements of the left and right wheels are calculated using the perfect
knowledge of the wheels’ linear velocities. In the rest of the code, the current robot
pose is computed from the previous pose and the odometric integration according to
the following equation [14]:
⎡ ⎤ ⎡ ⎤ ⎡ sr +sl −sl
⎤
xk xk−1 2
cos(θk−1 + sr2b )
⎣ yk ⎦ = ⎣ yk−1 ⎦ + ⎣ sr +sl sin(θk−1 + sr −sl ) ⎦ (1)
2 2b
sr −sl
θk θk−1 2b

where b is the distance between the two wheels, and sr and sl denote the displace-
ment of the left and right wheels, respectively. The Gazebo differential drive model
makes use of this pose update equation. As can be seen in Eq. 1, the odometry
230 E. Ozdemir et al.

update does not contain any noise term. However, since the odometry is required to
provide the corresponding 6×6 covariance matrix, the fixed variances (small vari-
ances for the diagonals related to x, y and rotation about Z axis, and large variances
for the diagonals related to z, rotation about X axis, rotation about Y axis) are
assigned to the diagonals of the matrix1 (between the lines 482–487 in the plugin
gazebo_ros_diff_drive).
As a result, by using this kind of idealized model in mapping, localiza-
tion, or navigation tasks, the odometry is calculated from noiseless and perfect
encoder data. Since the noiseless encoder does not model the real-life conditions
(i.e. wheel slips), the simulations cannot be used effectively to verify the devel-
oped algorithms before they are deployed in the field tests. Therefore, the plugin
gazebo_ros_diff_drive was modified to incorporate the noise. For this pur-
pose, the function UpdateOdometryEncoder is modified such that the noise is
added to sr and sl (lines 430–431):
double sl = vl∗(wheel_diameter_/2.0)∗seconds_since_last_update
+ GaussianKernel(0 , diffd_gaussian_noise . data . at (0))
∗diffd_noise_on_off . data . at (0);
double sr = vr∗(wheel_diameter_/2.0)∗seconds_since_last_update
+ GaussianKernel(0 , diffd_gaussian_noise . data . at (1))
∗diffd_noise_on_off . data . at (1);

where the noise is Gaussian noise N (0, σ 2 ) with zero mean and σ standard deviation
which can be set via a ROS topic. To generate a random number from N (0, σ 2 ), a
function called GaussianKernel (lines 231–243) is added.
In order to be able to change the noise related parameters in the
model, we add two new subscribers (lines 180–183 and lines 185–188)
to the plugin gazebo_ros_diff_drive. As a result, the plugin sub-
scribes to the following ROS topics: /diff_drive/noise_on_off and
/diff_drive/set_noise. We also add the associated callback func-
tions CallbackNoiseOnOff (lines 215–221) and CallbackNoise (lines
223–229) in order to set the variables diffd_gaussian_noise and
diffd_noise_on_off in the calculation of the wheel displacements. In the
original version of the plugin, the diagonals of the covariance matrix related to the
odometry are set to fixed values and cannot be changed during the simulation. These
values in the modified version (lines 535–541) depend on the noise standard deviation
which can be set by the user while the simulation is running.
All the changes were made in the following source files [4]:
noisy_diff_drive.h and noisy_diff_drive.cpp. With these changes,
the user can enable or disable the noise in the wheel encoders via the topic
/diff_drive/noise_on_off, and change the noise standard deviation via
the topic /diff_drive/set_noise.

1 The diagonals of the odometry covariance matrix correspond to x, y, z, rotation about X axis,
rotation about Y axis, rotation about Z axis [11].
High Fidelity IMU and Wheel Encoder Models for ROS Based … 231

2.2 Run-Time Adjustable Noise in Gazebo IMU Sensor

The Gazebo IMU sensor plugin gazebo_ros_imu_sensor includes Gaussian

noise as a plugin parameter [13]. However, this noise parameter can be set only
through xacro file. Therefore once the simulation starts running, this parameter can-
not be changed. To change the noise level at run-time, all the additions and modifi-
cations were made in the following source files [5]: noisy_imu_sensor.h and
noisy_imu_noise.cpp.
The published IMU data has three main fields: orientation, linear acceleration,
angular velocity. Since the orientation is in quaternion form, it has four sub-fields:
x, y, z, w. On the other hand, the linear acceleration and angular velocity fields have
three sub-fields: x, y, z. The code from the function UpdateChild in the modified
IMU plugin (lines 126–128) for the linear acceleration is as follows:
imu_msg. linear_acceleration .x = accelerometer_data .X()
+GaussianKernel(0 ,imu_gaussian_noise . data . at (7))
∗ imu_noise_on_off . data . at (7);
imu_msg. linear_acceleration .y = accelerometer_data .Y()
+ GaussianKernel(0 ,imu_gaussian_noise . data . at (8))
∗ imu_noise_on_off . data . at (8);
imu_msg. linear_acceleration . z = accelerometer_data .Z()
+ GaussianKernel(0 ,imu_gaussian_noise . data . at (9))
∗ imu_noise_on_off . data . at (9);

The similar code is added for the orientation and angular velocity. The variables
imu_gaussian_noise and imu_noise_on_off are set through the ROS top-
ics /imu/set_noise and /imu/noise_onoff. These variables are 10 dimen-
sional arrays in which the first four elements of these arrays are dedicated for the
orientation’s sub-fields, the next three elements for the angular velocity’s sub-fields,
the last three elements for the linear acceleration’s sub-fields. Since the plugin needs
to subscribe to these ROS topics, two additional subscribers (lines 74–78) and their
associated callback functions (lines 85–92 and lines 94–101) are added to the mod-
ified plugin. Hence, the user can activate or deactivate the noise and change the
standard deviation of Gaussian noise in the IMU sensor through these topics.

2.3 Modeling Sensor Uncertainties/Disturbances Using

Poisson Distribution

While AGVs are operating in their workspace, unexpected events may occur (as
discussed in Section 1), thereby increasing sensor uncertainty in wheel encoders and
IMU sensors. These unexpected events need to be integrated into the simulations, as
well. For this purpose, a ROS node called sensor_uncertainty was developed
in which Poisson distribution is used to decide when an unexpected event occurs [6].
The probability mass function for Poisson distribution is given as:
232 E. Ozdemir et al.

λk−1 e−λ
f (k; λ) = (2)
k!
where the parameter λ is the expected separation (interval) within which k events
occur. The reason why the Poisson distribution was chosen is that the Poisson dis-
tribution gives the probability of a given number of unexpected events occurring
in a fixed amount of time. By using this distribution, we can determine how many
unexpected events will occur within a certain period of time and how long they will
last. Therefore, we have an opportunity to test our algorithms, such as localization
and mapping algorithms, under varying noise settings. Here, the unexpected events
can be thought of as the changes in the noise level.
To simulate the errors (unexpected events) occurring in wheel
encoder and IMU sensors (the plugins gazebo_ros_diff_drive and
gazebo_ros_imu_sensor), we need to determine when these events occur.
For this purpose, we make use of NumPy library in Python to generate random
numbers from the Poisson distribution. The following code (lines 22–34) from the
ROS node sensor_uncertainty is used to change the noise level between low
and high noises for the IMU plugin:
while rospy . is_shutdown :
imu_poisson=poisson(imu_num_of_error_occurences / time_interval )
i f imu_poisson >= 1:
publisher . publish (high_msg)
start_time = counter

i f ( counter − start_time ) == imu_noise_duration :

publisher . publish (low_msg)

rate . sleep ()
counter += 1

Here we can define the number of error (unexpected event) occurrences (the variable
imu_num_of_error_occurrences) within a given time interval (the variable
time_interval) in seconds and the duration of how long the high noise level stays
active in seconds (the variable imu_noise_duration). If the random number
generated from the Poisson distribution is greater than or equal to 1, the noise level
is changed from low to high and kept high for imu_noise_duration. When the
time runs out, the noise level is changed from high to low. The same principle is
applied for the wheel encoders (lines 48–60).
The ROS node interacts with wheel encoder (noisy_diff_drive) and IMU
(noisy_imu_sensor) plugins through topics. This interaction can be seen in
Fig. 2. The node can enable or disable the noise in the wheel encoders and IMU via
the topics /diff_drive/noise_on_off and /imu/noise_on_off, and
change the noise standard deviations via the topics /diff_drive/set_noise
and /imu/set_noise.
We can specify all the necessary parameters for each sensor in the configuration
file sensor_uncertainty_config.json. There are three main fields for
the IMU sensor: orientation, angular velocity and linear acceleration. Each of these
High Fidelity IMU and Wheel Encoder Models for ROS Based … 233

Fig. 2 The relationship between the sensor_uncertainty ROS node and the modified Gazebo
plugins

fields has sub-fields: x, y, z for angular velocity and linear acceleration, and x, y, z,
w for orientation. Each sub-field is defined to have the standard deviations of low
and high noises. Therefore, there are the following 20 parameters in total for the
IMU sensor: imu_angular_velocity_[x/y/z]_[low/high]_noise in
rad/s, imu_linear_acceleration_[x/y/z]_[low/high]_noise
in m/s2 , and imu_orientation_[x/y/z/w]_[low/high]_noise
in quaternion form. As to the wheel encoders, there are two fields
(left and right) each of which has the standard deviations of low and
high noises. Therefore, there are four parameters for the wheel encoder:
diff_drive_[left/right]_wheel_[low/high]_noise in meters.
We can activate or deactivate the noises using the following parameters tak-
ing the values of “True” or “False”: imu and diff_drive. We have also
some parameters used in modeling the sensor uncertainties. The parame-
ters [imu/diff_drive]_num_of_error_occurrences) denote the
expected number of error occurrences for the IMU and wheel encoder sen-
sors within a certain time interval defined by time_interval in seconds.
[imu/diff_drive]_noise_duration gives the duration of how long the
high noise level stays active in seconds.

3 Simulation Studies

In this section, we aim to demonstrate the effect of the sensor plugins and explain how
to use them briefly. The details regarding the installation and usage of the plugins
can be found in the repository [3].
234 E. Ozdemir et al.

Fig. 3 AGV Gazebo model

Fig. 4 Gazebo factory environment used in the simulations

In order to test the sensor noise models, a realistic AGV model is created in
Gazebo environment. A differential drive based AGV dynamic model integrated with
an IMU sensor is generated via Solidworks URDF Export tool from CAD model.
Its wheelbase and wheel diameters are 0.51 m and 0.25 m, respectively. The physical
dimension of the AGV is 0.95 m (L) × 0.45 m (W) × 0.40 m (H). It can travel at a
maximum speed of 1 m/s. The Gazebo model of the AGV is shown in Fig. 3. The
AGV operates in the Gazebo factory environment as shown in Fig. 4.
Odometry calculation is done with the the modified version of the plu-
gin gazebo_ros_diff_drive and fused with IMU measurements using
robot_pose_ekf package [9] to improve the pose estimation.
Under the robot description package directory, “plugins” and “scripts” folders
were created. The files listed below are located in the “plugins” directory:
High Fidelity IMU and Wheel Encoder Models for ROS Based … 235

– gazebo_ros_utils.h and gazebo_ros_utils.cpp

– noisy_diff_drive.h and noisy_diff_drive.cpp
– noisy_imu_sensor.h and noisy_imu_sensor.cpp
The following files are located in the “scripts” directory:
– sensor_uncertainty_generator.py
– sensor_uncertainty_config.json
In order to use the noisy plugins, the libraries
libnoisy_diff_drive.so and libnoisy_imu_sensor.so are
selected in the robot description .xacro file. For these simulations,
p3d_base_controller - libgazebo_ros_p3d.so plugin is also
activated in order to compare the results with ground truth.
After setting up the robot description package, we spawned a robot in Gazebo
environment. Due to its real time capability, the sensor noise can be enabled or
disabled without restarting the simulation.
The file sensor_uncertainty_config.json is used to characterize the
Poisson distribution and sensor noise. This configuration file is used by the Python
script sensor_uncertainty_generator.py which is a ROS node changing
noise parameters in libnoisy_diff_drive and libnoisy_imu_sensor
plugins. During the simulation, the noise parameters can be changed to test localiza-
tion or mapping algorithms using varying sensor dynamics.
The simulations are conducted for the following two manoeuvres with the noisy
plugins activated:
– 3 m × 3 m square route
– Circular route with 1 m turning radius.

3.1 Square Route Test

The AGV is spawned at the pose (0, 0, 0) and moves 3 m forward and then turns left
with 90 degrees until it arrives its starting position. At the beginning of the simulation,
the noise is activated in the modified version of gazebo_ros_diff_drive with
the following settings:
– “time_interval”:10,
– “diff_drive_num_of_error_occurrences”:1,
– “rate”: 1,
– “imu”:“False”,
– “diff_drive”:“True”,
– “diff_drive_noise_duration”:0.5,
– “diff_drive_left_wheel_low_noise”:0.001,
– “diff_drive_right_wheel_low_noise”:0,
236 E. Ozdemir et al.

Fig. 5 Wheel odometry for square and circular routes without (red) and with noise (blue)

– “diff_drive_left_wheel_high_noise”:5.0,
– “diff_drive_right_wheel_high_noise”:0
In this configuration file, the left wheel encoder is intentionally set to be worse
compared to right wheel encoder. Since the route is a square, the odometry pose
should end up where it started. However some pose drifts as well as a distorted
square shape instead of a perfect square are observed. The trajectories for the ground
truth and the odometry can be seen in Fig. 5(left).

3.2 Circular Route Test

The AGV is located at (0, 0, 0) and the following message is published to /cmd_vel
topic to obtain a circular trajectory:
– linear x = 0.5
– angular z = −0.5
The parameters in the configuration file sensor_uncertainty_config.json
are set as follows:
– “time_interval”:10,
– “imu_num_of_error_occurrences”:3,
– “diff_drive_num_of_error_occurrences”:1,
– “rate”: 1,
– “imu”:“True”,
– “diff_drive”:“True”,
– “imu_noise_duration”:0.5,
– “diff_drive_noise_duration”:0.2,
– “imu_orientation_x_low_noise”: 0.1,
– “imu_orientation_y_low_noise”: 0.1,
– “imu_orientation_z_low_noise”: 0.1,
High Fidelity IMU and Wheel Encoder Models for ROS Based … 237

Fig. 6 IMU angular velocity output without noise

– “imu_orientation_w_low_noise”: 0.1,
– “imu_angular_velocity_x_low_noise”:0.1,
– “imu_angular_velocity_y_low_noise”:0.1,
– “imu_angular_velocity_z_low_noise”:0.1,
– “imu_linear_acceleration_x_low_noise”: 0.1,
– “imu_linear_acceleration_y_low_noise”: 0.1,
– “imu_linear_acceleration_z_low_noise”: 0.1,
– “imu_orientation_x_high_noise”: 0.5,
– “imu_orientation_y_high_noise”: 0.5,
– “imu_orientation_z_high_noise”: 0.5,
– “imu_orientation_w_high_noise”: 0.5,
– “imu_angular_velocity_x_high_noise”:0.5,
– “imu_angular_velocity_y_high_noise”:0.5,
– “imu_angular_velocity_z_high_noise”:0.5,
– “imu_linear_acceleration_x_high_noise”: 0.5,
– “imu_linear_acceleration_y_high_noise”: 0.5,
– “imu_linear_acceleration_z_high_noise”: 0.5,
– “diff_drive_left_wheel_low_noise”:0.001,
– “diff_drive_right_wheel_low_noise”:0.001,
– “diff_drive_left_wheel_high_noise”:0.3,
– “diff_drive_right_wheel_high_noise”:0.3
In this configuration file, the odometry and IMU noise levels are set to be medium
based on the field experience. From the simulation results shown in Fig. 5(right),
the odometry error accumulation is observed in negative x and positive y directions.
Similarly, the effect of noise on IMU measurements can be seen in Figs. 6 and 7.
The same test is conducted again in order to see the noise effect on localization
performance. For localization task, noisy odometry and IMU data are fused in the
238 E. Ozdemir et al.

Fig. 7 IMU angular velocity output under noise

Fig. 8 Effects of noisy odometry and IMU data on the localization using EKF and AMCL

robot_pose_ekf package. Then 2D laser scanner data and EKF output are fed
into Adaptive Monte Carlo Localization (AMCL) algorithm. Figure 8 shows the
effect of noise on Extended Kalman Filter and AMCL particle filter. The ground
truth position data is also plotted as a reference. It can be seen that noisy IMU
and wheel encoder measurements cause both distortion and shift on EKF output.
However, this effect is mostly corrected by the particle filter because odom_alpha
values are set to low values (0.15 for this case).
High Fidelity IMU and Wheel Encoder Models for ROS Based … 239

4 Conclusion

In this chapter, we consider the problem of how AGVs are simulated considering real-
istic disturbances and noises in their sensors. These disturbances or noises on wheel
encoder and IMU sensors are modeled to have simulation results consistent with the
real-life scenarios. Hence, we develop a ROS node and Gazebo plugins to incorpo-
rate the disturbance/noise into the sensors and model the error occurrences using
Poisson distribution. The proposed method is tested in Gazebo environment with
two different routes. The internal or environmental effects like wheel slip, encoder
mechanical coupling problems, uneven encoder counting issues cause errors or sud-
den drifts on odometry calculation, which is a significant problem for localization
tasks. This effect is demonstrated on Extended Kalman Filter and Adaptive Monte
Carlo Localization algorithms. Using the proposed methodology, these effects can
be simulated in the Gazebo environment.

References

1. Amirsadri, A., Kim, J., Petersson, L., Trumpf, J.: Practical considerations in precise calibration
of a low-cost mems imu for road-mapping applications. In: American Control Conference
(ACC), pp. 881–888 (2012)
2. Barreda Pupo, L.: Characterization of errors and noises in MEMS inertial sensors using Allan
variance method. Master’s Thesis, Universitat Politècnica de Catalunya (2016)
3. Bayram, H.: High Fidelity Sensor Models (2021). https://github.com/hlkbyrm/
HighFidelitySensorModels
4. Bayram, H.: The modified version of DiffDrivePlugin (2021). https://github.com/hlkbyrm/
HighFidelitySensorModels/blob/main/turtlebot3_highfidelityWheelEncoderIMU/plugins/
noisy_diff_drive.cpp
5. Bayram, H.: The modified version of Gazebo IMU Sensor plugin (2021). https://github.com/
hlkbyrm/HighFidelitySensorModels/tree/main/turtlebot3_highfidelityWheelEncoderIMU/
plugins
6. Bayram, H., Bozma, H.I.: Assistance networks for dynamic multirobot tasks. Auton. Robots
40(4), 615–630 (2016)
7. Goel, R., Gupta, P.: Robotics and industry 4.0. In: A Roadmap to Industry 4.0: Smart Production,
Sharp Business and Sustainable Development, pp. 157–169. Springer (2020)
8. Hsu, J., Koenig, N., Coleman, D.: gazebo_ros_pkgs - gazebo_rosv_diff_drive. https://github.
com/ros-simulation/gazebo_ros_pkgs/blob/kinetic-devel/gazebo_plugins/src/gazebo_ros_
diff_drive.cpp (2019 - commit c89b250)
9. Meeussen, W.: robot_pose_ekf. https://github.com/ros-planning/robot_pose_ekf (2021 - com-
mit fd6cef3)
10. Open Source Robotics Foundation: DiffDrivePlugin. https://github.com/osrf/gazebo/blob/
gazebo11/plugins/DiffDrivePlugin.cc (2020 - commit 3e5ffc9)
11. Open Source Robotics Foundation: geometry_msgs/PoseWithCovariance Message (2021).
https://docs.ros.org/en/melodic/api/geometry_msgs/html/msg/PoseWithCovariance.html
12. Ouyang, W., Wu, Y., Chen, H.: INS/odometer land navigation by accurate measurement mod-
eling and multiple-model adaptive estimation. IEEE Trans. Aerosp. Electron. Syst. 57(1), 245–
262 (2020)
13. Settimi, A.: gazebo_ros_imu_sensor in Simulation Tools In ROS. https://github.com/ros-
simulation/gazebo_ros_pkgs/tree/noetic-devel/gazebo_plugins/src/gazebo_ros_imu_sensor.
cpp (2020 - commit 05d6cb4)
240 E. Ozdemir et al.

14. Siegwart, R., Nourbakhsh, I.R., Scaramuzza, D.: Introduction to Autonomous Mobile Robots,
2nd edn. MIT Press, Cambridge (2011)
15. Skog, I., Handel, P.: In-car positioning and navigation technologies - a survey. IEEE Trans.
Intell. Trans. Syst. 10(1), 4–21 (2009)
16. Ullrich, G.: Automated Guided Vehicle Systems, 1st edn. Springer, Berlin (2015)
17. Yang, Y., Geneva, P., Zuo, X., Huang, G.: Online IMU intrinsic calibration: is it necessary? In:
Proceedings of Robotics: Science and Systems (RSS) (2020)

Emre Ozdemir is an Autonomous Mobile Robots Development Engineer at Ford Otosan, Turkey
for 7 years where he focuses on dynamic simulation, control system design and path tracking algo-
rithms. He is a Ph.D. student at Yildiz Technical University, focusing on Advanced Robust Control
Systems.

Hakan Gokoglu is an Autonomous Mobile Robots Development Engineer at Ford Otosan,

Turkey. He worked as an Engine Test Engineer at dynamometers for 9 years, focusing on engine
testing and automated component test rig real time controller design and development. For two
years, he focuses on autonomous mobile robot control systems design and software development.

M. Koray Yilmaz received the B.Sc. degree in Electrical and Electronics Engineering from
Sakarya University, Sakarya, Turkey, in 2011. He is currently a graduate student at Istanbul
Medeniyet University, Istanbul, Turkey. He is a member of Field Robotics Laboratory. His
research interests include aerial robots and target localization.

Umut Dumandag is an undergraduate student in Electrical and Electronics Engineering, Istanbul

Medeniyet University. He is also a member of Field Robotics Laboratory. His research focus is on
robotic simulations and aerial robots.

Ismail Hakki Savci received the Ph.D. degree in mechanical engineering from Marmara Univer-
sity, Istanbul, Turkey in 2015. He is a technical manager at Ford Otosan for 13 years. He special-
izes internal combustion engines, experimental studies and mobile robot applications.

Haluk Bayram received the Ph.D. degree in electrical and electronics engineering from Bogazici
University, Istanbul, Turkey in 2013. He was a Post-Doctoral Associate at Robotic Sensor Net-
works Laboratory, Department of Computer Science and Engineering, University of Minnesota
between 2014 and 2017. He is an Assistant Professor with Istanbul Medeniyet University, Istan-
bul, Turkey. He is also the director of Field Robotics Laboratory. His research interests include
target localization and tracking, field robotics, aerial and surface robotics, and automated guided
vehicles.