See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/375423446

Automation, Robotics & Communications for Industry 4.0

Research Proposal · November 2023

DOI: 10.13140/RG.2.2.17879.44961

CITATIONS READS

0 198

1 author:

Sergey Y. Yurish
International Frequency Sensor Association (IFSA)
125 PUBLICATIONS 594 CITATIONS

SEE PROFILE

All content following this page was uploaded by Sergey Y. Yurish on 07 November 2023.

The user has requested enhancement of the downloaded file.

 Automation, Robotics &
Communications for Industry 4.0

Proceedings of the 2nd Winter IFSA Conference

on Automation, Robotics & Communications
for Industry 4.0 (ARCI' 2022)

2-3 February 2022

Andorra la Vella, Andorra

Edited by Sergey Y. Yurish

Sergey Y. Yurish, Editor
Automation, Robotics & Communications for Industry 4.0
ARCI’ 2022 Conference Proceedings

Copyright © 2022
by International Frequency Sensor Association (IFSA) Publishing, S. L.
E-mail (for orders and customer service enquires): [email protected]
Visit our Home Page on http://www.sensorsportal.com
All rights reserved. This work may not be translated or copied in whole or in part without the written permission
of the publisher (IFSA Publishing, S. L., Barcelona, Spain).
Neither the authors nor International Frequency Sensor Association Publishing accept any responsibility or
liability for loss or damage occasioned to any person or property through using the material, instructions, methods
or ideas contained herein, or acting or refraining from acting as a result of such use.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identifies as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary
rights.

ISSN: 2938-4788
ISBN: 978-84-09-37741-1
BN-20220201-XX
BIC: TJFM
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Contents

Foreword ........................................................................................................................................................... 5

Measure of Complexity of the Spatial Environment of a Mobile Object ..................................................... 6

A. N. Karkishchenko, V. Kh. Pshikhopov
Identification and Discrete Inversion of Multi-Mass Systems as Part of a Disturbance Observer ........ 13
C. Schöberlein, M. Y. Liu, A. Schleinitz, H. Schlegel and M. Dix
Development and Validation of a Model for Online Estimation of Process Parameters
for Adaptive Force Control Algorithms........................................................................................................ 19
M. Norberger, A. Sewohl, S. Sigg, H. Schlegel and M. Dix
Simulation of Automated Handling in Textile Manufacturing of US Military Apparel
to Improve Efficiency and Quality ................................................................................................................ 25
Z. B. Rosenberg, J. A. Joines and J. S. Jur
Multi-Robot Cooperative SLAM Using Panoramas .................................................................................... 31
J. Y. Feng and Z. XuanYuan
The Autonomous Pollination Drone .............................................................................................................. 38
D. Hulens, W. Van Ranst, Y. Cao and T. Goedemé
“They got my keys!”: On the Issue of Key Disclosure and Data Protection in Value Chains .................. 42
A. Mosteiro-Sanchez, M. Barcelo, J. Astorga and A. Urbieta
Virtual Commissioning of an Automotive Station for Door Assembly Operation ................................... 46
R. Balderas Hill, J. Lugo Calles, J. Tsague, T. Master and N. Lassabe
A Model Driven and Hardware Agnostic Approach of Virtual Commissioning ..................................... 50
S. Marchand, H. Alhousseini, R. Bres, F. Dumas, M. Lachaise, L. Poulet de Grimouard
and M. Stieglitz
Development of an AI Maturity Model for Small and Medium-sized Enterprises .................................. 57
B. Schmidgal, M. Kujath, S. Kolomiichuk, M. Rentzsch and S. Häberer
Management and Path Planning Solution for Parking Facilities using Dynamic Load Balancing ........ 64
F. D. Sandru, V. I. Ungureanu and I. Silea
Switching Propulsion Mechanisms of Tubular Catalytic Micromotors..................................................... 70
P. Wrede, M. M. Sánchez, V. M. Fomin, and O. G. Schmidt,
Stability Margins for Linear Periodically Time-Varying Systems ............................................................. 73
Xiaojing Yang
Obstacle Segmentation for Autonomous Guided Vehicles through Point Cloud Clustering
with an RGB-D Camera ................................................................................................................................. 79
M. Pires, P. Couto, A. Santos and V. Filipe,
Competency-based Education of the Mechatronics Engineer in the Transition
from Manufacturing 3.0 to Industry 4.0 ....................................................................................................... 84
Eusebio Jiménez López, Francisco Javier Ochoa Estrella, Gabriel Luna-Sandoval,
Flavio Muñoz Beltrán,, Francisco Cuenca Jiménez and Marco Antonio Maciel Monteón
Simulation of a Collision and Obstacle Avoidance Algorithm for Cooperative Industrial
Autonomous Vehicles ..................................................................................................................................... 88
J. Grosset, A.-J. Fougères, M. Djoko-Kouam, C. Couturier and J.-M. Bonnin
Artificial Intelligence and Measurements ..................................................................................................... 92
R. Taymanov, K. Sapozhnikova, and A. Shutova
Intelligent Sensors Networks for Monitoring and Controlling Complex Systems
under Conditions of Uncertainty ................................................................................................................... 96
S. V. Prokopchina

3
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Methods and Technologies of Bayesian Intelligent Measurements for Human Resources

Management in Industry 4.0 ........................................................................................................................ 100
S. V. Prokopchina, E. S. Tchernikova
Intelligent Acoustic Monitoring of Underground Communications ........................................................ 104
V. P. Koryachko 1 , V. G. Sokolov 2 , S. S. Sergeev 3

4
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Foreword

On behalf of the ARCI’ 2022 Organizing Committee, I introduce with pleasure these proceedings with
contributions from the 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0
(ARCI’ 2022), 2-3 February 2022.

According to the modern market study, the global Industry 4.0 market will reach US$ 165.5 Billion by 2026
growing at the CAGR of slightly above 20.6% between 2021 and 2026. The Industry 4.0 means the usage of an
integrated system, which consists of an automation tool, robotic control and communications. The key factors
fuelling the growth of the industry 4.0 market include rapid adoption of Artificial Intelligence (AI) and Internet
of Things (IoT) in manufacturing sector, increasing demand for industrial robots, rising government investments
in additive manufacturing, and growing adoption of blockchain technology in manufacturing industry.

Industry 4.0 represents the 4th industrial revolution that marks the rising of new digital industry. It is defined as
an integrated system that comprises numerous technologies such as advanced robotics control, automation tools,
sensors, artificial intelligence, cloud computing, digital fabrication, etc. These technologies help in developing
machines that will be self-optimized and self-configured. It helps in enhancing asset performance, technology
usage, material usage and other industrial processes that are involves in various industries. Numerous benefits are
offered by these technologies such as low operational cost, improved productivity, enhanced customer
satisfaction, improved customization, and increased efficiency. The Industry 4.0 holds a lot of potentials and is
expected to register a substantial growth in the near future. There are several conferences on automation, robotics
and communications, but they are not meet the Industry 4.0 challenges.

The series of annual ARCI Winter IFSA conferences have been launched to fill-in this gap and provide a forum
for open discussion of state-of-the-art technologies related to control, automation, robotics and communication -
three main components of Industry 4.0. It will be also to discuss how to adopt the current R&D results for Industry
4.0 and to customize products under the conditions of highly flexible (mass-) production.

The conference is organized by the International Frequency Sensor Association (IFSA) - one of the major
professional, non-profit association serving for sensor industry and academy more than 20 years, in technical
cooperation with media partners – journals: MDPI Processes (Switzerland), MDPI Machines (Switzerland) and
Soft Measurements and Computing (Russia). The conference program provides an opportunity for researchers
interested in signal processing and artificial intelligence to discuss their latest results and exchange ideas on the
new trends.

I hope that these proceedings will give readers an excellent overview of important and diversity topics discussed
at the conference.

We thank all authors for submitting their latest works, thus contributing to the excellent technical contents of the
Conference. Especially, we would like to thank the individuals and organizations that worked together diligently
to make this Conference a success, and to the members of the International Program Committee for the thorough
and careful review of the papers. It is important to point out that the great majority of the efforts in organizing the
technical program of the Conference came from volunteers.

Prof., Dr. Sergey Y. Yurish,

ARCI’ 2022 Conference Chairman

5
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(002)

Measure of Complexity of the Spatial Environment of a Mobile Object

A. N. Karkishchenko 1, V. Kh. Pshikhopov 2
1
Southern Federal University, 2 Research and Design Bureau for Robotics and Control Systems,
2 Shevchenko Str., 347928 Taganrog, Russia
Tel.: + 7 8634371694, fax: +7 8634681894
E-mail: [email protected]

Summary: Formal determination of the complexity indicators of the environment of a mobile object operating in three-
dimensional space in the presence of obstacles is considered. The mathematical substantiation of the method for calculating
the complexity is given. The concepts of local and integral complexities of the environment are introduced. Formulas for
calculating the complexity are given.

Keywords: mobile object, scene, triangulation, local complexity, integral complexity.

1. Introduction In [5], an approach to determining the complexity

of scenes from the point of view of the visual visibility
The planning of the movement of autonomous of the surfaces forming is considered, as well as under
mobile objects (MO) in uncertain environments is conditions of diffuse illumination of these surfaces. In
currently of considerable interest [1, 2]. In particular, [6], a method for estimating the complexity of
special requirements are imposed on such algorithms polygonal scenes based on reachability graphs is
if the environment has elements of unpredictability and given. The work [7] is devoted to the introduction of
parametric uncertainty [3]. Experimental studies have the concept of the complexity of the visible part of a
shown that the choice of strategies and planning three-dimensional scene, depending on the point of
algorithms significantly depends on the characteristics observation. In [8], a method is given for assessing the
of the environment in which MO operate. In many complexity of a scene during animation, i.e., an
cases, it is advisable to use different planning attempt was made to develop methods for assessing the
algorithms at different stages of MO movement. It is complexity of changing scenes. At the same time, in
shown in [3] that in complex environments this can [6], a geometric approach to describing complexity is
lead to a significant, up to 50 %, improvement in used, which is computationally laborious with a large
performance indicators. At the same time, the use of number of elements on the scene, and in [5, 7, 8], a
the same planning methods in simple environments computationally simpler statistical approach is used to
leads to an insignificant, about 10 %, change in quality introduce the concept of complexity from the point of
indicators. Therefore, the key concept is the view of information theory. At the same time, all these
complexity of the environment, which is difficult to methods are generated by and associated with
formalize; nowadays this concept is often defined and computer graphics problems and have little relation to
used intuitively. As a result, this does not allow assessing the complexity of the environment in which
formalizing the procedure for choosing one or another an autonomous mobile object operates.
approach to planning a movement corresponding to the Previously, the authors considered the problem of
complexity of the environment and, therefore, determining the complexity measure of the
increasing the efficiency of the MO. environment of a mobile object operating on a plane in
The purpose of this work is to construct a measure the presence of obstacles [9]. This work generalizes for
of the complexity of the environment, which allows the spatial case the results obtained earlier.
optimal selection of motion control algorithms.
The concept of complexity arises in various fields,
2. Problem Statement and Assumptions
but there is no uniformly understood definition, since
specific tasks lead to the need to introduce specific An autonomous mobile object moves in space with
definitions of this concept. “The meaning of this the task of getting to the point that is the target. At the
quantity should be very close to certain measures of same time, there may be other stationary or moving
difficulty concerning the object or the system in objects in the area of movement of the MO, which
question: the difficulty of constructing an object, the make it difficult to plan the trajectory of movement,
difficulty of describing a system, the difficulty of giving rise to the risk of a potential collision. Objects
reaching a goal, the difficulty of performing a task, and that interfere with the movement of a moving object
so on. The definition of complexity cannot be unique, will be called obstacles. To simplify the model, it is
simply because there are many different ways to assumed that all such obstacles are replaced by the
quantify these difficulties …” [4]. minimal balls representing them. If the real obstacle is

6
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

not such, we will assume that it is replaced by a simple, if there are no obstacles at all on the scene, up
minimal ball containing this obstacle inside. to maximum, if obstacles block the target and do not
It is assumed that the MO has range sensors that allow the MO to move to the target. In order to
allow scanning a spatial area in the direction of reasonably choose a strategy for the formation of the
movement. Geometrically, the scanned area is a trajectory, it is necessary to have a quantitative
pyramid S with the top that coincides with the MO. measure  (S ) of the complexity of the scene S .
The pyramid is always symmetrical about the axis
of movement of the MO. Moreover, it has a fixed apex
angle equal to 2 h in the horizontal plane, and equal 3. General Requirements for the Measure
to 2 v in the vertical plane (Fig. 1). The pyramid with of Complexity
the obstacles indicated in it and, possibly, the target,
we will sometimes call the scene and also denote S . The choice of a measure of complexity is, generally
speaking, not so much a mathematical question as a
question reflecting the physical content of what is
meant by the complexity of a real scene. It is possible
to present different requirements to the complexity of
Z this measure, meaning various parameters of the scene,
characterized and/or taken into account by this
measure. Therefore, the introduction of such a measure
can be carried out in many ways. In particular, taking
Y into account that dynamically changing scenes are
generally considered, the degree of complexity can
v depend on time t , i.e.,   S (t )     S , t  .
h
It should also be noted that such a measure will
only relatively characterize the complexity of the
Scene S scene, in the sense that the attainability of a given
MO
target will depend not only on the parameters of the
scene itself, in particular, the size and number of
X obstacles, but also on the size of the mobile object
itself. The scene, which is relatively simple for a small
Fig. 1. Scene S and scanning angles. MO, can be difficult for a large mobile object, since
the target may be unattainable for it due to the lack of
sufficiently wide passages between obstacles.
We will assume that the faces forming the pyramid
At the same time, it is possible to formulate a
are always flat, while the base opposite to the top of the
general requirement, the fulfilment of which is
pyramid in reality can have an arbitrary shape,
naturally required from any measure of complexity.
including kinks and breaks, since it is the result of
The measure should be positive and preferably
scanning the real space in front of MO. Nevertheless,
for the sake of simplicity, we will assume that it has normalized, i.e., 0    S(t)  1 for any S and t . In
the shape of a flat rectangle (Fig. 1), at a distance H this case, equality to zero should correspond to the
from the vertex (the length of the centerline of the simplest scene, on which there are no obstacles, and
pyramid). the trajectory can be a segment of a straight line. The
We also assume that there is an external coordinate most difficult scene, on which it is impossible to lay a
system that allows to accurately position the object and trajectory to the target, should have a difficulty equal
the target. However, further constructions will be to one.
considered in the coordinate system associated with
the MO, in which the origin of coordinates coincides
with the MO center, the ordinate axis – with the MO 4. Triangulation and Spatial Partitioning
axis in the direction of its movement, and the abscissa
axis is directed to the right, so that together with the We will assume that there are n obstacles
applicate axis, the coordinate axes form right triplet B1, B2 ,..., Bn on the scene S . Each obstacle in space
(Fig. 1).
We will assume that the space outside the scanned will be characterized by four real numbers
pyramid is completely occupied by obstacles, i.e. it is Bi   xi , yi , zi , ri  , where xi , yi , zi are the coordinates
impossible to build trajectories outside the pyramid. In
of the center of the obstacle, and ri is its radius. For
other words, all possible trajectories of the object's
movement lie strictly inside the pyramid. convenience, the center of the i -th obstacle will also
Depending on the position of the target and MO, as be denoted pi (xi , yi , zi ) . We will also designate by
well as the parameters of the scene (position, number,
size of obstacles), the difficulty of achieving the target P   p1, p2 ,..., pn  – a set of obstacle centers located
may vary. This "difficulty" can vary from extremely in space.

7
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Consider the vertices of the pyramid S as new dimensional space of a four-dimensional polyhedron
virtual obstacles BA , BB , BC , BD , BE and construct the of a special type. Linear relations for the number of
faces of different dimensions of this polyhedron are
spatial Delaunay triangulation on expanded set of described by the Dehn-Sommerville equations [11]. If
obstacles [10]. With this construction, all internal
we denote by f 1 the number of edges (one-
regions of the resulting partition will be tetrahedra, and
its surface will be the convex hull of the triangulation dimensional faces) of the Delaunay polyhedron, then
(Fig. 1). with the help of some additional constructions it can be
shown that the following relations hold:

BC  M  2N  3  0 ,
 (1)
BB n  f1  N  7  0 .

As it will be shown below, the parameters of the graph

ultimately determine the complexity of the reachability
computation.
BD The point defining the location of the target (if
any) will be inside a certain area of spatial partition,
BA therefore it will also be identified with some vertex of
the constructed graph. However, as follows from the
construction, the point at which the MO is located is a
ПО
BE virtual obstacle BE ; therefore, it is identified with all
areas for which BE is the vertex of the corresponding
Fig. 2. Triangulation of an expanded set of obstacles. tetrahedron.
If the target and the MO turn out to be identified
Any trajectory of a MO to the target can be with the same vertex of the graph, then the complexity
associated with a sequence of regions from the is zero. The solution in this case is a trajectory, which
partition, which this trajectory intersects. Such a is a segment of a straight line connecting the MO and
trajectory and the corresponding sequence of regions the target. Therefore, we will assume that the mobile
can be called conjugate. Conversely, any sequence of object and the target correspond to different vertices of
regions in which any two adjacent regions are adjacent, the graph.
the first of which contains the MO, and the last is the Any trajectory of the object's motion corresponds
target, generates a family of trajectories conjugate to it. to a conjugate sequence of partition regions, and to it,
We will assume that all trajectories satisfy the in turn, a path on the graph G from some vertex
constraints that are natural for the problem under corresponding to the MO to the vertex corresponding
consideration: to the target is in one-to-one correspondence. To obtain
1) They cannot cross the border of each area in constructive results, this model requires metric
more than two points; refinement, since different paths can be very different
2) They cannot pass through points that belong to in terms of the safety of their passage.
three or more partition areas.
6. Path Width. Target Reachability
5. Topological Description of Triangulation
When moving, a mobile object intersects the
imaginary tetrahedron faces in the Delaunay partition.
The resulting partition of the pyramid S will be The faces have different sizes and, therefore, different
described by a topological graph G   X , U  with a complexity of passing. This complexity can be
set X of vertices, X  N , and a set U of edges, considered as the throughput, hereinafter called
"width", and measured as the area of the region, which
U  M . The vertices of the graph are in one-to-one
is the locus of points, possessing the following
correspondence with N areas of the pyramid property: if the center of the object is at such a point,
partition. Two vertices are considered adjacent if and then the object passes unhindered through this face
only if the corresponding areas have a common border (Fig. 3).
of non-zero area. All points lying inside some open Let an arbitrary face be formed by three obstacles
area of the partition are mapped to the same vertex of with centers at points p1 , p2 , p3 and, respectively,
the graph. radii r1 , r2 and r3 . Then, using geometric reasoning,
There is a certain relationship between the number
it can be shown that the width w of such an area is
N of graph vertices, the number M of edges and the
determined by the expression
number n of obstacles, which is determined by the
particular qualities of the Delaunay partition. The 1 1 3
Delaunay partition can be viewed as a net in three- w  p1 p2 , p1 p3     ri     2 i ,
2   2 i 1

8
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

where [] is the vector product,  – radius of the For any i and j all paths from x i to x j fall into
mobile object,  i are the angles in radian measure at disjoint classes of paths of topological length 1, 2, …
the corresponding vertices of the face,   0 is a However, some of these classes may be empty.
certain "gap" of safety. The more  , the safer the Therefore, finding the width of the safest path is
passage. reduced to choosing the safest path among the safest
paths in each class, i.e., max rij( k ) .
k I 

r3 Taking into account the reflexivity of the relation

 3 set by the graph G , when calculating the maximin
"degrees" of a matrix, it is sufficient to restrict
 S
r1 ourselves only to matrices R (1)  R , R (2) ,..., R ( N 1)
 since R ( N 1)  R ( N )  R ( N 1)  ... . Therefore, if, for
1
r2 example, xMO  xi , and xT  x j then
2
  xT   max max rij( k ) .
xi  X MO k I N 1

Fig. 3. The region of passage of the object through the face Note that the matrix R has size N  N , and by
of the two regions of the partition.
virtue of relations (1), equality N  f1  n  7 holds.
Thus, to determine the exact size of the matrix R , it is
The entered widths can be considered as a weight necessary to know the number f1 of edges of the
function defined on the edges of the graph. Thus, we resulting partition. However, it can be seen that the size
consider a weighted topological graph G   X , U , w  of the matrix is proportional to f1 . Meanwhile, an
with a marked set X MO of vertices corresponding to asymptotic estimate is known [12], namely, any
the position of the object, and a vertex xT triangulation in a space of dimension d contains
corresponding to the target.
The width of a path on a graph is a value equal to

O n 
d 2
 simplices, where  is the nearest integer
the smallest width among all the edges that form this greater than or equal to that element, so in our case it
path. Consider all the paths on the graph that connect is O  n 2  . Thus, for large n , there is a quadratic
the object to the target. Reachability  of the target dependence of the matrix size on the number of
refers to the maximum width among all such paths. obstacles. In this case, the number of edges of a
topological graph is related to the number of vertices
by the relation M  2N  3 , therefore, the number of
7. Matrix Procedure for Computing nonzero elements of the matrix R is exactly
Reachability determined by its size and by virtue of symmetry is
equal to 4N  6 . Therefore, at large N , the matrix R
To calculate the reachability   xT  , one can use a is very sparse, which can be used to optimize
simple matrix procedure. Consider the adjacency computations.
matrix R  rij of a graph G   X , U , w  , where Reachability  as a characteristic of a scene is
inconvenient, since, on the one hand, it is expressed in
 w(uij ), if xi and x j are adjacent , absolute units of distance, i.e., depends on the chosen
rij   scale, and, on the other hand, it does not satisfy the
 0, otherwise.
axiomatic requirements for a measure of complexity.
Next, we will consider an expression for a complexity
Now define the maximin composition R  R  R
(2)
measure associated with the reachability and devoid of
of the matrix R with itself according to the rule the indicated disadvantages.


R(2)  R  R  rij(2) , where rij(2)  max min  rik , rkj  ,
kI N

(2) 8. Local Complexity Measure
where I N  {1, 2,..., N }. Then rij is equal to the
maximum width among all widths of all paths of length Note that with growth  it is natural to assume that
2 connecting the vertices x i and x j . In particular, this the complexity  ( ) should decrease. Moreover, if 
means that there is at least one maximally "safe" path is large enough, and accordingly  ( ) small, then
(2)
of length 2 from x i to x j , the width of which is rij . further increase will only slightly reduce the
complexity. On the contrary, for small reachability, the
Next, inductively define R  R  R  rij
(3) (2) (3)
, complexity of the scene should be large, and small
increases of  should lead to a rapid decrease in
…, R( n1)  R(n)  R  rij(n1) , …
complexity. Thus, we can assume that the rate of

9
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

change of complexity at any value of reachability  is are reported to the mobile object in the process of
inversely proportional to the value of complexity at the moving it on the scene, or they can change at different
same value, i.e.,  ( )   ( ) , where  is some times. An integral measure can be constructed as
follows.
positive constant characterizing this dependence.
For simplicity, we consider the case when all
Hence, we obtain that  ( )  Ce   , where C is the obstacles are point, i.e. have a zero radius. Let us
constant of integration, which can be established on the assume for definiteness that the tetrahedron in which
basis of the following considerations. the mobile object is located has number 1. Let us
When   0 the target is unattainable for the MO, denote by 1 ( p ) the complexity of reaching the point
and the complexity of such a scene is maximal. Taking p ( x, y )  S by the mobile object located at the top of
into account the axiomatically imposed requirement
that the complexity measure be normalized, i.e., the pyramid. Then, as an integral characteristic of the
complexity of the scene, we can take the averaging
0   ( )  1 , we obtain  (0)  С  1 and, therefore
over the local measures of the complexity of all points
 ( )  e   . of the scene:
The parameter  determines how quickly the
complexity of the scene decreases with increasing  . 1
S  
m  S  S
1 ( p)dp ,
The choice of  can be made in various ways, for
example, as follows. Since the limiting value
 ( )  0 is formally reached only at    , then for where m S  is the volume of the pyramid S .
real problems it can be assumed that the complexity is It is easy to see that the function 1 ( p ) is
"practically" equal to 0, if, say,  ( )   , where  is piecewise constant on S . If we denote by the Sk - k
a small positive number. On the other hand, it can be -th region of the spatial partition of the pyramid, and
assumed that the complexity of the implementation of
the movement of the MO is practically zero if the path by m Sk  its volume, then
width (i.e., reachability) exceeds the area of the
diametrical section  2 of the mobile object by more 1 N N
m  Sk   r1k
S    
m  S  k 1 Sk
 1

( p ) dp  
k 1 m  S 
e .
than a factor k . In other words, we can assume that
2 ln 
  e   k  , from where    . With this in
k  2 Here the quantities m  S  and m  Sk  are easily
mind, the expression for the complexity measure takes calculated from geometric considerations.

This formula is obtained under the assumption that
the form  ( )   k 2
. Assuming, for example the obstacles are point. In this case, it admits of a

 simple probabilistic interpretation. If the appearance of
  0 .0 1 and k  20 , we get  ( )  10 10 2
. a point specifying the target is a random variable
m  Sk 
The reachability value depends on the coordinates uniformly distributed in S , then is the
of the target, so the last expression should be mS 
understood as a function probability of the target appearing in the area Sk ,
therefore, the last expression is the mathematical
   ( p )   e   ( p ) , expectation of the local complexity function.

Therefore, it is reasonable to call this expression a

local measure of the scene complexity. 10. Modeling

Simulation in MATLAB for the convenience of

9. Integral Complexity Measure visualization was carried out for the plane case. In this
case, the pyramid turns into a triangular sector, and the
The resulting expression is an estimate of the obstacles are represented by the minimum circles
complexity of the scene with the selected target, i.e. covering them. Below Fig. 4 – Fig. 7 show examples
with the given coordinates of the target. The question of randomly generated scenes for the same sector with
arises how to characterize the complexity of the scene obstacles of different radius, for which the local (LC)
in general, i.e., irrespective of the given target of the and integral (IC) complexities are calculated. Target
movement. One of the possible approaches is to position x  32, y  64 , the size of the mobile object
introduce some characteristic that takes into account is 3, the positions of obstacles and their radii within
the complexity of reaching each point of the scene, 3 - 11 were formed randomly. The number of obstacles
which may be a potential target. It is natural to call such in the experiments shown is 10. The calculation time
a measure an integral measure of the complexity of the included generating the scene and obstacles,
scene. A similar problem arises, for example, if the calculating all parameters, displaying the picture on
coordinates of the target are not known in advance, but

10
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

the screen, calculating the local and integral 11. Conclusions

complexities and was approximately 0.1 sec.
The paper proposes an approach to determining the
complexity of the spatial environment of a mobile
object. This concept is based on the safety of
movement of an object when planning the path of
movement to a given target. The results obtained are
constructive and allow real-time calculations. The
results of modeling confirmed the constructiveness
and computational efficiency of the proposed
measures of the complexity of the environment.
The calculation of complexity measures is done
quickly enough, so the considered method can be used
cyclically, applying it to dynamically changing scenes,
i.e., with movable obstacles. In other words, in such
conditions, based on sensory information, it is possible
Fig. 4. LC = 1.0000, IC = 0.9791. to continuously monitor the complexity of the
environment and select an algorithm for controlling a
mobile object depending on the prevailing conditions.
The development of this study can be a generalization
of the results for the case when the assessment of the
complexity of the environment is based on several
indicators that characterize not only the safe
attainability of the target, but also the quality of the
trajectories of movement.

Acknowledgements
This work was carried out at the Research and Design
Fig. 5. LC = 0.5371, IC = 0.6282. Bureau for Robotics and Control Systems with the support
of the Russian Science Foundation, project no. 22-29-00533.

References
[1]. M. Hoy, A.S. Matveev, A.V. Savkin, Algorithms for
collision-free navigation of mobile robots in complex
cluttered environments: a survey, Cambridge
University Press, 2014.
[2]. Xingjian Jing, Behavior dynamics-based motion
planning of mobile robots in uncertain dynamic
environments, Robotics and Autonomous Systems,
Vol. 53, Issue 2, 2005, pp. 99-123.
[3]. Path Planning for Vehicles Operating in Uncertain 2D
Environments, Ed. V. Pshikhopov, Elsevier,
Fig. 6. LC = 0.3670, IC = 0.4503. Butterworth-Heinemann, 2017.
[4]. W. Li, On the Relationship Between Complexity and
Entropy for Markov Chains and Regular Languages,
Complex Systems, Vol. 5, Issue 4, 1991, pp. 381-399.
[5]. M. Feixas, E. del Acebo, Ph. Bekaert, M. Sbert, An
Information Theory Framework for the Analysis of
Scene Complexity, EUROGRAPHICS’99, Vol. 18,
Issue 3, 1999.
[6]. L. Niepel, J. Martinka, A. Ferko, P. Elias, On Scene
Complexity Definition for Rendering, Winter School
of Computer Graphics and Visualization 95
(WSCG’95), Plzen, 1995, pp. 209–217.
[7]. D. Plemenos, M. Sbert, M. Feixas, On Viewpoint
Complexity of 3D Scenes, in Proceedings of the
International Conference GraphiCon, 2004, Moscow,
Russia, (http://www.graphicon.ru)
Fig. 7. LC = 0.9889, IC = 0.9805.
[8]. J. Rigau, M. Feixas, M. Sbert, Visibility Complexity of
a Region in Flatland, EUROGRAPHICS, 2000.

11
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

[9]. A. Karkishchenko, V. Pshikhopov, On Finding the [11]. A. Brøndsted, An Introduction to Convex Polytopes,
Complexity of an Environment for the Operation of a Springer Verlag, 1983.
Mobile Object on a Plane, Automation and Remote [12]. R. Seidel, The upper bound theorem for polytopes: an
Control, Vol. 80, Issue 5, 2019, pp. 897-912. easy proof of its asymptotic version, Computational
[10]. Siu-Wing Cheng, Tamal K. Dey, Jonathan Shewchuk, Geometry, Vol. 5 Issue 2, 1995, pp. 115–116.
Delaunay Mesh Generation, Chapman and Hall/CRC,
2013.

12
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(004)

Identification and Discrete Inversion of Multi-Mass Systems

as Part of a Disturbance Observer
C. Schöberlein 1, M. Y. Liu 1, A. Schleinitz 1, H. Schlegel 1 and M. Dix 1
1
Institute for Machine Tools and Production Processes, Chemnitz University of Technology
Reichenhainer Straße 70, 09126 Chemnitz, Germany
Tel.: + 4937153130505 fax: + 49371531830505
E-mail: [email protected]

Summary: In the context of Industry 4.0 and the proclaimed digitalization of production towards autonomous and self-
monitoring systems, an increasing demand for novel monitoring and diagnostic functions can be derived. Especially in the
area of production technology, continuous monitoring and evaluation of the manufacturing processes is of particular interest
for developing autonomous production systems. A contribution can be made by machine internal drive systems and their
integrated sensors and measurement functions. In this paper, a partial aspect of an observer structure for estimating load and
process forces based on motor currents and axis positions of electromechanical drive systems is presented. The key objective
is the identification of the required transfer functions of the mechanical transfer behaviour between load application point and
servomotor. The methodology is based on the exclusive use of drive internal excitation and signal sources. An identification
routine automatically determines the system order based on frequency responses and calculates a corresponding model. The
optimization of the model parameters is carried out by applying a Nelder-Mead-optimization algorithm. Subsequently, the
optimized models are inverted and compared to the original system. In addition, all model equations are discretized to enable
their implementation on discrete sampled computing systems like the machine control.

Keywords: Disturbance observer, Parameter estimation, Transfer function, Discrete inversion, Multi-mass system.

1. Introduction the other hand, the simultaneous measurement of

motor and load side position signals is also discussed
Process monitoring systems can contribute to in various publications (e.g. [6, 7]). The investigated
increasing the productivity and flexibility of system characteristic is limited to second order
production systems as well as the quality of the systems. A simulative comparison of the mentioned
manufactured products. For numerous production methods and an evaluation under variation of model
processes (e.g. metal cutting, deep drawing, order and its parameters was performed in [8]. In
burnishing, shear cutting), the machining forces addition to open-loop methods, numerous publications
represent a significant quantity for evaluating and focus on approaches in which the estimated value is
monitoring the process itself. Besides an installation of continuously fed back into the model. For example,
additional sensors, these forces can be estimated by Aslan [9] parameterizes an extended Kalman filter
utilizing the already installed position and current based on frequency response measurements. The main
sensors of the main and auxiliary drives of the advantage is an appropriate handling of non-
machine. minimum-phase multi-mass systems.
The majority of common applied disturbance However, all model-based methods have in
estimation methods is based on a more or less complex common that a model of the electromechanical system
model of the drive systems as a disturbance observer. is required. The bandwidth and accuracy of the
Ohnishi [1] presents a basic structure that subsequently disturbance estimation increases with the level of
serves as the origin for other observer types. The detail of the underlying models. The purpose of this
author models the mechanical subsystem as a first paper is to determine appropriate models for an
order system with an additional low-pass filter to observer approach called transfer function-based
eliminate resonance effects. In [2] the author considers disturbance observer (TFDOB), which was already
the mechanical system as an elastically coupled multi- introduced in [10]. Its fundamental structure as well as
mass oscillator. Nonetheless, a first-order system is an exemplary multi-mass system is shown in Fig. 1. In
ultimately applied, since a directly driven linear axis is the figure, the parameters C and D denote the stiffness
used for the experimental investigations. and damping values of the i-th partial oscillator with
In order to avoid the modeling of complex multi- the corresponding moment of inertia J. The angular
mass mechanics, a position measuring system on the velocity ω and position φ as well as the motor torque
load side offers another opportunity. Modern machine Tm are the input values for the observer. The index m
tools usually provide such a direct measurement denotes all motor related parameters, whereas the
system for reasons of position accuracy. A so-called index l identifies all measured variables on the load
load disturbance observer is applied by Yamato and side. Note that the number of partial oscillators
Sato to estimate the process forces in milling [3-5]. On between motor side and load side (index i) is not

13
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

restricted. Eventually, the observer estimates the torque Ta as well as the transfer function of the
unmeasurable load torque Tl . An approach for the mechanical part Gmech (z) must be determined.
automatic identification and subsequent validation of In the following section, the procedure for the
the models for friction compensation (Tf,m , Tf,i and automatic identification and inversion of the required
Tf,l ) and axis weight correction Tg were already discrete models is presented. Subsequently, the
investigated in [11]. For the complete observer, the experimental validation of the proposed method is
inverted transfer functions in the indirect case Gp,m (z) carried out on a feed axis of an exemplary machining
and direct case Gp,l (z) for estimating the acceleration center. The paper concludes with a summary of the
results and an outlook.

Fig. 1. Signal flow chart of multi-mass system and transfer function-based disturbance observer (TFDOB)
for drive-based disturbance estimation.

2. Methodology the same time, the angular velocities on motor side and
load side are recorded. Subsequently, the signals are
The fundamental sequence for determining the discrete transformed into frequency domain and their
inverse model equations is shown in Fig. 2. In the first magnitude and phase parts are calculated. To avoid
module, the system is excited on the motor side with a distortions of the frequency response in the range of
Pseudo-Random Binary Signal (PRBS). The main the controller bandwidth, the speed controller should
advantage over other established identification be parameterized as a backup controller during signal
routines, e.g. in [9, 12] is the renunciation of external recording.
excitation sources like impulse hammer or shaker. At

Fig. 2. Flow chart of the discrete model estimation and inversion procedure.

14
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Based on the magnitude response, the model values. Therewith, the value of parameter a is set equal
parameters are estimated in module two. The to one (cf. Eq. (3)). If, on the other hand, a phase
frequency responses for the indirect case Gp,m (z) and rotation of -180 ° takes place without subsequent
direct case Gp,l (z) are characterized by the total mass reversal, this indicates a partial oscillator according to
moment of inertia in the range of low frequencies Eq. (3) with the parameter a set to zero. The associated
according to Eq. (1). In addition, depending on the damping values dr,i and df,i are initialized with default
number of resonance and antiresonance frequencies values of 0.01.
ωr,i and ωf,i , a defined number of partial oscillators can In the second part of module two, the optimization
be detected. These are calculated using Eq. (2). For the of the model parameters is performed using Nelder-
direct case, an adjustment of the partial oscillator Mead method [15]. The previously determined
characteristic can be made by adapting parameter a parameters Jtot , ωr,i , ωf,i , dr,i and df,i are substituted
{0,1} (cf. [10, 15]). into the corresponding Eq. (3) or (4) and the deviation
between measured and modeled magnitude response is
1 minimized. In addition, the method is allowed to vary
GJ s = (1) the frequency values in the range of ± 10 %. The
Jtot ∙s
damping values may vary between 0 and 1. Eventually,
1 2 2 2df this results in a continuous-time model for the indirect
a∙ ∙s + ∙s+1 case and direct case (Eq. (3)) as well as the mechanical
ωf ωf
GPO s = (2) part according to Eq. (4).
1 2 2 2dr
∙s + ∙s+1 In module three, the transformation of all
ωr ωr
continuous-time models into discrete-time models is
By multiplying GJ s with the product of all partial performed using the z-transformation [16]. The cycle
oscillators GPO,i s in Eq. (3), the actual model time Tcyc of the installed control system is used for the
equations for Gp,m (s and Gp,l (s are derived. sampling time TA. Furthermore, there may occur large
differences between the largest and smallest exponents
n when calculating discrete transfer functions. This can
Gp,ml s =GJ s ∙ GPO,i s (3) ultimately lead to inaccuracies when determining the
i=1
corresponding poles and zeros. In order to avoid this
effect, the transfer functions are initially transformed
The model of the mechanical transmission into the discrete state space. Subsequently, a
behavior Gmech (s), which can also be calculated as the transformation of the system matrices into an
quotient of Gp,m (s) and Gp,l (s), is calculated solely by equivalent system is performed using a transformation
multiplying all i partial oscillators (Eq. (4)). matrix. The underlying procedure is described in detail
in [17].
n In module four, the inversion of the model
Gmech s = GPO,i s (4) equations is carried out. Especially in case of direct and
i=1 mechanical transfer functions (Eq. (2) and (3)), the
resulting models may have a non-minimum phase
As illustrated in the next section, a separate character. This means that the transfer function has one
estimation for all three model equations is necessary to or more unstable zeros, which is expressed in the pole-
achieve correct model parameters. This can be justified zero diagram by their location outside the unit circle.
by minor differences in frequency and damping values By inverting the model equation, all zeros become
depending on the location of the direct measurement poles and vice versa. Consequently, the inverted
system and the constructive design of the system. transfer function would have the same number of
The underlying methodology for the actual unstable poles, which basically result in an unstable
determination of the model parameters was already system itself. In this case, the transfer function is
presented in [13, 14] and partially extended. The basic decomposed into a minimum phase component and an
idea is to generate a non-parametric model by all-pass component. The basic procedure is explained
connecting several partial oscillators in series and in detail in [18]. For the subsequent inversion, only the
multiply them with the proportion for the total mass product of the minimum phase component and the
moment of inertia. Therefore, in the first part of magnitude of the all-pass component is used. Although
module two, Jtot is determined by calculating the this results in a phase shift of the inverted transfer
gradient at lower frequencies. At the same time, the function (cf. Fig. 5), however, deviations in the phase
resonance and antiresonance frequencies ωr,i and ωf,i are usually tolerable for numerous applications. On the
are determined based the phase characteristic. It is other hand, the course of the magnitude response is not
necessary to distinguish between two cases affecting affected.
the value of parameter a of Eq. (2). If a previously Furthermore, another obstacle arises when
defined phase threshold value (e.g. 50 °) is exceeded in inverting the model equations. Due to the fact that the
the negative or positive direction and there is a order of the denominator n is always larger than the
subsequent reversal of the phase to the original value, order of the numerator m for all model equations, a
the frequencies are close to the crossed threshold system with differentiating character is obtained by

15
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

performing a direct inversion. However, such a system 850V of the company DMG Mori. The tests were
cannot be implemented in reality for reasons of carried out on the horizontally arranged x-axis of the
causality [19]. A possible solution was already machine. Its structural design is illustrated in Fig. 3.
presented in [10] and applied for a second-order The rotational motion of the servomotor is converted
system. The idea is to extend the inverted transfer into a linear movement of the carriage via a belt gear
function with a defined number (n-m) of high- and a ball screw drive. Position sensors are located on
frequency poles until the denominator order the servo motor and on the carriage, whose time
corresponds at least to the numerator order. For a derivation serves as the input signals for the
continuous system, this is calculated as follows: identification procedure. By taking into account the
spindle pitch hsp and gear ratio ig the load speed is
-1
1 1 converted onto the motor shaft. The connection of the
Gp s = ∙ n-m (5) PRBS excitation signal to the motor torque Tm is
Gp s 1+TAT ∙s
implemented by a drive-internal function generator.
Note that the multiplication with high-frequency The results at the output of module two (model
poles also leads to a lowering of the phase response of identification) are illustrated in Fig. 4. All identified
the inverted transfer functions. With respect to the and optimized parameters are listed in Table 1. For all
magnitude, however, an adequate inversion of the three modell equations, a total number of three partial
model equation is obtained. oscillators was detected. Note that all values for
resonance and antiresonance frequency were
converted from ( rad∙s-1 ) to (Hz). Regarding the
frequency responses, the identified continuous models
3. Experimental Results (orange) match the measured signals (blue) over a
broad range of frequencies. Only the cycle time of the
The presented approach is validated on a real controller (Tcyc = 8 ms) limits the bandwidth of the
machine axis of a three-axis machining center DMC recorded signals up to 125 Hz.

Fig. 3. Structure of the linear feed axis.

10 1 10 1 10 1

10 1 10 1 10 1

measurement identified model optimized model deviation between measurement and optimized model

Fig. 4. Frequency response of measured signals and identified and optimized models.

16
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Table 1. Optimized parameters of the identified model equations

System Total moment of Partial oscillator Partial oscillator Partial oscillator

function inertia No. 1 No. 2 No. 3
parameter value parameter value parameter value parameter value
ωf in Hz 23.51 ωf in Hz 47.04 ωf in Hz 87.60
Gp,m(s) df 0.18 dr 0.11 dr 0.10
kgm2
Jtot in 0.0058
s2 ωf in Hz 28.94 ωf in Hz 61.13 ωf in Hz 99.82
dr 0.14 dr 0.07 dr 0.07
ωf in Hz 25.18 ωf in Hz 59.65 ωf in Hz 99.33

kgm2
dr 0.23 dr 0.09 dr 0.03
Gp,l(s) Jtot in 0.0057
s2 ωf in Hz 28.82 ωf in Hz 58.95 ωf in Hz 99.33
dr 0.19 dr 0.06 dr 0.07
ωf in Hz 26.18 ωf in Hz 63.04 ωf in Hz 85.83

kgm2
dr 0.17 dr 0.09 dr 0.26
Gmech(s) Jtot in -
s2 ωf in Hz 23.69 ωf in Hz 47.09 ωf in Hz 85.83
dr 0.16 dr 0.10 dr 0.09

In the next step, all identified and optimized models one all-pass. Due to the phase-minimum character, this
are transferred to their discrete-time representation and procedure is not required for the indirect case as well
the subsequent inversion takes place. The results can as the mechanical part. The magnitude response shows
be seen in Fig. 5. The frequency responses of all three that the all-pass component (purple) has basically no
discrete models (blue), their discrete inversion influence on the minimum-phase component (yellow).
(orange) as well as the compensated value (green) are Only in the phase signal an additional drop is
plotted individually. Initially, a phase drop for all recognizable. Nevertheless, a phase reduction can be
models due to the discretization procedure is observed for all transfer functions. This corresponds to
recognizable. By reducing the sampling time TA, the other methods, for example the closed-loop approach
bandwidth of the models may be further enhanced. For in [9]. Apart from that, it can be stated that especially
the direct case in the center part of the figure, the in the magnitude response, an adequate discrete
algorithm has detected one unstable zero and inversion is performed for all three model equations.
consequently performs an all-pass decomposition with

10 1 101 10 1

10 1 10 1 10 1

discrete model discrete, inverted model allpass part phase-minimal model compensated signal

Fig. 5. Frequency response for discrete modeled and inverted transfer functions.

17
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

4. Summary and Conclusion screw-Driven Stage, International Journal of

Advanced Manufacturing Technology, Vol. 87, 2016,
pp. 3337-3348.
The paper presents a methodology for discrete
[7]. Yamada Y., Kakinuma Y., Mode decoupled Cutting
modeling and inversion of transfer functions for multi- Force Monitoring by applying Multi Encoder based
mass systems. The main advantage of the approach is Disturbance Observer, in Procedia CIRP, Vol. 57,
the automatic determination of the system order as well 2016, pp. 643–648.
as all necessary parameters. The modeled discrete [8]. Schöberlein C., Norberger M., F., Schlegel H.,
transfer functions serve as origin for a new observer Putz M., Simulation and Disturbance Estimation of
structure that is not limited to lower-order systems (e.g. Speed-controlled Mechatronic Drive Systems, in
two-mass systems). Future studies should aim to MATEC Web of Conferences, Vol. 306, 2020, 04001.
validate the complete observer structure under [9]. Aslan D., Altintas Y., Prediction of Cutting Forces in
Five-axis Milling using Feed Drive Current
machining conditions. In addition, it should be
Measurements, IEEE/ASME Transactions on
analyzed to what extent the axis position in the Mechatronics,Vol. 23, Issue 2, 2018, pp. 833–844.
working frame influences the values of individual [10]. Schoeberlein C., Schleinitz A., Schlegel H.,
frequencies in the frequency response. Putz M., Simulative Investigation of Transfer
Function-based Disturbance Observer for Disturbance
Estimation on Electromechanical Axes, in
Acknowledgements Proceedings of the 17th Int. Conf. on Informatics in
Control, Automation and Robotics, 2020, pp. 651–658.
[11]. Schoeberlein C., Sewohl A., Schlegel H.,
Dix M., Modeling and Identification of Friction and
Weight Forces on Linear Feed Axes as Part of a
Disturbance Observer, International Journal of
Funded by the Federal German Ministry for Mechanical Engineering and Robotics Research,
Economic Affairs and Energy 2021, in press.
[12]. Brecher C., Rudolph T., In process identification of
cutting condition using digital drive signals, in
Proceedings of the CIRP 2nd International Conference
References Process Machine Interactions, 2010.
[13]. Münster R., Walther M., Schlegel H., Drossel W.,
[1]. Ohnishi K., Matsui N., Hori Y., Estimation,
Experimental and Simulation-based Investigation of a
Identification, and Sensorless Control in Motion
Velocity Controller Extension on a Ball Screw System,
Control Systems, in Proceedings of the IEEE, Vol. 82,
in Proceedings of the 14th Mechatronics Forum
Issue 8, 1994, pp. 1253–1265.
International Conference MECHATRONICS 2014,
[2]. Kempf C. J., Kobayashi S., Disturbance observer and
2014, pp. 226–234.
feedforward design for a high-speed direct-drive
[14]. Hipp K., Schöberlein C., Schlegel H., Neugebauer R.,
positioning table, IEEE Transactions on Control
Simulation based Optimization for Controller
Systems Technology, Vol. 7, Issue 5, 1999,
Parametrization of Machine Tool Axes – advanced
pp. 513–526.
Application, Journal of Machine Engineering, Vol. 17,
[3]. Yamato S., Sugiyama A., Suzuki N., Irino N.,
Issue 1, 2017, pp. 57–68.
Imabeppu Y., Kakinuma Y., Enhancement of Cutting
[15]. Nelder J., Mead R., A Simplex Method for Function
Force Observer by Identification of Position and
Minimization, The Computer Journal Advance, Vol. 7,
Force-amplitude dependent Model Parameters,
1965, pp. 308–313.
international Journal of Advanced Manufacturing
[16]. Graf U., Applied Laplace Transforms and Z-
Technologies, Vol. 104, 2019, pp. 3589–3605.
Transforms for Scientists and Engineers: A
[4]. Yamato S., Imabeppu Y., Irino N., Suzuki
Computational Approach Using a Mathematica
N., Kakinuma Y., Enhancement of Sensor-less Cutting
Package, Switzerland, Birkhäuser Basel, 2012.
Force Estimation by Tuning Observer Parameters from
[17]. Park S., Altintas Y., Dynamic Compensation of
Cutting Test, in Procedia Manufacturing, Vol. 41,
Spindle Integrated Force Sensors with Kalman Filter,
2019, pp. 272–279.
ASME Journal of Dynamic Systems Measurement and
[5]. Sato T., Yamato S., Imabeppu Y., Irino
Control, Vol. 126, 2004, pp. 443–452.
N., Kakinuma, Y., Precise Cutting Force Estimation by
[18]. Schilling R. J., Harris S. L., Digital Signal Processing
Hybrid Estimation of DC/AC Components, in
Using MATLAB, USA, Cengage Learning, 2016.
Proceedings of the 2020 International Symposium on
[19]. Schröder D., Elektrische Antriebe – Regelung von
Flexible Automation, July 8–9, 2020, V001T02A005.
Antriebssystemen, Springer Vieweg. Heidelberg, 4th
[6]. Yamada Y., Kakinuma Y., Sensorless Cutting
edition, 2015.
Force Estimation for Full-closed Controlled Ball-

18
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(006)

Development and Validation of a Model for Online Estimation of Process

Parameters for Adaptive Force Control Algorithms
M. Norberger 1, A. Sewohl 1, S. Sigg 2, H. Schlegel 1 and M. Dix 1
1
Institute for Machine Tools and Production Processes, Chemnitz University of Technology,
09126 Chemnitz, Germany
2
Fraunhofer Institute for Production Systems and Design Technology, 10587 Berlin, Germany
Tel.: + 4937153136970, fax: + 49371531836970
E-mail: [email protected]

Summary: Production technology is characterized by the use of electromechanical feed axes, for which the concept of cascade
control has become established. The concept is based on linear control engineering. It is not suitable for the control of process
forces, which is associated with nonlinearities. Here, adaptive control algorithms from the field of higher control engineering
represent a promising approach. The basis for adaptation is the estimation of process parameters. In this paper, the development
of a model for online parameter estimation is presented. Key components are data pre-processing, parameter estimation based
on a recursive least squares (RLS)-algorithm, and data post-processing. The functionality of the parameter estimation is
demonstrated in simulation and validated by means of experiments on a test setup with modern industrial motion control. In
addition, various influencing factors are examined and their effects are evaluated.

Keywords: Force control, Adaptive control, Parameter estimation, Motion control.

1. Introduction challenges. The most accurate possible a priori

information of the system behavior are used for the
In the area of production engineering, there are design and parameterization of the control of technical
ongoing efforts to improve manufacturing strategies systems [4], [5]. For this purpose, e.g. construction
and processes in terms of stability, quality and data but also non-invasive identification methods can
efficiency. One possibility for ensuring stable process be used [6]. In practice, some information may not be
conditions and reducing rejected parts is closed loop available a priori. Basically, this can be caused on the
control of quality determining parameters [1]. one hand by insufficient knowledge of process
However, their control is not trivial and requires parameters in the case of a time-invariant process
precise knowledge as well as correspondingly real- behaviour and on the other hand by a time-variant
time-capable sensor technology. At this point, the process behavior [4]. This can be remedied by the
process force is a suitable and very relevant variable. It estimation of variable process parameters, which
is often the limiting factor for the design of the forms the basis for adaptation. The performance of
processes and the choice of parameters. As a controlled adaptive control depends significantly on the accuracy
variable, the process force is predestined to ensure and real-time capability of the estimation model.
stability and safety of many processes [1, 2]. It This publication focuses on the development of a
contains important information regarding the process model for online parameter estimation. Key
state and allows conclusions to be drawn about components are data pre-processing, parameter
deviations in the manufacturing process, the machine, estimation based on a RLS-algorithm, and data post-
the tool, the workpiece or the material. However, there processing. The estimation is performed in real time
are many challenges and requirements associated with based on measured actual values and data from the
force control. The process itself is part of the controlled control system. The functionality of the parameter
system, so that deviations of the controlled system and estimation is demonstrated in simulation and validated
nonlinearities occur more frequently. With classic PID by means of experiments on a test setup with modern
controllers, this results in poor performance or even industrial motion control.
instability. PID-control forms the basis of the In the next chapter, the basic concept of parameter
established cascade control for electromechanical feed estimation is described. Different possibilities are
axes, which is also known as servo control [3]. Control discussed and selected on the basis of the use case. The
of process forces in production machines with chosen experimental test-setup for the validation is
electromechanical feed axes is still a developing field presented in the third chapter. Subsequently, the
and offers space for potential improvement. The use of structure of the parameter estimation unit as well as the
higher control concepts is recommended for conducted experiments are explained and evaluated in
controlling non-linear systems. In particular, adaptive chapter four. Finally, the results and conclusions are
algorithms that can react to deviations of the controlled summarized in the last chapter.
system represent a promising approach to meet the

19
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

2. Parameter Estimation of the covariance leads to a fast estimation of the

stiffness when the parameter estimation unit is
To achieve a high control performance, the online switched on. The forgetting factor 𝜆 achieves a
estimation of process parameters must be convergent. successive reduction of the weighting of old
Otherwise, poor control behavior up to instability is measurement data. The memory of the RLS estimator
possible [7]. Online parameter estimation, both for can be approximated as a function of 𝜆 to:
feed axes and in general, is a very wide field. For
general information and an overview of different 2
𝑁 (7)
algorithms see for example [4, 7, 8] and for online 1 𝜆
parameter estimation on feed axes see for example [6],
[9, 10]. Basically, there are recursive and non- where N is the number of measured values in the
recursive parameter estimation methods. With regard estimator [8]. For 𝜆 0.96 the estimation is therefore
to the memory requirements and the computing time as based approximately on the last 50 measured values,
well as from the point of view of the cyclic availability for 𝜆 0.5 only the last 4 measured values are taken
of new measurement data in the process image of a into account. Small values for 𝜆 thus lead to a fast
programmable logic control (PLC), recursive methods adaptation of the parameter estimation to changing
should be preferred [6, 8]. The basic method for parameters, but bear the risk of a fast estimator blow-
recursive parameter estimation is RLS [6], which is a up and can thus lead to a strongly noisy parameter
recursive computational prescription of the well- estimation [8]. The RLS-algorithm can be influenced
known least squares method. If the process parameters by means of the forgetting factor. Typical values for 𝜆
are time-invariant, a distinction can be made between are in the range of 0.95 … 0.99 [4], [8]. When using a
the cases of an immediate parameter change and a variable forgetting factor, the control is done via the
continuous parameter change [8]. In the case of an adjustment factor 𝛾.
immediate change, the variables of the algorithm can
be reset. In the case of a continuous change, it is useful
to implement a forgetting factor 𝜆 [11]. 3. Test Setup
The task of the parameter estimation model for an
adaptive force control is to estimate the effective For the experiments and validation, a test-setup of
stiffness 𝐾 online as quickly as possible and with as an electromechanical feed axis was selected, which is
minimal noise as possible. 𝐾 is defined as the change designed for loads up to 10𝑘𝑁. The basic structure of
of the process force per change of the actual position. the test-setup corresponds to a portal construction.
Due to the significant noise of the force measurement However, only one drive is used to generate the
and the fact that both 𝛥𝐹 and 𝛥𝑥 can be zero, it is not movement. Position control is implemented in
useful to calculate 𝐾 as a numerical derivative. cascaded structure. It is also possible to switch to a
Analogous to [13], the use of an RLS algorithm proves force control on the same level. The mechanical
to be more appropriate. With the approach: construction and control engineering structure are
described in more detail in [12]. When performing load
𝛥𝐹 𝑡 𝐾 𝑡 ∗ 𝛥𝑥 𝑡 𝜀𝑡 (1) tests it has been shown that the force controller can
become unstable. These instabilities result from
the calculation rule for the recursive least squares deviations and non-linearities in stiffness, which is
method results in: essential for the design of the force control. The
parameter estimation method described in this paper is
able to detect unexpected changes in stiffness. A
𝜀𝑡 𝛥𝐹 𝑡
𝛥𝑥 𝑡 ∗ 𝐾 𝑡 1 (2)
modular and exchangeable spring package was
𝑃 𝑡 1 ∗ 𝛥𝑥 𝑡 designed for the experiments as shown in Fig. 1. This
𝐾𝑡 (3)
𝜆𝑡 1 𝛥𝑥 𝑡 ∗ 𝑃 𝑡 1 allows systematic replication and investigation of
𝜆𝑡 1 𝛾∗𝜀 𝑡 1 𝐾 𝑡 ∗ 𝛥𝑥 (4) changes and deviations in the system stiffness.
1 𝐾 𝑡 ∗ 𝛥𝑥 𝑡 ∗ 𝑃 𝑡 1
𝑃𝑡 (5)
𝜆𝑡
𝐾 𝑡 𝐾 𝑡 1 𝐾 𝑡 ∗𝜀 𝑡 (6)

where 𝜀 is the prediction error, 𝐾 is the gain, 𝑃 is the

covariance and 𝐾 is the estimation of the effective
stiffness. The parameter estimation is further affected
by the initialisation through the start value
𝐾, 𝐾 0 and the start covariance 𝑃 𝑃0.
However, the influence of these two initialisation
values gets lost quickly, so that they are not considered
in more detail in the further analysis. 𝐾 , is set to
Fig. 1. Modular spring package for investigating changes in
300 and 𝑃 to 1000. A high initialisation value stiffness in experiments with force control.

20
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Variable load characteristics can be initiated with high points 𝑥 , 𝐹 and new data are provided cyclically.
reproducibility by a movement of the axis against the Averaging is performed as long as the following abort
spring package. The use of spacer elements can conditions are not met:
provoke high steps in the stiffness. The stiffnesses of
the springs are known, which gives a reference value ∣𝑥 𝑥 ∣ 𝜀 𝑜𝑟 ∣ 𝐹 𝐹 ∣ 𝜀 (8)
for parameter estimation. A manufacturing tolerance
of 10 % per spring must be taken into account. When the position limit 𝜀 or the force limit 𝜀 is
exceeded, the process is terminated and the mean value
4. Implementation and Validation of the interval is transferred to the ring buffer of the
input data memory. In addition, force and position
4.1. Structure of the Parameter Estimation Model values in the following cycle 𝑡 1 , are defined as
the new reference data points and averaging starts
As shown in Fig. 2 the parameter estimation model again. This corresponds to a digital FIR low pass
consists of three components. The data pre-processing, filtering whose filter length is controlled by the abort
the actual parameter estimation with the help of the condition (8). Thereby it is ensured that a model
RLS-algorithm and data post-processing. Core excitation is present and no invalid data sets with
component is the parameter estimation, which is 𝛥𝑥 0 are used. The generation of the input data for
explained in chapter 2. The basic structure of this the RLS-algorithm with the cycle time 𝑡 is based on
algorithm is described in more detail in [4], [7] and [8].
the mean values 𝑥 𝑡 and 𝐹 𝑡 and it is shown
Once the algorithm has been chosen, it can be adapted
graphically in Fig. 3(b). Overlapping data points are
by using the forgetting factor, depending on the
specific implementation. The design of the data pre- used for a balance between high 𝛥𝐹 -excitation and fast
response to stiffness changes. This overlap is achieved
processing and the choice of suitable design
parameters is considerably more extensive than the by means of the ring buffer of length 𝐿 in the input data
RLS-algorithm. The purpose of data pre-processing is memory. It corresponds to the difference between the
to improve the signal-to-noise ratio and to provide the newest and oldest mean value, so that 𝐿 2 values are
RLS-algorithm with signals with a high excitation. skipped. In this way, suitable RLS input data 𝛥𝑥 𝑡
Therefore, suitable data sets of 𝛥𝑥 and 𝛥𝐹 for the and 𝛥𝐹 𝑡 can be generated if the measurement data
stiffness estimation must be generated. The discrete are strictly monotonically decreasing or increasing
data of the measured force 𝐹 𝑡 , the actual position over sufficiently long time periods. Typically, this
𝑥 𝑡 , the actual speed 𝑣 𝑡 and the speed setpoint applies to the position and the correlated force. In a
𝑣 𝑡 are made available to the parameter estimation final step, after parameter estimation, data post-
processing can take place by low-pass filtering.
unit as an input signal from the motion control in the
base cycle of the PLC 𝑡 1𝑚𝑠. A sign check of the
4.2. Design of the Parameter Estimation Model
feed rate is performed against the 𝐿 previous
measurement data in the input data memory, which has
First, the effects of individual influencing factors in
the cycle time 𝑡 . This allows the detection of standstill
the data pre-processing are investigated. In the
phases and measurement noise and serves for data pre- following, a suitable design parameter set for the
filtering. Subsequently, the data is low-pass filtered parameter estimation unit will be determined.
and stored in the input data memory. The sequence of Synthetic data are used for this purpose, since they can
filtering in the data preprocessing is shown in Fig. 3(a). be used to generate exact stiffness curves with known
With the initialization of the algorithm, the data points parameters.
𝑥 𝑥 𝑡 and 𝐹 𝐹 𝑡 are defined as reference

Fig. 2. Structure of the presented method for estimation of the stiffness.

21
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

a) b)

Fig. 3. Sequence of averaging in data pre-processing from measured data (a) and generation of the input data for the RLS-
algorithm with an input data memory length of 𝐿 3 (b).

The setpoint and actual force 𝐹 and 𝐹 , the 𝜎 0.513𝑁 . The position signal at standstill shows
actual position 𝑥 as well as the setpoint and actual noise with a variance of 𝜎 1.9585 ∗ 10 𝑚𝑚 .
velocity 𝑣 and 𝑣 are required in the estimation Furthermore, the actual velocity shows a noise with a
model. These signals have to be provided simulatively
variance of 𝜎 3.4985 ∗ 10 . To simulate
with a correct sampling frequency, a correct
quantization and a correct noise by the machine model. this noise using Matlab Simulink, a stochastic process
Furthermore, 𝐹 is calculated in the test setup from of mean-free, Gaussian-distributed pseudo-random
the moving average of the current and the last nine numbers with a sampling time of 1𝑚𝑠 is added to the
measured values. The measured values show noise and actual values. The test data for the investigation of
are quantized to 1 𝑚𝑁. This is also taken into account influencing factors consist of movements at a feed rate
in the model. To simulate the behavior of the parameter of 0.2 , 0.4 and 1 into a spring with a
estimation unit, a realistic modeling of this signal is stiffness of 100 . The stiffness is set to 500
particularly relevant. In order to determine the variance
after one second. Then the feed rate is kept constant for
of the force signal, the machine is loaded from 0𝑁 to
500 𝑚𝑠. The sampling frequency is 1 𝑘𝐻𝑧. The results
5𝑘𝑁 at a typical stiffness value 𝐾 500 . The are illustrated in Fig. 4. It can be seen that varying the
force set points of 1𝑘𝑁, 2𝑘𝑁, 3𝑘𝑁, 4𝑘𝑁 and 5𝑘𝑁 are filter length 𝐿 of the ring buffer has little effect and the
held for 2𝑠 each. From the steady state within these estimation behavior improves with higher values. A
holding periods, the variances were determined. The similar effect can be observed for the position limit 𝜀 .
mean variance of the quantized measurement signal is
a) b)
-80 -80
500 500
-100 -100
400 -120 400 -120
300 -140 300 -140

200 -160 200 -160

-180 -180
100 100
-200 -200
1000 1050 1100 1150 1200 1000 1050 1100 1150 1200
Time [ms] Time [ms]
c) d)
-80 -80
500 500
-100 -100
400 -120 400 -120
300 -140 300 -140

200 -160 200 -160

-180 -180
100 100
-200 -200
1000 1050 1100 1150 1200 1000 1050 1100 1150 1200
Time [ms] Time [ms]

Fig. 4. Influence of different settings of the parameter estimation unit on the behaviour of the estimation. a) Variation of the
length 𝐿 of the input data memory; b) Variation of the position limit 𝜀 ; c) Variation of the force limit 𝜀 ; d) Variation of the
average filter length 𝑚.

22
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

The parameter estimation model reacts faster and results with recorded measurement data from the
there is a better approximation to the nominal value for machine are illustrated in Fig. 5. A ramp-shaped force
smaller force limits 𝜀 . This can also be observed for profile with standstill phases is used. The influence of
the average filter length 𝑚 . The estimation of 𝐾 the adjustment factor 𝛾 is investigated here. Larger
agrees after 40-50ms in nearly all cases with the values lead to a faster estimation of the stiffness (see
setpoint jump of 𝐾 to 90%. Fig. 5(b)), but this also increases noise effects (see
Fig. 5(c)).

4.3 Validation Steps

Table 1. Determination of the design parameter set for the
The aim of the validation of the parameter RLS algorithm.
estimation unit is to analyse whether the stiffness 𝐾
estimated on the basis of the input data agrees with the 𝜀 𝜀 𝐿 𝑚
real stiffness of the test setup 𝐾 . The validation is 0.323 µ𝑚 1.294𝑁 18 5
carried out in three steps and thus three different
procedures: (I) in Matlab using synthetic data with a
precisely known stiffness; (II) in Matlab using data A good compromise is achieved with 𝛾 0.03. It
from the real test machine recorded in advance with the can also be seen that the algorithm detects standstill
conventional force controller; (III) online in the phases and provides reliable values for a constant
Beckhoff motion control system using real-time data stiffness, stiffness jumps and even when the direction
from the experiments. For cases (I) and (II) the of motion is reversed (see Fig. 5(a)).
parameter estimation unit is used on a development PC In step (III), the model was verified with real-time
in Matlab Simulink. For case (III) it is implemented as data on the test setup during ongoing operation with
a TcCOM object (Matlab Simulink model in real-time one constant stiffness. Here the spring package was
environment of TwinCat) on the machine controller. In loaded with a force of 3kN and then unloaded again.
step (I), individual influencing factors were first varied The force-position curve and the stiffness estimated
and examined. The tests and results are described in with the model from the actual values 𝐾 are shown in
chapter 4.2 and essentially served to constrain the Fig. 6. The actual stiffness 𝐾 is determined from 𝛥𝑥
possible settings of the parameter estimation unit. and 𝛥𝐹 in the range from the start position 𝑃 to the
The determined values thus provide an orientation. end position 𝑃 . The results are summarized in Table 2.
However, these parameters also influence each other The stiffness profile in the area of 4-6mm results
and are additionally dependent on the feed rate. For from deviations of the spring lengths and the non-
step (II), a suitable combination was selected in the linear range until all springs have contact. It can be
simulation with an empirical variation of the seen that the estimated stiffness 𝐾 settles to a constant
parameters. The final parameter set for data pre- value with slight fluctuations after the contact area and
processing is summarized in Table 1. The simulation the learning phase of the estimation model.

a)
600 0

-1000
400
-2000
200 -3000

-4000
0
-5000
4 5 6 7 8 9 10 11
Time [ms] 10
4
b) c)
200 5 510
0 -2420
150 500
-5
-2440
100 -10 490
-15
50 480 -2460
-20
0
-25 470 -2480
4.75 4.752 4.754 4.756 4.758 4.76 6.14 6.145 6.15 6.155 6.16
Time [ms] 4 Time [ms] 4
10 10

Fig. 5. Comparison of the parameter estimation 𝐾 for different adjustment parameters 𝛾 using real measurement data of the
machine (simulation with pre-recorded measurement data). a) Overview of the entire stiffness estimation; b) Parameter
estimation at the first stiffness jump 𝐾 ; c) Noise of the parameter estimation 𝐾 .

23
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

design of an adaptive force control has been created.

0
200 However, it is noted that the estimated stiffness 𝐾 is
delayed from the real stiffness between 30-50ms
150
-1000 depending on the design parameters of the data pre-
processing, the feed rate and the cycle time of the
100 control. The implications for adaptive control will be
-2000 considered in future research. Furthermore, an
50 optimization of the empirically set design parameters
is also intended in further work.
0
-3000
5 10 15 20 25
Position [mm]
References

Fig. 6. Parameter estimation on the test setup. [1]. J. M. Allwood, et al., Closed-loop control of product
properties in metal forming, CIRP Annals –
Manufacturing Technology, Vol. 65, 2016, pp. 573-
596.
A comparison between the estimated stiffness 𝐾 and
[2]. X. Yao, et al., Machining force control with intelligent
the actually calculated stiffness 𝐾 using the values in compensation, The International Journal of Advanced
Table 2 shows very good agreement. Open-loop tests Manufacturing Technology, Vol. 69, No. 5-8, 2013,
with several abrupt changes in stiffness were also pp. 1701-1715.
carried out on the test setup. Here, as in the simulation, [3]. W. Leonhard, Control of Electrical Drives - Power
the estimation model provided reliable values for Systems, 3rd edition, Heidelberg, Germany: Springer,
stiffness steps. Berlin Heidelberg, 2012.
[4] I. D. Landau, et al., Adaptive control - Algorithms,
analysis and applications, Communications and
Table 2. Data for verification of the parameter estimation control engineering, 2nd edition, London, Springer,
unit with real measured values on the test setup. 2011.
[5] K.-P. Schulze and K.-J. Rehberg, Entwurf von
adaptiven Systemen - Eine Darstellung für Ingenieure,
Data Plateau 𝑷 Vol. 1, Berlin, VEB Verlag Technik, 1988.
Start position, [6] R. Neugebauer, et al., Non-invasive parameter
𝑃 10, 𝑃 26.37, identification by using the least squares method,
End position, 𝑚𝑚
𝛥𝑥 𝛥𝑥 16.37 Archive of Mechanical Engineering, Vol. 58, 2011,
pp. 185-194.
Start force, End 𝐹 809, 𝐹 [7] R. Isermann, Identification of dynamic systems: An
force, 𝛥𝐹 𝑁 introduction with applications, Springer, Berlin, 2011.
3000, 𝛥𝐹 2191 [8] K. J. Åström and B. Wittenmark, Adaptive Control,
Stiffness 𝐾 2nd edition, Dover Books on Electrical Engineering.
determined from 𝑁 Newburyport, Dover Publications, 2013.
𝐾 133.84
𝛥𝑥 and 𝛥𝐹 𝑚𝑚 [9] S. Beineke, Online-Schätzung von mechanischen
Approximated Parametern, Kennlinien und Zustandsgrößen
geregelter elektrischer Antriebe, Dissertation,
estimation 𝐾 of 𝑁
𝐾 130 … 140 Universität Paderborn, Fortschritt-Berichte VDI Reihe
the parameter 𝑚𝑚 8, Meß-, Steuerungs- und Regelungstechnik,
estimation unit
Düsseldorf, VDI Verlag, 2000.
[10] R. Neugebauer, et al., Time-Based Method for the
Combined Identification of Velocity-Loop Parameters,
5. Summary and Conclusion Archive of Mechanical Engineering, Vol. 58, 2011,
pp. 175-184.
In this paper, a model for estimating the process [11] D. W. Clarke and Ch. J. Hinton, Adaptive control of
stiffness was presented. It is composed of the three materials-testing machines, Automatica, Vol. 33,
Issue 6, 1997, pp. 1119–1131.
components data pre-processing, parameter [12] A. Sewohl, et al., Performance analysis of the force
estimation, which is based on an RLS-algorithm, and control for an electromechanical feed axis with
data post-processing. The verification of the model industrial motion control, in Proceedings of the 17th
was done in three steps by simulations and International Conference on Informatics in Control,
experiments. The influence of different design Automation and Robotics 2020, Paris, France, Vol. 1,
parameters was also investigated. It has been found 2020, pp. 667-674.
that the estimated stiffness 𝐾 agrees sufficiently [13] D. W. Clarke, Adaptive control of servohydraulic
accurate with the actual stiffness 𝐾 for both real and materials-testing machines: a comparison between
black- and grey-box models, Annual Reviews in
synthetic data. In addition, the developed model Control 25, 2001, pp. 77–88.
provides reliable results even in the face of various
challenges, such as stiffness steps, reversal of the
direction of motion and standstill phases. With the
methodology described in this publication and the
developed parameter estimation unit, the basis for the

24
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(007)

Simulation of Automated Handling in Textile Manufacturing of US

Military Apparel to Improve Efficiency and Quality
Z. B. Rosenberg, J. A. Joines and J. S. Jur
Department of Textile Engineering Chemistry and Science, North Carolina State University, Raleigh, NC, USA
E-mail: [email protected]

Summary: In this paper, we simulate implementing automated handling for transferring fabric part pieces throughout the
construction process of US military-relevant textile products. The current infrastructure of the textile industry would benefit
from integrating new automated technology in the textile production line to replace manual operations. Using Simio software,
we simulated the current production process for the cummerbund in the US Army’s Modular Scalable Vest (MSV). The current
production is done through batch production with manual transferring of the batches of parts to the next workstation and the
loading/ unloading of part pieces into sewing workstations. Using SIMIO simulation software, we can model the implications
on time and labor of replacing the manual operations with automated robotic handling to enable continuous flow
manufacturing. Incorporating automation in the manufacturing of the cummerbund increases efficiency, throughput, and
quality of production.

Keywords: Automated handling, Materials handling, Textile manufacturing, Digital simulation, Industry 4.0.

1. Introduction piece of equipment going down, allowing other

workstations to continue, and preventing the entire
There is growing demand to deliver textile production line from stopping.
products, such as apparel, to the customer faster. Some automated equipment has been inserted into
Automation can enable manufacturing on-demand and production floors; however, such equipment still
shorter lead times by using robotics to construct depends on operators (human-assisted automation),
products. Using robotics replaces labor, allowing for making it challenging to implement floor plans with
continuous production and increased throughput. continuous flow production and on-demand
Additionally, automated equipment provides more manufacturing [4]. The automated textile equipment
consistent construction than manual assembly, being employed most broadly consists of sewing
generating improved uniformity. machines that automatically move fabric pieces
The assembly of textile products remains largely through the sewing operation. However, the output of
dependent on manual operations [1]. Minimal the automated sewing machines is limited by the time
automated processes have been inserted into the needed manually to load and unload the machine
garment construction process due to the challenge of quickly and accurately. The operator can incorrectly
developing proper textile handling equipment as well load the sewing machine where the textile parts are not
as the economic feasibility of such an investment. The accurately placed, causing a wrinkle or misalignment
textile industry has remained economically viable in the component and leading to a defective product.
without automation because skilled, inexpensive labor Because of the complexities in handling fabric, this
is widely available in areas such as Central America equipment is primarily available for sewn flat pieces or
and Southeast Asia, making it difficult to justify sewing simple, repeated folds such as the pleat in the
manufacturing in high labor cost countries [2]. back of a button-down shirt. Using automated handling
Implementing automated processes provides the for the loading process could improve the construction
prospect to manufacture textile products in high labor quality of the products and increase the throughput of
cost countries and to increase quality and consistency the manufacturing process.
in the final product. The assembly process of a cummerbund, which is
Typical production facilities for textile products part of the US Army’s Modular Scalable Vest (MSV)
have multiple sewing workstations that perform a already utilizes five types of human-assisted
single assembly step for the product. Production is automated equipment. Additionally, the MSV fabric
done in a batch process, where a “batch” of products is part pieces can be handled with a robotic arm because
completed at each step before moving to the next of its high stiffness and dense woven structure. We
production step [3]. Batch processing is more effective have modeled the production process using Simio to
with the current infrastructure since the operator does simulate the implications of including additional
not need to move individual pieces between automated procedures in the assembly. Because of the
construction steps and instead can move a grouping, integration of automated handling, continuous flow
making production more time-efficient. Batch manufacturing becomes a viable construction method
production also allows for flexibility in the case of a to improve production efficiency.

25
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

2. Methods cummerbund assembly. In this paper, we explored four

alternate production process variations of the factory
In our research, we used the discrete event floor with various levels of automation. The models
simulation software SIMIO to simulate the included implementing automated handling with batch
implications of adding automated handling and production, automated handling with continuous
procedures in the assembly of a cummerbund. The manufacturing, automated handling with continuous
manufacturer has provided data about their current manufacturing and redistributed labor, and all
production that is used to drive and validate the automated processing.
simulation. Various production strategies of inputting
a robotic arm to replace an operator were modeled to 2.3. Timing of Automated Handling
determine the most effective implementation of robotic
handling. The theoretical production time was calculated for the
automated processes with implemented automated
2.1. Manufacturing Process of MSV Cummerbund handling. Employing robotic handling decreases the
production time required for each automated step by
The current assembly of the MSV cummerbund removing the time it takes for the operator to grasp the
consists of 12 construction steps. These steps all occur part pieces, position them accurately, and then move
at different workstations manned by operators of the equipment into place and then place the completed
various skill levels depending on the operation. The part in the pile for the next step. The theoretical
assembly is done through batch production in batches production time was determined through video
of 20 pieces, meaning 20 pieces are assembled before analysis of the current construction process. The
the batch can move to the next workstation. The theoretical production time was defined as the time for
manufacturer provided the average times to complete the machine to start and finish the sewing process,
each step and the time required to transfer the batch to removing the time of handling, grasping, placing, and
the next workstation. These assembly steps and times transferring the part pieces. These times are shown in
can be found in Table 1. The current assembly process Table 2. The time for handling, placing, and
incorporates five pieces of human-assisted automated transferring the part piece was specified in the SIMIO
sewing equipment, which still requires an operator to model as taking five to ten seconds for each automated
load, position, and unload the part pieces into the assembly operation. This time accounts for both the
machine and transfer the completed batch to the next timing of the robotic arm operations as well as
assembly station. This equipment reduces the encompassing the extra time in case a robotic arm
necessary skill level of the operator. The human- encounters issues or were to malfunction or need
assisted automated operations are indicated with repair.
an (A).
Table 2. Production time of automated assembly steps of
Table 1. Assembly steps of cummerbund with current cummerbund when using robotic handling. Automated
production and transfer times. Automated processes processes indicated with an (A).
indicated with an (A).
Cummerbund Assembly Production Time
Cummerbund Step (seconds)
Production Transfer Time of C (A) 28
Assembly
Time (seconds) Bundle (seconds) D (A) 27
Step
A 63.5 15 E (A) 10
B 14.6 45 F (A) 6
C (A) 50.9 20 H (A) 7
D (A) 66.0 11
E (A) 33.0 14
F (A) 60.0 15
The transfer time for the automated processing with
G 86.8 15 automated handling is dependent on how the
H (A) 56.7 10 cummerbund is manufactured, whether it is batch or
I 58.7 8 continuous. In batch production, the time to transfer
J 36.0 8 remains the same because the completed parts are
K 98.6 8 stacked into piles and still need to be moved as a batch
Total Time 10 mins 24.8 secs 2 mins 49 secs to the next workstation, despite the automated loading
and unloading of the fabric. When the production
changes to continuous manufacturing, the transfer time
2.2. Simio Model of Cummerbund Assembly becomes negligible because the sequential production
steps would need to be positioned closely together for
The factory floor was modeled in Simio to simulate the robotic arm to reach between the steps. The robotic
the current production process. We ran twenty arm transfers the part from the completed assembly
replications of the model and averaged the results to workstation into the next assembly operation. The
determine the daily production and timing of the model accounts for the transfer time as the five to ten

26
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

seconds added to the previously mentioned robotic components to enter the operation and those exact
assembly operations to represent the loading and components to be in a completed cummerbund. The
unloading of parts. long time in the system is attributed to the bottlenecks
within the assembly. The modeled output aligns with
the manufacturer’s current production capabilities that
3. Results and Discussion were provided. Even optimizing the system by adding
an additional operator and station at two bottleneck
3.1. Simio Model of Current Production operations, the number produced per day can only
of Cummerbund Assembly increase 419.8 units/day with the average time in the
system 150.8 minutes.
The current factory layout and results from the
simulation are shown in Fig. 1 and Table 3,
respectively. In this simulation model, each assembly 3.2. Scenario 1: Automated Handling with Batch
step was given a 10 % variance in the production time Production
using a Beta Pert distribution. The variance is added to
account for the variability of a human operator having Robotic arms were added to the model to simulate
some faster and some slower production times. The automated loading and unloading of the fabric pieces
variance of a human operator can range depending on into the automated equipment shown in Fig. 2.
the operator’s experience, the difficulty of the
operation, and fatigue.

D E A B/C
F

G
Fig. 2. Simio model of current factory facility with an
integrated robotic arm for handling fabric.
H
JJ
K The implementation of robotic arms reduced the
I average production times to the times provided in
Table 2. The variance was reduced to 5 % for the
automated processing steps to reflect the consistency
Fig. 1. Current production layout. and reliability that automated handling offers. The
production remained in batch production, where 20
units had to be completed before transferring to the
Table 3. Production output of current factory layout
in simulation. next workstation. The automated handling enabled an
increased workday of sixteen-hours to account for the
Current Production Output reduced labor that could be redistributed to two shifts.
Number of Operators 11 This is assuming the manufacturer is able to employ
Number of Robots 0 one additional operator, or reallocate an operator from
Hours in Work Day 8 a different production line. The results of automated
Average Time in System 660.43 minutes handling within the workstation are shown in Table 4.
Number Produced per Hour 39.72 units/hour
Average Numbers 317.79 units/day
Table 4. Production output of current production with
The simulation of the current production accounted inserted robotic handling to load/unload automated
equipment.
for an eight-hour workday. For this production line, the
manufacturer employs eleven operators, or one shifts Automated Batch Production Output
worth, due to the labor shortage of seamstresses in the Number of Operators 6
United States. Currently, the manufacturer is not able Number of Robots 5
to hire two shifts worth of sewing operators due to the Hours in Work Day 16
scarcity. The current production line produces 317.79 Average Time in System 628.38 minutes
cummerbunds a day, or 39.72 units per hour. On Number Produced per Hour 39.17 units/hour
average, the cummerbund is in the system for 660.43 Average Numbers Produced 626.67 units/day
minutes. This is the time it takes for the fabric

27
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

manufacturing, there is less leeway for inconsistent

The daily production of cummerbunds increased to timing because each construction step depends on the
626.67 units, nearly doubling the current production, previous step to produce in time. Continuous or lean
and a minor improvement in the time in the system to manufacturing requires a reliable supply of materials
628.38 minutes, signifying a slight reduction in the to keep the assembly process running.
bottlenecks of the system. However, the cummerbunds Switching to automated material handling offers
produced per hour marginally decreased to 39.17 units consistent production to enable continuous
per hour. This is a result of the fabric needing to be manufacturing. The cummerbund assembly has five
handled twice, both by the robotic arm for the automated processes that all occur during the early
automated assembly operation as well as by the stages of the construction process. The results of the
operator to transfer the batch. Even though the timing continuous automated assembly are shown in Table 5.
of the assembly steps is reduced with automated
handling, having to transfer each part piece twice
minimizes the effectiveness of robotic handling. Table 5. Production output of continuous production with
Nonetheless, the daily production nearly doubled inserted robotic handling and automated loading/ unloading
because robotic handling enabled a sixteen-hour at automated workstations.
workday, improving daily output.
Automated Continuous
Production Output
Number of Operators 6
3.3. Scenario 2: Automated Handling Number of Robots 5
with Continuous Manufacturing Hours in Work Day 16
Average Time in System 1282.55 minutes
The model with implemented robotic handling was Number Produced per Hour 34.97 units/hour
then modified for continuous manufacturing. In the Average Numbers Produced 559.54 units/day
continuous manufacturing model, each unit moves
individually through the assembly process. Continuous
manufacturing provides a more suitable method of With robotic handling and continuous
production with robotic handling. The transferring and manufacturing, the daily production produces
placing of the part pieces into the assembly operations 559.54 units per day and 34.97 units per hour. This is
are combined, and the part pieces can move an improvement from the current production but a
continuously throughout the production. For reduction from the automated handling in batch
continuous flow, the machines have to be located production. This diminished value of introducing
directly next to one another so the robotic arm can automated handling is represented in the increase of
reach both workstations for transferring the fabric part. the average time in the system, which increases to
The continuous manufacturing simulation layout is 1282.55 minutes owing to the bottlenecks in the
shown in Fig. 3. system. In this production process, the automated
processes all occur during the early stages of
production. The later steps remain manual and have
longer manufacturing times. Because each operation is
dependent on receiving the completed material from
the prior operation, it must wait for that operation to
finish. Since the assembly operations all take a
different amount of time to complete, the shorter
operations are held up by the longer operations. The
bottlenecks in continuous manufacturing have a higher
impact than batch production because the operations
are more dependent on one another for efficient
production.
Fig. 3. Simio model of the facility with integrated robotic
arms at the automated workstations for handling and 3.4. Scenario 3: Automated Handling and
transferring fabric.
Redistributed Labor with Continuous
Manufacturing
Automated handling provides more consistency
and reliability than manual operations. This facilitates The output for continuous manufacturing is lower
controlled timing of the production steps to enable a than the output for batch production, which is
continuous process flow. Batch production is more attributed to the bottlenecks in the system. However,
suitable for manual operations because it allows continuous manufacturing reduces the overall handling
flexibility in the timing of the production. If one time by combining the assembly operation handling
workstation is delayed, whether the equipment goes with the transfer handling. We redistributed three
down or the operator needs a break, the other operators from the original operations and placed them
workstations can continue to function. In continuous at the bottleneck process or longer operation steps to

28
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

improve continuous production. We modeled having equipment that can operate continuously without
two operators at operations G, I, and J. However, with breaks. There is one operator at K, which can be filled
employing more workers on the production assembly, by three shifts to account for the 24-hour workday. The
the assembly operation returned to one shift, or an results of continuous automated production are shown
eight-hour workday. This is assumed due to the in Table 8.
likelihood of not being able to hire or reallocate seven
additional operators. The results are shown in Table 6.
Table 7. Production time of automated assembly steps of
cummerbund when using robotic handling and automated
Table 6. Production output of continuous production with equipment.
inserted robotic handling at automated workstation and an
additional operator at operations G, I, and J. Cummerbund Production Time
Assembly Step (seconds)
Automated Continuous Production A (A) 5
with Redistributed Labor Output B (A) 5
Number of Operators 9 C (A) 28
Number of Robots 5 D (A) 27
Hours in Work Day 8 E (A) 10
Average Time in System 971.83 minutes F (A) 6
Number Produced per Hour 47.28 G (A) 15
Average Numbers Produced 378.28 H (A) 7
I (A) 20
K 33
Redistributing the labor showed improvements in Total Time 221.6 seconds
production output and a reduction in bottlenecks. The
cummerbunds produced per hour increased to
47.28 units and the average time in systems decreased Table 8. Production output of continuous production with
to 971.83 minutes. However, due to the eight-hour robotic handling and automated assembly at operations A-I.
workday, the daily output fell to 378.23 units a day, an
All Automated Processing Output
improvement from the current output, but a regression
Number of Operators 1
from both automated assemblies with sixteen-hour Number of Robots 9
workdays. Hours in Work Day 24
Average Time in System 4.23 minutes
Number Produced per Hour 97.2 units/hour
3.5. Scenario 4: All Automated Processing Average Numbers Produced 2332.8 units/day

The overall goal is to automate the textile assembly

process, which enables more consistency in Daily production increases to 2,332.8 units with the
production, reduced labor, faster production times, and implementation of all automated equipment. The
the ability to manufacture on demand because the bottlenecks are alleviated, representative by the time in
equipment can run 24/7 and requires less lead time to system decreasing to 4.23 minutes, similar to the sum
hire and train production workers. of the assembly operations, signifying efficient
Automating the full assembly would require the production. The units produced per hour more than
development of automated equipment for the doubles to 97.2 units. Additionally, by implementing
additional operations. We determined the timing of the automated handling, we can infer the quality of the
theoretical automated equipment that would need to be product will improve from the consistency of robots
produced based on the video analysis process done in compared to a human operator. However, actualizing
section 2.3. The updated assembly times are shown in an autonomous assembly of a textile product would
Table 7. require significant endeavors and investments in
Operation J is removed when the entire process is automated equipment.
automated because it is no longer necessary with
automated equipment. Operation K is not able to be 3.6. Comparison of Automated Insertion Scenarios
automated since it is the final inspection, however, we
assume that the time will be reduced by approximately Robotic arms were inserted to fully automate the
a third, due to the consistency and accuracy of current human-assisted automated assembly steps in
automated equipment. the production of an MSV cummerbund. A
The all automated process was modeled with the summarization of the output for the different types of
times shown in Table 7 and 10 seconds for the robotic scenarios discussed in this paper is shown in Table 9.
handling to place, handle, and transfer the parts at all Introducing robotic handling in the assembly of the
the assembly operations. Each production step was MSV cummerbund reduces the number of operators
given a 5 % variability, and the workday was modified required for the assembly process. However, robotic
to 24 hours of production since the assembly handling does not necessarily increase the speed of
operations are conducted by primarily robotic production, but instead allows labor to be redistributed,

29
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

enabling two shifts of operators, doubling the remaining four assembly steps, autonomous
production hours. Batch production serves as a more continuous flow production is possible in producing
suitable production process when there are longer 97.2 units per hour and 2332.8 units per day due to the
human operations still in the assembly process. If capability to run twenty-four hours. Automated
available, labor should be redistributed to the longer handling of textiles is essential to automating the
operations to decrease the bottleneck effects. However, assembly of textile products. Automating the handling
if redistributing the operators reduces production to of textiles improves efficiency for production by
one shift, the daily output will decrease despite an reducing assembly times and enabling longer
increase in the hourly production. An all automated production hours. Automation in textile handling is
production process has the most efficient assembly and most beneficial when the entire process can be
operates for twenty-four hours thus significantly automated, alleviating the bottlenecks that come with
increasing the daily output. manual operations in a continuous production line.
However, automated equipment, including machinery
and robotic arms, requires a significant investment, so
Table 9. Comparison of the production output of continuous it essential to assess the value of adding automation to
production with robotic handling and for the current the production process before investing.
manufacturing process and the four alternate scenarios.
Scenario Current 1 2 3 4
Operators 11 6 6 9 1
Acknowledgments
Robots 0 5 5 5 9
Units/Hour 39.72 39.17 34.97 47.28 97.2 This work was funded through the University of
Units/Day 317.79 626.67 559.54 378.28 2332.8
North Carolina Systems Office and the United States
Army Combat Capabilities Development Command.

4. Conclusions References

Simio software was used to simulate the [1]. International Labour Office- Geneva, The future of
implications of inserting automated handling in the work in textiles, clothing, leather and footwear,
assembly of an MSV cummerbund. When automated International Labour Office, Sectoral Policies
Department. – Geneva: ILO, 2019. Working Paper:
handling is employed at the automated equipment
No. 326.
operations in the current batch production process, the [2]. R. Handfield, H. Sun, L. Rothberg, Assessing Supply
daily production of cummerbunds nearly doubles to Chain Risk for Apparel Production in Low Cost
626.67 units from improved efficiency and introducing Countries Using Newsfeed Analysis, Supply Chain
a sixteen-hour work day. Using automated handling in Management, Vol. 25, No. 6, 2020, pp. 803–21.
continuous flow production is less effective producing [3]. K. Kulkarni and R. Adivarekar, Developments in
559.54 units per day. Reallocating the saved labor can Textile Continuous Processing Machineries, in
improve the bottlenecks in continuous manufacturing Advances in Functional Finishing of Textiles, Springer
by increasing hourly production from 34.97 units per Singapore, 2020.
hour to 47.28 units per hour, but due to the increased [4]. M. Suh, Automated Cutting and Sewing for Industry
4.0 at ITMA 2019, Journal of Textile and Apparel,
labor, an eight-work hour day must be employed, Technology and Management, Special Issue: ITMA,
reducing daily production to 378.28 units per day. 2019.
With the creation of automated equipment for the

30
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(008)

Multi-Robot Cooperative SLAM Using Panoramas

J. Y. Feng and Z. XuanYuan
1
Beijing Normal University - Hong Kong Baptist University United International College, ZhuHai, China
Tel.: +86-131-43116416
E-mail: [email protected], [email protected]

Summary: Simultaneously Mapping And Localization (SLAM) systems are essential in robotics systems. Using a single robot
to map a large scene is time-consuming. Therefore, a map can be constructed using a group of robots working together. Visual-
based loop detection in cooperative SLAM is essential, which can help to efficiently and accurately merge and construct global
maps among robots working independently. However, most visual SLAM algorithms focus on loop detection along the same
trajectory direction. These algorithms cannot handle multi-robot cooperative systems well because, in such systems, multi-
robot trajectory direction may often form loops along opposing or perpendicular directions. This paper proposes a multi-robot
cooperative SLAM system based on panoramic images. Each robot uses a camera as a sensor and a local map algorithm to
construct a local map. All local maps are merged into a global map when loops are detected. The panoramic images provide
scene information in all directions for loop detection. Experiments show that the loop detection accuracy improves in the
proposed system, and the time for constructing a global map is significantly reduced using a multi-robot cooperative system.

Keywords: Visual SLAM, Cooperative mapping, Panoramic image.

1. Introduction with all trajectory directions. The main contributions

of this work include the following:
In different application scenarios such as 1. A loop detection algorithm for a multi-robot
autonomous driving and robot navigation, it is usually SLAM system based on panoramas is proposed to
necessary to do mapping and self-localization in large enhance the accuracy and efficiency of the cooperative
scenes, such as a whole city scale. In this case, running mapping tasks.
SLAM with a single robot is time-consuming. In 2. A multi-robot cooperative mapping system has
addition, in map building, the pose errors will been designed and implemented to test the efficacy of
accumulate as the traveling distance increases. Maps the loop detection algorithm on different datasets.
can be divided into several sub-maps and built using a
group of robots, which can then merge into a global
consistent map by sharing information when loops are
detected. In this case, the distance of each robot is
shorter, the running time is shorter, and the
accumulated error is also reduced.
Fig.1 shows the comparison of sample images in
the same and different trajectory directions from the
KITTI dataset [1]. The appearance of two images in the
same trajectory directions is similar, but the
appearance of two images in different trajectory
directions is quite different. This leads to the fact that
although monocular visual SLAM can complete map
building efficiently and accurately, it can only perform Fig. 1. Comparison of images in the same and different
loop detection along the same trajectory direction, but trajectory directions in KITTI Dataset [1].
not in opposing or perpendicular directions, which is
not a practical and efficient assumption for multi-robot
cooperative systems because the path planning for 2. Related Work
efficient multi-robot cooperation may result in
rendezvous (loopback) of robots along with different In the following, we independently conclude with
trajectory directions and viewpoints. SLAM Algorithm a review of visual SLAM, fisheye-based visual SLAM,
that is robust to viewpoint variation is crucial for multi- visual-based cooperative SLAM, and place
robot cooperative systems. recognition.
This paper proposes a cooperative visual SLAM
system based on panoramic images, using multiple 2.1. Visual SLAM
robots to collect scene maps and build global
maps. With panoramic images as input, 360-degree The early visual SLAM algorithms are mainly
scene information can be obtained. On this basis, it can based on filtering [2], [3]. The filtering-based method
effectively solve the problem of loop detection along is inefficient and easy to accumulate linearized errors.

31
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Afterward, keyframe-based approaches were FAB-Map 2.0 [16] can use a vocabulary as large as
proposed. It only uses the selected frame as the 10,000 words.
keyframe to estimate the map. PTAM [4] was the first Some neural network-based methods have been
keyframe-based SLAM algorithm proposed to separate applied to place recognition. NetVLAD [17] is
the two functions of tracking and mapping in the modified based on VLAD [18] and directly embedded
SLAM system as two threads. ORB- SLAM2 [5] is one in the trainable CNN(convolutional neural network)
of the most popular visual SLAM. It adopts the multi- architecture. [19] used the Edge Boxes object proposal
threaded framework of PTAM, but the ORB[6] method combined with a mid-level CNN[20] to
features are extracted. Loop detection thread is added, identify and extract landmarks to complete place
which uses an efficient search algorithm BoW(Bag-of- recognition.
Words) [7]. In the past few years, feature-based methods have
become one of the main technologies in place
recognition. Its low computing cost and effectiveness
2.2. Fisheye Based Visual SLAM can be applied to low processing and storage capacity
devices [21]. The application scenarios of the multi-
Because a single camera’s FoV (Field of View) is robot cooperation visual SLAM system are complex
limited, extracting feature points and detecting the and huge. It increases the difficulty of pre-training of
correct loop in some indoor and outdoor scenes is not the neural network-based methods. The neural
easy. The fisheye camera has a wide FoV and can network-based approach requires large computation.
collect more scene information. The challenge of using In a centralized system, the central server needs to
the fisheye camera is how to solve the fisheye camera’s process a large amount of data from the client robots,
distortion effectively. In [8], cubemap is used to and the use of neural network-based methods will
improve the distortion of the fisheye camera. The increase its burden.
method uses five planes: up, down, left, right and front,
and then projects the spherical image separately. [9] is
a multi-fisheye camera SLAM system. This method 3. Methodology
requires the calibration of a polynomial model of the
fisheye camera. The camera parameters are used for 3.1. System Framework
connection transformation between multiple cameras.
[10] uses a multi-fisheye camera to obtain 360-view This cooperative SLAM system uses a visual
images and input them into the SLAM system for odometer based on a panoramic camera as a sensor and
mapping. a global place recognition based on panoramic images.
The use of panoramic images solves the problem of
loop detection in different trajectory directions in a
2.3. Visual Based Cooperative SLAM multi-robot system.
As shown in Fig. 2, it is the workflow of the multi-
Generally, multiple-robot cooperative SLAM can robot cooperative SLAM system. The system contains
be a centralized system [11]. In a centralized system, a central server and multiple robots. Suppose robots in
multiple robots equipped with sensors, communication the system are equipped with a panoramic camera and
components, and processors are used to complete the wireless communication interfaces. Each robot starts
mapping task in a centralized system. Each robot can from its initial position and independently constructs
run visual SLAM independently. The computation for its local map. The robot’s initial position is random,
some tasks, such as map merging, is performed by a which depends on the user. The robots do not know the
specific robot on the team or by an external agent [12]. position and coordinate system of other robots. The
The specific robot or the external agent communicates visual SLAM algorithm used by each robot is similar
with other robots as the system’s center. Other robots as the algorithm described in [10]. The difference is
are independent and do not communicate with each that we have added a data transmission part in the loop
other. The central server receives data sent by other detection. The central server communicates with
robots and provides required data and feedback. [13], multiple robots in the system. Each robot will send its
[14] are cooperative systems that use this structure. map data to the central server through its wireless
communication channel in robot movement. The
central server maintains a globally consistent map and
2.4. Place Recognition a corresponding sub-map for each robot. After the
central server receives the data of each robot, it will
Place recognition [15] is of great significance to reconstruct the keyframes and map points consistent
visual SLAM. One method is to extract local features. with each robot and add them to the sub-map
Hundreds of local features may be extracted from each corresponding to each robot. And the central server
image. BoW [7] clusters discrete extracted features and will do place recognition among the sub-maps. When
uses many images to train offline to generate a it is detected that different robots have come to
vocabulary containing multiple words. A typical common areas, the sub-maps corresponding to these
vocabulary usually contains thousands of words, but robots will be merged to get a globally consistent map.

32
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 2. The framework of the cooperative SLAM

3.2. Single Robot Mapping 3.3. Communication Model

Each robot is equipped with a LadyBug3 camera as We transmit data mainly about keyframes and map
a sensor. The panoramic images taken by the points and their corresponding relationships. We use
LadyBug3 camera are fed to local panoramic SLAM keyframes instead of all captured frames. For example,
which has been modified based on [10]. Every robot a dataset containing 498 frames, a map constructed by
will generate and maintain its own map. See an a single robot only contains 181 keyframes. Only
overview Fig. 3, there are four main threads in the local transmitting keyframes greatly reduces the data that
panoramic SLAM, tracking, local mapping, loop needs to be transmitted and ensures the map
closing, and global optimization. The first three information’s integrity.
threads typically run in parallel. Global optimization is We use LCM (Lightweight Communications and
enabled after a loop is detected. Marshalling) [22] to complete data transmission. LCM
The tacking and the local mapping work together can encapsulate the data to be transmitted into a whole,
to complete the visual odometer function, which uses facilitating data transmission and management.
continuous panoramic images to produce a sparse map. Our goal is to continuously rebuild the map of each
This map shows the robot’s trajectory and is used to robot using the transmitted data in the central server,
achieve localization. The loop detection is in charge of so there is no tracking module in the central server. The
place recognition. It detects whether the robot is at the loop detection module only uses the information of
same place it has been before. If the loop is detected, keyframes and map points and does not update those
the local map near the current keyframes is optimized information. Therefore, we chose to fetch the relevant
using the two keyframes of place recognition matches. data in the loop thread of each robot and transmit it.
Next, the global BA (Bundle Adjustment) thread is In the transmission process, packet loss is inevitable
enabled. It will apply the optimization to all the due to the bottleneck of network communication. The
keyframe poses, and map point coordinates in the map. central server can associate newly received data with
These processes are used to reduce the accumulated the current map to reduce the impact of packet loss.
errors in the robot movement.

3.4. Loop Detection and Map Merging

3.4.1. Loop Detection on Panoramic Images

The loop detection in panoramic SLAM uses the

method mentioned described in [15]. This method
extracts the image’s LDB (Local Difference Binary)
descriptor. It extracts a binary string P for each sub-
panorama I in a panoramic image I. The binary string
of a panoramic image is calculated by concatenating
the binary strings of n sub-panoramas. Calculating the
similarity between panoramic images i and j can be
done by associating the elements of P , P and
generating a distance metrics M.
The loop detection in panoramic SLAM sets a fixed
Fig. 3. The framework of the single robot mapping.
threshold to filter the appearance of keyframes. If the

33
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

distance metrics M calculated for the two keyframes is covisibility keyframes, loop keyframes, and non-loop
less than the threshold, then they are similar enough to keyframes. We can get that the calculated distance
be considered as a loop. This loop candidate pairs are range of the loop keyframe is closer to the covisibility
then further verified using geometry-based validation. keyframes than the non-loop keyframes varies greatly.
We set 20 different thresholds for the loop detection Based on these observations, we believed that
of monocular images and panoramic images in the applying a fixed threshold to all keyframes for loop
Kashiwa dataset [23], respectively. The process of loop detection is not optimal to deal with complex scenes.
detection does not include geometry-based validation. Therefore, we propose a distance filtering method
Fig. 4 describes the comparison of recall obtained from based on an adapted threshold.
panoramic and monocular images. It can be seen that
the recall obtained by using panoramic images is
higher, which means more loops can be detected by
using panoramic images.

Fig. 6. The comparison of distance ranges calculated from

the keyframe (id = 270) with its covisibility keyframes, loop
keyframes, and non-loop keyframes in the ground truth.

3.4.2. The Adaptive Threshold Method

As mentioned before, we believe that the threshold

Fig. 4. Comparison of recall between panoramic images for filtering distance should be related to a keyframe’s
and monocular images. covisibility keyframe. To verify this idea, we set up the
following experimental steps:
1. Obtain a certain number of keyframes near the
Our experiments showed that the loop detection in current keyframe, for example, 10.
the original panoramic SLAM can detect loops in the 2. Calculate the distance metrics M between the
same trajectory direction, but it is sometimes current keyframe and the keyframes in neighbors,
challenging to detect loops in different trajectory respectively. Then store all the distance matrices M
directions. The reason is that the fixed threshold set is into the set C.
unsuitable for complex scenarios. As shown in Fig. 5,
we calculated the distance between different C M , ,M , ,M , ,...,M , ,...M , (1)
keyframes and found that their loop frames fluctuates
greatly in terms of distance. The smallest distance is 3. To compare precision and recall, we set 20
less than 55,000, and the largest distance is greater than gradually increasing thresholds.
90,000.
C C
T~ C m
10
C C
T ~ C m
10
m 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 (2)

According to Fig. 7, we found that before the 12th

threshold, as the threshold increases, the recall
increases relatively large, and the amount of frames to
be verified does not increase much. After the 12th
threshold, as the threshold increases, the increase in the
recall is relatively small, but the amount of frames to
be verified increases a lot. Therefore, we believe that
Fig. 5. The red line represents the value of distance
the 12th threshold is the most appropriate.
calculated by several keyframes and their corresponding loop
keyframes. The green line represents a fixed threshold of
As shown in Fig. 8, it compares the performance of
64,000. adaptive threshold and fixed threshold. The blue line
represents the calculated adaptive threshold T of all
keyframes with loops. The orange line represents the
As shown in Fig. 6, it is the calculated distance fixed threshold of 64000. From Fig. 5 and Fig. 8, we
ranges between a keyframe (id=270) and its found that the optimal adaptive threshold is varies

34
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

greatly thus further proved the fixed threshold method maps of the robots are merged into the global
to be sub-optimal. consistent map, the above RT matrix is applied to add
the subsequent keyframes and map points of the robot
into the global map. In this way, the global consistent
map can be built continuously.

4. Evaluation

4.1. Datasets

We evaluated our system based on the Kashiwa

dataset [23]. The dataset contains a total of 498 frames.
Each frame was captured using a panoramic camera.
This dataset contains loops in the same and different
directions.
Fig. 7. Comparison between the amount of data to be
verified and recall of 20 threshold. 4.2. Loop Detection Results

A total of 79 loops matched pairs are in the ground

truth in the Kashiwa dataset. The number of loops
matched pairs that can be detected using the adaptive
threshold, and the fixed threshold is shown in Table 1.
The number of keyframes that can find loop keyframes
with the fixed threshold is very small. More loops can
be found using adaptive thresholds.

Fig. 8. The comparison of adaptive threshold and fixed Table 1. Comparison of detection results between adaptive
threshold. The blue line represents the calculated adaptive threshold and fixed threshold.
threshold of the keyframes with loops. The orange line
represents the fixed threshold. Detected Precisio
Recall
loop n
Adaptive
51 64.56 % 100 %
3.4.3. Map Merging threshold
Fixed threshold 13 16.46 % 100 %
Loop detection is in charge of detecting loop
matched pairs between different sub-maps. After
detecting a loop, a 4 4 matrix of the loop matched 4.3. Cooperative SLAM System Performance
pairs are obtained by RANSAC iterations, like (3). It
represents a similar transformation between the pose of Fig. 9 shows the designed cooperative SLAM
the current keyframe and the pose of the detected loop system based on two robots and merging two sub-maps
keyframe. Using M we can obtain the relationship in the same trajectory direction (a) and different
of rotation and translation between the poses of these trajectory direction (b). The red arrow indicates the
two keyframes. direction of movement of robot1. The yellow arrow
indicates the direction of movement of robot2. The
R T purple circle represents the loop detected between
M (3) robot1 and robot2. The central server uses the
0 1
transmitted data to reconstruct the sub-map (1) and (2)
Then, we can use the (3) to merge the sub-map corresponding to each robot. Then loop detection with
corresponding to the two robots by a rigid adaptive threshold will detect loops in each sub-map.
transformation, like (4). It transforms every keyframe In the experiments, both of the loops in the same
and map point in two local coordinate systems to a trajectory direction and different trajectory directions
global coordinate system. are detected. Then, the two corresponding sub-maps
are merged, and the global consistent maps are
M M 𝑀, (4) constructed (a)-(3) and (b)-(3).
Table 3 shows the comparison of time-cost between
where M and M are the poses of the keyframes and single robot mapping and cooperative mapping. The
the coordinates of the map points in the two coordinate mapping of a single robot used the frames from 1 to
systems respectively. 280, and it consumed 161.915s. In the cooperative
The global consistent map can be obtained by system, robot 1 used the frames from 1 to 140 and
merging the sub-maps of the two robots. Once the sub- consumed 84.9719s. Robot 2 used the frames from 141

35
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

to 280 and consumed 76.751s. The central server seen that the central server without feature extraction
received the data from robot-1 and robot-2 and and tracking has very high computational efficiency. It
reconstructed the map, run loop detection, and merged is more efficient to use multiple robots to map in the
sub-maps. It consumed a total of 10.084s. It can be same scene.

Fig. 9. Cooperative SLAM system based on two robots and the merging of two sub-maps
in the same trajectory direction (a), and different trajectory directions (b).

In terms of accuracy, we use APE (Absolute Pose map. We improved the method by keyframes, and map
Error) as the metric. Table 2 shows the APE points were added. Our system can associate
comparison between single robot and multi robots. We keyframes and map points with existing maps even
can see that compared with single robot mapping, when packets are lost in network communication. We
cooperative mapping can reduce the cumulative error also improved the loop detection part of panoramic
and increase the accuracy of mapping. SLAM. We propose three adaptive thresholds instead
of fixed thresholds to filter candidate loop keyframes.
Experiments show that the filtering effect of the
Table. 2. Various parameters of APE(m). adaptive threshold is better than that of the fixed
threshold. The adaptive threshold can be used to find
rmse mean std sse more loop matching pairs and correct loops. Both of
Single Robot 0.83 0.69 0.46 45.23 them improve the efficiency and robustness of loop
Multi Robots 0.24 0.21 0.11 3.28 detection. Experiments show that our system can
complete multi-robot transmission and map merging
well in the same and different trajectory directions.
Table. 3. The comparison of time-cost between single robot
mapping and cooperative mapping.

Frame 141- 1-140+ References

1-280 1-140
range 280 141~280
The [1]. A. Geiger, P. Lenz, C. Stiller and R. Urtasun, Vision
Single meets robotics: The KITTI dataset, The Intl. J. of
Mapping Robot1 Robot2 central
robot Robotics Research, 2013.
server
[2]. Davison A J, Reid I D, Molton N D, et al.
Time-
161.91 84.97 76.75 10.08 MonoSLAM: Real-time single camera SLAM, IEEE
cost(s)
Transactions on Pattern Analysis and Machine
Intelligence, 29, 6, 2007, pp. 1052-1067.
[3]. E. Eade and T. Drummond, Scalable monocular
5. Conclusions SLAM, in Proceedings of the IEEE Computer Society
Conference on Computer Vision and Pattern
We propose a multi-robot cooperative system Recognition (CVPR), Vol. 1, New York City, USA,
based on panoramic SLAM. Panoramic images are June 2006, pp. 469–476.
used to accurately detect whether independent robots [4]. Klein, G. and Murray, D., Parallel tracking and
mapping for small AR workspaces, in Proceedings of
meet each other in Omni-directions and then merge the

36
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

the International Symposium on Mixed and Augmented [15]. Arroyo R. et al., Bidirectional Loop Closure Detection
Reality (ISMAR), IEEE, 2007, pp. 225-234. on Panoramas for Visual Navigation, in Proceedings
[5]. Raúl Mur-Artal, and Juan D. Tardós, ORB-SLAM2: an of the IEEE Intelligent Vehicles Symposium (IV), 2014,
Open-Source SLAM System for Monocular, Stereo 14466969.
and RGB-D Cameras, IEEE Transactions on Robotics, [16]. M. Cummins and P. Newman, Appearance-only
2016. SLAM at large scale with FAB-MAP 2.0, Int. J. Robot.
[6]. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, Res., Vol. 30, No. 9, 2011, pp. 1100-1123.
ORB: an efficient alternative to SIFT or SURF, in [17]. R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and
Proceedings of the IEEE International Conference on J. Sivic, NetVLAD: CNN architecture for weakly
Computer Vision (ICCV), Barcelona, Spain, November supervised place recognition, in Proceedings of the
2011, pp. 2564-2571. IEEE Conf. on Computer Vision and Pattern
[7]. D. Galvez-López and J. D. Tardós, Bags of binary Recognition, 2016, pp. 5297-5307.
words for fast ´ place recognition in image sequences, [18]. H. Jégou, M. Douze, C. Schmid, and P. Pérez,
IEEE Transactions on Robotics, Vol. 28, No. 5, 2012, Aggregating local descriptors into a compact image
pp. 1188-1197. representation, in Proceedings of the IEEE Computer
[8]. Wang Y., Cai S., Li S.-J., Liu Y., Guo Y., Li T., Cheng Society Conference on Computer Vision and Pattern
M.-M., CubemapSLAM: A Piecewise-Pinhole Recognition, 2010.
Monocular Fisheye SLAM System, in Proceedings of [19]. N. Sünderhauf, S. Shirazi, A. Jacobson, F. Dayoub,
the 14th Asian Conference on Computer Vision, Perth, E. Pepperell, B. Upcroft, and M. Milford, Place
Australia, December 2–6, 2018, pp. 34-49. recognition with ConvNet land-marks: Viewpoint-
[9]. Urban, S., Hinz, S., MultiCol-SLAM-A modular real- robust, condition-robust, training-free, in Proceedings
time multi-camera SLAM system, 2016. of the Robotics: Science and Systems Conf., Rome,
arXiv:1610.07336. Italy, Jul. 2015.
[10]. Ji, S. et al., Panoramic SLAM from a multiple fisheye [20]. Krizhevsky, I. Sutskever and G. Hinton, ImageNet
camera rig, ISPRS Journal of Photogrammetry and classification with deep convolutional neural networks,
Remote Sensing, 159, 2019, pp. 169-183. in Proceedings of the Annual Conference on Neural
[11]. Leung K. Y. K., Cooperative Localization and Information Processing Systems, 2012, pp. 1090-1105.
Mapping in Sparsely-Communicating Robot Networks. [21]. Garg S., N. Suen De Rhauf and M. Milford, Don't Look
Ph. D. Dissertation, University of Toronto, Toronto, Back: Robustifying Place Categorization for
ON, Canada, 2012. Viewpoint- and Condition-Invariant Place Recognition,
[12]. Saeedi, Sajad, Trentini, et al., Multiple - Robot in Proceedings of the IEEE International Conference
Simultaneous Localization and Mapping: A Review, on Robotics and Automation (ICRA’ 2018), 2018.
Journal of Field Robotics, Volume 33, Issue 1, 2016, [22]. Huang, A. S., E. Olson, and D. C. Moore, LCM:
pp. 3-46. Lightweight Communications and Marshalling, in
[13]. Forster S. Lynen, L. Kneip and D. Scaramuzza, Proceedings of the IEEE/RSJ International
Collaborative monocular slam with multiple micro Conference on Intelligent Robots and Systems (IROS),
aerial vehicles, in Proceedings of the IEEE/RSJ Int. 2010.
Conf. on Intelligent Robots and Systems, 2013, [23]. Kashiwa (http://gpcv.whu.edu.cn/data/panslam.html)
14002391.
[14]. P. Schmuck and M. Chli, Multi-UAV collaborative
monocular slam, in Proceedings of the IEEE Int. Conf.
on Robotics and Automation, 2017, 17058387.

37
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(009)

The Autonomous Pollination Drone

D. Hulens 1*, W. Van Ranst 1*, Y. Cao 2 and T. Goedemé 1
1
EAVISE, KU Leuven, Saint-Katelijne-Waver, Belgium
2
Magics Technologies NV, Geel, Belgium
E-mail: [email protected], [email protected], [email protected],
[email protected]

Summary: In this paper we will supplement the declining bee population by the development of a small drone that is
able to autonomously pollinate flowers. The proposed solution uses a DJI Tello drone carrying a Maix Bit processing board
capable of running all processing on-board. Additionally, the drone is equipped with a small color camera and a distance sensor
to detect and approach the flower. We developed a two stage algorithm that is able to detect the flower, steer the drone towards
the flower and makes the drone touch the flower to pollinate it.

Keywords: Pollination drone, Two-stage approach, Neural network.

1. Introduction In the remainder of this paper, we investigate how

we create such a pollination drone.
World food consumption in projected to increase in In section 2 we lay out the related work on robot
the coming decades [1], this together with a declining pollination methods, autonomous drone technology
insect population [2] and the roles bees play in the and the computer vision techniques we use to steer the
pollination of crops [3], is a big concern for world food drone. In section 3 we go in depth into the inner
security. To have a successful crop, a farmer might rely workings of our pollination drone. In section 4 we
on the surrounding ecosystem, artificially introduce a explain the experiments we did to evaluate the
bee colony to pollinate their crops or rely on some effectiveness of our pollinator drone and discuss our
other biological pollination method. However, due to promising results. We conclude in section 5.
collapse of surrounding ecosystems, regulatory
difficulties such as protection from invasive species, or 2. Related Work
the circumstances of an artificial factory environment,
interest in human made pollination methods has In this section we will first investigate existing
grown. Currently, pollination robots are already being artificial pollination solutions and then go into
used [4, 5]. hardware platforms that can be used for on board
These methods use mobile robot arms together with computer vision based navigation. As mentioned
some kind of pollination brush to go from flower to earlier, world food consumption is projected to
flower, distributing pollen from plant to plant. Using increase significantly in the coming decades [1]. To
such a robot arm however does have some limitations: address this challenge, more efficient ways of
the way a farm is laid out might for instance not easily producing food should be developed to feed this
allow a robot to pass between different crops, the type growing demand. A possible solution using
of produce might not lend itself to be touched by a technological means is to aid nature with the
robot arm, or the field might be too big or steep to build pollination of crops and plants. In recent research,
infrastructure for such a mobile ground robot. robots were already developed to automatically
Nowadays, some crops are even planted above each pollinate crops and flowers by using a robotic arm on
other in vertical farms, which makes it impossible for a base equipped with wheels [7, 5]. For the detection
a wheeled robot to pollinate [6]. of the flower, they use Inception-V3 [8] together with
All of this means that there is a lack and a certain color segmentation. This is a common approach for
need for a more generic and versatile pollination detecting flowers as seen in [9] where they combine a
method. CNN and SVM to predict an even more accurate
In this paper we developed a mini drone equipped segmentation of the flowers. The downside of a robotic
with a camera and on-board processing to arm on a wheeled platform is its size and
autonomously pollinate flowers using a two-stage deep maneuverability. In [10] a drone was used to pollinate
learning approach. As a preliminary use case, we focus the flowers. This drone has multiple advantages over a
on sunflowers. wheeled robot, but was way to big (50cm x 50cm) to
The three main novelties of this paper are: do the extremely precise job of pollinating flowers. In
1. Embedded on-board real-time computer vision. this work we developed a small drone (10cm x 10cm)
2. Trained with a partial synthesized dataset. with on-board processing power which can estimate
3. A two-stage flower approaching visual servoing the position and angle of the flower. Furthermore, this
(successfully demonstrated).

38
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

drone can autonomously fly towards the flower and low power Kendryte K210 hardware platform which is
precisely pollinate the flower. able to run quantized Mobilenet based networks
We investigated many hardware platforms that mounted on a commercially available DJI Tello Talent
would fit our criteria for on board computation of Deep drone. A two stage flower approaching technique is
Learning based computer vision tasks. Our developed to successfully touch and pollinate the
requirements for the platform were the following: flower. In the first stage, we detect the flower from a
1. The platform should be light weight; distance between 8 and 0.8 m using a convolutional
2. The platform should be low-power; detection network. In the second stage (0.8 - 0 m) we
3. Easy to deploy and integrate with an existing use an end-to-end network to predict the center of the
drone; flower in the camera feed. To guide the drone towards
4. Able to run state-of-the-art computer vision the center of the flower, a PID control loop is used to
algorithms with a sufficiently high frame rate. convert the location of the flower to steering
For the most part this rules out higher power mini- commands for the drone. In this section we explain all
computer devices such as the NVIDIA Jetson series. of these components one by one.
The STM32F746 or other ARM Cortex M7 based
platforms provide a deep learning library (CMSIS-NN) 3.1. Flower Dataset
to run deep learning applications on the CPU
architecture. The Ambiq Apollo3 Blue development To train the computer vision neural networks, we
board is able to run Tensorflow Lite models mostly created an artificial dataset based on images of real
targeted at voice and gesture recognition not computer sunflowers, artificial (synthetic) sunflowers and
vision. The Greenwave GAP8 is a RISC-V processor virtually rendered sunflowers. An important aspect we
optimized to run deep learning models in a multi take into account is that for the drone to successfully
threaded way. The Kendryte K210 processor is also pollinate a sunflower, it should approach the sunflower
based on a RISC-V core, but in addition to that also in a line perpendicular to the sunflower. In order to be
contains a deep learning accelerator that is able to run able to do this, we need to estimate the angle of gaze
YOLO [13] models at up to 20 fps. Because of its orientation of the sunflower to enable us to correct for
performance and relative light weight, we decided that this when we are planning the drone's trajectory. This
the Kendryte processor best fits our use case. means that in each image the angle of the flower w.r.t.
From a machine learning point of view, we need the camera should also be annotated.
models that are able to do two types of tasks: detection For the virtual flowers, we simply render them at
and classification (direct steering). These areas are a different frontal angles. For the real and synthetic
very active field of research, and many pipelines are sunflowers, we made a recorder device that is able to
available, each with their own trade-offs. Discussing rotate the flower automatically while taking pictures of
all current techniques would lead us too far from the the flower. The generated images are then pasted on
scope of this paper. Instead, we will discuss the images from the Places dataset, more specific on
techniques that are available on the chosen platform images from Japanese gardens [17] to use as
here. The Kendryte processor has support for the background. A total of 5660 annotated images were
Mobilenet [11] family of backbones. This is a generated this way. An example of these images can
backbone especially optimized for low-power mobile be seen in Fig 1.
devices, and is able to save a tremendous amount of
computing power by using depthwise convolutions.
Since the publication of the Mobilenetv1 paper, many
additions to the network have been proposed as well
[12]. However, due to the platform we choose, we are
limited to Mobilenetv1. The classification head uses a
simple cross entropy loss. In the detection stage we use
a modified detection network, together with a
Mobilenet to do fast inference. The detector makes
bounding box predictions with only a single pass over
the network, as opposed to region based detectors like
Faster-RCNNN [14] and Masked-RCNN [15]. Due to Fig. 1. Left: 3D generated sunflowers for end-to-end
their nature, generally speaking, single stage detectors approach. Right: Generated images for direct visual
are faster than region based ones. servoing approach.
The authors in [16] did a complete survey on the
current state of object detection algorithms. 3.2. Hardware Platform

3. Autonomous Drone Pollination Since the precise steering the drone has to perform,
processing of the video feed should happen on-board
In this section we go further into detail about the such that the delay between receiving images and
inner workings of our pollination drone. We create an correcting the drone's position is minimal. This implies
artificial sunflower dataset containing images of real, the need for a small and light-weight processing board
as well as artificial and virtual sunflowers. We use the capable of performing real-time image processing. We

39
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

choose to use a Maix Bit development-board which dataset. We altered the CNN architecture to
contains a Kendryte K210 (RISC-V) processor. This additionally output the angle of the flower. This
board measures 53 mm x 25 mm and weights 25 g. predicted angle is used to steer the drone towards a 0°
Furthermore, this board is equipped with a camera, angle w.r.t. the flower. The position of the flower is
resulting in a minimal image receiving delay. used to steer the drone such that the flower is always
The camera feed is directly read by the Maix Bit on in the center of the drone’s view. In this stage the size
which we either run the end-to-end network or the of the detected flower is used as a distance
detection network. The Maix Bit communicates with measurement. When the drone approaches the flower
the Tello drone via its on-board ESP32 and the size of the flower exceeds 60 % of the frame-
microcontroller. In addition to the camera, we also height, the flower becomes too big to be reliably
have a ToF depth sensor (VL53L1X) to measure the detectable and we switch to the second visual servoing
distance from the flower in the final approach. A stage.
detailed overview of our setup is shown in Fig. 2.
Fig. 3 shows the drone carrying the Maix Bit. 3.3.2. Direct Visual Servoing

In this stage we use an end-to-end network based

again on a Mobilenetv1 architecture trained for
classification that directly outputs steering commands
(up, down, left, right or center). This network is trained
on zoomed-in images of a flower. Here, we use a ToF
distance sensor to measure the distance between the
drone and the flower with high accuracy. The
pollination stick in front of the flower measures 8cm,
when the distance becomes smaller than 8cm we
assume the stick touched the flower, and we fly
backwards to restart the first stage and find a new
Fig. 2. Overview of our hardware setup. random flower.

3.4. PID Control Loop

To transfer the position of the flower into steering

commands for the drone, four different PID loops are
used, one for each of the drone's axes. The goal of the
PID loops is to center the flower in the drone's view
and steer the drone such that the angle between the
drone and the flower reaches zero degrees (ideal for
pollination). To position the drone such that the flower
is in the center of the screen, the Y-coordinate of the
flower is used to move the drone up or down (Altitude)
while the X-coordinate is used to steer the drone to the
Fig. 3. Our pollination drone carrying the Maix Bit. left or right (Roll). The angle of the flower is used to
control the drone's rotation around its Z-axis (Yaw).
When the Yaw is changed, consequently, the flower
3.3. Hybrid End-to-end and Detection Approach will also move to the left or right in the image, which
will be again compensated by controlling the Roll. To
As explained before, the vision part of our approach the flower, the size of the detection is used to
approach plan is divided into two stages. In the first move the drone forwards or backwards (Pitch). When
stage we run a CNN for flower pose estimation: the distance becomes smaller than 0.8 meter, we
position, size (indication of the distance) and switch to an end-to-end stage where the time of flight
orientation. This model makes the drone fly towards a distance sensor is used to control the Pitch instead.
position close to the flower (approximately 80cm),
directly facing it. In the second stage, we trained an 4. Experiments and Results
image-based visual servoing network in an end-to-end
approach such that it directly outputs steering In the first experiment, we evaluated the average
commands towards the sunflower’s position. This precision of the detection results of our first model.
model is used for the final approach, the pollination We reached a mAP of 0.36 on our test dataset (note
touchdown. that the different angle classes in our dataset make our
dataset more challenging, when doing real-world test
3.3.1. Detection Stage this accuracy proved sufficient).
The second experiment we conducted was
In the first stage (distance of 8m to 0.8m) we use a evaluating the success-rate of pollinating a sunflower.
Mobilenet [11] detector trained on our own flower We repeated this experiment 20 times were the drone

40
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

took off from a distance of 5m, flew to the flower, References

touched it and flew back. We reached a success-rate of
85 %. In the other 15 % of non-success, a false
detection was the cause of failure. [1]. Michiel van Dijk et al., A meta-analysis of projected
The maximum frame rate of which images could be global food demand and population at risk of hunger
for the period 2010–2050, Nature Food, 2, 2021,
processed on the Maix Bit is approximately 30 fps for
pp. 494–501.
the direct steering model and 20 fps for the detection [2]. David L Wagner, Insect declines in the Anthropocene,
network. We also tested the minimum frame rate Annual Review of Entomology, 65, 2020, pp. 457-480.
needed to smoothly steer the drone towards the flower [3]. Shaden A. M. Khalifa et al., Overview of bee
since a slower frame rate results in less power pollination and its economic value for crop production,
consumption. The minimum framerate needed to Insects, 12, 8, 2021, p. 688.
smoothly approach a flower is measured at 12 fps. A [4]. Ting Yuan et al., An autonomous pollination robot for
demo of our working pollination drone can be seen in hormone treatment of tomato flower in greenhouse, in
[18] and Fig. 4 shows a successful pollination Proceedings of the International Conference on
sequence. Systems and Informatics (ICSAI), 2016, pp. 108-113.
[5]. Nicholas Ohi et al., Design of an Autonomous
Precision Pollination Robot, in Proceedings of the
IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2018, pp. 7711 – 7718.
[6]. Fatemeh Kalantari et al., A review of vertical farming
technology: A guide for implementation of building
integrated agriculture in cities, Advanced Engineering
Forum, 24, 2017, pp. 76-91.
[7]. Jared Strader et al., Flower Interaction Subsystem for
a Precision Pollination Robot, in Proceedings of the
IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2019.
[8]. Christian Szegedy et al., Rethinking the inception
architecture for computer vision, in Proceedings of the
IEEE Conference on Computer Vision and Pattern
Fig. 4. Successful pollination sequence. Recognition, 2016, pp. 2818 - 2826.
[9]. Philipe A. Dias, Amy Tabb, and Henry Medeiros.,
5. Conclusion Apple flower detection using deep convolutional
networks, Computers in Industry, Vol. 99, August
In this paper we developed a pollination drone that 2018, pp. 17-28.
[10]. Peter C. Guglielmino et al., Autonomous Drone
detects flowers, estimates the angle and flies towards
Pollination, Worcester Polytechnic Institute, 2021.
the flower in a two stage approach. Different [11]. Andrew G. Howard et al., Mobilenets: Efficient
experiments were performed to evaluate all the parts of convolutional neural networks for mobile vision
this approach, yielding promising results. We managed applications, 2017, arXiv:1704.04861.
to run all processing on-board, resulting in a fully [12]. Andrew Howard et al., Searching for mobilenetv3,
autonomous drone with a minimal delay between Proceedings of the IEEE/CVF International
taking images and the control of the different degrees Conference on Computer Vision, 2019.
of freedom. Real world test demonstrate a success rate [13]. Joseph Redmon and Ali Farhadi., YOLO9000: better,
of 85 %. faster, stronger, in Proceedings of the IEEE
Conference on Computer Vision and Pattern
Furthermore, in this work we demonstrate that,
Recognition, 2017.
using currently available hardware, it is possible to [14]. Shaoqing Ren et al., Faster R-CNN: towards real-time
autonomously fly a drone towards a flower and object detection with region proposal networks, IEEE
maneuver it in a way similar to what a bee would do to Transactions on Pattern Analysis and Machine
pollinate a flower. It remains to be seen how well Intelligence, Vol. 39, 2016, pp. 1137-1149.
methods like these would scale up to replace actual bee [15]. Kaiming He et al., Mask r-cnn, Proceedings of the
populations, and what other environmental impacts a IEEE International Conference on Computer Vision.
dwindling insect population can have on the survival 2017, pp. 2980 – 2988.
of humanity, and the surrounding ecosystems it relies [16]. Li Liu et al., Deep learning for generic object
detection: A survey, International Journal of
on. A discussion should be had on how much we want
Computer Vision, 128, 2020, pp. 261–318.
to rely on technological solutions for our own food [17]. Bolei Zhou et al., Places: A 10 million image database
security, and what role humanity should play in for scene recognition, IEEE Transactions on Pattern
interfering in existing ecosystems. Analysis and Machine Intelligence, 40, 6, 2018,
pp. 1452 - 1464.
Acknowledgements [18]. https://youtu.be/quX5HhVyR3g, Youtube, 2021.

This research received funding from the Flemish

government (Flanders AI program).

41
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(010)

“They got my keys!”: On the Issue of Key Disclosure and Data Protection
in Value Chains
A. Mosteiro-Sanchez 1,2, M. Barcelo 1, J. Astorga 2 and A. Urbieta 1
1
Ikerlan Technology Research Center (BRTA), Arrasate/Mondragón, Spain
2
University of the Basque Country, Bilbao, Spain
E-mail: [email protected]

Summary: Value chains exchange massive volumes of data. Interconnection between companies makes them vulnerable to
value chain attacks and data breaches. Information is encrypted before uploading it to shared databases, and its hash is stored
in distributed data solutions like Distributed Ledger Technology (DLTs) to ensure data auditability. Key secrecy is vital for a
security system based on an encryption algorithm. However, keeping encryption keys secret is vital to the system’s security.
Thus, accidental disclosures compromise the system security, as does the lack of a periodic key renovation system. This paper
presents a solution based on Ciphertext-Policy Attribute-Based Encryption (CP-ABE). Our proposal enforces key renewal
after each security event and guarantees it via time-based encryption, performed by Smart Contracts. We thereby ensure users
refresh their keys periodically, attackers do not access new information once the key disclosure is discovered, and that original
information does not need to be re-encrypted.

Keywords: Value chain, CP-ABE, DLT, Smart contract, Industry 4.0, Key disclosure.

1. Introduction Several schemes have been proposed to achieve the

aforementioned one-to-many encryption. Fiat and
Smart manufacturing and Industry 4.0 imply Naor outlined the first version of broadcast encryption
massive information exchange across value chains. [3]. This cipher has been further enhanced by Boneh et
Information exchange increases operational efficiency al. [4]. Following the same idea of one-to-many,
and improves value chain management. Furthermore, Waters proposed Ciphertext-Policy Attribute-Based
it bolsters decision-making and company Encryption (CP-ABE) [5]. CP-ABE protects
competitiveness. information according to access policies, which are
However, interconnections between members of defined by attributes. For instance, a piece of data can
the value chain make the security of the entire chain be encrypted according to an access policy such as
reliant on the security of each member. A security “CompanyA_Engineer OR External Auditor”. An
breach in one member can have repercussions for the authority provides users with secret keys to decrypt the
rest. Data breaches resulting from cyber-attacks on messages. This authority generates users’ secret keys
value chains [1] have a high monetary and reputational according to users’ attributes in the system. This way,
cost for the affected companies [2]. Thus, establishing only users whose secret keys fulfil the access policy
a secure value chain infrastructure is essential for a can read the encrypted data.
thriving Industry 4.0 development. After successful data protection, information must
Encryption schemes can guarantee industrial data be distributed to the different value chain members.
confidentiality in value chains. However, encryption For this purpose, distributed data sharing systems such
algorithms have a high computational cost, and it is as Distributed Ledger Technology (DLT) have proven
essential to balance performance and security. This their suitability for data sharing in supply chains [6].
balance is relevant when data is generated and The information is stored in a distributed database,
encrypted by IIoT devices with limited capabilities. while its hash is stored in a DLT. This allows all value
Regarding security, encryption schemes require the chain members to keep track of data modifications.
generation of cryptographic keys. These keys must be This way, data control and trust are shared among all
renewed periodically to ensure the system’s security, supply chain members. Data sharing via DLTs can be
and thus a key revocation and renewal system must be combined with encryption systems, including one-to-
implemented. Key disclosures compromise sensitive many encryption systems. For this purpose, the
information since any attacker in possession of the information must be encrypted before being stored in
disclosed key can read the encrypted data. Therefore, the shared database, and the hash recorded in the DLT
security systems designed for value chains must ensure must match the encryption.
that keys are renewed after a security event. To recover the original information, partners need
Concerning performance, IIoT devices benefit from a valid decryption key. However, because of the
solutions that reduce the computational load while particularities of value chains, new users coexist with
maintaining the security properties achieved with old users, and access privileges may change. Similarly,
encryption. Value chain scenarios with multiple secret keys being disclosed pose a threat to system
participants requiring the same information benefit confidentiality. There needs to exist a key management
from solutions that allow one-to-many encryption. system resilient against key misuse—i.e., keys

42
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

obtained by nefarious means, accidentally disclosed Encryption ensures information confidentiality but
keys or legitimate keys that should have been renewed. does not procure user identification. Therefore, even if
Traditional key revocation systems require lists the data breach is discovered, there is no mechanism to
identifying secret keys that cannot be used again. prevent attackers from reusing the stolen keys. Current
However, this implies that at some point, the encryptor key revocation systems rely on lists to record revoked
and decryptor must negotiate new keys to continue the keys. Each time a data consumer uses their key, it is
information exchange. In addition, data protected with compared to those in the list. Although this prevents
the old keys have to be re-encrypted. Hence, the key the use of disclosed keys, it implies list management.
revocation system designed for a value chain must In addition, the keys are linked to the CTs, which must
provide a solution where IIoT devices do not need to be re-encrypted. A new symmetric key must also be
renegotiate keys or re-encrypt information. generated and securely transmitted in symmetric
This paper proposes a time-based encryption encryption systems. In the case of asymmetric
system based on CP-ABE to protect the system from encryption algorithms, the encryptor and decryptor
key misuse. Users must own secret keys generated have to renegotiate the key pair.
after a particular timestamp that reflects the last Every key disclosure implies the risk of attackers
detected security event. This way, users with invalid using stolen keys to access the system. Therefore, there
keys or old keys are automatically denied access to is a need for a revocation system compatible with one-
information and have to authenticate themselves again. to-many encryption systems and which does not have
The solution foregoes the revoked key lists used by to re-encrypt every CT generated during the system
traditional revocation systems. In addition, the IIoT lifetime. The system must also not renegotiate keys
devices do not need to reencrypt the information, nor between the IIoT device that encrypted the information
do they have to renegotiate secret keys with value and the chain member that retrieved it. To this end,
chain partners. applying a new encryption layer whenever data is
This paper has been structured as follows. retrieved can limit key abuse. In this paper we propose
Section 2 of this paper defines the identified issue with to link this encryption to a security timestamp
traditional key revocation methods. Section 3 reflecting the last known security event. This
describes the proposed solution, and Section 4 guarantees that the system rejects expired or leaked
summarizes the conclusions of the proposal. keys without maintaining a revocation list. The
solution, based on CP-ABE, is explained in the next
section.
2. The Issue of Key Disclosure

Key disclosure poses a threat to information 3. Proposed Solution

confidentiality. To ensure one-to-many encryption,
value chain members protect the data at the source and 3.1. CP-ABE-based Time Encryption
upload the resulting ciphertexts (CTs) to a shared
database. The hash of the CT is stored in the DLT so The proposed solution implements an encryption
that every chain member can verify its integrity. The system that protects information against key misuse.
auditability provided by DLTs makes it possible to The solution ensures regular refreshing of partners’
detect modifications to the information and trace which keys and prevents attackers from using stolen keys. It
value chain partner has made them. However, also exempts IIoT devices from data re-encryption and
regarding CT retrieval, anyone who has a key created key renegotiation. To explain how this is achieved,
during the system’s lifetime can retrieve the original first, we introduce the concept of security events.
message. As shown in Fig. 1, if there are no key With the term “security event” we define any
management systems, data consumers can use old and situation that affects the system’s security state—for
disclosed keys to access data. example, a periodic key renewal or a security incident.
An event would also be restoring the system after
detecting a security breach. Our solution must account
for different events and maintain one-to-many
encryption. Thus, we base our solution on the
discretionary data access provided by CP-ABE.
As explained, CP-ABE protects information
according to access policies and guarantees that only
partners whose attributes comply with the policy can
read encrypted data. Thus, it achieves one-to-many
encryption without the need to identify each user. Any
CT to which we add a new CP-ABE encryption layer
can obtain this property. Our previous work [7]
explored the concept of encryption layers and can be
applied for time-based encryption.
Fig. 1. With no key revocation, no longer valid keys (in In our time-based CP-ABE solution, the first time
red) can still obtain the information. value chain partners want to retrieve information from

43
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

the shared database, they request a secret key (SK) to the CT, the PPs, and the system restoration timestamp
an authority. The authority authenticates the partner, (ETMSP). The SC uses ETMSP to define an access policy
timestamps the request, and considers the timestamp such that AP = (TKTMSP > ETMSP). The SC uses this AP
(TKTMSP) as the partner’s attribute. Thus, the system and the PPs to encrypt the CT requested by the
generates each user’s secret key based on the TKTMSP consumer. The newly encrypted CTtime is then sent to
of their key request. Previously, the system has the consumer. Consumers who do not comply with the
generated another timestamp reflecting the last system AP defined by the SC cannot obtain the original
restoration after a security event (ETMSP). Using CP- information. The encryption and decryption processes
ABE properties, the requested information can be are detailed below and are depicted in Fig. 3.
encrypted at the time of the request according to a
policy that requires a secret key more recent than the
security event. This way, users with old keys or known
disclosed keys cannot access the information. Instead,
they have to identify themselves again and authenticate
their identity to the authority to obtain a new key.

3.2. Message Exchange

This section discusses how the proposed system

deploys time-based encryption by combining it with
the DLTs mentioned in Section 2. The proposed time-
based encryption is represented in Fig. 2. It shows the Fig. 1. Secret key request and generation.
inclusion of the time encryption module, so the
consumer interacts directly with it instead of with the
database and the DLT. 1. Consumers request a secret key according to the
AA.
2. The AA records the request timestamp, TKTMSP.
3. The AA generates the secret key using TKTMSP as
the attribute.
4. The consumer receives the requested secret key.

After obtaining the SK, consumers interact with the

SC to retrieve the CTs. In Fig. 2, we present how the
Smart Contract performs the Time-Encryption before
returning data to the consumer who requested it.

Fig. 2. Time encryption denies decryption to data

consumers with no longer valid keys (in red).

The Attribute Authority (AA) generates some

Public Parameters (PP) and a Master Secret Key
(MSK) during system setup. The PPs are made
available to all value chain members, while the MSK
is known only to the authority. Value chain members
store the CTs in a shared database, while the hash of
the CT is stored in the DLT. Meanwhile, data Fig. 2. Time Encryption.
consumers receive the secret keys (SKs) based on
TKTMSP from the AA. The authority uses the MSK to
1. Consumers request a CT from the Smart Contract.
generate these SKs taking TKTMSP as the attribute.
2. The Smart Contract uses an oracle to retrieve the
Consumers interact with a Smart Contract (SC) to
PP, the ETMSP, and the required CT from the shared
retrieve the CTs. By its nature, SC can only work with
database.
the information stored in their associated DLT. Thus,
3. The SC encrypts the CT according to an access
for the SC to retrieve data stored outside the DLT, it
policy that requires a TKTMSP newer than ETMSP.
needs the help of an oracle. Oracles [8] are services
capable of providing information for smart contracts.
𝐴𝑃 𝑇𝐾 𝐸
They act as the layer between data outside the DLT and
SCs. With the help of the oracle, the SC can retrieve 𝐸 𝐶𝑇, 𝐴𝑃 → 𝐶𝑇

44
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

4. The Smart Contract sends CTtime back to the disclosed key misuse. Finally, IIoT devices are not
consumers. overloaded since they do not have to re-encrypt
5. Consumers decrypt CTtime using their key. If TKTMSP information or renegotiate keys.
fulfils the condition expressed in the AP, they
access the information. Meanwhile, they recover ⊥
if the key is older than the last event. Acknowledgments

Thus, consumers must request a new key from the This work was financially supported by the
AA whenever they recover ⊥. This strategy ensures European commission through ECSEL-JU 2018
that consumers keep their keys up to date, and the program under the COMP4DRONES project (grant
authentication system prevents attackers from agreement Nº 826610), with national financing from
obtaining valid keys. Furthermore, IIoT devices are France, Spain, Italy, Netherlands, Austria, Czech,
exempt from further interactions beyond the original Belgium, and Latvia. It was also partially supported by
encryption, as they do not have to renegotiate keys or the Ayudas Cervera para Centros Tecnológicos grant
re-encrypt information. And since at no point is the of the Spanish Center for the Development of
original CT decrypted by anyone other than the Industrial Technology (CDTI) under the project
consumer, data confidentiality is guaranteed during the EGIDA (CER-20191012), and by the Basque Country
whole process. Government under the ELKARTEK program, project
TRUSTIND - Creating Trust in the Industrial Digital
Transformation (KK-2020/00054).
4. Conclusions

Large-scale information exchange in value chains References

increases the competitiveness of the involved
companies. However, it also makes them vulnerable to [1]. ITRC, Q1 2021 Data Breach Analysis, 2021.
attacks and data breaches. Distributed storage [2]. IBM, The 2020 Cost of a Data Breach, 2020.
solutions combining databases and DLTs enable chain [3]. A. Fiat and M. Naor, Broadcast Encryption, in
members to trace data tampering. This storage can be Proceedings of the 13th Annual International
Cryptology Conference Advances in Cryptology -
combined with one-to-many encryption solutions to
CRYPT0 ’93, California, USA, 1993, pp. 480-491.
reduce the risk of data breaches. However, users’ [4]. D. Boneh, C. Gentry and B. Waters, Collusion
secret key management and revocation remains a Resistant Broadcast Encryption with Short Ciphertexts
challenge that needs to be addressed in an efficient and and Private Keys, in Proceedings of the 25th Annual
scalable manner. International Cryptology Conference Advances in
For the system to be secure, it is necessary to Cryptology -- CRYPTO 2005, Santa Barbara,
guarantee periodic key renovation and establish California, USA, 2005, pp. 258-275.
mechanisms preventing the use of compromised keys. [5]. B. Waters, Ciphertext-Policy Attribute-Based
For this purpose, this paper proposes the application of Encryption: An Expressive, Efficient, and Provably
Secure Realization, in in Proceedings of the 14th
time-based encryption. The solution is based on CP-
International Conference on Practice and Theory in
ABE and is achieved using a Smart Contract combined Public Key Cryptography, Taormina, Italy, 2011,
with oracles. For this purpose, the authorities provide pp. 53-70.
consumers with CP-ABE decryption keys that match [6]. T. M. Fernández-Caramés, O. Blanco-Novoa, I. Froiz-
the key generation timestamp (TKTMSP). Smart Míguez and P. Fraga-Lamas, Towards an Autonomous
Contracts use the oracles to retrieve the CTs, public Industry 4.0 Warehouse: A UAV and Blockchain-
parameters and the last security event timestamp, Based System for Inventory and Traceability
ETMSP. They use these parameters to encrypt de CTs Applications in Big Data-Driven Supply Chain
according to access policy AP = (TKTMSP > ETMSP). Management, Sensors, Vol. 19, No. 10, 2019, p. 2394.
[7]. A. Mosteiro-Sanchez, M. Barcelo, J. Astorga and
This new ciphertext is sent to the consumers.
A. Urbieta, Multi-Layered CP-ABE Scheme for
Since decryption requires a key generated after the Flexible Policy Update in Industry 4.0, in Proceedings
security event, the system guarantees that only users of the 10th Mediterranean Conference on Embedded
with updated keys can access the information. As for Computing (MECO’2021), Budva, Montenegro, 2021,
the attackers, they need to steal a new key each time pp. 1-4.
the system is restored. Since they must bypass the [8]. C. Smith, Ethereum: Oracles, 3 January 2022.
authentication system, data is protected against (https://ethereum.org/en/developers/docs/oracles/)

45
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(011)

Virtual Commissioning of an Automotive Station for Door

Assembly Operation
R. Balderas Hill, J. Lugo Calles, J. Tsague, T. Master and N. Lassabe
Capgemini Engineering, France
E-mail: {rafael.balderas-hill, jesus-hiram.lugo-calles, junior.tsague, tobiah.master, nicolas.lassabe}
@capgemini.com

Summary: Industry 4.0 has established new standards in manufacturing and quality processes, which are mainly based on the
type and efficiency of the different components of an industrial system. Virtual commissioning (VC) is an approach that allows
the user to propose a commissioning solution using computer models of industrial systems to speed up and improve the
traditional processes. This paper presents a VC solution for an industrial automotive assembly line process using toolchains of
standard and off-the-shelf software and hardware components, including physical PLC and HMI. The virtual assembly line
environment was developed using digital mock-ups of industrial robots, smart conveyors, controllers and sensors. The
toolchains share most of their components, virtual and physical, and can be classified as pure software or software and hardware
applications. The connection between the elements of the toolchain is done using industrial protocols like Modbus TCP and
APIs, ensuring complete data transfer. After several tests, the results have shown that by performing the simulated process
with the VC platform, it is possible to verify automation hardware and software, with a representative digital robotic cell,
thanks to the applied simulation technique.

Keywords: virtual commissioning, virtual engineering, automotive industry.

1. Introduction automotive industry, requires a lot of engineering and

planning effort.[1, 3, 10-12] VC is the best option for
Nowadays, simulation systems are recommended these scenarios, reducing ramp-up time, resulting in
for engineering and decision-making tasks. [1] shorter product's time to market.
Simulation models from different fields are combined, This technology allows to verify the functionality
using multi-domain simulation tools, to create a of systems, through the testing of off-line programs for
complete model of the system specifications and specific devices, and avoid unexpected expenses due
behaviors, considering the different physical to inadequate component selection or damage during
interactions and communication protocols. [2, 3]. testing. [13] VC can be seen as a part of the modern
A digital factory, also known as Smart Factory or approach to Product Lifecycle Management (PLM),
Cyber-Physical Production Systems, is the generic being mostly used for product design inception
term for a network of digital models. It considers as through its manufacture. [14]
well methods and tools integrated in the life cycle Research has been done reagind the standardization
phases of a production system, characterized by their of VC to identify its level of complexity, detailing
scalable and modular structure based on the idea of functions, with the main goal to ease its
concurrent engineering and computer integrated implementations and business model. [15] This
manufacturing. [1, 2, 4-7] This approach eases the standard categorizes VC into 5 levels, 1 to 5, according
integration and replacement of production lines whilst to the functionality and accuracy of the solutions,
being flexible to disruptions and failures. This type of being level 5 the highest. This paper presents first the
simulation generates model-based copies, known as description of a level 5 VC Toolchain and the
digital twins, that can be defined as virtual automotive use case; secondly, the simulation and
representations of a physical assets enabled through results are studied; lastly, conclusions are discussed.
data for real-time prediction, optimization, monitoring,
controlling, and improved decision making [8].
Virtual Commissioning (VC) focuses on the 2. Virtual Commissioning Toolchain
optimization and validation of automatic systems,
increasing quality and efficiency in production As it was previously introduced, VC consists in
engineering whilst decreasing required time, through modeling functional and 3D kinematized models to do
using Smart Factories and commission them in a Verification and Validation (V&V) of automation
simulation environment. [4, 9] As mentioned in [1], hardware and software. The advantage of doing such
VC can be considered as the quality gate of the virtual engineering techniques, is that the V&V of
mechatronic, robotics and automation engineering automation solutions can be done offline by
results, and the first step on the development process connecting the simulation platforms to the operational
that ensures their interoperability. technologies, e.g., PLCs. Leading to not impact the
Increasing the production of complex and/or production in the assembly lines, since the production
customized products with short life-cycles, like in the process does not require to be interrupted. Depending

46
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

on the phases within the workflow from engineering the real world is expected to be in the control
until real commissioning, one might need to implement components area of the shopfloor.
different simulation technologies. Because of the
previous, we show an implementation of three
simulation approaches permitting to validate
requirements during different project phases. The
targeted architecture can be shown in Fig.1. The
modeling and simulation tools are based on:
 ControlBuild: For functional and behavioral
modeling of the mechatronic components of the
automotive assembly station
 Delmia: For modeling 3D kinematized
components.

Fig. 2. Automotive cell 2-D layout.

A brief recall of the processes performed in this

use case is shown in Fig. 3 and listed below:
 Start cycle: Absences of faults are acknowledged
in the station, afterwards, a signal is sent to start the
production process.
 Production mode: The production sequence is
launched. Signals are sent in parallel to the
different devices in the cell to begin their
processes.
 Switching car: The conveyor brings the car body
into the assembly area. The conveyor linear
velocity and position are controlled throught the
Fig. 1. Targeted Virtual Commissioning toolchains. use of position sensors and a motor drive unit.
 Check buffer: Right and left buffers rotate to bring
the body sides to the position where the two pick-
The operational technologies are based on Siemens
and-place robots can perform the assembly. The
and Schneider editors, with their appropriated
angular positions of the buffers are measured by
programming environments, respectively TIA Portal,
absolute encoders.
and Control Expert. To apply such global architecture
into the automotive use case, in the next section we  Side car positioning: The robots assemble the
show the application of the simulation platform for a body sides to the chassis. When the task is finished,
door assembly process of an automotive use case, the car continues along the conveyor to the
extensively described in [10]. Additionally, the three following process, outside of the current robotic
types of simulation approaches, i.e., Model-in-the- cell.
Loop (MiL), Software-in-the-Loop (SiL), and  Safety flag system: Safety sensors, emergency
Hardware-in-the-Loop (HiL), are shown, thus buttons, and circuit breaker systems are monitored
validating high and low-level automation and robotics during the whole process. This task possess the
requirements. highest priority of the whole operation, whenever a
single alarm and/or fault is detected, the process
enters in safety mode and stops the production.
3. Simulation and Testing Specification

The automotive assembly cell consists of a smart

conveyor (used to carry the car body through the cell),
a couple of robots (LRo and RRo), buffers (LB and
RB), positioning sensors (WA1 and WA2), safety gate
sensors (LSG and RSG), and safety barrier sensors
(SLBX). All of those components are arranged in a
mirror manner, as shown in Fig. 2. Control and power
electronic components, like PLC, drives, reles, robot
controllers, industrial cabinets, etc., physical models Fig. 3. Door assembly operation flowchart
are omitted in the 3D environment as their location in for a single cycle.

47
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

To V&V the automation process, we focused on the communication is done via a coupling driver
use of two validation environments: Siemens and respectively, using the PLCSimAdv API of Siemens,
Schneider automation software. According to Fig. 4, for TIA Portal, and Modbus TCP for ControlExpert.
there are two toolchains permitting to perform both Additionally, in order to associate the variables
SiL, and HiL techniques. On the upper right of the between the PLC and ControlBuild, output and input
toolchain in Fig. 4, the system to be commissioned is a mapping tables are created in order to properly match
Siemens virtual PLC 1515F-2 PN, and a virtual HMI the variables, ensuring thus, a bidirectional
KTP700F. These automation components are communication. It is worth noticing that safety and
connected to the simulation platform via shared control signals, as well as the information generated by
memory, permitting to exchange data bidirectionally, simulated sensors in Delmia and Controlbuild provide
to V&V the programs and routines computed in the the required data for closing the control loop with the
virtual PLC and HMI through SiL. On the bottom right PLCs (physical or simulated), meaning that the proper
of the Fig. 4, a physical HMI is connected to the mapping of these variables along the toolchain is in
simulation platform. In this case a virtual PLC high regard and data loss must be minimized.
Modicon 580 and a physical HMI Harmony STU855
from Schneider Electric are used. The communication
between the simulation platform and the execution
layer based on Schneider automation software is done
via Modbus TCP. From the system setup, it has been
accomplished the end-to-end communication and
digital continuity from the simulation to the execution
layer, permitting thus to improve the commissioning
process for an automotive scenario.

Fig. 5. Network setup for HiL with physical HMI.

4. Conclusions

VC implementations allow the user to develop,

simulate, and test different industrial scenarios. These
simulations can be as complex and detailed as the tests
required, always providing enough information for the
Fig. 4. System to be commissioned through MiL, SiL, V&V process. The automotive use case studied in this
and HiL. research was tested with SiL and HiL toolchains,
demonstrating the feasibility to simulate the operation
In terms of network and infrastructure architecture, with different control components without losing any
Fig. 5 shows a schematic of the communication of its features. It is worth noticing that the 3D
protocols among the several components. The simulation environment requires a computer with
communication between the simulation PC and the enough graphical and processing resources to perform
physical HMI is done via a physical connection to the a fluid simulation, being this the biggest bottleneck of
local network of both components establishing an the toolchain. Regarding the control devices and
exchange through Modbus TCP. The virtual PLC runs software, due to the fact that the process was well
in the simulation PC, which is virtually connected to analyzed and interpreted, the logic behind the solution
ControlBuild via Modbus TCP. Finally, the was implemented in an optimal manner, allowing the
communication between ControlBuild, and Delmia, virtual PLCs to run it without any setback.
for the visualization of the 3D kinematized
components is done via FMU.
Fig. 6 permits to visualize the mechanism of Acknowledgements
variable exchange performed between the PLC’s
(From TIA Portal and ControlExpert) and This work was conducted and funded with the
ControlBuild. It is worth noticing that the support of the Research Tax Credit (CIR in French).

48
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 6. Communication and mapping.

References International Conference on Emerging Technologies

and Factory Automation (ETFA), 2019, pp. 820-825.
[1]. S. Weyer, T. Meyer, M Ohmer, D. Gorecky, D, Zühlke, [10]. A. Kampker, S. Wessel, N. Lutz, M. Reibetanz
Future Modeling and Simulation of CPS-Based and M. Hehl, Virtual Commissioning for Scalable
Factories: An Example from Automotive Industry. Production Systems in the Automotive Industry:
IFAC-PapersOnLine, Vol. 49, Issue 31, 2016, Model for evaluating benefit and effort of virtual
pp. 97-102. commissioning, in Proceedings of the 9th International
[2]. C. Scheifele, A. Verl, O. Riedel, Real-time co- Conference on Industrial Technology and
simulation for the virtual commissioning of production Management (ICITM), 2020, pp. 107-111.
systems, Procedia CIRP, Vol. 79, 2019, pp. 397-402. [11]. R. Balderas-Hill, J. Delbos, S. Trebosc, J. Tsague,
[3]. S. Süß et al., Test methodology for virtual G. Feroldi, J. Martin, T. Master, N. Lassabe,
commissioning based on behaviour simulation of Improving interoperability of Virtual Commissioning
production systems, in Proceedings of the 21st IEEE toolchains by using OPC-UA-based technologies, in
International Conference on Emerging Technologies Proceedings of the 26th IEEE International Conference
and Factory Automation (ETFA), 2016, pp. 1-9. on Emerging Technologies and Factory Automation
[4]. T. Lechler, E. Fischer, M. Metzner, A. Mayr, (ETFA), September 2021, Vasteras, Sweden,
J. Franke, Virtual Commissioning – Scientific review pp. 01-07.
and exploratory use cases in advanced production [12]. S. Makris, G. Michalos, G. Chryssolouris, Virtual
systems, Procedia CIRP, Vol. 81, 2019, Commissioning of an Assembly Cell with Cooperating
pp. 1125-1130. Robots, Advances in Decision Sciences, Vol. 2012,
[5]. S. T. Mortensen, O. Madsen, A Virtual Commissioning 2012, Article ID 428060.
Learning Platform, Procedia Manufacturing, Vol. 23, [13]. Ružarovský, Roman, Holubek, Radovan
2018, pp. 93-98. and D. Sobrino, Virtual Commissioning of a Robotic
[6]. T. Breckle, M. Kiesel, J. Kiefer, N. Beisheim, The Cell Prior to its Implementation Into a Real Flexible
evolving digital factory – new chances for a consistent Production System, Research Papers Faculty of
information flow, in Proceedings of the Conference on Materials Science and Technology Slovak University
Intelligent Computation in Manufacturing of Technology, Vol. 26, no. 42, 2018, pp. 93-101.
Engineering, Naples, Italy, 2018, p. 252. [14]. A. Philippot, B. Riera, V. Kunreddy, S. Debernard.
[7]. D. Sobrino, R. Ružarovský, R. Holubek, K. Velíšek, Advanced Tools for the Control Engineer in Industry
Into the early steps of Virtual Commissioning in 4.0, in Proceedings of the IEEE International
Tecnomatix Plant Simulation using S7-PLCSIM Conference on Industrial Cyber-Physical Systems
Advanced and STEP 7 TIA Portal, in Proceedings of (ICPS), 2018, Saint Petersburg, Russia, pp. 555 – 560.
the Modern Technologies in Manufacturing (MTeM [15]. Albo, Anton & Falkman, Petter. A standardization
2019) MATEC Web Conf., Vol. 299, 2019, 02005. approach to Virtual Commissioning strategies in
[8]. A. Rasheed, O. San, T. Kvamsdal, Digital twin: complex production environments, Procedia
Values, challenges and enablers from a modeling Manufacturing. 51, 2020, pp. 1251-1258.
perspective, IEEE Access, 8, 2020, pp. 21980-22012.
[9]. A. Fernández, M. A. Eguía and L. E. Echeverría,
Virtual commissioning of a robotic cell: an educational
case study, in Proceedings of the 24th IEEE

49
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(012)

A Model Driven and Hardware Agnostic Approach

of Virtual Commissioning
S. Marchand, H. Alhousseini, R. Bres, F. Dumas, M. Lachaise,
L. Poulet de Grimouard and M. Stieglitz
Capgemini Engineering, ER&D, 350 Avenue JRG GAUTIER de la LAUZIERE,
13593 Aix-en-provence, FRANCE
Tel.: +33 4 84 93 44 76
E-mail: [email protected]

Summary: With the possibility to interconnect almost every systems, the Industry 4.0 has brought a lots of possibilities to
improve lead-time and cost management while dealing with the design or the evolution of assembly lines. New tools and
concepts are developed, such as Virtual Commisionning or digital twins, to improve the process to upgrade an existing
assembly line or create a new one. These tools allows to design, simulate and validate changes without implying costly and
time-consuming tests on real infrastructures. This paper describe an approach, and its implementation, to facilitate the use of
Virtual Commisionning tools. It uses standard OPC UA communication and information modeling to implement a model
driven and hardware agnostic approach to be the boundary between the virtual and the physical world. The solution can reuse
models, such as AutomationML, from the design phase, can be used with simulation tool to develop and test the
implementation in a virtual environment, and be used in the target hardware infrastructure for integration tests.

Keywords: Cyber-physical Systems, Model transformation, Design principles in Industry 4.0, Smart Factories, Smart
Manufacturing and Technologies, Digital Production and Virtual Engineering.

1. Introduction  The message model to interact between

applications.
Smart factories and Industry 4.0 have enhanced a  The communication model to transfer the data
lot connectivity in factories. The goal is not only to between end-points.
connect production systems together, but to share all  The information model to represent structure,
their knowledge with other systems. Using the data, behaviour and semantics.
failure can be anticipated with predictive maintenance,  The conformance model to guarantee
changes can be tested on virtual environments, etc. All interoperability between systems.
those new possibilities lead to substantial decrease in These four pillars are the base of the OPC UA
risks and costs compared to traditional development, interoperability standard. They define the standard
testing, and commissioning phase. way to exchange data between applications, and the
From an engineering point of view, the data way to universally model it.
collected is a goldmine. It can be used to reproduce Moreover, OPC UA has been proven to be a secure
real-world problems in digital twin and continuously protocol [5] and it is more and more integrated as a
improve processes. standard way to acquire data by software editors.
However, some issues are yet to be solved to fully
take advantage of all these data. First of all industrial
communication protocols are numerous, among which 1.1. Data Acquisition Using OPC UA Aggregation
PROFIBUS, MODBUS, Fieldbus, HART, ASi, Server
LonWorks, DeviceNet, ControlNet, CAN Bus, and
Industrial Ethernet are the most famous according to As stated by its specification [4], OPC UA is
[1, 2]. Moreover, those protocols are often purely designed to be a platform independent, reliable and
industrial protocols, not designed to go out of the secure protocol which can be used from plant-floor
factory, which causes side effects, such as poor PLC (Programmable Logic Controller) to enterprise
security handling. servers. Its conformance units and profiles
Then modeling has to be handled in a unified way, organization [6], allows it to be scalable from an
so that every system speaks the same language. Once embedded sensor to an enterprise server. For these
again, the variety of languages to model assets is wide, reasons, OPC UA has been widely adopted by device
with for example UML, SysML, AutomationML [3], vendors (PLC manufacturers, robots manufacturer,
OWL, etc. etc.) as well as industrial software platforms: SCADA
These facts tend to be solved by the maturation of (Supervisory Control And Data Acquisition), MES
OPC UA (Open Platform Communication Unified (Manufacturing Execution System), ERP (Enterprise
Architecture). Indeed, OPC UA standard defines in its Resource Planning), etc. OPC UA is now part of the
specification [4] 4 axes: RAMI4.0 (Reference Architecture Model for Industry

50
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

4.0), as the preferred protocol for its communication the same manufacturer, they will all define the type
layer [7]. representing this robot. The first approach would be to
Modern factories can now use OPC UA as differentiate types depending on their source server.
communication protocols at every level of the process. Using this approach, there is no risk to mix different
Recent PLC are OPC UA servers, gateways exist to versions of the same type, or 2 types named the same.
transform classic industrial communication protocol to The second one would be to merge common types,
OPC UA, and software can be fed with data using their because the type of the 3 robots is the same, it should
OPC UA clients. For those reasons, data acquisition be defined and handled so. Both approaches are
using OPC UA looks like a foregone conclusion. possible based on configuration of the aggregator.
OPC UA aggregation servers have been studied for
a few years now, and several sample exist to serve as a
starting point for data collection. As described in [8] 1.2. Data Modeling
several challenges must be faced to successfully setup
an OPC UA aggregation server. The steps identified IEC62541 defines OPC UA’s meta model [4, 9].
for a generic approach of aggregation are: This meta model defines the way to represent data
1. Type aggregation: consists in getting all the types using an object oriented model, see Fig. 1. In OPC UA
from an underlying server. To do so, the Types model, objects are composed of objects, variables and
node of the underlying server is browsed methods. The model describes types as well as
recursively, and the result is merged with already instances.
known types in the aggregation server.
2. Instance aggregation: consists in getting all
instances in the underlying server. Nodes
aggregated from source servers are then evaluated
with specified rules to be put in the right place in
the aggregation server’s address space.
3. Service mapping: In the proposed architecture, the
service calls on the aggregation server are
forwarded to the source server on which the node
concerned by the call is located.
This generic approach lays the foundations of the
construction of an aggregation server, and is used in
implementations, such as the sample application of the
OPC foundation. However, it has some drawbacks:
 In a vertical multi-level aggregation, the Fig. 1. OPC UA Object model.
propagation of the service call from the client to the
base source server might be long and cause latency The set of existing types and instances are
or timeout. represented in a server as a graph, with nodes (types
 There is no local version of nodes. Every request and instances) bound by references, see Fig. 2. Nodes
must be forwarded to the underlying source server. have attributes, that define what they are, and
If a group of nodes must be read by several clients, references that link them to other nodes.
it would generate a lot of unnecessary Read
requests to the underlying server. That can be a
problem on some industrial networks, which
should not be saturated.
To deal with these problems, a second kind of
aggregation (Data Warehouse mode) has been
implemented, based on the OPC foundation’s sample
aggregation server. Basic steps defined previously still
remain true, the server starts by the discovery of types
contained in underlying servers, then it gets instances,
and services are mapped with underlying server’s
services when it is necessary. Nodes are copied in the
aggregation server’s address space, and subscriptions
to corresponding nodes are created. Doing so, some
services, such as Read do not need to be mapped, and
propagated. For some services like method call, there
is no other way than propagating the call to obtain the Fig. 2. OPC UA Node model.
desired behaviour.
Another limit to be tackled with is the redundancy An OPC UA server can load models using 2
of namespaces. Some namespaces are defined by standard methods. The first consists in loading a
vendors, and duplicated in each of their devices. For standard Nodeset2 file that describe the address space
instance, if a shop floor is equipped with 3 robots from of a server. The second is based on code. UA server

51
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

solutions generally comes with a code generator that paradigm. The use of AML format by OPC UA server
converts models to usable code (specific to the server). has been standardized [12, 13]. It defines a way to
Both ways are complementary and have pros and cons. transform AML file in OPC UA model loadable by an
Modeling is an important point of OPC UA OPC UA server.
specification and its meta model allows users to model However, the wide variety of existing models,
any of their industrial assets. On top of the meta model whether a standard translation exist or not, are only
defined by the OPC UA specification, some working models and do not define how the data and the model
groups have formed to standardize the use of OPC UA are bond.
modeling in their fields. Those initiative gave birth to
companion specifications. The list of these
specifications [10] includes several information 2. Architecture
models aiming at the description of machines such as
OPC UA for Machinery, OPC UA for Robotics, OPC To try to go a step further and solve issues
UA for MachineTools, etc. Some of these models are mentionned in Part 1, using a generic approach, the
dedicated to modeling language adaptations such as I4SSM (Industry 4.0 Semantic and Security
OPC UA for I4 Asset Administration Shell, Management) has been developed. The main piece of
ISA-95 Common Object Model, OPC UA for the solution consists in an OPC UA server, which
AutomationML, etc. implements an architecture with 3 functions, described
AutomationML (AML) is a data exchange standard in Fig. 3:
[11] used for production system engineering. It gives a  Aggregation.
way to exchange data between applications to model  OPC UA operation server.
the system and represents it using an object oriented  Semantic layer.

Fig. 3. I4SSM architecture.

2.1. OPC UA Operation Server The operation server also manages the loading of
models. They can be loaded using 2 formats: Nodeset2
The operation server is a standard OPC UA server or compiled libraries.
which exposes and manages interfaces to configure the The main difference with state of the art servers
aggregation module and the semantic layer. [14] relies in the fact that an interface and reflection
The operation server handles OPC UA client calls mechanism is used here so that a compiled model can
and handles it with the help of other components if be used without having to rebuild the whole server.
needed.
Acquisition of data is managed by the aggregation
module, which discovers and updates nodes from 2.2. Aggregation Layer
underlying servers. These nodes are exposed to clients
using the operation server. The link between both The aggregation layer of I4SSM is designed to be
modules is done via an internal event and messages generic. The implementation of the aggregation
mechanics. module is based on the conclusions drawn from

52
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Part 1.1. This article focuses on the Data warehouse particular aspect, a collaboration with Intel has enabled
mode as default implementation, which creates a local the use of different hardware configuration enhanced
equivalent to nodes discovered from underlying data with software or hardware add-ons and accelerators.
sources. Moreover, the aggregation module exposes an Some common actions have been implemented on a
interface to develop specific acquisition drivers, in test environment facilitating test execution. This
case a non OPC UA connection to data source is environment enabled the possibility to test the
required. solution’s scaling, both horizontally (more aggregated
As for previous implementation [8], the servers) and vertically (more layers of aggregation
aggregation server is an OPC UA client, and uses servers) to ensure the possibility to use the solution
Browse standard service to discover its underlying with an architecture as close as possible from real-life
source server address space. It begins with the applications. The test of the functionalities of I4SSM
discovery of types and continues with instances. both separately and in parallel has given feedbacks on
Discovered types are added to aggregation server’s the implementation of the solution. For instance
types, i.e. they are added in the aggregation’s server Table 1 shows improvement of performances between
Type tree. Then the server creates a copy of underlying campaign 1 and 2, concerning the number of servers
servers instances, and organizes it in the aggregator’s that can be aggregated. As a result, the feedback on the
address space by putting them in folders corresponding first 2 tests campaigns has reinforced the need to
to their source server, as shown in Fig. 4. include in the project life cycle the stress tests to
anticipate performances issues and find bottlenecks in
order to facilitate code optimization by design.

Table 1. Success rate of aggregation tests for N source

server with 5k variables each.

5K VARIABLES
Source servers 1st campaign 2nd campaign
1 100 % 100 %
2 100 % 100 %
3 60 % 100 %
4 0% 100 %
5 0% 100 %
10 0% 100 %
15 0% 100 %

2.3. Semantic Layer

The Semantic layer plays a major role in I4SSM

Fig. 4. Address space organization. architecture. It implements the link between models
loaded using the operation server and data acquired
using the aggregation layer.
The discovered nodes are assigned a new and To execute this task, in a generic, and as standard
unique NodeId in the aggregation server. A mapping as possible way, the OPC UA meta model is used to
of the local nodes with remote nodes (in the underlying define what a SemanticLink is. The address space of
server) is maintained, so that the remote node can be the server is created with a SemanticLinkSet object
accessed for service calls that cannot be fulfilled which lists all semantic links instanciated in the server
locally. Subscriptions are created by the client and has methods to add or remove links, as shown in
embedded in the aggregator to update local nodes. The Fig. 5.
Sampling interval of monitored items and Publish The model for semantic link definition is
interval of the subscription are configurable. extensible, to do so a base abstract type defines
Once the discovery is completed, Read, Browse common parameters of all semantic link types. This
and subscription can be made on the aggregator base type is defined in Table 2. A semantic link makes
address space by a client, without interacting with the a link between at least 2 variables, a source and a
underlying server. However, services like Write and target. The relation can be one to one, or many to
Call must be forwarded to the underlying server to be many. A link can define simple links, such as “this
effective. Indeed the writing of a parameter in an target variable represents this data”, it can also do some
underlying server must be done in the real server to be operation on the fly to specify that “this target variable
effective. is the same as this source data with a different unit”. It
Because plants and shopfloor are dealing with can also be used to perform basic arithmetic
potentially important data volume, scaling up is a key operations, such as addition, power of ten, or affine
point to manage Data Model as well as upload and function. Moreover, an interface is available for a user
download information to the shopfloor. To address this to extend the set of links, by adding its own types of

53
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

link and implementing its associated behavior. This still be up and working in case of a server restart.
mechanism is implemented so that a simple However, this matter is not trivial to solve because of
configuration of the server is needed to add a type of the automatic discovery of underlying servers address
link to the server, no rebuild of the server is required. spaces.
Nodes discovered in underlying sources are
mapped with a local node, which NodeId is uniquely
generated in the server. The mapping table and the
NodeIds must be persisted, so that it can be retrieved
after a restart or a crash. It is important, so that existing
semantic links can find their sources and targets back.
The same is also true for external client applications
that often store NodeIds and base their internal
mappings on that.
To be able to set the server in its prior shutdown
state, the following items of interest have been
identified:
 Nodes added with the AddNode service.
 Nodes updated with the Write service.
 References added with the AddReferences service.
 User roles.
 Sematic links.
 Aggregated nodes’ NodeId mappings.
These elements are identified as key elements that
Fig. 5. SemanticLinkSet in I4SSM address space. need to be persisted. Nodes added to the server must
be recreated, as well as the references. Updates on
nodes must be persisted to be able to retrieve
Table 2. SemanticLinkType definition. parameters as they were before the stop of the server.
For example a parameter that defines an alarm
SEMANTICLINKTYPE ATTRIBUTES threshold should not be rewritten by the user when the
Attribute Value server restarts. The same idea is also true with user
BrowseName SemanticLinkType roles configuration.
IsAbstract True
SEMANTICLINKTYPE REFERENCES
Modelling
3. Use case: Virtual Commissioning
References Node Class BrowseName
Rule and Automotive Cell
SubtypeOf ObjectType BaseObjectType
3.1. Automotive Cell Components and Process
HasComponent Method AddSource Optional
HasComponent Method AddTarget Optional
The Virtual Commissioning simulation platform,
HasComponent Method RemoveSource Optional described in [15], is composed of two software
HasComponent Method RemoveTarget Optional allowing to perform functional models of mechatronic
components of the station (ControlBuild), and a 3D
visualization environment for the kinematized
The modeling and implementation using OPC UA components (3DX Delmia). Additionally, these two
meta model offers 2 major advantages: a standard way simulation tools are connected via OPC UA to the
to design and model semantic links, and the possibility I4SSM server introduced previously. Fig. 6 shows an
to use a simple OPC UA client to add, remove and overview of the proposed toolchain.
modify semantic links in the server. Indeed, any OPC
UA client that can perform method call is able to
manage semantic links. That way, scripting can be
used to massify the creation of links, to fit better with
real use case in which thousands of semantic links are
needed to map data from a production line to its high
level model.

2.4. Fault Recovery and Persistency

Fault recovery and persistency are major concerns

for I4SSM. First of all, after having done the mapping
of thousands of nodes, a user does expect that it will Fig. 6. Toolchain.

54
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

The automotive cell consists in several components This use case demonstrates the possibility to create
allowing to emulate an assembly process for the body a digital twin of an automotive assembly line, using
sides of a car. The assembly station is composed of OPC UA and taking advantage of a unified model. In
mechatronic components, such as robots, sensors and this scenario, Delmia is responsible for the 3D
safety components, as shown in Fig. 7, and listed as visualization of the line, ControlBuild manages its
follows: simulation. I4SSM is responsible for implementing the
 Car body hardware abstraction layer, by the exposition of a
 Car conveyor model that the simulation tool can understand, and by
 Rotational buffers linking this model to the real data. With that
 6-Degree-of-Freedom robots abstraction, simulation can be performed and
 Presence sensors visualized in 3D, moreover tests can be done using
 Safety components, such as safety gates and light virtual or real PLCs, without changing the
barriers. configuration of the simulator. Using this, engineers
can virtually simulate a change of parameters, validate
a new PLC program using virtual PLC and even test a
modification on the physical environment.

4. Conclusions

This paper presents a way to deal with the

boundary between virtual and physical world and
illustrates it in a practical use case. The I4SSM tool set
presented is implemented and running following the
ideas described in the article. It implements the basic
OPC UA server aggregation needed to get data, it adds
on top of this a set of objects to makes it possible to
Fig. 7. Automotive station. link acquired data with models, in a generic server,
without need to rebuild it when the model evolves. It
also offers several possible ways to load models, using
Additionally, the variables governing the full AML to Nodeset2 converter, custom converter, or
assembly process are described by various datatypes, loading libraries built from models and tools made
thus allowing to show the cyber-model translation available on GitHub by the OPC Foundation [16].
capabilities in what follows. The various types data Additionally to the classical acquisition server use
translated into an OPC UA model contains therefore case, I4SSM can be used in the virtual world, or as a
information related to sensors, robots positions and boundary between virtual and physical environment,
velocities, allowing the station to operate adequately. as proven with the automotive cell use case. However,
I4SSM can still be improved and completed. Next
steps would be to use it OPC UA Historical Access
3.2. Model Translation and Mapping (HA) functionalities to replay some failure cases in
collaboration with a simulation tool, performances
The model loaded in I4SSM is translated from a could be improved again to be able to deal with even
format exported by ControlBuild to a Nodeset2 format. bigger amount of data, and the implementation of the
Doing so, ControlBuild can be interfaced with the OPC PubSub OPC UA specification would open new
UA server using its OPC UA client and connect its possibilities, such as edge to cloud communication and
internal variables to variables in I4SSM. aggregation.
ControlBuild can then be used as a simulation and
validation tool without having to bother about
underlying implementation. That is to say that the Acknowledgements
simulation can be run with I4SSM only, with virtual
Programmable Logic Controller (PLC), or using real This work was conducted and funded with the
PLC. Since ControlBuild is mapped with the loaded support of the French tax system, thanks to the
model, nothing has to be done at its level to switch Research Tax Credit (CIR in French).
from a full virtual test to a test implying real hardware. We would like to thank INTEL, and especially Jan
If PLCs (whether virtual or physical) are used, VAN OFFEREN for the support, the test platform and
I4SSM aggregation layer copies their address space. help on performance improvement. We also thank
That way, PLCs can be from different vendors, Jacques MEZHRAHID, Nicolas NGUYEN, Eric
implying potentially different address spaces, without OURSEL and Sylvain PLAZANET for their support
consequence for ControlBuild. Using built-in and expertise. Finally, we would like to thank Rafael
SimpleMappingSemanticLinkType offered by I4SSM, BALDERAS HILL, the Virtual Commisionning team,
the model suiting ControlBuild can be linked with data and N. LASSABE for their expertise and their support
discovered automatically from underlying sources. on the redaction of this paper.

55
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

References [11]. Rainer Drath, et al., AutomationML in a Nutshell, in

Handbuch Industrie 4.0, Vogel-Heuser B.,
[1]. P. Bondeson, S. Liss, Roadmap of virtual Bauernhansl T., ten Hompel M. (Eds.), Springer
commissioning inertia, Thesis, KTH Royal Institute of Reference Technik. Springer Vieweg, Berlin,
Technology, Stockholm, Sweden, 2018. Heidelberg, Bd. 2, pp. 213-258.
[2]. I. Gonzalez, A. J. Calderon, A. Mejias, J. M. Andujar, [12]. DIN SPEC 16592, Combining OPC Unified
Novel Networked Remote Laboratory Architecture for Architecture and Automation Markup Language,
Open Connectivity based on PLC-OPC-LabVIEW- December 2016.
EJS Integration. Application in Remote Fuzzy Control [13]. Robert Henßen, Miriam Schleipen, Interoperability
and Sensors Data Acquisition, Sensors, 16, 11, 2016, between OPC UA and AutomationML, Procedia
p. 1822. CIRP, Vol. 25, 2014, pp. 297-304.
[3]. Lüder A., Hundt L., Keibel A., Description of [14]. Stefan Profanter, Benedict Simlinger, From modelling
manufacturing processes using AutomationML, in to execution – OPC UA Information Model Tutorial
Proceedings of the 15th IEEE Conference on Emerging (https://opcua.rocks/from-modelling-to-execution-
Technologies and Factory Automation (ETFA), 2010, opc-ua-information-model-tutorial/), 2020.
pp. 1-8. [15]. R. Balderas-Hill, J. Delbos, S. Trebosc, J. Tsague,
[4]. OPC UA - Part 1: Overview and Concepts Release 1.04 G. Feroldi, J. Martin, T. Master, N. Lassabe,
Specification, 2017. Improving interoperability of Virtual Commissioning
[5]. Federal Office for Information Security (BSI), OPC toolchains by using OPC-UA-based technologies, in
UA Security Analysis, 2017. Proceedings of the 26th IEEE International Conference
[6]. OPC UA - Part 7: Profiles Release 1.04 Specification, on Emerging Technologies and Factory Automation
2017. (ETFA), September 2021, Vasteras, Sweden,
[7]. Platform Industry 4.0, RAMI4.0 – a reference pp. 01-07.
framework for digitalisation, 2018. [16]. OPC foundation GitHub,
[8]. Johansson Markus, Vyatkin Valeriy, Aggregating OPC https://github.com/OPCFoundation
UA Server for Generic Information Integration, 2017.
[9]. OPC UA - Part 3: Address Space Model Release 1.04
Specification, 2017.
[10]. OPC Foundation website (https://opcfoundation.org)

56
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(013)

Development of an AI Maturity Model for Small

and Medium-sized Enterprises
B. Schmidgal, M. Kujath, S. Kolomiichuk, M. Rentzsch and S. Häberer
Fraunhofer Institute for Factory Operation and Automation IFF,
Sandtorstr. 22, 39106 Magdeburg, Germany
Tel.: + 49391 4090140
E-mail: [email protected]

Summary: The relevance of Artificial Intelligence (AI) for manufacturing and service operations in the industrial sector is
growing rapidly. Various economic actors, including small and medium-sized enterprises (SME) that are the backbone of the
European economy, are already well aware of specific use cases of AI in the industrial context and demand effective AI
solutions to their challenges. However, existing approaches and models for identifying the appropriate AI use case for
efficiency improvement potentials can cover the growing market demand to a limited extent only. Uncertainties about the
actual capabilities, a lack of a strategic focus, missing success metrics, poor data quality, or false expectations of AI systems
can all lead to failure of AI projects. To counter this, an AI maturity assessment can assist companies to overcome challenges
on the way to a sustainable planning, deployment and adaptation of AI in industrial applications. This paper examines the need
for a holistic method and presents the Fraunhofer IFF AI Maturity Model – a method based on the needs of SME in the
manufacturing industry.

Keywords: Artificial intelligence, AI-supported business model innovation, Use case-specific assessment, Maturity level,
Feasibility analysis, Decision support.

1. Introduction in large and multi-national companies and can infer the

wear and tear of machines from certain data such as
Artificial intelligence (AI) is regarded as an pressure, temperature, acoustic noise, power
important technology that is indispensable for consumption or vibrations. Business-relevant patterns
preserving Germany's economic performance. The and key figures can be derived from this and used as
German government has recognized the relevance of decision support for technicians.
this key technology as well as its potential for
additional economic growth, and has developed a 2. Background and Motivation
framework for action to promote and fully exploit this
potential. Essential framework conditions have been While certain AI applications, such as interaction
merged into a national “AI Strategy”, which is with customers (chatbots) and automated data
designed as a learning strategy that needs to be collection and evaluation have become almost routine
continuously readjusted jointly by politics, science, in the industrial context, German SMEs are reticent
industry and civil society (The German Federal yet. A survey on the use of AI conducted by Bitkom
Government 2020). In particular, the course is to be set e.V. in 2020 – the industry association of the German
for the expansion of competencies and expertise, the information and telecommunications sector – shows
promotion of young researchers and knowledge that 9 percent of German industrial companies with
transfer, as well as for modern research infrastructures 100 to 199 employees use AI in their company in the
and international networking. Thus, AI is expected to context of Industry 4.0, whereas 11 percent of German
not only change the way people work, but also the industrial companies with 200 to 499 employees use
individuals’ everyday life and society in general. AI (Bitkom 2020). Consequently, one can estimate an
Ministries, economic actors and major German adaptation rate of AI in the industrial context in
associations agree: A successful digital transformation German SMEs of around 10 percent.
is of existential importance for prosperity, growth and However, the same survey with same respondents
innovative strength. By leveraging large data sets (Big revealed that the benefits of AI in the context of
Data), businesses are increasingly striving to create Industrie 4.0 are widely known and companies
new value for their clientele and develop new, recognize economic potentials. According to the study,
innovative business models (German Engineering 43 percent cited that AI can provide advantages for
Association 2020). AI-driven business model predictive maintenance as the most important benefit
innovation already extends to various industrial of AI in the context of Industry 4.0, followed by
application fields, such as warehousing and sorting by productivity enhancement, and optimization of
autonomous vehicles, analytics for product production and manufacturing processes with 41 and
development, or quality control by means of AI-driven 39 percent accordingly. Hence, even though the
image recognition in the field of machine and plant advantages and use cases are mostly known, AI-based
manufacturing. Learning algorithms are already used solutions are still barely used.

57
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

As one of the objectives of application-oriented which existing or easily captured customer data or
research, it is necessary to identify the gap between the process data combined with AI methods form the basis
state of knowledge and state of implementation of AI for the development of new value propositions. If these
and its economic possibilities within the SME are integrated into corresponding revenue mechanics
landscape. This gap is due to existing barriers to use and value chains, innovative, AI-based business
AI. A survey of AI experts by The Scientific Institute models can be developed.
for Infrastructure and Communication Services However, to realize such business models, in
revealed that a lack of expertise and skilled workers is addition to domain knowledge – the knowledge of how
a strong (36 %) or very strong (64 %) obstacle to the the industry works, what customers want, and the
implementation of AI solutions in SME established processes – methodological knowledge in
(Wissenschaftliches Institut für Infrastruktur und the context of AI is required above all. Therefore, it is
Kommunikationsdienste 2019). This reflects the extent necessary to identify the status quo of the resource
of the general shortage of skilled workers in SME: IT endowment and requirements typical for the company
specialists in particular are rare on the labor market. size class. Therefore, the Fraunhofer IFF developed a
SME are often unable to keep up with the salaries paid third-party assessment of the current maturity level in
by large companies. order to derive necessary capacities and resources for
Inadequate availability of data in sufficient empowering employees and exploiting potential by
quantity and quality is another major obstacle. Due to means of AI. The assessment was designed with the
the smaller size of the company, the possibilities for claim to be objective, holistic and realistic. The
data collection and data utilization are more limited maturity model as a component of a use case analysis
than in large companies. Many of the SME are not is predominantly aimed at companies in the
sufficiently digitized to ensure an adequate collection manufacturing sector.
of production data and machine data, among other First of all, it is important to understand why the
things, by means of software and sensor technology. maturity model approach was preferred for making
Since the presence of suitable datasets along the value companies capable of acting in the field of AI. On the
chain is a key prerequisite for training AI systems, one hand, maturity models can be used to describe the
successful deployment of AI is directly dependent on change in the object under consideration (assessment)
the company's level of digitization. Data security in order to derive recommendations for action based on
concerns, lack of acceptance among employees, and this, which are aimed at achieving the next higher
limited financial resources are further obstacles that level. In addition to performance evaluation and
were also identified and analyzed by AI experts and improvement, maturity models are also used for
trainers. internal and external comparisons. They provide a
benchmark for best-in-class companies or the entire
industry (Bruin et al. 2005). In the corporate context,
3. The AI Maturity Model maturity models can be helpful tools, especially for the
responsible managers, to be able to classify the current
Based on these findings, concrete needs were development status of the company in relation to the
defined, which are to be covered by a methodical past and the future. Maturity models focus the
approach. The goal is to derive a standardized, holistic assessment on a specific area, but can be applied to a
procedure for identifying relevant use cases for AI broad field (Häberer et al. 2017). Figure 1 depicts the
deployment and providing a decision template. Fraunhofer IFF AI Maturity Model that represents an
Existing AI applications mostly focus on incremental development trajectory required to
increasing efficiency. However, AI offers SME greater achieve a networked and fully automated production
potential through service or product innovations, in and logistics environment with the help of AI use.

Fig. 1. The AI Maturity Model – five stages to AI maturity and their respective characteristics.

58
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

The model consists of five stages of AI integration: technologies; AI experts and interdisciplinary project
 Stage 1: Manual data processing structures.
 Stage 2: Networked data utilization Example: Commercial vehicle manufacturer
 Stage 3: Integrated intelligence collects process data at every step of the manufacturing
 Stage 4: Smart data process. Data pre-processing and initial analysis takes
 Stage 5: Factory of the future place on the edge. Through standardised data exchange
The following stage characteristics were interfaces, information is exchanged between the shop
established to make a clear differentiation between the floor and central PPS system. The collected data is then
individual maturity levels so that the differences to the processed in the cloud. AI is used to forecast
next higher or next lower stage are distinct: completion times, delays and possible disruptive
1. Manual data processing: Recurring and events in production. The algorithms suggest the
manual data entry; Distributed, local data storage and optimal production planning and exchange the data
availability; Unstructured, non-value-adding data; with upstream and downstream processes (supply
Lack of a scalable system for deploying AI chain management, transport of finished products).
applications; Lack of AI strategy, skills, and resources Thus, the control of these processes is also data-based
in-house. on the actual state of production and existing
Example: A paper-based, non-automated data production state forecasts for defined forecast
capture and paper-based process control system in horizons.
screw manufacturing is present. In production 5. Factory of the future: Quantifiable value
planning and control, data evaluation is dominated by creation through data; Trustworthy and explainable AI;
humans and data availability is only given in individual Independent system monitoring and self-calibration;
processes. Enabling, interoperable and performant systems;
2. Networked data utilization: Structured and Industry-leading AI competencies and diversified AI
partially automated data acquisition; partially teams.
networked data storage; context-related and Example: High-tech electronic chip manufacturer
retrospective data evaluation; standards and optimized has fully automated production lines. Production
ICT; embryonic AI initiatives and resource allocation. planning is carried out by AI algorithms depending on
Example: A building materials manufacturer order situation, deadline chain, material availability.
considers its data to be relevant to the business because Quality control is carried out inline using current
data is evaluated on a rules-based approach and used production data, evaluation of images of the products
to support employees in their decision-making. The and semi-finished products by AI-based systems.
team leader in the construction department can Quality deviations are automatically compensated for
evaluate his shift planning in a partially automated and by the production control system. Human intervention
algorithm-supported manner and store it in a is only required in cases that are not clearly defined and
knowledge database, which the management can in exceptional circumstances. The supporting
access. processes (logistics, infrastructure) are also data-
3. Integrated Intelligence: Continuously driven and depend directly on the main process
networked data storage; Networked product and (production). Through the external data exchange,
process data; Central data management and production capacities as well as external service
availability; AI frameworks and scalable IT providers and suppliers are controlled across all
infrastructure; Internal AI competencies, specific AI locations.
budgets and project groups. Based on the assumption that one level of AI
Example: Data scientists at a turbine integration is characteristic for each stage of
manufacturing company use data mining analysis development, the individual AI levels of a use case,
techniques to evaluate and interpret extensive and with increasing maturity, merge into one another, but
complex data sets for error detection in production are not mutually exclusive. Therefore, the distinction
series. As part of a dedicated AI strategy, a data between AI levels for the respective use case levels
warehouse has already been built as an should be regarded as intersecting.
interdepartmental data storage and analytics platform. The main purpose is that an organization should not
Analysts in development, production, quality only discover at which level its areas of application are,
assurance and warranty processing have access to this but in particular what kind of solutions are goal-
data warehouse to perform ad hoc analyses and create oriented in order to get to the next higher level. This
standard reports supported by machine learning gives the organization an indication of what it needs to
algorithms. After particularly complex calculations, achieve if it wants to reach the next higher level of AI
the company can now forecast for each part and each maturity. Thus, it should not be the goal of an SME to
product variant the frequency with which certain achieve stage 5 but rather the next stage of
defects occur as a rule. development.
4. Smart data: Automated pipelines for data The maturity model does not fulfil the claim of
evaluation; Communicable, consistently available evaluating the entire company in terms of a uniform AI
data; Networked product and process structure and maturity, substantiated by a single numeral degree.
infrastructure; Configurable interfaces and cloud Instead, it aims to evaluate individual use cases within

59
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

the company, representing a business process or  What areas of application are there for AI in
function that describes a sequence of individual my company and where can AI specifically be
activities that are carried out successively to achieve a helpful?
business or operational result with a discernible value  Which AI tools exist and which of them can fit
(IBM 2017). As process-related components of an my business use cases?
organization, use cases are seen as complex workflows  What data do I need at all and how do I need it
that are intended to contribute to business success. For to train my AI?
this reason, the objective is to identify the Secondly, operational use cases are to be identified
economically most prospective business use cases for and prioritized. On-site inspections (plant visits) and
ensuring a successful and structured introduction of AI personal interviews form the basis for a structured
within a company. approach and solution development. The goal is to
successfully carry out a company-specific use case
4. The Fraunhofer AI CheckUp determination for various alternative actions. For this
purpose, a questionnaire based on the fields of action
The maturity model is one component of an overall of AI in the industrial context was developed to enable
service that aims to make manufacturing companies a systematic evaluation of AI use cases. Since the
ready for the use of artificial intelligence – the so- maturity model is adapted for the requirements, goals
called AI CheckUp. The AI CheckUp is a method and challenges of each individual use case to be
developed for the analysis of processes to identify assessed, an assessment is always use case-specific and
suitable use cases for AI-based solutions within SME. benchmarking of use cases does not appear to be
The AI CheckUp arose from the development need of purposeful. Therefore, use case-specific and general
a modular structure to cover different scientific and fields of action were developed that are relevant to
economic requirements for AI in an industrial context: cover a company’s needs for actions in a holistic way.
 Evaluation of the existing database (quality, The AI CheckUp focuses on the following seven fields
availability, etc.); of action of AI in the industrial context:
1. Quality of the collected data
 Identification of improvements in the existing
2. Handling / Processing of data
data management and utilization (problem
3. Infrastructure and access to technologies and
monitoring);
tools
 A credible evaluation system of use cases and
4. AI strategy and leadership
realistic references of AI solutions
5. Employees and AI acceptance
implemented to date;
6. AI evaluation and impact assessment
 Manufacturer-independent orientation and
7. Data sovereignty and collaboration
consulting;
Figure 2 depicts the seven fields of action that are
 Third-party assessment of the AI and data to be inquired and assessed by the experts. Each field
management competencies available in the of action has one or several central questions, intended
company through a holistic multi-method to guide the experts' data collection, but do not limit
approach (interviews, workshops, analysis of the field of action to these key questions alone. Based
the database, etc.); on the corresponding answers from the interviews with
 Estimation of quantitative benefits; management, skilled workers and specialists, as well
 Checklist with minimum requirements for a as the information gathered through observations and
project; process analyses during the plant tours, each of the
 Identify new research needs. seven fields of action can be evaluated and classified
The approach is to compare the as-is situation with according to the level characteristics of the maturity
the requirements that an AI-based solution requires, model at level 1 to 5 for the respective use case.
identify existing data sources, and design the to-be data Hence, the AI maturity level is determined. The
workflows for the AI-based solution. Eventually, an application areas are classified objectively in terms of
identification of required resources (professional, the requirements and prerequisites for AI. The aim is
technological, methodological) and well defined to realistically assess the potential of AI to achieve
success criteria is to be established. these goals. An effort-benefit assessment of the
The first step is to raise awareness and educate the measures is carried out so that problem definitions can
employees of a company around the topic of AI and be derived and aligned with the corporate strategy. The
clarify the potentials and restrictions of AI claim is a realistic assessment of where the identified
implementation. There is also a focus on the business and prioritized use cases of the company stand in
goals and strategy in the sense of concretizing what a relation to AI and which AI-supported solutions make
company wants to achieve independently of AI. economic sense. Based on the maturity degree of the
Experts in the field impart fundamental knowledge respective use case, necessary development steps can
about AI and provide answers to questions such as: be derived and prioritized, taking into account an
 Why should I get involved with AI? integrated risk assessment (such as dependencies on
 How do I find out whether AI is the right partners and suppliers).
technology for me?

60
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 2. Seven fields of action of AI and their key questions for information acquisition.

What follows is an AI use case development. Based forces) (Osterwalder 2010). Approaches for
on the information intake and the determined maturity optimizing the economic mechanics of a specific
degree, efficiency improvement potentials can be business use case can thus be determined and
evaluated and individual measures for increasing the developed. As a result, new product and service ideas
AI maturity of the respective use case can be derived. as well as efforts towards process innovation can be
The measures can be of technical, organizational or initiated. The AI Use Case Canvas consists of 14
methodical nature. Together with the employees, the elements and aims to holistically visualize a specific
experts derive operational use cases with potential for use case in the company:
increasing efficiency and reducing costs, and match the 1. Value proposition: It describes the essence of
problems with possible solutions (problem-solution- the use value that the use case provides to the user.
fit). 2. Internal customer segments: Involved,
Ultimately, a decision template is developed and affected or interested internal parties and
provided. For the purpose of decision support of the organizations, which exert influence on the use case
SME in possible investment projects, the Fraunhofer and influence each other (in particular own employees,
IFF developed a management tool for systematically shareholders and service providers).
developing identified use cases further, visualizing its 3. External customer segments: Target groups
elements, and serving the decision makers as a decision outside of the company to whom the use case is
support template – the so-called AI Use Case Canvas. intended to add value (in particular customers, the
Its development is based on the Business Model general public, authorities and lawmakers).
Canvas by Alexander Osterwalder which is an 4. Business model: The as-is logic of how a
established “strategic management tool to describe company generates added value and secures earnings.
how an organization creates, delivers and captures The revenue mechanism are in the foreground here.
value” (Osterwalder 2021). Comparable approaches 5. Data situation / data quality: The data
with similar structuring principles for business process situation / data quality describes how well data are
modeling of AI applications are available on the suited to fulfil their purpose for the use case.
market, such as OwnML on the website 6. Technology and infrastructure: The set of
www.ownml.co (OwnML 2022). The Business Model tools, technologies (software and hardware),
Canvas arranges the business model elements in nine infrastructure, and methods used to provide the use
basic building blocks and establishes a uniform case with the required resources and to achieve (create)
structure, with the aim to show the logic of how a the value proposition.
company intends to make money. Similarly, the AI 7. Domain and method knowledge: Existing
Use Case Canvas was developed to describe business domain knowledge of how the industry works,
use cases in a structured way. Decision makers can customer needs, and established processes, as well as
discuss, evaluate and develop the building blocks, and methodological knowledge in the context of AI, data
discover correlations between them. This can serve as management and business models.
a template for decision support for the further 8. Key partners: The partnerships that are
development of existing and future use cases. The tool created through networking with individuals and
can be integrated into the company's strategic planning organizations that can contribute to the business
and provides an overview of a company’s business use success of the use case.
case within its external environment (market forces, 9. Key resources: The key assets for the
industry forces, key trends, and macroeconomic successful implementation of the use case, or the

61
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

resources available for the use case with which measures and solutions that employees can trust and
operations can be executed effectively. accept. By means of the identification of business use
10. Key activities: Processes that are essential for cases for AI, the analysis of fields of action and a use
the success of the business model or use case. Key case-specific classification, an evaluation of efficiency
activities describe which activities are of particular improvement potentials can be made so that individual
importance for the success of the use case. measures for increasing the AI maturity can be
11. Risks: Risks associated with the AI use case derived. In the end, an SME can benefit from decision
project, possible negative outcomes, possibility of loss support including an effort-benefit analysis as tangible
and failure. The risk-based assessment of AI describes added value. Thus, in the context of digitization, SMEs
the extent and quality of addressing system criticality can undertake an incremental development of their
as an expression of the damage potential of an business model, with manufacturers evolving from a
algorithmic system. pure product sale to an as-a-service business model, for
12. Cost structure: The main expenses and costs example, and be able to keep up with multinational
associated with this use case (in particular fixed costs companies.
such as salaries, wages, rent, and infrastructure
expenses, variable costs (proportional to the volume of
goods or services produced), economies of scale, and Acknowledgements
economies of scope).
13. Revenue streams: Revenues that can be The development of the Fraunhofer IFF AI Maturity
generated from each customer segment, such as sales Model was supported by Christian Löwke, Andreas
of goods or services, user fees, membership fees, Herzog and Lina Lau. We would like to thank you for
leasing, renting, lending, royalties, and advertising your professional contribution.
fees.
14. Key performance indicators for success of
failure: Business metrics that reflect the success of the References
use case in various areas.
[1]. Bitkom. Industrie 4.0 – so digital sind Deutschlands
Fabriken, May 19, 2020, p. 8. https://www.bitkom.org/
sites/default/files/2020-05/200519_
bitkomprasentation_industrie40_2020_final.pdf,
Accessed: 14.11.2021, in German.
[2]. De Bruin, T., Rosemann, M., Freeze, R., Kulkarni,
U., Understanding the Main Phases of Developing a
Maturity Assessment Model, in Proceedings of the 16th
Australasian Conference on Information Systems
(ACIS), 2005 p. 3.
[3]. German Engineering Association (VDMA), Leitfaden
Fig. 3. The Fraunhofer IFF AI Use Case Canvas Künstliche Intelligenz – Potenziale und Umsetzungen
im Mittelstand, VDMA Bayern, Munich, Bavaria,
2020.
The AI Use Case Canvas is developed and https://www.vdma.org/documents/34570/1052572/Le
illustrated together with the managers and specialists itfaden+K%C3%BCnstliche+Intelligenz-
Potenziale+und+Umsetzungen+im+Mittelstand.pdf/c
of the company, so that decision makers always have e38a591-68cb-9775-101e-
the possibility to continue working with this structure d7cad064b149?t=1615364023575). Accessed:
and use it as a strategic decision template. The 07.01.2021 (in German).
objective is to accompany the SME in its [4]. Engineering Lifecycle Management / 6.0.4.
transformation into a future- and AI-capable company Anwendungsfälle definieren, IBM Corporation, 2017.
and to make it capable of acting in the field of AI in a https://www.ibm.com/docs/de/elm/6.0.4?topic=requir
number of steps. ements-defining-use-cases. Accessed: 07.01.2022
(in German).
[5]. Häberer S., Kujath M., Flechtner E., Industrie 4.0-
5. Conclusions CheckUp, wt Werkstattstechnik online, wt Jahrgang
107, 2017, Heft 03 , pp. 141-142.
[6]. OwnML, The Machine Learning Canvas,
In order for companies to profitably exploit the https://www.ownml.co/machine-learning-canvas
potential of new AI technologies, it is essential to Accessed: 13.01.2022.
determine the current state of its business use cases. [7]. Osterwalder, A., Strategyzer AG, Löwenstrasse
This is where the maturity model as a component of the 2 CH, 8001 Zürich, Switzerland 2021,
AI CheckUp can provide support for SMEs. The aim https://www.alexosterwalder.com/,
of the AI Maturity Model is to support companies in Accessed: 21.12.2021.
taking advantage of the economic opportunities [8]. Osterwalder, A., Pigneur, Y. Business Model
offered by AI and machine learning in particular. In Generation. A Handbook for Visionaries, Game
order to make the most of the potential, it is important Changers, and Challengers, John Wiley & Sons, 2010,
pp. 200-210.
to pursue a human-centric AI approach and to develop

62
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

[9]. The German Federal Government. Strategie Mittelstand – Relevanz, Anwendungen, Transfer.
Künstliche Intelligenz der Bundesregierung: WIK GmbH, Rhöndorfer Straße, 6853604 Bad
Fortschreibung 2020, 2020. https://www.ki-strategie- Honnef. 2019. https://www.mittelstand-digital.de/
deutschland.de/files/downloads/201201_Fortschreibu MD/Redaktion/DE/Publikationen/kuenstliche-
ng_KI-Strategie.pdf. Accessed: 07.09.2021 intelligenz-im-
(in German). mittelstand.pdf?__blob=publicationFile&v=5,
[10]. Wissenschaftliches Institut für Infrastruktur und Accessed: 14.11.2021 (in German).
Kommunikationsdienste. Künstliche Intelligenz im

63
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(014)

Management and Path Planning Solution for Parking Facilities

using Dynamic Load Balancing
F. D. Sandru, V. I. Ungureanu and I. Silea
Automation and Applied Informatics Department, University Politehnica Timisoara, Timisoara, Romania
E-mail: [email protected], vlad.ungureanu @upt.ro, [email protected]

Summary: The growth of personal vehicle ownership has had a direct impact on the need to accommodate vehicles into
structures that haven’t been conceived to handle such requirements of usage. One impacted area by this change is the optimal
usage of existing parking facilities, considering an increase in parking space needs, existing facilities will be filled faster than
before. While navigating empty or near-empty facilities results in a reduced effort in finding available spaces, the moment a
driver must navigate the existing infrastructure almost completely or multiple times has negative effects both economically
and for the environment. This paper presents a method of managing the position of vehicles in parking facilities based on a
multivariable approach with a load balancing component. The presented solution provides minimal functionality even if data
sources are unavailable thus allowing permanent service availability.

Keywords: Parking, Route optimization, Reduced travel time, Load balancing, distance, Dijkstra, SUMO.

1. Introduction The navigation component (guidance of the vehicle

from start to destination) is most associated with map-
One of the aspects that impact both vehicle owners based systems to a minima-based search in a graph also
and infrastructure providers both private and public is called the shortest path problem. A review study of
the storage of the vehicle while it is not in use. various algorithms is made available in [5], for this
Infrastructure providers need to plan for the current paper the Dijkstra algorithm was chosen as the basis
and future demand of the offered facilities while for implementation due to its good performance output
needing to provide appropriate maintenance and regarding computational power and previous proof of
management for existing solutions, guidelines for this concepts being available. The Dijkstra algorithm can
are available in works like [1] where already proven be improved by reinterpreting the cost for a specific
strategies are laid out. Vehicle owners need to use case, one example of such extension is made in [6]
determine the optimal cost (travel time, monetary) where the cost is associated with the fuel consumption
while choosing a parking solution. While the of vehicles traveling on the road infrastructure, a
individual decision regarding the monetary factor has similar approach is made also in this paper but
only an indirect effect on infrastructure providers (loss considering as a cost the total travel time and the
of income, this can be mitigated by designing the congestion generated by vehicles.
facility to the right capacity), offering vehicle drivers Congestion issues are typically seen in computer
the choice of a specific parking space can impact the networks and computing tasks where the non-optimal
infrastructure by generating traffic congestions on use of resources can cause serious problems, one
more desired parts of the facility. A significant number mitigation method is the use of load balancing. Load
of research articles have as a scope management balancing consists of the distribution of tasks among
solution for parking facilities, a distinction needs to be the available system resources. A characterization of
made in between the type of solutions they offer, while load balancing types centered is made in [7], even
papers like [2] provide specific solutions for part of the though cloud computing centered the methods
management process (access control) other papers like described can be applied to the optimal vehicle routing
[3], make a holistic approach and provide an all-in-one process. Few papers have explored this methodology
solution for the given infrastructure. Considering in a parking management context, the examples
advances in mobile computing solutions (smartphones, include [8] and [9] where the distribution was made on
dedicated navigation systems), the driver is presented the scale of multiple parking facilities.
the option of getting real-time traffic information and
adjusting driving behavior accordingly. Some of the
problems associated with this solutions are presented 2. Solution Description
in [4], according to the article users of such systems
might end up in more difficult driving scenarios or The goal of the solution is to provide the optimal
traffic congestions will be artificially created because travel time from a facility and not an individual’s point
the route chosen might not be able to handle the of view. This is achieved by using a multi-parameter
increase in traffic, part of the problem has its cause due approach based on the information that can be made
to competition in-between solution providers, in case available inside the parking facility.
of an enclosed and unique management system, this is
not the case.

64
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

2.1. Definition of Input Data lseg and 𝑠 having the same meaning as before.
The load factor is expressed using eq. 3:
The basic data inputs were considered the ones
with the least technological implication: the road ∑
infrastructure consisting of entry and exit points, the , 𝑟𝑒𝑠 1
𝑙 , (3)
connecting roads with characteristics regarding length, 0, 𝑟𝑒𝑠 1
and the total amount of parking spaces and allowed
maneuvering. where:
More complex types of data are considered for the lveh = length of the vehicles traveling on the segment;
advanced feature of the algorithm, those data types res = usage condition, the computed values is used
include access and entry times for dedicated only if the vehicle demand exceeds the available
infrastructure nodes, the number of vehicles targeting infrastructure.
the parking facility, the destination when exiting the This approach allows to inflate the cost in case the
parking facility, and the congestion of each traffic road structure usage is above the road capacity, the
element. One important factor in regards to this type of factor is only applied once the strain is above the
dynamic data is that their acquisition will increase both capacity.
the initial cost of the facility due to the sensors needed Additionally to this, to better capture the parking
or infrastructure needed to acquire them and the specific maneuver downtime on the road network, a
general computing requirements. time penalty is added for vehicles that would be
parking or exiting the parking space Δtm [s].
Thus the actual cost function becomes the one
2.2. Algorithm Description
described in eq. 4:
As previously mentioned, the developed algorithm _
can be seen as an extended version of the Dijkstra
algorithm, in the following ways: the cost function of 𝑐 𝑡 𝑑 ∆𝑡 (4)
the starting point is not zero as with the standard
approach as it reflects the entry time to the facility, it
considers subdivisions of the graph without generating Another specificity added to the algorithm is the
new point, it features the balancing component by fact that the check for capacity and infrastructure entry
increasing the cost for paths that already have traffic time is done before allowing vehicle access to the
assigned to them. given infrastructure.
In a common navigation system, the minima is The complete flow can be seen in Fig. 1.
computed based on the physical distance between the
start and the trip destination, the proposed algorithm
deviates from this definition and defines the primary 3. Simulation and Results
minima condition as the smallest amount of time spent
in traffic. This change is supported as map providers A medium-sized parking lot was designed to be
include besides the road length also information about used as a demonstrator for the deployment, in the
various speed limits, thus the first component used for initial conception phase a dedicated computer-aided
evaluation becomes the one described in eq. 1 design software (LibreCAD) was used, the output of
the projection phase is visible in Fig. 2.
𝑡 𝑙 /𝑠 , (1) The parking facility features the following
elements: a single one-way street connecting the two
where: entry and exit points. The parking lot is divided into
tseg = necessary time to traverse a map segment [s]; parking areas consisting of the parking spaces
lseg = legnth of the segment [m]; connected to a road segment. The parking space
sseg = the maximun allowed speed to traverse the dimensions were chosen to be 2.5 m for with and 5m
segment [m/s]. for length. A general speed limit of 15Km/h is
The information obtained by this step is purely considered for the traffic inside the parking facility
static and doesn't take into account the real-world while the access road has a speed limit of 50 Km/h.
environment like infrastructure use resulting in a The simulation necessities, network definition,
degradation of the projected arrival cost. static and dynamic information extraction, route
Adding the dynamic characteristic data about the generation, and vehicle control were done using the
other traffic participants is needed, this can be either SUMO (Simulation of Urban MObility) package
sourced from measurement systems or is known in features, developed by the German Aerospace Center,
case the path planning is done by a centralized this is “an open-source, highly portable, microscopic
mechanism. This component is presented in eq 2. and continuous traffic simulation package designed to
handle large networks” [10], while its primary scope is
𝑑 𝑙 𝑙 /𝑠 , (2) not the simulation of parking facilities the features it
offers were of great help in implementing the
where: simulation scenarios and solution by extending and
lf = load factor for the segment; relying upon it in the developed code.

65
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 1. Flow chart representing the proposed algorithm.

Fig. 2. Parking facility designed for validation.

As the SUMO tooling requires a specific input used by the solution scripting for parameters like
format for being able to simulate the environment, the junctions, direction change restrictions, segment
initial schematic was recreated in the netedit utility length, max segment speed, and the association
denoting the network nodes and the associated between the parking area and the access road. Specific
restrains. The network provides direct information to the simulation is the computation of a parking space

66
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

offset, due to the association being made on parking using the TraCI API, Traci allows this by creating a
area level with each parking space the with of the TCP-based connection to the SUMO application and
individual space needs to be accounted for when using that connection to pass data. The high-level
computing the distance travelled. Another specificity overview of the SW components used is visible in
of the simulation is the slit of the parking area due the Fig. 4. Application internals where specifically
presence of a junction, this is visible in the lateral developed for this study while existing elements are
parking areas thus increasing the number of areas that marked separately.
need to be monitored. The parking facility netedit The simulation phase considered 3 different
specific design is visible in Fig. 3. scenarios for choosing a parking space for a vehicle
next in queue:
a) Guidance based solely on proximity to the target;
b) A randomized allocation of parking spaces with
guidance instructions;
c) The proposed extension to the algorithm to balance
the network usage.
The evaluation criteria considered was the least
time spent idling, this value is taken from the
simulation and as per TraCI API [11] is defined as
“The waiting time of a vehicle is defined as the time
(in seconds) spent with a speed below 0.1 m/s since the
last time it was faster than 0.1 m/s.”
Fig. 3. Parking facility design translation for simulation.
As a general path target, vehicles always originate
in the right outmost extremity of the access road and
Currently, this step has been done manually, the their destination is a parking space within the facility.
available scripting allows the direct import only of In a direct comparison, the 3 scenarios have clear
OSM (Open Street Map) format, the authors are differences from a performance point of view, the
considering the development of a script that would time-based criteria show the ability of the developed
make it possible to do type conversions DXF file solution to reduce the general time spent in traffic, by
format to the XML file format used by SUMO. having the same demand this is visible in Fig. 5.
The control of the simulation is done from a
python-based implementation of the parking manager

Fig. 4. Software overview.

Fig. 5. Solution performance in loading the complete parking facility.

67
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

The individual max idle time per solutin are presented 4. Proposals for the Physical Realization
in Fig. 6, Fig. 7 and Fig. 8. of the System

Human-machine interface: Depending on the level

of automation of the vehicle human interaction might
or might not be needed, in the case of a non-
autonomous vehicle, the information about parking
space availability, the assigned position, and the
navigation can either be done using an application
running on a smartphone or directly on a vehicle
integrated navigation system.
Access control: While the enforcement of the entry
and exit rights is usually done using physical barriers
this approach would only lead to delays, considering
this a solution where the access rights are checked by
the management system is the desired one. As the
parking space is assigned by the system any occupancy
Fig. 6. Idle time for strategy a). of a non-assigned parking space could be considered a
violation in the terms of usage in case an unauthorized
vehicle is occupying a parking space blocking that
vehicle on the used parking space till the service
payment is made is considered the preferred approach.
The monitoring is proposed to be done using camera
systems capable of detecting the presence of vehicles
and associating their position to the model of the
facility.
Vehicle navigation: The maneuvering of the
vehicle inside the parking structure can be one of the
most challenging aspects of operation, the map data of
the parking structure would either need to be made
available beforehand and associated to a general
navigation map or merged upon the proximity to the
entry position and deleted upon exit to reduce the
Fig. 7. Idle time for strategy b). storage requirements, another approach could consist
in using a dedicated reduced map consisting only of
the parking facility inside a specific application on a
smartphone. Positioning is another key factor, while a
combination of a GNSS system with a map matcher
component would provide a certain degree of
precision, this might not meet the constraints to be
within a few meters. In this case, a solution based on
using inertial data like described in [12] proves itself
useful but would not be able to compensate for a
complete loss of GNSS signal throughout a complete
trip. A possible solution could consist in the usage of a
camera-based system that would be able to recognize
specific signage and associate it to the corresponding
infrastructure elements.
Fig. 8. Idle time for strategy c).

5. Conclusions
Allocation based on the minimal driving distance,
this approach generates excessive strain on the first The solution can determine based on a given
connection points to the parking area and results in infrastructure the necessary parking lot information
congestion. Waiting times observed during these and provide access control features in case the capacity
simulations were the highest while choosing a random is reached while outperforming classic navigation
parking area among the available structures yields solutions in terms of time spent in traffic.
better results than the closest available area. Future extensions should consider more simulation
Higher waiting times are still visible this is due to the scenarios and the inclusion of more complex driving
bottlenecking of the entryway. behavior (like take over).

68
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

The authors are planning after the extended weight-Dijkstra algorithm, IET Intelligent Transport
validation phase to also realize a physical Systems, 10, No. 7, 2016, pp. 495-502.
implementation of the solution. [7]. Katyal, Mayanka, and Atul Mishra, A comparative
study of load balancing algorithms in cloud computing
environment, International Journal of Computer
References Applications, 117, 24, 2014, pp. 33-37.
[8]. Kim, Oanh Tran Thi, Nguyen H. Tran, Chuan Pham,
[1]. Todd Litman, Parking management: strategies, Tuan LeAnh, My T. Thai, and Choong Seon Hong,
evaluation and planning. Victoria, BC: Victoria Parking assignment: Minimizing parking expenses and
Transport Policy Institute, 2016. balancing parking demand among multiple parking
[2]. E. Hassania Rouan and A. Boumezzough, RFID Based lots, IEEE Transactions on Automation Science and
Security and Automatic Parking Access Control Engineering, 17, no. 3, 2019, pp. 1320-1331.
System, Business Intelligence, 2021, pp. 434–443. [9]. A. Souza, Z. Wen, N. Cacho, A. Romanovsky,
[3]. M. Owayjan, B. Sleem, E. Saad and A. Maroun, P. James and R. Ranjan, Using Osmotic Services
Parking management system using mobile application, Composition for Dynamic Load Balancing of Smart
in Proceedings of the Sensors Networks Smart and City Applications, in Proceedings of the IEEE 11th
Emerging Technologies Conference (SENSET’ 2017), Conference on Service-Oriented Computing and
2017, pp. 1-4. Applications (SOCA’ 2018), 2018, pp. 145-152.
[4]. IEEE Spectrum: Technology, Engineering, and [10]. P. A. Lopez et al., Microscopic Traffic Simulation
Science News, 2021. using SUMO, in Proceedings of the 21st International
https://spectrum.ieee.org/computing/ Conference on Intelligent Transportation Systems
hardware/your-navigation-app-is-making-traffic- (ITSC’ 18), 2018, pp. 2575-2582.
unmanageable (accessed Jun. 12, 2021). [11]. Python: module traci._vehicle, Sumo.dlr.de, 2021.
[5]. Madkour, Amgad, et al., A survey of shortest-path https://sumo.dlr.de/daily/pydoc/traci._vehicle.html
algorithms, International Journal of Applied (accessed Jan. 2, 2022).
Engineering Research, Vol. 13, No. 9, 2018, [12]. Sandru F. D., Nanu S., Silea I. and Miclea R. C.,
pp. 6817-6820. Kalman and Butterworth filtering for GNSS/INS data,
[6]. Zhang, Jin-dong, Yu-jie Feng, Fei-fei Shi, Gang Wang, in Proceedings of the 12th IEEE International
Bin Ma, Rui-sheng Li, and Xiao-yan Jia, Vehicle Symposium on Electronics and Telecommunications
routing in urban areas based on the oil consumption (ISETC’2016), 2016, pp. 257-260.

69
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(015)

Switching Propulsion Mechanisms of Tubular Catalytic Micromotors

P. Wrede 1,2, M. M. Sánchez 1, V. M. Fomin 1,3,4 and O. G. Schmidt 1,5,6
1
Institute for Integrative Nanosciences (IIN), Leibniz Institute for Solid State and Materials Research (IFW)
Dresden, Dresden, Germany
2
Max Planck Institute for Intelligent Systems and Max Planck ETH Center for Learning Systems,
Stuttgart, Germany
3
Institute of Engineering Physics for Biomedicine, National Research Nuclear University “MEPhI”,
Moscow, Russia
4
Laboratory of Physics and Engineering of Nanomaterials, Department of Theoretical Physics,
Moldova State University, Chişinău, Republic of Moldova
5
Material Systems for Nanoelectronics, TU Chemnitz, Chemnitz, Germany
6
Center for Materials, Architectures and Integration of Nanomembranes (MAIN), TU Chemnitz,
Chemnitz, Germany
Tel.: +49 711-689-3424, fax: + 49 711-689-3412
E-mail: [email protected]

Summary: Different propulsion mechanisms have been suggested for a variety of chemical micromotors. Their high efficiency
and thrust force enable several applications in the fields of environmental remediation and biomedicine. In particular, bubble-
recoil based motion has been modeled by three different phenomena: capillary forces, bubble growth, and bubble expulsion.
However, these models have been suggested independently based on a single influencing factor (i.e., viscosity), limiting the
understanding of the overall micromotor performance. Here the combined effect of medium viscosity, surface tension, and
fuel concentration on the micromotor swimming performance is analysed. Hence the dominant propulsion mechanisms
describing its motion are accurately identified. Using statistically relevant experimental data, a holistic theoretical model is
proposed for bubble-propelled tubular catalytic micromotors that includes all three above-mentioned phenomena. The
proposed model provides deeper insights into their propulsion physics toward optimized geometries and experimental
conditions. These findings pave the way for highly optimized catalytic micromotors for various applications.

Keywords: Microrobots, Chemical propulsion, Viscosity sensing, Micromachines, Micro-/nanotechnology.

1. Introduction inside the tube, leading to a uni- or bidirectional

movement, depending on the micromotor geometry
For more than a decade researcher are developing [6].
new chemical micro-devices, including micromotors. To reach a fundamental understanding of the
Owing to their small size these highly controllable propulsion physics of such micromotors, we fabricate
man-made machines enable precise applications for micromotors with different lengths and semi-cone
targeted medical treatments [1, 2] and environmental angles using two-photon lithography. Based on these
remediation [3, 4]. Moreover, micromotors are structures, the influence of different concentrations of
interesting model systems for studying motion fuel, surfactant and the viscosity of the environment as
mechanisms as well as other physical phenomena at well as geometric parameters on the propulsion mode
low Reynolds number regimes. A particularly is analysed [7]. We show that different propulsion
interesting type of chemically-driven propulsion is the mechanisms can be observed individually or
bubble-recoil mechanism. Micromotors utilising this simultaneously in the same experiment. This provides
specific mechanism show excellent motion new insights into the propulsion mechanisms and the
performance in terms of speed and thrust force with possibility to further optimize and simulate the
simple designs. They are mainly made of asymmetrical behaviour of bubble propelled micromotors for
tubes coated with an inner surface of a catalytic operating in more complex media.
material (e.g., Pt, Ag, Pd, enzyme) and an outside
functional layer (e.g., Au, Fe, SiO2, TiO2) for their
guidance, in situ reactions or biofunctionalization 2. Simulation and Switching of Propulsion
purposes [5]. A chemical reaction between the fuel Mechanis
liquid and a catalyst coating the inner surface of the
micromotor converts chemical energy into mechanical The motion of conical bubble propelled
motion. The most commonly used reaction is the micromotors is described by three established
decomposition of H2O2 into O2 and H2O using Pt as mechanisms, the capillary force [8], bubble growth [9]
catalyst. Bubbles are formed by O2, which is produced and jet-like propulsion [10]. Our experiments imply

70
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

that, depending on the composition of the fuel fluid, a

switching of propulsion mechanisms occurs [7]. The
use of a surfactant, here SDS, reduces the surface
tension of the medium hence reducing the drag force
acting on the bubble during its motion through the
micromotor. This leads to an increased micromotor
velocity for SDS concentration rising from 1.25 % till
10 %. This effect reverses for SDS concentrations
larger than 10 %. The amount of surfactant present
absorbable by the micromotor scales with the available
surface area. After reaching the threshold of 10 % SDS,
when the used micromotors reveal their maximum
SDS absorbance, the positive effect of the surfactant
on the micromotors speed diminishes. A dominant
influence of the bubble growth mechanism is observed
for the concentrations higher than 20 % SDS (at 2.5 %
H2O2) as well as for the concentrations of 0.4 % MC
and higher (with viscosity above 0.004 Pa⋅s) at 2.5 %
H2O2 and 5 % SDS (Fig. 1). In those cases, insufficient Fig. 1. Measured and simulated micromotor speeds
force is exerted on the bubble to detach it from the tube. for various fluidparameters. (After Wrede et al. [7]).
This causes the bubble to keep growing much bigger
Hence, for our simulations, all three mechanisms
than the larger opening of the micromotor. Due to the
are taken into account, in particular, for the
motion of the bubble in the micromotor, a relatively
H2O2 concentrations lower than 20 %, while as for the
small contribution of the capillary force is also present.
H2O2 concentration above this value, only the bubble
Anyway, it can be neglected for a sufficiently long
expulsion mechanism functions. Considering all
time because no second bubble is formed inside the
mechanisms in our model, the simulated speed as a
micromotor. With increasing MC concentration, the
function of the H2O2 concentration (ranging from
viscosity of the fluid, and therefore the drag force
2.5 % to 30 %) is in good agreement with the
counteracting the propulsion force of the micromotor,
experimental data. All three propulsion mechanisms
increase. This leads to an increase in propulsion speed.
are also considered for the whole range of the MC
Additionally, the MC forms a passivating layer around
concentrations (from 0.05 % to 0.25 %, corresponding
the Pt inside the micromotor. This results in a reduction
to viscosities from 1.3 to 6 mPa⋅s), as well as for the
of the chemical activity of platinum. Hence, less H2O2
whole range of the SDS concentrations (from 1.25 %
is decomposed and thus less O2 is produced. As a
to 10 %). The growth mechanism is dominant for the
consequence, the bubble release frequency decreases,
MC concentrations of 0.4 % and 0.6 % as well as for
allowing the bubbles to collect more O2, which results
the SDS concentrations of 20 % and 30 %.
in an increased bubble diameter. Our experimental data
Additionally, we evaluate the effect of different
suggest that starting from the concentration of about 20
geometric parameters on the performance of the here
% H2O2 (the specific value depends on the system
presented micromotors, scaling the micromotor semi-
under study), the bubble expulsion mechanism
cone angle and length. In particular, 50 µm long
dominates over the other two mechanisms (Fig. 1) by
micromotors with the varying semi-cone angle (2.5°,
two reasons. Firstly, the bubbles move along the tube
5°, and 10°), as well as micromotors with the semi-
so quickly, that their radii remain smaller than that of
cone angle of 5° and varying lengths of 25, 50, and
the tube. Accordingly, there is no contact between the
100 µm, are analysed. An increase of the micromotor
bubble and the inner wall to induce a capillary force.
length from 25 to 100 µm results in an increase of the
Secondly, due to their small size, the bubbles leave the
micromotor speed, while an increase in the semi-cone
micromotor without adhering, so that no growth force
angle slows the micromotor down. To better
emerges. The main reason for these observations is the
understand this behaviour, in our simulations, we
increased amount of the available H2O2 when
assume certain dependencies: when increasing the
increasing concentration. This, in turn, leads to an
semi-cone angle of the micromotor, we keep the
increased production of O2, and hence, to a quicker
smaller radius constant, therefore the larger radius
formation of bubbles. Accordingly, the bubble release
increases. The same applies for an increase in length.
frequency increases, and so does the micromotor
Such an increase in the larger opening of the tube leads
speed. Due to this increased frequency and the
to the production of high-volume bubbles. Using our
associated shorter time, during which the bubbles stay
experimental data for a micromotor with the length of
in the tube, they can also collect less oxygen leading to
50 µm and the semi-cone angle of 5°, the bubble radius
a decreased bubble radius and a smaller rise in speed
is twice as large as the opening radius for different
for H2O2 concentrations over 10 %. For all remaining
H2O2 concentrations (up to 10 %). For larger
fluid parameters, all three mechanisms need to be
H2O2 concentrations, the bubble radius is similar to the
considered to describe the micromotors motion
opening radius. This effect is caused by a higher
(Fig. 1).
bubble expulsion frequency. The latter depends on the

71
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

active catalytic area of the micromotor, which causes 9, 4, 2018, pp. 303–316.
an increase in the oxygen production rate. An increase [2]. Wang, B., Kostarelos, K., Nelson, B. J. and
of both the semi-cone angle and the length of the Zhang, L., Trends in micro-/nanorobotics:
micromotors leads to an increase in the catalytic materials development, actuation, localization,
surface area. Particularly, an increase of the semi-cone and system integration for biomedical
angle from 2.5° to 5° for a conical micromotor with a applications, Advanced Materials, 33, 2021,
length of 50 µm results in a surface area decrease of 20 pp. 1–44.
%. Additionally, larger semi-cone angles increase the [3]. Srivastava, S. K., Guix, M. & Schmidt,
drag force acting on the micromotor. In summary, an O. G., Wastewater mediated activation of
increase in bubble expulsion frequency and bubble micromotors for efficient water cleaning, Nano
radius is predicted for increasing micromotor length Letters, 16, 2016, pp. 817–821.
and semi-cone angle. At the same time, an increased [4]. Jurado-Sánchez, B. & Wang, J., Micromotors for
speed is predicted for smaller semi-cone angles and environmental applications: A review.
longer micromotors. Environmental Science: Nano, 7, 2018,
pp. 1530–1544.
3. Conclusion [5]. Zhang, Y., Yuan, K. & Zhang, L., Micro/
Nanomachines: From functionalization to
A physical explanation of the bubble-propelled sensing and removal, Advanced Materials
micromotor motion is performed using the established Technologies, 4, 2019, pp. 1–22.
capillary force, bubble growth, and jet-like propulsion [6]. Mei, Y. F., Naeem, F., Bolaños Quiñones,
mechanisms. Switching of propulsion mechanisms is V. A., Huang, G. S., Liao, F., Li, Y., Manjare,
for the first time unveiled at certain values of the H2O2, M., Naeem, S., Solovev, A. A. and Zhang,
MC, and SDS concentrations. Besides, it is observed J., Tubular catalytic micromotors in transition
that for certain experimental parameters, a concerted from unidirectional bubble sequences to more
action of all three propulsion mechanisms is observed. complex bidirectional motion, Applied Physics
A theoretical model is proposed to calculate the speed Letters 114, 2019, pp. 1–16.
of the bubble-propelled micromotors, including the [7]. Wrede, P., Medina-Sánchez, M., Fomin,
contributions of all three known propulsion V. M. and Schmidt, O. G., Switching propulsion
mechanisms, resulting in good agreement with the mechanisms of tubular catalytic micromotors,
obtained experimental data. This offers new Small, 17, 2021, pp. 1–11.
possibilities to optimize and predict the performance of [8]. Klingner, A., Khalil, I. S. M., Magdanz,
bubble-propelled micromotors for different V., Fomin, V. M., Schmidt, O. G. and Misra,
applications in various working environments, S., Modeling of unidirectional-overloaded
including new applications, such as, in-situ sensing of transition in catalytic tubular microjets, Journal
the medium viscosity, as well as for fundamental of Physical Chemistry C, 121, 2017, pp. 14854–
understanding of the impact of various geometrical 14863.
parameters and experimental conditions (length, semi- [9]. Manjare, M., Yang, B. and Zhao, Y. P. Bubble-
cone angle, surface properties) on the propulsion of propelled microjets: Model and experiment.
conical catalytic micromotors. Journal of Physical Chemistry C, 117, 2013,
pp. 4657–4665.
References [10]. Li, L., Wang, J., Li, T., Song, W. and Zhang,
G., Hydrodynamics and propulsion mechanism
[1]. Medina-Sánchez, M., Xu, H. and Schmidt, of self-propelled catalytic micromotors:
O. G. Micro- and nano-motors: The new Model and experiment, Soft Matter, 10, 2014,
generation of drug carriers, Therapeutic Delivery, pp. 7511–7518.

72
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(019)

Stability Margins for Linear Periodically Time-Varying Systems

Xiaojing Yang
School of ASE, Beihang University, 37 Xueyuan Rd., 100191 Beijing, China
Tel.: + 86 13240488262, fax: + 86 13240488262
E-mail: [email protected]; [email protected]

Summary: Singular Perturbation Margin (SPM) and Generalized Gain Margin (GGM) are Phase Margin (PM) and Gain
Margin (GM) like stability metrics. In this paper, the problem of SPM and GGM assessment for Linear Periodically Time-
Varying (LPTV) systems is formulated. Chang transformation makes it possible to reduce the SPM analysis for Hill equations,
which is essentially a stability problem of higher order LPTV systems due to the SPM gauge introduced dynamics, to the
second order LPTV systems. Based upon Floquet Theory, the SPM and the GGM assessment methods for the second order
and the general order LPTV systems are established, respectively.

Keywords: Stability margin, Singular perturbation, Regular perturbation, Linear system, Periodically time-varying systems,
Hill equation.

1. Introduction perturbation. Due to the distinguishing feature of the

exponential stability of the equilibrium point, i. e. the
Research on periodically time-varying systems can robustness to the nonvanishing regular perturbations,
be traced back to M. Faraday [1] in 1830s; the first the concepts of SPM and GGM have been investigated
detailed theory study for Linear Periodically Time- towards to the establishment of the quantitative
Varying (LPTV) systems is given by E. Mathieu [2]; G. exponential stability metrics for general Multi Input
Floquet established Floquet theory [3] in 1883, which Multi Output (MIMO) Nonlinear Time Varying
forms the basis of a great many of the descriptions of (NLTV) systems, defined by the maximum singular
parametric behaviors [4]; one of the most significant perturbation parameter, denoted by  max , and the
earlier papers on periodically time-varying systems maximum and minimum regular perturbation
was G. W. Hill [5], whose name has been given to the parameters, denoted by kmin and kmax that render the
general class of the second order periodic differential
perturbed closed-loop system at onset of instability to
equations, and his research is the first investigation of
gauge the capability in accommodating singular
a practical problem in such field. Due to a growing
perturbations (parasitic dynamics) and regular
number of applications of LPTV systems in different
perturbations (parametric dispersions). Furthermore,
fields, since the first half of the 20th century, more
the long term effort of such research is the
study effort was put on Meissner equation [6], Mathieu
development of the theoretically based and practically
equation [7], the solutions [8] and the stability behavior
efficient SPM and GGM analysis methodology to
of the above equations [9], etc. Through the continuous
facilitate the nonlinear and time-varying control
hard work of scholars for the recent 60-70 years, the
system design.
research methodology of LPTV systems has been
In this paper, the problem of SPM and GGM
formed a relatively mature body of knowledge, and can
analysis for LPTV systems is formulated in Section
be roughly divided into three categories: 1, find the
Objectives; the main results: decoupling technique of
time invariant equivalent system for the periodically
the singularly perturbed LPTV systems; SPM gauge
time-varying system and further discuss the problems
design; SPM and the GGM assessment methods,
of stability and pole assignment [10-13]; 2, analyze the
examples etc., are listed in Section 3; Section 4
controllability and oberservability of LPTV systems
concludes the whole paper's with some insightful
based on the state transition matrix of one period
remarks.
[14, 15]; 3, Lyapunov methods [16-19].
Stability is a key problem in the design of
automatic control systems, and in engineering 2. Objective
applications, not only the question whether the system
is stable is must known information, but also the The SPM and GGM analysis for a closed-loop and
quantitative criteria of stability is pivotally demanded. well-designed LPTV control system (also called
Recently, Singular Perturbation Margin (SPM) and Nominal System)
Generalized Gain Margin (GGM) [20] have been
proposed as the classical Phase Margin (PM) and Gain x  Anom (t ) x , (1)
Margin (GM) like stability metrics for Single Input
Single Output (SISO) Linear Time Invariant (LTI) and
n
Nonlinear Time Invariant (NLTI) systems [21] from where x is the state vector, Anom (t ) is bounded
the view of the singular perturbation and the regular and continuously differentiable, satisfying

73
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Anom (t )  Anom (t  T ) (2) Design the SISO LTI SPM gauges at the perturbation
sites, which might have the equation forms as [20]
minimum-phase gauge, zero-error gauge or
with T the system period, is referred as to the
approximately-linear phase gauge, etc., and the
quantitative assessment of the capability of the
singularly perturbed system is given by
exponentially stable null equilibrium (origin) in
accommodating singular perturbations and regular
perturbations: x  A 11 (t ) x  A12 (t ) z
, (4)
(1) Design the SPM gauge and GGM gauge for the  z  A21 (t ) x  A22 z
specific perturbations;
(2) Determine the stability of the perturbed system where the singular perturbation parameter  describes
using stability criterion when different singular
the time-scale separation, A11 (t ), A12 (t ), A 21 (t ), A22
perturbation parameters and gain parameters are
given, respectively; are function matrices and parameter matrix that are
(3) Obtain the SPM and GGM by the marginally dependent on the perturbation site and the SPM gauge
design, and z 
m
stable situation of the perturbed system; is state vector of the fast
(4) Reveal the relation between the characteristics of dynamics.
the nominal system (1) and its exact SPM and
GGM.
3.2. Decoupled Slow and Fast Systems

3. Perturbed Hill Equation The singularly perturbed system (4) can be totally
decoupled to slow and fast system by Chang
3.1. Stability Criterion and SPM/GGM Gauges transformation.

We first introduce a necessary and sufficient Lemma 2 With the SISO LTI SPM Gauge, the
exponential stability condition for LPTV systems that decoupled slow system of the singularly perturbed Hill
will be used in the sequel. equation (4) is a Hill equation with the same period to
Lemma 1 [4] The natural response of an LPTV the nominal system.
system is stable, if and only if no eigenvalues of the Proof. Consider the two perturbation positions,
Discrete Transition Matrix (DTM) have magnitudes Site 1 and Site 2, in the loop of Fig. 1.
greater than one and that an eigenvalue with unity Site 1: With the SISO LTI SPM Gauge, the system
magnitude is not degenerate. matrices of the singularly perturbed system in form of
The eigenvalues of the DTM is also called Floquet (4) are given by
characteristic multipliers of the periodic system. The
 ... 0 ,
A11   0 1  , A12 (t )   0  a  2 q (t ) C
transition matrix of a general order LPTV system is
analytically intractable in Lemma 1, but for the second-  0 0     f 
order cases, the general form
A21  B f 1 0  , A22  A f ,
x  1 (t ) x   0 (t )  0
where A f , B f , C f are the parameter matrices of the
can be converted into an expression without a first SISO SPM gauge. Chang transformation [22] is given
derivative, and the classical form is by

y  (a  2q (t )) y  0,  (t )  (t   ) , (3)      I   H ( t ) L ( t )  H ( t )   x   S ( t )  x 

   L( t ) I   z   z 
where function  (t ) has a period of  and a, q are
constant parameters, is identified as Hill equation. Hill where function matrices L(t) and H(t) satisfy
equation is not only the model in many applications, Differential Riccati Equation (DRE) and Differential
but also has significant meaning for general second- Sylvester Equation (DSE)
order equations. When the singular and regular
perturbations are considered in the closed-loop, the 1 1
block diagram of Hill equation with corresponding  L (t )  L(t ) A11  A21  L(t ) A12 (t ) L(t )  A22 L(t )
perturbation sites is shown by Fig. 1.
 
 H ( t )  A11 H ( t )  A12 ( t )  H ( t ) L( t ) A12 ( t )
1
 H ( t ) A22  A12 ( t ) L ( t ) H ( t )


For Chang transformation, the iterative solutions of

Fig. 1. Block Diagram of Hill Equation with Perturbations. the DRE and DSE in form of

74
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra
 
which can be denoted as x   2 (t ) x  1 (t ) x  0 .
L ( t )    j L j ( t ), H (t )    j H j (t )
j 0 j 0 Then, by transformation

 1 t 
where x ( t )  exp     2 (t )dt  y (t )
 2 0

L0  A221 A21 , H 0  A121 A22
Lk 1  A221  L k  Lk  A11  A12 Lk   , k  0,1, 2,... Eq. (7) is transformed to be
 
H k 1    H k  H k Lk A12   A11  A12 Lk  H k  A221
y ( t )   1 (t )   2 (t )   22 ( t )  y (t )
1 1
 
k  0,1, 2,...  2 4 

Then, the slow system is given by which is a Hill equation. 

Aslow  A11  A12 (t ) L(t )

3.3. SPM Assessment of Hill Equations
 0 1 
  a  2 q (t ) C L (t )
  f 1  a  2q (t )  C f L2 (t )  By Lemma 2, it is known that with an SISO LTI
SPM Gauge, the singularly perturbed Hill equation can
where L1 (t ), L2 (t ) are the column vectors of L(t ) , be decoupled to a fast system and a slow system. The
following Theorem 1 is a direct result of the definition
and the second order equation becomes
for SPM and Lemma 1.
Theorem 1 The SPM of the LPTV nominal system
x   2q (t )  a  C f L2 (t ) x   2q (t )  a  C f L1 (t ) x  0 (5) is equal to the maximum singular perturbation
parameter of the slow system satisfying

By transformation  max  sup  :   (0,  )

 1 t  when   (0,  ) no eigenvalues of the DTM of the slow

x ( t )  exp     2 q ( t )  a  C f L2 dt  y ( t ) (6)
 2 0
 system have magnitudes greater than one.
Eq. (5) is transformed to When the perturbation is added in Site 1, design the
LTI SPM gauge as a second order minimum-phase
y    2 q ( t )  a  C f L1 ( t )  q ( t )C f L2 ( t ) fast system, which has a transfer function
1 2
  2 q ( t )  a   C f L1 ( t )   y
2
02
4  L fast ( s ) 
( s ) 2  20 ( s )  02
which is a Hill equation with the same period to
function  (t) . Consider a Mathieu equation, which is a special
case of Hill equation.
Site 2: With the SISO LTI SPM Gauge, the system y   a  2 q cos(2t )  y  0
matrices of the singularly perturbed system become
Numerically calculate the DTM for Mathieu
A11   0 0 , A (t )  0 C f 0 , equation and check the eigenvalues of the DTM for the
  a  2 q (t ) 0  12  0 ... 0  stability situation (Lemma 1). The stability diagram is
A21  B f  0 1 , A22  A f , shown by Fig. 2.

By Chang transformation, the slow system is given by

Aslow (t )  A11 (t )  A12 L(t )

 C f L1 (t ) C f L2 (t ) 

   a  2 q ( t )  0 

and the second order slow system becomes

 C L
 
x  C f L1  C ff L22 x  C f L1 
C f L2 C f L1
C f L2
(7) Fig. 2. Stability diagram for the Mathieu equation. Blank
  2 q (t )  a  C f L2  x  0 regions correspond to stable solutions and shaded regions to
unstable solutions. The diagram is symmetrical about a axis.

75
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

In the expression of the slow system, matrix L(t) order LPTV system to obtain the singular perturbation
can be obtained by the iterative expression and the margin.
slow system is given by 3.5. GGM Assessment of Hill Equations

Aslow  A11  A12 ( t ) L(t ) Meissner Equation is a well-known Hill equation,

 
 0 1 
 0
0 0   2 q cos(2t )  a  02
0
0 
L(t ) when the function  (t ) is unit rectangular waveform
in (3), which is shown by Fig. 3, and can be viewed
as the pair of constant coefficient equations

L( t )    j L j ( t )
j 0

L0  A221 A21 , Lk 1  A221  L k  Lk  A11  A12 Lk 

 
k  0,1, 2,...

Fig. 3. Meissner Equation and Impulse

The parameters are: a  3, q  1, 0  2,   0.2 . Coefficient Hill Equation.
Numerically calculate the DTM for the slow
system and check the eigenvalues of the DTM for the
stability situation (Lemma 1). The stability situation is x   a  2 q  x  0, 0  t 
different with different singular perturbation parameter x   a  2 q  x  0,  t 
 , and the results are shown by Table. 1.
The DTM of Meissner equation is given by
Table 1. Stability of the slow system with different singular
perturbation parameters (Site 1)  ( , 0) 
 1 
Max magnitude of  cos d (   ) d
sin d (   ) 
No.  the eigenvalues of Stability   d sin d (   ) cos d (   ) 
the DTM
 1 
1. 0.01 1.0 yes
 cos c c
sin c  ,
2. 0.03 1.0 yes   c sin c cos c 
3. 0.04 1.0 yes
4. 0.05 1.05 no where d   a  2q  , c   a  2q  .
5. 0.06 1.06 no
Case 1. With constant regular perturbation
parameter k in the closed loop, either Site 1 or Site 2 in
According to Table. 1, the definition of SPM and Fig. 1, the DTM of the regularly perturbed system is
Theorem 1, it is obtained that  max  0.04 . given by the above ( ,0) , albeit the parameters k
dependent. Denote
3.4. SPM Assessment of General Order LPTV
Equations d  a  2q  k , c  a  2q  k
The DTM, denoted by ( ,0) , is a key point for The characteristic multiplier can be written as
the stability margin assessment, and according to
Floquet theory
r / 2
2
1,2  r / 2  1
x ( t  m )   ( , 0) m x ( t )
and

let t  0 and m  0,1, 2,..., n in the above equation,

r  trace  pert ,k ( , 0)  2 cos d (   ) cos c
through simulation to know the value of
x(0), x( ),..., x(n ) , and then the elements of d c 
    sin d (   ) sin c
c d 
( ,0) becomes an n2 algebraic equation.
Due to the numerical methods for DTM, the By Lemma 1, the stability diagram for Meissner
algorithms of section 3.3 can be generalized to any equation with GGM parameter k is shown by Fig. 4.

76
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

In Fig. 4, the blank area represents the stable solutions equations, which is essentially a stability problem of
of Meissner equation with the parameter couple higher order LPTV systems due to the SPM gauge
(q, a) . When the nominal system with parameter introduced dynamics, to second order LPTV systems.
Based upon Floquet Theory, the SPM and GGM
couple ( q0 , a0 ) , is perturbed by regular perturbation assessment methods for the second order and general
parameter k, then ( q , a ) will move along the line order LPTV systems are established, respectively.
a / q  3 , until the couple ( q f , a f ) first touches the
boundary of the blank area, i. e. the regularly perturbed
Acknowledgements
system becomes marginally stable, which is shown by
the legend square. Then, the GGM k m ax and k m in are
The author is grateful of the financial support from
obtained. National Natural Science Foundation of China No.
61803010 and Aeronautical Science Foundation of
China No.2018ZA51002.

References
[1]. M. Faraday, On a peculiar class of acoustical figures;
and on certain forms assumed by groups of particles
upon vibrating elastic surfaces, Proceedings of the
Royal Society of London, 3, 1830, pp. 229-318.
[2]. E. Mathieu, Memoire sur le movement vibratoire d'une
membrane de forme elliptique, J. Math. Pure Appl., 13,
1868, p. 137.
[3]. G. Floquet, Sur les equations differenctiales lineaires.
Ann. L'Ecole Normale Super., 12, 1883, p. 47.
[4]. J. A. Richards, Analysis of Periodically Time- Varying
Systems. Springer, 1983.
Fig. 4. Stability Diagram for Meissner Equation and GGM [5]. G. W. Hill, On the part of motion of the lunar perigee
Assessment. which is function of the mean motions of the sun and
moon, Acta Mathematica, 8, 1886, pp. 1-36.
[6]. E. Meissner, Uber Schuttelerscheinungen im System
Case 2. When Meissner equation is regularly mit periodisch veranderlicher Elastizitat, Schweiz.
Bauztg., 72, 1918, pp. 95-98.
perturbed on parameter q. The parameters d and c [7]. J. R. Carson, Notes on theory of modulation, Proc. IRE,
in the DTM of the regularly perturbed system become 10, 1922, pp. 57-64.
d  a  2qk , c  a  2qk . In Fig. 4, when
[8]. H. Jeffreys, Approximate solutions of linear
differential equations of second order, Proc. Lon. Math.
a  20 , the legend circles show boundaries of stability Soc., 23, 1924, p. 428.
solutions, and with q  1 as the nominal system [9]. B. van der Pol, M. J. O. Strutt, On the stability of
solutions of Mathieu's equation, Philos. Mag., 5, 1928,
parameter (point A), the GGM of Meissner equation is pp. 18-39.
k m a x  1 .6 , k min   1.6 . [10]. W. Y. Yan, R. R. Bitmead, Decentralized control
techniques in periodically time-varying discrete-time
Case 3. When Meissner equation is regularly control systems, IEEE Transactions on Automatic
Control, 37, 10, 1992, pp. 1644-1648.
perturbed on parameter a, the parameters d and c [11]. M. S. Chen, Y. Z. Chen, Static output feedback control
in the DTM become d  ak  2q , c  ak  2q .
for periodically time-varying systems, IEEE
Transactions on Automatic Control, 1999, 44, 1,
In Fig. 3, the legendtriangle represent the boundaries pp. 218-222.
of stable solutions. [12]. S. Kalender, H. Flashner, A new approach for control
of periodically time varying systems with application
to spacecraft momentum unloading, in Proceedings of
3.6. GGM Assessment of General Order Equations the AIAA Guidance, Navigation, and Control
Conference and Exhibit, 2006.
Due to the numerical methods for DTM, the [13]. J. Zhang, C. Zhang, Robustness of discrete periodically
algorithems of section 3.5 can be generalized to any time-varying control under LTI unstructured
perturbations. IEEE Transactions on Automatic
order LPTV system to obtain the GGM.
Control, 45, 7, 2000, pp. 1370-1374.
[14]. P. Brunovski, Controllability and linear closed-loop
controls in linear periodic systems, Differential
4. Conclusions Equations, 6, 4, 1969, pp. 296-313.
[15]. Y. T. Hagiwara, Periodically time-varying controller
The problem of SPM and GGM assessment for synthesis for multiobjective control of discrete-time
LPTV systems is formulated. Chang transformation systems and analysis of achievable performance."
makes it possible to reduce the SPM analysis for Hill Systems & Control Letters, 6, 2, 2011, pp. 296-313.

77
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

[16]. S. Bittanti, 30 years of periodic control-form analysis with bounded parameter variation, Iet Control Theory
to design, in Proceedings of the 3rd Asian Control & Applications 3, 10, 2009, pp. 1448-1461.
Conference, Shanghai, 2000, pp. 1253-1258. [20]. X. Yang, J. J. Zhu, A. S. Hodel, Singular perturbation
[17]. X. Su, M. Lv, H. Shi, Q. Zhang, Stability analysis for margin and generalized gain margin for linear time
linear time-varying periodically descriptor systems: a invariant systems, International Journal of Control,
generalized Lyapunov approach, in Proceedings of the 88, 1, 2015, pp.11-29.
IEEE World Congress on Intelligent Control & [21]. X. Yang, J. J. Zhu, Singular perturbation margin and
Automation, 2006. generalized gain margin for nonlinear time invariant
[18]. X. Xie, J. Lam, P. Li, A novel h∞ tracking control systems, International Journal of Control, 89, 3, 2016,
scheme for periodic piecewise time-varying systems, pp. 1-18.
Information Sciences, 484, 2019, pp. 71-83. [22]. P. V. Kokotovic, H. K. Khalil, J. O’Reilly, Singular
[19]. R. C. L. F. Oliveira, M. C. De Oliveira, P. L. D. Peres, perturbations methods in control: analysis and design,
Special time-varying Lyapunov function for robust Academic Press, New York, 1986.
stability analysis of linear parameter varying systems

78
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(020)

Obstacle Segmentation for Autonomous Guided Vehicles through Point

Cloud Clustering with an RGB-D Camera
M. Pires 1, P. Couto 1,2, A. Santos 3 and V. Filipe 1,4
1
University of Trás-os-Montes e Alto Douro (UTAD), 5000-801 Vila Real, Portugal
2
CITAB, Quinta de Prados, 5000-801 Vila Real, Portugal
3
Active Space Technologies, Parque Industrial de Taveiro, Lote 12, 3045-508 Coimbra, Portugal
4
INESC Technology and Science (INESC TEC), 4200-465 Porto, Portugal
Tel.: + 351259350356
E-mail: [email protected]

Summary: The movement of materials in industrial assembly lines can be done efficiently using Automated Guided Vehicles
(AGVs). However, visual perception of industrial environments is complex due to the existence of many obstacles in pre-
defined routes. With the INDTECH 4.0 project, we aim to develop an autonomous navigation system, allowing the AGV to
detect and avoid obstacles based on the processing of depth data acquired with a frontal depth camera mounted on the AGV.
Applying RANSAC algorithm and Euclidean Clustering to the 3D point clouds captured by the camera, we are able to isolate
obstacles from the ground plane and separate them into clusters. The clusters give information about the location of obstacles
with respect to the AGV position. In the experiments conducted in outdoor and indoor environments the results revealed that
the method is effective, returning high percentages of detection for most tests.

Keywords: Autonomous navigation, Obstacle avoidance, AGV (Automated Guided Vehicle), ROS (Robot Operating
System).

1. Introduction Ground plane and obstacle detection are essential

tasks for collision free navigation of autonomous mobile
Automation is one of the fastest developing fields of robots [6]. One of the most common methods to detect
robotics. With the ever-growing interest in automating the ground plane in a scene is based on processing depth
processes, both for efficiency or safety, the ability to information. After ground plane extraction, objects can
provide robots with autonomous navigation capabilities be detected in tridimensional data by using point cloud
is of paramount significance [1, 2]. segmentation, which is the method explored in this work.
Several sensor technologies and types of navigation An example of this kind of methodology is presented in
systems are available for the autonomous navigation of [7] where the authors present three different point cloud
mobile robots. A key part of mobile autonomous robots segmentation methods and compare their performance
is the ability to detect objects in their path and to be able with the k-means segmentation algorithm. In [8], the
to navigate freely with minimum risk of collision. authors explored the idea of combining different
Because of this, in industrial environments, AGVs are segmentation methods and post-processing algorithms
usually provided with laser scanners to prevent them for point cloud segmentation of aerial images of urban
from running into people or things. However, the areas. A 3D point cloud segmentation survey is presented
effectiveness of these systems can be affected by adverse in [9] where a comparative analysis on various state of
weather conditions such as direct sunlight, rain, snow, or the art point cloud segmentation methods is made.
fog. Even considering the use of laser scanners prepared On the project “INDTECH 4.0”, developed by a
to operate outdoors, other vehicles (e.g., forklifts) may consortium of companies and universities, one of the
not be detected, thus leading to collisions. As an work packages aims to study and develop autonomous
alternative, 3D point clouds captured with a depth navigation solutions, based on vectors of flexibility,
camera could be used in these complex conditions. collaboration, adaptability and modularity, to be
There are different ways to perform object detection implemented in the car assembly industry. In this
using 3D point clouds. Most solutions tackle this context, an Intel RealSense D435i camera was mounted
problem by using neural networks adapted to this kind of on the front of an AGV to collect depth data. The 3D
data. An important paper in this type of methods presents point clouds acquired are continuously processed to
the VoxelNet algorithm [3] that works with sparse 3D ensure obstacle avoidance, which is an essential task for
point clouds, without the need of manual feature real-time outdoor autonomous navigation of AGVs.
engineering. Further studies lead to the development of In this paper, we present a method to detect potential
algorithms like the one presented in [4] that uses RGB obstacles and to measure their relative position to the
images to generate detection proposals for 3D detection robot using point cloud data provided by the depth
and PointRCNN [5] that uses a bottom-up 3D proposal camera. This way, the AGV is guided through vision
generation method. technology, without any previous knowledge of the

79
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

environment that can be static or dynamic, indoor or approximated by their centroid. This process results in a
outdoor, in a process called natural navigation. point cloud with less points, but that still describes the
The remainder of this paper is structured as follows. original surface accurately.
Section 2 is a description of our methodology, detailing In our tests, we defined the limits of the distance
the various processing phases of our method. Section 3 window in the interval [1.5, 5] m along the z-axis for the
presents and analyses the results according to various pass-through filter and a voxel edge length of 0.07 m for
metrics relevant for this kind of algorithm. Section 4 the voxel grid.
presents the conclusions of this paper and suggestions of
future work.
2.2 RANSAC Segmentation

2. Proposed Methodology To successfully identify obstacles in the camera field

of view, it is necessary to segment the volume of the
A method to detect obstacles and their locations by point cloud that corresponds to the ground plane. This is
means of 3D point clouds provided by the Intel done to deliver a higher degree of separation between
RealSense D435i depth camera is proposed. The method objects, facilitating the process of grouping points into
processing sequence is shown in Fig. 1. clusters. For this purpose, the RANSAC (Random
Sample Consensus) algorithm is used [11].
RANSAC is an iterative algorithm for fitting
mathematical models to experimental data. This method
takes a number of arbitrary points from the experimental
set and instantiates a model using those points. Having a
candidate model defined, the number of points that are
within a defined tolerance to the plane is counted
(inliers). When the number of inliers exceeds a threshold,
the candidate model is chosen. If the number of inliers is
too low, the process is repeated until a plane is identified.
The PCL implementation of RANSAC further
Fig. 1. Processing Flowchart. includes the option of constraining the plane fit to a
specified axis within an angular tolerance. This helps in
identifying the ground plane by restricting the detection
The method consists in a ROS (Robot to horizontal planes.
Operating System) node that subscribes to the point Once the ground plane is identified, inliers are
cloud provided directly by the depth camera in removed, and the resulting point cloud solely contains
the ”/camera/depth/color/points” ROS topic. objects that may represent obstacles.
This point cloud is subjected to three different stages
of processing, resulting in the separation of obstacles into
point clusters. Each of these clusters represents a 2.3. Euclidean Clustering
different obstacle and, through the clusters, we can obtain
the position of each obstacle relative to the AGV. After After removing the ground plane, Euclidean
processing, all detected clusters are merged in a point Clustering is carried out to identify the various clusters
cloud that is published in the “total_clusters” ROS topic. of points present in the point cloud.
In addition to ROS, functionalities provided by the Euclidean Clustering consists in separating sets of
Point Cloud Library (PCL) [10] have been used in all points according to the Euclidean distance between those
processing steps. points. If two points are within a predefined distance
threshold, they are considered to be in the same cluster.
However, if that distance is higher than the threshold, the
2.1 Pass-through Filter and Voxel Grid points are segregated into different clusters.
The PCL implementation of Euclidean Clustering
The first processing stage consists in applying a pass- also contains the ability to define the minimum and
through filter. This filter is used to restrict the point cloud maximum cluster sizes to mitigate the detection of
limits to a relevant distance window. undesirable sets of points.
The ultimate use of this algorithm is to apply it in a Once all the clusters are defined, we can visualise
navigation system of a low speed AGV for load them and measure the position to each obstacle. For the
transportation. Thus, it is possible to limit the interest of avoiding AGV collisions, our method returns
computational effort while conserving a high relevance the coordinates of the closest point of each cluster.
of the results since, due to the low speed, we do not need
to detect obstacles that are far away from the robot.
Another processing tool used to save computational 3. Results
time is the voxel grid to down sample the input point
cloud. This filter divides the point cloud into voxels with Throughout this work, various tests have been carried
user-defined size. The points present in each voxel are out to optimise this algorithm. These tests were

80
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

performed in the premises of Active Space Technologies For each ROS bag, the point from each cluster closest
and consisted in recording ROS bags using the D435i to the camera is obtained and published in a ROS topic
depth camera with various types of obstacles located at (cluster_closest_point) for analysis. This topic contains
different distances. We tested the algorithm for both the xyz coordinates of the closest point of each object.
static and mobile obstacles and in indoor and outdoor The total number of point clouds analysed by the
environments. algorithm is also counted.
To represent static obstacles, chairs placed at 1.8 m, A python script is used to subscribe to the
2.5 m and 4.5 m were used. The distance to the chairs (cluster_closest_point) topic and the information is
was measured taking the closest part of the chair into assembled and plotted to a 3D graph. This graph gives us
account. This set up allows us to evaluate the algorithm a sense of the position of obstacles in the corresponding
performance for different distances across the defined test.
depth window ([1.5, 5] m). For each of the three In addition to visualising the data, a clustering
distances, a ROS bag is recorded with two chairs side- algorithm is applied to this set of points, originating
by-side, indoors and outdoors. clusters of points that correspond to detections of the
Another test was made with three chairs, one at each same object. Using the clusters of closest points, we
distance mentioned above, to assess simultaneous count the number of points that represent each obstacle
detection of various obstacles at different distances. The detected by the camera. That value determines how many
final test with chairs consisted in moving the camera times each obstacle is detected.
perpendicularly to the three chairs, in the direction of the By comparing these values with the number of point
chair furthest away. This test simulates object detection clouds processed by our object detection algorithm we
of static obstacles with a circulating AGV. are able to calculate the effectiveness of the object
For mobile obstacles, we performed tests in which detection.
people moved in front of the camera in various Each ROS bag has a duration of approximately
directions. A final test was made in which the camera was 15 seconds, which at 30 fps represents a total of
also moving among wandering people, simulating a more 450 frames for each test run. The process described
complex scenario that would be common for an industrial above is repeated 10 times for each ROS bag and the
AGV. results are presented in Tables 1 and 2.
Every test mentioned above has been done indoors
and repeated outdoors.
This batch of tests enables us to get a sense of the Table 1. Indoor detection effectiveness.
algorithm’s performance in different conditions
considered relevant for the project. Figure 2 represents Distance (m) Indoor Detection Effectiveness [%]
the typical outputs of this algorithm concerning the three 1.8 92.72 ± 5.04
major stages of processing. 2.5 70.59 ± 6.98
4.5 51.02 ± 0.70

Table 2. Outdoor detection effectiveness.

Distance (m) Outdoor Detection Effectiveness [%]

1.8 99.05 ± 0.25
2.5 99.12 ± 0.86
4.5 89.22 ± 1.51

From these results we conclude that the method is

very effective, returning high percentages of detection
for most tests. Also, the percentages of detection of both
Fig. 2. Results for the four stages of processing. From top-left objects are very similar for all distances.
to bottom-right: original point cloud, down sampled and range It is seen that the detection rates drop more
limited point cloud, point cloud after ground plane removing significantly for indoor obstacles that sit farther away
and point cloud with detected clusters (two chairs).
from the camera. This is due to the poorer luminosity and
the fact that for farther objects, the point cloud includes
3.1 Object Detection Effectiveness parts of the ceiling, extending the dimensions of the point
cloud and making the object detection process more
difficult. In this case, the performance bottleneck is
To measure the object detection effectiveness, we
mostly due to the increased size of the point cloud, hence
counted the number of times that each object is detected
explaining why, for outdoor conditions, an efficacy drop
and compared it with the number of frames of each test.
is not as drastic.
This was done using the ROS bags corresponding to the
We can also conclude that, albeit detection rates vary
placement of chairs side-by-side at various distances and
according to environmental conditions, there is not a
using indoor and outdoor measurements.
variance in detection efficacy between the two obstacles.

81
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Thus, detection success decreases with distance to the whole test regardless of the movement of the AGV and
obstacle, but does not appear to change for obstacles at of the person.
the same distance. The percentage of obstacle detection
is also significantly more stable in outdoor conditions, as
can be concluded by the lower standard deviation values.

3.2 Distance Measurements

A major requirement of this work is not only to detect

obstacles in front of the robot but also to measure their
position relative to the camera.
Using the same ROS bags, with two chairs side-by-
side at various distances, we analysed the distance
measurements obtained with our algorithm.
The results in Tables 3 and 4 show the mean and
standard deviation values for the distances obtained in Fig. 3. Moving obstacle detection (wandering pedestrian).
each test, for indoor and outdoor conditions. In this test, both the person and the AGV are moving.
Top: original point clouds; bottom: detected clusters

Table 3. Indoor distance measurements analysis.

3.4 Frame Rates
Indoor Measurements
Average Standard
Distance (m) Since this algorithm is intended for real-time obstacle
Distance (m) Deviation (m)
1.8 1.85 0.03
avoidance, it is also important to analyse the frame rate.
2.5 2.60 0.09 Using a computer with an Intel(R) Core(TM) i7-
4.5 5.01 0.11 6700HQ CPU @ 2.60 GHz, we do not register significant
frame rate drops relatively to the native frame rate of the
point cloud ( 30 fps), with the maximum frame rate drop
Table 4. Outdoor distance measurements analysis. being of 58 % in one of the tests, but that still results in
a rate of 12 fps for obstacle detection. That being said,
Outdoor Measurements most tests performed had a frame drop inferior to 10 %
Average Standard relative to the native frame rate of the camera, with the
Distance (m)
Distance (m) Deviation (m) average frame drop in all tests being of 18 %. These
1.8 1.91 0.03 values mean that our method is applicable in a slow
2.5 2.56 0.13 moving AGV for real-time object detection.
4.5 4.95 0.20

4. Conclusions
For distance measurements, we are able to obtain
reasonable values, with an expected drop in accuracy for In this work, an object detection algorithm using
farther objects. These measurements are entirely point cloud data from an Intel RealSense D435i is
dependent on the accuracy of the Intel RealSense D435i proposed. This algorithm is meant to be applied in an
depth camera, and, as demonstrated, we can rely on the autonomous navigation system with obstacle avoidance
good accuracy provided by the depth sensor for our capabilities for AGVs in dynamic industrial
application. environments.
The presented results show this method can detect
obstacles and measure their position relative to the AGV
3.3 Moving Object Detection in various environmental conditions. This is very
important in a SLAM (Simultaneous Localization and
For moving obstacles, the performance of the Mapping) context to enable robust autonomous
algorithm shows favourable results, although the analysis navigation of robots. This method is also able to function
of these tests has been made merely by visual inspection. in real-time, which is crucial for autonomous navigation.
The proposed method is able to segment and follow The algorithm detects objects with high efficacy and
moving obstacles smoothly with high degree of accuracy returns results with inconsequential frame drops from the
and with high frame rate without apparent fail in camera’s native frame rate. The resulting frame rates of
detection. detection are adequate for a slow moving AGV, as
Figure 3 shows two frames from the mobile intended for our application.
AGV/wandering person test where it is clear that the The obstacle position measurements returned by our
person is segmented accurately and there are no false method also present high degrees of precision that
positives. The detection remains accurate during the improve with illumination and as obstacles get closer to

82
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

the camera, characteristics that are innate to the Intel [3]. Zhou Y. and Tuzel O. Voxelnet: End-to-end learning for
RealSense D435i depth camera. point cloud based 3D object detection, in Proceedings of
Further developments of this algorithm may provide the IEEE Conference on Computer Vision and Pattern
the capability to create a system that generates odometry Recognition, 2018, pp. 4490-4499.
[4]. Qi C. R., Liu W., Wu C., Su H. and Guibas
measurements using depth camera data, such as L. J., Frustum pointnets for 3D object detection from
implementing a visual odometry system for SLAM. Such RGB-D data, in Proceedings of the IEEE Conference on
algorithms are useful to perform sensor fusion with an Computer Vision and Pattern Recognition, 2018,
inertial measurement unit or can even replace this kind pp. 918-927.
of sensors. Finally, the methodology can be extended to [5]. Shi S., Wang X. and Li H. PointRCNN: 3D object
process RGB images, acquired simultaneously by the proposal generation and detection from point cloud, in
Intel camera, for object detection and classification. Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, 2019, pp. 770-779.
[6]. Kircali D. and Tek F. B. Ground plane detection using an
Acknowledgements RGB-D sensor, in Proceedings of the 29th International
Symposium on Computer and Information Sciences,
This work was funded by Project “INDTECH 4.0 – 2014, pp. 69-77.
new technologies for smart manufacturing”, POCI- 01- [7]. Habermann D., Hata A., Wolf D. and Osório F. S.
0247-FEDER-026653, financed by the European 3D point clouds segmentation for autonomous ground
vehicle, in Proceedings of the III IEEE Brazilian
Regional Development Fund (ERDF), through the Symposium on Computing Systems Engineering, 2013,
COMPETE 2020 - Competitiveness and pp. 143-148.
Internationalization Operational Program (POCI). The [8]. Vosselman G. Point cloud segmentation for urban scene
work is also funded by national funds from FCT - classification, Int. Arch. Photogramm. Remote Sens.
Portuguese Foundation for Science and Technology, Spat. Inf. Sci, Vol. 1, No. 257-262, 2013, p. 1.
under the project UIDB/04033/2020. [9]. Nguyen A. and Le B. 3D point cloud segmentation: A
survey, in Proceedings of the 6th IEEE Conference on
Robotics, Automation and Mechatronics (RAM), 2013,
References pp. 225-230.
[10]. Rusu R. B. and Cousins, S. 3D is here: Point cloud library
[1]. Pires M. A., Natural navigation solutions for AMRs and (PCL), in Proceedings of the IEEE International
AGVs using depth cameras, MSc diss., Universidade de Conference on Robotics and Automation, 2011, pp. 1–4.
Coimbra, 2021. [11]. Fischler M. A. and Bolles R. C., Random sample
[2]. Lynch L., Newe T., Clifford J., Coleman J., Walsh consensus: a paradigm for model fitting with applications
J. and Toal D. Automated ground vehicle (AGV) and to image analysis and automated cartography,
sensor technologies - a review, in Proceedings of the 12th Communications of the ACM, Vol. 24, No. 6, 1981,
IEEE International Conference on Sensing Technology pp. 381–395.
(ICST’18), 2018, pp. 347-352.

83
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(023)

Competency-based Education of the Mechatronics Engineer in the Transition

from Manufacturing 3.0 to Industry 4.0
Eusebio Jiménez López 1, Francisco Javier Ochoa Estrella 2, Gabriel Luna-Sandoval 3,
Flavio Muñoz Beltrán 2 , Francisco Cuenca Jiménez 4 and Marco Antonio Maciel Monteón 3
1
Universidad Tecnológica del Sur de Sonora-ULSA Noroeste, CIAAM,
Dr. Norman E. Borlaug Km. 14. Cd Obregón, Sonora, Mexico
2
TecNM/ Instituto Tecnológico Superior de Cajeme,
Carretera Internacional a Nogales Km. 2 s/n, Cd Obregón, Sonora, Mexico
3
Universidad Estatal de Sonora, Carretera,
San Luis Río Colorado - Sonoyta km. 6.5, Parque Industrial, San Luis Río Colorado, México
4
Universidad Nacional Autónoma de México, Av. Universidad 3004, Col, Copilco Universidad,
Coyoacán, 04510, Ciudad de México, CDMX, Mexico
Tel.: + 52 (644) 414-8687, fax: + 52 (644) 414-8687
E-mail: [email protected]

Summary: The changes in industrial processes that are being introduced by the new industrial revolution called "manufacturing
4.0" have broad repercussions on engineering education. Engineering education depends on the competencies required by the
industrial sector; however, the technological transition is not constant and homogeneous in companies or in productive regions,
especially in developing countries such as Mexico, so there are still many productive systems that operate under the philosophy of
manufacturing 3.0. It is necessary to design an adequate profile and determine an appropriate educational model for the mechatronic
engineer in the context of the transition between manufacturing 3.0 and industry 4.0, which allows him/her to have the necessary
competences to face the current and future challenges of the companies. This paper proposes Competency Based Education (CBE)
as the basis for training mechatronic engineers.

Keywords: Mechatronics, Industry 4.0, Education, Manufacturing 3.0, Competences, Engineering.

1. Introduction existence of obsolete educational technology, teachers

who are not updated and do not have a good command of
Industry 4.0 not only presents a radical challenge to computer tools, unconsolidated educational models, the
companies but also extends to education, especially to weakness of national value chains and the lack of an
engineering education. The incursion of disruptive effective public technological policy, among others.
technologies (Big Data, the Internet of Things, Artificial Although those countries that are at the forefront of
Intelligence, Cloud Computing, Machine Learning, the implementation of Industry 4.0 can afford a better
Augmented Reality, etc.) in industrial processes evolution in education, in developing countries such as
motivates the training of highly qualified and intelligent Mexico, this transition is not continuous or
professionals who are able to be trained with multiple homogeneous, so it is necessary to study and propose
skills, such as the ability to work in different business appropriate educational methods or approaches to train
units, strategic thinking, computer skills, centralization the engineer according to the local reality (mainly
capacity, general aptitude, leadership, culture protection focused on Manufacturing 3.0) and according to the new
and information security, among others [1]. challenges posed by Industry 4.0.
Similarly, the engineer must acquire (core or The competency-based approach can be a way to
transversal) process and content competencies, social, train new engineers, especially mechatronic engineers,
technical, resource management and systems since it fosters comprehensive training, flexibility and
competencies. These include the ability to interact with self-management, promotes active learning, develops
machines, data analysis, planning and programming; technical and social skills and encourages engineers to
negotiation, collaboration, team building, flexibility, solve problems in complex situations, among other
rapid learning ability, problem detection and resolution, relevant characteristics.
and autonomous initiative among others [2].
However, although numerous authors propose
changes in engineering education due to the accelerated 2. Technological Transition: Technical Aspects
incursion of Industry 4. 0 the skills and competencies
suggested in engineering education are difficult and The current technological transition implies a new
complex to acquire, mainly in developing countries due challenge for mechatronic engineering as machinery,
to numerous factors, such as: the lack of an effective processes and factories are evolving from traditional
relationship between companies and universities, the automation to intelligent industry. Today, concepts such

84
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

as manufacturing cell or CIM (Computer Integrated service industries [3]. One of the crucial infrastructures
Manufacturing) have evolved to cyber-physical systems that must be valued in the training of the mechatronics
and digital twins, integrated with artificial intelligence, engineer is computational as it is constantly changing and
advanced simulation and autonomous and cooperative has a wide influence on the technologies supporting
robots. Similarly, the design and manufacture of various Industry 4.0, so constant updating of curricula can help
parts and components are now performed by advanced the computational infrastructure not to be overtaken by
software and simulation, as well as by additive changes in processing speed or become obsolete in a
manufacturing systems. Information is stored and short period of time. In fact, mechatronics has gone from
processed by Big Data and cloud computing techniques, being purely associated with essentially autonomous
and protected by cybersecurity methods. Training is systems, such as robots, to providing the intelligent
aided by virtual and augmented reality techniques. To objects and systems that are the building blocks of cyber-
these challenges must be added the great improvements physical systems and, therefore, of systems based on the
in interconnection and communication provided by the Internet of Things and the cloud, and all this has been
Internet of Things and the improvement of IT integration made possible due to the evolution of computer
systems in companies. All these technological technologies, so the importance of these technologies in
improvements bring great challenges only on the the education of the mechatronics engineer is
technical side for mechatronics engineers. transcendental.
With regard to the various companies, especially in Mechatronics was born at the height of the third
Mexico, the technological transition (from industrial revolution and, consequently, together with
manufacturing 3.0 to industry 4.0) has become a serious computing, informatics and robotics, it brought a
problem, as they face the challenge of deciding whether significant technological impact to the industrial world
a technological upgrade (almost total change of over a period of at least 50 years. Today, the education
production systems) that implies a considerable of the mechatronics engineer faces the major challenges
economic investment or a technology reconversion that of transforming CIM systems (core of manufacturing
involves a substantial improvement to the production 3.0) to modern cyber-physical systems (CPS) that
systems they already have and, therefore, less economic integrate various Digital Twins (DT) and are the basis of
investment, is relevant. This type of analysis and decision Industry 4.0.
making undoubtedly involves mechatronics engineers, Cyber-physical entities are described as any entity
so it will be necessary to use reverse engineering composed of physical and cyber-physical elements that
techniques for the evaluation of the technologies interact autonomously with each other, with or without
available to the companies if a technological human supervision [4]. The main characteristic of a
reconversion criterion is to be applied. Likewise, it will Cyber-Physical System is its ability to merge the physical
be necessary to be involved in the technology transfer and virtual worlds, in particular, through its embedded
process, which implies knowledge of the purchase, software.
installation and start-up of the systems acquired and One of the technologies on which cyber-physical
knowledge of intellectual property for the study of systems are supported is simulation, described as the
patents and licenses in the event that a technological imitation of the behavior of the properties of a system.
update is chosen. One of the most important concepts within CPS are
The transition between two industrial revolutions Digital Twins that use simulation as part of their
generates an important series of scientific and processes.
technological challenges, not only for companies, but A digital twin is defined as "a formal digital
also for governments, the education sector, trade and representation of some asset, process, or system that
strategic competition between regions or countries. captures attributes and behaviors of that entity suitable
for communication, storage, interpretation, or processing
within a given context" by the Industrial Internet
3. Mechatronics Education in Industry 4.0 Consortium [5].
In general CPS are understood from the point of view
The synergic integration of Mechanics, Control, of integration that combines systems or processes that
Electronics and Computing in the development of occur in reality with computational processes, while
products, machines, processes and production systems Digital Twins or digital copy are used in real-time
has always had great challenges for the mechatronics improvement and optimization processes. This implies
engineer, so now we must consider that to the traditional that the Digital Twin relates to its physical counterpart in
challenges of mechatronics we must add various two ways: 1) It receives from it information generally
requirements and technical requirements demanded by from sensors and 2) The valuable or processed
industry 4.0, especially the various techniques of information from the digital copy is sent to the physical
artificial intelligence, digitization and the internet part to achieve specific objectives. Cyber-physical
of things. systems and Digital Twins are two of the central concepts
The fourth industrial revolution requires that must necessarily be taken into account for the design
multidisciplinary knowledge (as in the case of or redesign of mechatronics career curricula and for the
Mechatronics) and expertise. Multidisciplinary skills are design of multidisciplinary competencies.
mandatory and real-time collaboration will be more On the other hand, the most significant
important than ever for success in the manufacturing and transformation related to the way products are

85
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

manufactured is digitalization. The fourth industrial evidence of the level of development of the defined set
revolution aims to optimize the computerized third of competencies [9].
industrial revolution (Industry 3.0). This requires the On the other hand, with the purpose of making
development of intelligent equipment with access to students access active and meaningful learning, active
more data, thus becoming more efficient and productive methodologies were introduced. Prince and Felder
by making decisions in real time [6]. In this sense, the compiled six most innovative teaching methods, namely,
different educational models around mechatronics inquiry learning, problem-based learning, project-based
should consider as a priority two fundamental aspects for learning, case-based learning, discovery learning, and
the training of engineers: 1) The formal study of the just-in-time teaching. These teaching methods are
various algorithms and tools of Artificial Intelligence and learner-centered and motivate students with real-world
2) The conformation of intelligent teaching and learning problems, cases, and observations, which triggers
environments. There is a range of tools that support students desire to learn knowledge [10].
Artificial Intelligence, such as Deep Learning, Machine There are several proposed taxonomies for
Learning, Neural Networks, Fuzzy Logic, Genetic engineering education. One of them consists of seven
Algorithms, Natural Language Processing, Knowledge categories [11]: 1) Engineering specific work includes
Engineering, Expert Systems, Data Mining and others. general engineering work, specialized engineering work,
These tools will be commonly used in industries in the and engineering combined with other work, 2) Non-
coming years. engineering work specific work includes general,
Faced with the constant challenges of manufacturing supervision and leadership, and management and
4.0, mechatronics and mechatronics education must administration, 3) Communication 4), Interpersonal
reinvent themselves. To this end, designers of interactions include personal and teamwork, 5) Personal
mechatronic systems of all varieties and types must be dispositions include general and discipline 6), Adaptive
aware of a set of fundamental principles if they are to dispositions include personal development, lifelong
ensure success. Some of these principles are as follows: learning and change management, and 7) Advanced
technology; complexity; operational software; hardware; dispositions include achievement orientation, impact and
reliable and rechargeable power supplies; innocent influence, conceptual thinking, analytical thinking,
human error; connectivity; privacy, dependency, initiative, self-confidence, interpersonal understanding,
ubiquity, and hybrid society, among others [7]. concern for order, information seeking, teamwork and
cooperation, experience and service orientation.
The competency model may be ideal for the
comprehensive training of the mechatronics engineer
4. Competency-Based Approach
whose professional activity will be to solve problems in
the transition from manufacturing 3.0 to industry 4.0, as
The training of the new engineer not only demands this model offers students the possibility to hone their
an update of the curriculum but also requires universities ability to recognize, build and manage their skills. It
to have a strong connection with companies and the allows students to evaluate and improve their
adoption of new educational approaches, such as performance, solve difficulties, plan innovative
Competency-Based Education (CBE) and active learning strategies and interpret diverse situations in an ethical
methodologies, as well as a rational contextualization of and responsible manner. The competencies integrate a
the conditions in which the technological transition is response to society, the culture of quality, globalization
taking place in each country and in each productive and business competitiveness.
region. Competencies are understood as a combination of Although in Mexico, as in other Latin American
skills, abilities, knowledge, aptitudes, attitudes and countries, CBE has been implemented for two decades,
values to perform well in complex and authentic this educational approach has not been homogenized nor
contexts. has it been effective in engineering education. Some
Competency-Based Education is both goal-oriented universities in other countries have been applying the
and outcome-oriented. It is an approach embedded in competency-based approach and have even adopted
adult learning theory, which posits that adult learners are active methodologies such as Project Based Learning
more likely to engage in learning that is focused on a [12] and Learning Factories [13] to train engineers in the
specific goal or outcome. This means that both teaching current industrial context. It is necessary that CBE be
and learning in CBE are oriented toward the development established in an integral way in engineering careers in
of an explicitly stated and described skill, and measured Mexico and in other Latin American countries, since in
by observing how the learner performs this skill [8]. A this way it will be possible to design effective curricula
Competency-Based Education approach focuses on the and learning strategies, as is the case of Mechatronics
development and demonstration of competencies by a Engineering, in accordance with the demands of the new
learner. Thus, a CBE perspective puts the learner at the industrial revolution.
center of the model and provides a comprehensive
education oriented to the competencies demanded in the
world. CBE has strong implications in the teaching- 5. Conclusions
learning process. This approach alters the curricular
organization, the way content is taught, the learning The current profile of the mechatronic engineer must
environments, the assessment instruments and the be designed taking into account the context of each

86
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

country and the implications generated in the transition [4]. DeSmit Z., Elhabashy A. E., Lee J. Wells L. and Camelio
from manufacturing 3.0 to industry 4.0. The objective of J. A., Cyber-Physical Vulnerability Assessment in
their training should be the acquisition of competencies Manufacturing Systems, Procedia Manufacturing, 5,
and technical and social skills that allow them to solve 2016, pp. 1060-1074.
[5]. Jian D., Tian M., Qing Z., Zhen L. y Ji‑Y., Design and
problems in both technological approaches.
application of digital twin system for the blade‑rotor test
The CBE is a good option for the formation of the
rig, Journal of Intelligent Manufacturing, 2021.
mechatronic engineer in the context of the current
[6]. Souza, M. L. H., C. Andre da Costa, G. de Oliveira
technological transition, because it is possible to design Ramos, Rodrigo da Rosa Righi, A survey on decision-
and evaluate the necessary competencies in engineering making based on system reliability in the context of
students for each requirement needed by the industry. It Industry 4.0, Journal of Manufacturing Systems, 56,
is necessary to incorporate in the curricula and study 2020, pp. 133–156.
plans of the mechatronics career special topics of cyber- [7]. Russell D., Reinventing Mechatronics - Final Thoughts,
physical systems, digital twins and artificial intelligence in Reinventing Mechatronics, Xiu-Tian Yan, David
with the aim of making inroads into the base technologies Bradley, David Russell and Philip Moore (Eds.),
of industry 4.0, and promote the development of Springer, 2020, pp.179-186.
[8]. Mace K. L., Welch C. E., The future of health professions
competencies and skills in programming and computing
education: considerations for competency-based
in students, since this topic is of great relevance and education in athletic training, Athletic Training
importance in the current context of technological Education Journal, 14, 03, 2019, pp. 215–222.
applications. [9]. Félix L. C., Rendon A. E. and Nieto J. M., Challenge-
Mechatronics as an area of technological application based learning: an I-semester for experiential learning in
and education of the same, face the difficulties and Mechatronics Engineering, International Journal on
challenges involved in transforming CIM systems that Interactive Design and Manufacturing, 13, 2019,
represent the technologies of manufacturing 3.0 to cyber- pp. 1367–1383.
physical systems that make up the core of industry 4.0. [10]. Prince M. J. and Felder R. M., Inductive teaching and
learning methods: Definitions, comparisons, and research
bases, Journal of Engineering Education, Vol. 95, 2006,
pp. 123-138.
References [11]. Woollacott L. C., Validating the CDIO syllabus for
engineering education using the taxonomy of engineering
[1]. Kaur R., Awasthi A., Grzybowska K. Evaluation of Key competencies. European Journal of Engineering
Skills Supporting Industry 4.0—A Review of Literature Education, 34, 6, 2009, pp. 545–559.
and Practice, in Grzybowska K., Awasthi A., Sawhney R. [12]. Atmojo U. D., Project-based Learning at Aalto Factory of
(Eds.), Sustainable Logistics and Production in Industry the Future - A Flexible Production-based Industrie 4.0
4.0. EcoProduction (Environmental Issues in Logistics Learning Factory, in Proceedings of the 11th Conference
and Manufacturing). Springer, Cham. 2020. on Learning Factories (CLF’2021), 1–2 July, 2021,
[2]. BRICS skill development working group: Skills Graz, Austria, 2021.
development for Industry 4.0. 2018. [13]. Montoya A., Guarín A. and Mora J., Advantages of
[3]. Stankovski S., Ostojić G., Zhang X., Baranovski I., Learning Factories for Production Planning Based on
Tegeltija S., and Sabolč Horvat. Mechatronics, Shop Floor Simulation: A Step towards Smart Factories
Identification Tehnology, Industry 4.0 and Education. in Industry 4.0, in Industry 4.0 - Impact on Intelligent
in Proceedings of the 18th IEEE International Symposium Logistics and Manufacturing, Tamás Bányai (Ed.),
INFOTEH-JAHORINA, 20-22 March, 2019. Chapter 2, IntechOpen, 2020.

87
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra
(024)

Simulation of a Collision and Obstacle Avoidance Algorithm

for Cooperative Industrial Autonomous Vehicles
J. Grosset 1,2, A.-J. Fougères 2, M. Djoko-Kouam 2, C. Couturier 1 and J.-M. Bonnin 1
1
IMT Atlantique, IRISA, Rennes, France
2
ECAM Rennes, Louis de Broglie, Campus de Ker Lann, Bruz, Rennes 35091, France
E-mail: [email protected]

Summary: Industry 4.0 leads to a strong digitalization of industrial processes, but also a significant increase in communication
and cooperation between the machines that make it up. This is the case with intelligent autonomous vehicles (IAVs) and other
cooperative mobile robots which are multiplying in factories, often in the form of fleets of vehicles, and whose intelligence
and autonomy are increasing. The implantation and deployment of IAVs fleets raises several challenges: acceptability by
employees, vehicle location, traffic fluidity, vehicle perception of changing environments. Simulation offers a good framework
for studying solutions for these different challenges. Thus, we propose in this paper the extension of a collision detection
algorithm to deal with the obstacle avoidance issue. The conclusive simulation will allow us to experiment in real conditions.

Keywords: Simulation, Autonomous vehicles, Cooperative collision avoidance.

1. Introduction IAVs more autonomous, we can particularly note the

location and positioning of vehicles [4], as well as the
One of the challenges of Industry 4.0, is to avoidance of other vehicles or obstacles [5]. This last
determine and optimize the flow of data, products and problem can be solved by the cooperation between
materials in manufacturing companies. To realize these IAVs [2, 6].
challenges, many solutions have been defined [1] such The capacity to exchange information between the
as the utilization of automated guided vehicles (AGVs), different IAVs of a fleet should improve this autonomy
intelligent autonomous vehicles (IAVs) and other [7, 8]. The study [2] proposed a cooperation strategy
cooperative autonomous robots. The implantation and based on the exchange of messages to determine the
deployment of intelligent autonomous vehicle fleets priority to pass an intersection between IAVs. The
raises several challenges: acceptability by employees, solution requires the vehicle to know its own position,
vehicle location, traffic fluidity, vehicle perception of and to be able to communicate with the other vehicles.
changing environments (dynamic). In this context, The collision avoidance algorithm presented in [2]
autonomy is reduced to predetermined trajectories. allows IAVs to communicate and cooperate using
To improve the autonomy of a fleet, one way it to different types of messages.
develop a collective intelligence to make the behaviors The communication between IAVs is done with 3
of vehicles adaptive. We will focus on a class of different types of messages:
problems faced by IAVs related to collision and • Hello_msg: message to indicate its presence with its
obstacle avoidance. This occurs when two vehicles position
need to cross an intersection at the same time, known • Coop_msg: message before an intersection area to
as a deadlock situation. But also, when obstacles are determine priority.
present in the aisles and need to be avoided by the • Ack_msg: message to confirm receipt of a
vehicles safely. Coop_msg.
In this paper, we will propose an enhancement to The European Institute of Telecommunications
the collision avoidance algorithm experimented in the Standards (ETSI) has published a standard for this kind
study [2], in order to handle the problem of obstacle of cooperative awareness messages (CAM) (ETSI EN
avoidance. 302 637-2 standard [9, 10]) and decentralized
environmental notification message (DENM) (ETSI
EN 302 637-3 standard [11]). These specifications and
2. State of the Art messages are approved and constitute building blocks
for the safety of future intelligent transport systems
For the consumer sector, autonomy of vehicles is (ITS) [12]. The purpose of CAM messages is similar to
well determined with 6 levels of autonomy [3]. Hello_msg in [2]. They make it possible to locate
However, no such scale exists in the industrial context. vehicles in real time relative to each other. DENM
Our objective is to improve the IAV autonomy messages are alert messages. They are issued at the
integrated in a fleet based on collective intelligent time of an unexpected event in order to cooperate, warn
strategies. Among the problems to be solved to make and disseminate information in the geographical are

88
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra
concerned. They are important messages that would autonomy. The individual autonomy of the IAVs
complete the range of possible messages to be strengthens their collective autonomy.
exchanged to cooperate and avoid collisions for the The messages exchanged by the different IAVs
IAVs in the Bahnes et al. algorithm [2]. remain consistent with the Bahnes’s algorithm. We just
propose two new message types for the collaborative
perception (added to the three messages defined by
3. Algorithm Improvement Bahnes et al.: Hello_Msg, Coop_Msg and Ack_Msg):
• Obstacle_msg: a message sent by an IAV agent to
The collision avoidance algorithm of [2] makes it the other IAV agents circulating in the warehouse
possible to deal with the priority of different vehicles to indicate the presence of a perceived obstacle.
when approaching an intersection. However, it does not • Alert_msg: a message sent by an IAV agent to the
deal with the problems of detection, communication other IAV agents circulating in the warehouse to
and avoidance of fixed or moving obstacles (e.g. indicate an unavoidable obstacle.
human operators).
We extend the algorithm of Bahnes et al. to handle
the presence of fixed or moving obstacles in Fig. 1. 4. Simulation Results
Then, we simulate the algorithm staying within the
framework of the three scenarios proposed by [2]. The traffic plan presented in Fig. 3. It shows the
These simulations rely on an agent-based model where different scenarios that we consider as a benchmark
IAVs are identified [13, 14]. Indeed, agent-based plan to compare results. Ten IAVs are distributed over
simulation for IAVs is the most common in the same 3 circuits: the red IAVs on the first circuit, the blue ones
way as simulations based on discrete events or robotics on the second and the yellow ones on the third.
software [15]. It involves different intersections, where vehicles
IAV agents have the ability to exchange messages can arrive from different sides like in a warehouse (4
and are equipped with radar. This allows them to detect intersections are shown in Fig. 3.). Thus, it provides the
vehicles in front of them. For instance, given an IAV different characteristics of an industrial environment
agent ai, if another IAV agent aj in front of it is stopped and allows us to realize simulated experimental tests in
or travelling at a slower speed, the IAV agent ai can line with realistic scenarios.
detect it with its radar and stop accordingly to avoid We notice in the simulation that the avoidance is
hitting it, as in Fig. 2. well respected and the obstacles are perceived by the
To improve the collective autonomy of the IAVs it IAVs. Therefore, the simulation validates the extended
is essential that they have a good capacity for individual Bahnes’s algorithm with collision avoidance and fixed
or dynamic obstacle detection processing.

Fig. 1. Improvement of Bahnes’s algorithm to treat the problem of collision and obstacles

89
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 2. Simulation of radar use: (a) at the top of the picture: one blue and three yellow IAVs arrive near the intersection, (b)
while waiting for the yellow IAV to pass the intersection, the radar of the blue IAV and the two other yellow IAV allow
them to stop and keep their distance to avoid colliding

Fig. 3. Simulation of the scenarios: (a) in the center of the picture: a blue and yellow IAV arrive at an intersection,
(b) the yellow IAV passed the intersection after communicating with other IAVs, (c) on the left side of the picture: a red IAV
perceives a fixed obstacle in front of him, (d) a red IAV avoided the obstacle.

90
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

5. Conclusions and Perspectives Artificial Intelligence (ASPAI 2021), Porto, Portugal,

November 17-19, 2021, pp. 80-86.
[5]. A. M. Nascimento, L. F. Vismari, C. B. S. T. Molina,
In an Industry 4.0 context, many actors cross paths
P. S. Cugnasca, J. B. Camargo, J. R. de Almeida
in different areas of a warehouse: vehicles, operators, and A. Y. Hata, A systematic literature review about
obstacles (objects that fall or are left in aisles may the impact of artificial intelligence on autonomous
appear). vehicle safety. IEEE Transactions on Intelligent
The algorithm in [2] proposed a message-based Transportation Systems, 21, 12, 2019, pp. 4928-4946.
communication protocol between vehicles to prioritise [6]. M. Hafner, D. Cunningham, L. Caminiti, and D. Del
passage through an intersection. We extended it in Vecchio, Cooperative collision avoidance
order to have the possibility to handle the detection of atintersections: Algorithms and experiments. IEEE
Transactions on Intelligent Transportation Systems,
these fixed and mobile obstacles (Fig. 1). 14, 3, 2013, pp. 1162-1175.
We have validated the algorithm by a simulation [7]. C. Medrano-Berumen and M. İlhan Akbaş, Validation
approach with the traffic plan presented in [2]. of decision-making in artificial intelligence-based
As an extension of the work and the aim to propose autonomous vehicles, Journal of Information and
a cooperation protocol for a collaborative collision and Telecommunication, 5, 1, 2021, pp. 83–103.
obstacle detection, the messages will respect the [8]. A. Daniel, K. Subburathinam, B. Muthu, N. Rajkumar
standards defined by ETSI [9, 11]. and S. Pandian, Procuring Cooperative Intelligence in
Autonomous Vehicles for Object Detection through
In future work, we will work to simulate others
Data Fusion Approach, IET Intel. Transport Syst., 14,
levels of autonomy using collective strategies – 2020, 11, pp. 1410–1417.
cooperation between IAVs but also cooperation [9]. Intelligent Transport Systems (ITS); Vehicular
integrating the infrastructure and the environment of Communications; Basic set of Applications; Part 2:
the IAVs. We also plan to do real experimentations Specification of Cooperative Awareness Basic Service,
with robots. ETSI EN 302 637-2 V1.3.2, 2014.
[10]. N. Lyamin, A. Vinel, M. Jonsson and B. Bellalta,
Cooperative awareness in VANETs: On ETSI EN
302637-2 performance, IEEE Transactions on
References Vehicular Technology, 67, 1, 2017, pp. 17-28.
[11]. Intelligent Transport Systems (ITS); Vehicular
Communications; Basic set of Applications; Part 3:
[1]. H. Andreasson, A. Bouguerra, M. Cirillo, D.Nikolaev- Specification of Decentralized Environmental
Dimitrov, D. Driankov, L. Karlsson, A. J. Lilienthal, Notification Basic Service, ETSI EN 302 637-3 V1.2.1,
F. Pecora, J. Pekka Saarinen, A. Sherikov, 2014.
and T. Stoyanov, Autonomous transport vehicles: [12]. J. Santa, F. Pereniguez-Garcia, A. Moragуn
Where weare and what is missing. IEEE Robotics & and A. Skarmeta, Experimental evaluation of CAM and
Automation Magazine, 22, 1, March 2015, pp. 64–75. DENM messaging services in vehicular
[2]. N. Bahnes, B. Kechar, and H. Haffaf, Cooperation communications, Transportation Research Part C:
Intelligent Autonomous Vehicles to enhance Emerging Technologies, 46, September 2014,
containerterminal operations, Journal of Innovation in pp. 98–120.
Digital Ecosystems, 3, 1, June 2016, pp. 22-29. [13]. A.-J. Fougиres, A Modelling Approach Based on
[3]. H. Khayyam, B. Javadi, M. Jalili, and Reza N. Jazar, Fuzzy Agents, Int. J. of Computer Science Issues, 9, 6,
Artificial Intelligence and Internet of Things 2013, pp. 19-28.
forautonomous vehicles, in Nonlinear Approaches in [14]. C. M. Macal, Everything you need to know about
Engineering Applications: Automotive Applications of agent-based modelling and simulation, Journal of
Engineering Problems, R. N. Jazar, L. Dai (Eds.), Simulation, 10, 2, 2016, pp. 144-156.
Springer International Publishing, 2020, pp. 39-68. [15]. D. Bechtsis, N. Tsolakis, D. Vlachos, and J. S. Srai,
[4]. A. Ndao, M. Djoko-Kouam, and A.-J. Fougères, Matrix Intelligent autonomous vehicles in digital supply
Beaconing for the Location of Autonomous Industrial chains: a framework for integrating innovations
Vehicles on a Simulation Platform, in Proceedings of towards sustainable value networks, Journal of Cleaner
the Third Int. Conf. on Advances in Signal Processing Production, 181, 2018, pp. 60-71.

91
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(025)

Artificial Intelligence and Measurements

R. Taymanov, K. Sapozhnikova, and A. Shutova
D. I. Mendeleyev Institute for Metrology, 19, Moskovsky pr., Saint Petersburg, Russia, 190005
Tel.: + 78122519920, fax: + 78127130114
E-mail: [email protected]

Summary: The emergence of artificial intelligence (AI) increases human abilities in solving cognitive tasks, expands the
scope of measurement as well as forces a rethinking of the basic philosophical question about the material and the spiritual
(ideal). The development of AI-based systems will lead to mass production of their various types with relatively weak AI,
which, along with systems with strong AI, will change the life of society but create social problems. The paper describes the
effectiveness of using measurement systems with AI in manufacturing, medicine, transport, assessment of the product and
service quality, determination of human abilities, etc. The directions of activities that can accelerate the positive trends in the
new development stage but will delay the emergence and facilitate the solution to related social problems are highlighted.

Keywords: Artificial intelligence, Measurement, Weak and strong artificial intelligence, Development of civilisation, Voice
assistant.

1. Introduction desirable, we recommend turning to the opposition

“measurable – immeasurable”. This transition is
Any creature with a central nervous system regards reasonable since “when you can measure what you are
itself as a part of the environment where it takes action speaking about and express it in numbers, you know
to survive. In this environment, the creature has to something about it” as Sir W. Thompson, Lord Kelvin,
separate itself from the other environment components observed far back. (The concept of “immeasurable”
and distinguish components that have special features, and the mentioned Thomson's observation should be
e.g., they are potential food, pose a threat, or provide referred to the level of knowledge at the time of the
an increase in the population. statement formulation).
The emergence of human beings required the Both philosophy and measurement science should
ability to understand the environment much more contribute to enriching knowledge of the world and to
widely and recognize more specific differences in the increasing the effectiveness of activities aimed at its
environment components. It also turned out to be perfecting. Accordingly, the proposed opposition
necessary to identify certain abstractions in this places emphasis on the need to broaden the scope of
environment which were important for forming the measurements. Indeed, the possibilities of
view of the surrounding world. This view allowed measurements are increasing with time. One can notice
planning the life of humans themselves and their tribe it concerning measuring the properties of natural
for a long time. At first, the view took the form of intelligence and those of artificial intelligence.
religion and then that of philosophy. Both forms were
changing over time, and the diversity of their versions
was growing. 2. Vision of Artificial Intelligence, Its
It is now common in philosophy to divide the world Possibilities, and Prospects
into two categories: the material that determines the
content of space as well as time and the spiritual The number of papers dealing with AI has
(ideal), or in short, matter and consciousness. increased dramatically in the current century. There
Advances in artificial intelligence (AI) development, exist international standards in this area and dozens of
robotics, and additive technology are increasingly draft standards, both national and international,
leading to discussions on the concept of the spiritual. relating to AI. However, the known definitions of AI
It is possible to assume that the spiritual is our are contradictory [1]. Meanwhile, AI is mainly
perception of the material. Then, the set of design and examined in three ways: it can be a set of properties,
technological documentation related to some product set of technologies, and an engineering discipline.
is a form of the ideal vision of this product based on Within the frames of this paper, AI is considered to
knowledge allowing to produce the product. be a set of properties that allow analyzing data that
However, this knowledge is incomplete. For come from input channels, getting knowledge,
example, it does not include information related to constructing representations, forming concepts, and
some features of materials applied, which can result in reinterpreting them depending on the results of self-
changing product characteristics in the course of its learning.
operation. Taking into account that to emphasize the Various sensors and other measuring transducers
unknown in comparing the material and spiritual is can be input devices of the channels. Based on the

92
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

outcome of data analysis, the AI can identify socialization. A technical revolution of this kind is
correlations between the data, recognize images and inevitable, as it opens up new prospects for the
measure multidimensional quantities, classify them, development of civilization.
predict changes in the ratios of various parameters over Naturally, there will be a code of ethics for the AI-
time, and make decisions or propose them to a human based systems that should exclude undesirable
operator. consequences of their socialization. Significantly, in
Nowadays, as a rule, the human operator 2021, the Code of Ethics for Artificial Intelligence was
formulates the tasks. In many cases, the AI finds the published in Russia [2]. It establishes “general ethical
algorithm to solve them, sometimes without informing principles and standards of behaviour to guide
the operator about it. However, in a few years, humans participants in AI relationships ... as well as of
will only set strategic tasks. Tactical tasks, e.g., corresponding mechanisms for implementing the
suppression of interfering influences or provisions of this Code.” The Code applies to relations
troubleshooting, will be handled by the AI related to ethical aspects of the development (design,
independently. engineering, piloting), implementation, and use of AI
Thus, while in the past technical systems were technologies at all stages of the life cycle...” The Code
created to increase the capacity of individual human states that “human beings, their rights, and freedoms
organs (hands, feet, eyes, etc.) and to speed up the shall be regarded as the highest value”.
solution to simple mathematical problems, the However, it is hardly possible that such a document
development of AI opens up the prospect of increasing will be able to fundamentally change the direction of
the human abilities to solve tasks of a cognitive nature. robotics development. As the number of robots
Since the level of complexity and variety of tasks capable of emotional responses increases above some
to be performed by systems based on AI is different, it limit, more developed socialization is likely, possibly
is reasonable to produce a number of system types that involving interested humans.
differ in terms of AI level. Then we can expect the spontaneous selection of
It is now common to divide AI into weak (narrow) leaders. They will be able to set tasks for groups of
and strong (general). Systems based on weak AI robots. This does not exclude tacit correction of a robot
extract information from a limited set of data and are firmware with their participation. As a result, there
only able to cope with the specific tasks that they were have been assumptions about the possibility of
trained to perform. They are usually oriented towards conflicts between robots and humans [3]. The history
tasks that, when carried out by a human, do not require of civilization provides numerous examples of the
high qualifications but oblige him/ her to use regularly emergence of similar situations in the relationships
updated reference data. Often, communications of between human communities. Usually, they concern
similar systems with humans, i.e. conversational skills violations of ethical laws established by both religions
with a limited vocabulary, are also required. Such tasks and state laws.
include, e.g., working as a waiter/waitress, security
guard, tour guide, interpreter, storekeeper, etc. It
should be noted that the systems based on weak AI are 3. AI-based Measurement Systems
now relatively widespread, while the number of their
types and manufactured product output are growing The problem of preventing conflicts between
steadily. robots and humans is not yet a hot topic, although it
A system with strong AI is much more expensive. should not be forgotten. Metrology specialists now
It can reason, make judgements in the face of face a different challenge: efficiently apply AI for
uncertainty, plan, learn, integrate prior knowledge into measurement purposes.
decision-making, suggest new ideas, etc. It can be The first direction of work is the identification of
assumed that systems with strong AI will also be multidimensional quantities that characterize the
produced in several types, depending on the specific occurrence of deviations from nominal values,
tasks they solve. measurement of the parameters of these quantities, and
However, it is possible that in the process of the AI- determination of the deviation dynamics over time.
based system operation, such as mobile robots, there The next stage is the development of methods and
will happen a need to coordinate the actions of several means that allow obtaining a desirable result. The
robots to solve a new problem. In particular, it will be essence of the result depends on the task to be solved.
necessary to perform the actions simultaneously or There exist several groups of tasks here.
jointly process information coming from many One of them is to search for irregularities in the
sources, the number of which is greater than that of materials used in critical products, e.g., to detect a
channels a single system with AI has. hidden crack in the support of a construction.
Apparently, the robot firmware will allow for this Conventional methods of ultrasound analysis can
kind of cooperation. For example, a robot will detect the presence of irregularities using a panoramic
“realize” that on its own it cannot solve the task set by scanning. If the results of the multichannel
an operator and will initiate the cooperation. measurements made with the help of an AI-based
Essentially, this proactive request for help will mean system show a correlation of changes, in particular, in
an emotional response, which, if perceived by another many space points, under some impacts on the support,
robot, will mark the beginning of an era of robot

93
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

or over time, then the presence of a crack in the support particular job, the emotional expressiveness of the
is diagnosed. This fact makes it possible to prepare a musical composition performance, etc.
solution aimed at eliminating the hazard. However, in some cases, an expert can be biased,
Another group of tasks is related to medicine, i.e., i.e. interested in an assessment for reasons unrelated to
the detection of abnormalities in human physiological the materials submitted to him.
systems. In [4], on the basis of the electrocardiogram The trustworthiness improvement of
processing, Uspenskiy proposes to use AI for “measurements” being performed with experts’
analyzing the ratios between blood pressure pulse participation up to an acceptable level that excludes
parameters got at several points on the patient’s body crude errors is the second direction of metrological
surface. Statistics of clinical studies give the basis for research and development that relies on the capabilities
the automatic diagnosing of more than forty diseases of AI-based measurement systems.
by characteristic deviations of measured ratios from To minimize the role of subjective factors, expert
the nominal values fixed in healthy people. The judgements should be reduced to a common scale.
diagnosis, in its turn, determines the course of Selecting the experts, one should take into account the
treatment to restore the health of the patient. competence level bringing the expert number to the
This method also opens up additional number that depends on the task to be solved. This
opportunities. As we all know, certain music requirement can be met using AI. The useful method
influences the physiological processes taking place in is, e.g., the correlation analysis of experts’ answers to
the body [5]. By analyzing changes in the above ratios, abstract questions combined with a number of
while a patient is listening to selected music excerpts, indicators characterizing the features of experts’
it is possible to refine the diagnosis and treatment neurophysiological activity (brain biorhythms, voice
recommendations. sound, etc.) while making their assessments.
Diagnosing in this way is also technically relatively The Bayesian approach has been successfully
easy with exposure to force, vibration, heat, or applied in this field too. Examples are given in [7].
otherwise. This feature makes it possible to help The third direction is the measurement of abilities
patients having a significant likelihood of abnormal and properties of intelligence, including the emotional
physiological processes. sphere.
It is sufficient to arrange measurements of a Usually, the level of human intelligence is assessed
specially selected group of indicators, which form, by the intelligence quotient IQ. It enables the
using an AI-based system, an estimate of the value of comparison between individual results and average
the quantity indicating the approach of an attack. The ones for people of the same age. The assessment is
corresponding signal should be transmitted to the made with the help of tests, i.e., a series of
doctor if the patient is in hospital or to the patient progressively more difficult questions chosen by
himself/herself if he/she has a means of preventing a experts. There exist widespread tests, but the authors
seizure. Such methods can diagnose diseases at early are not aware of any standard in this field. The reason
stages too. seems to be the limited interpretation of intelligence in
The considered direction of AI-based measuring this method and its emphasis on logic.
systems can be effective in the automatic translation of However, the role of the emotional sphere in a
emotionally coloured speech [6]. Special speech human intellectual activity is significant. Analysis has
processing can single out infrasound multidimensional shown that AI-based systems make it possible to
modulation components and identify, e.g., the irony of identify abnormalities in intelligence development,
the said. e.g., from the sound of babies’ cries, i.e. his/her
AI can also be efficient in traffic management. The emotional reactions. The abnormalities can be either
relevance of work in this area increases with the negative, which indicates, e.g., the early stages of
appearance of driverless cars. But also for driving autism, or positive related to the accelerated
conventional cars in regular traffic flows, AI-based car development of imaginative thinking.
measuring systems will certainly be in demand. They The ineffectiveness of IQ in investigating human
will be able to track the direction and speed of abilities does not undermine the merits of the test
oncoming and passing cars, calculate the probability of method. The application of tests including not logical
collision with them or with other obstacles, and form components only but emotional ones may be efficient
signals correcting the car movement. in evaluating AI-based systems in case communication
The above examples concern the application of with humans is one of their primary functions.
measurements in areas where they were necessary Performing the duties of a waiter, salesperson,
before but could not provide the required efficiency security guard, or tour guide could be the examples
without AI. where the “voice assistant” function can be very
However, in some fields, the term “measurement” appropriate. An IQ assessment with such a test makes
can so far only be applied conventionally. This it possible to compare the quality of this function in
statement relates to the assessment of systems produced by different companies.
multidimensional quantities in which the role of an In particular, in 2019, Loup Ventures prepared 800
expert is essential. Such quantities include many questions assigned for testing “voice assistants”
indicators of the quality of goods and services, the Amazon Alexa, Apple Siri, and Google Assistant [9].
level of knowledge and skills of applicants for a They checked the understanding of the question and

94
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

whether the answer was correct. The questions References

concerned the address of an institution, an order to buy
something, a request to be reminded of something, etc. [1]. R. Taymanov, K. Sapozhnikova, The role of artificial
The percentage of correct answers given by Google intelligence in measuring systems, in Proceedings of
Assistant, Apple Siri, and Amazon Alexa was 93%, the XXXI International Scientific Symposium on
83%, and 80% accordingly. Comparison with the Metrology and Metrology Assurance (MMA), Sozopol,
results of 2015 (86%, 79%, and 61% for the same Bulgaria, 7-11 September 2021, pp. 1-5.
[2]. Code of Ethics for Artificial Intelligence
assistants) shows remarkable progress [9]. (in Russian) Web Portal (https://www.profiz.ru/upl/20
However, many AI-based systems are aimed at 21/Кодекс_этики_в_сфере_ИИ_финальный.pdf).
solving tasks where communication with humans is [3]. S. Hawking, Brief Answers to the Big Questions, John
minimal. In such cases, to use technical indicators to Murray, 2018.
assess product quality is appropriate [10]. [4]. V. M. Uspenskiy, Artificial intelligence in the
The fourth direction is the classification of diagnosis of diseases of human internal organs,
particular multidimensional quantities, e.g., acoustic Informational and Telecommunication Technologies,
signals or visual images. Quite often, the parameter Vol. 50, 2021, pp. 15-21 (in Russian).
ratios or even the parameters themselves, which are [5]. AMTA Web Portal (https://www.musictherapy.org/).
[6]. E. Zinovieva, Yu. Kuznetsov, M. Shakhmatova,
used as characteristic features in quantity identification I. Baksheeva, I. Danilova, K. Sapozhnikova,
and classification, are specified with a dominant R. Taymanov, Metrological approach to emotion
definitional uncertainty. recognition in sounding speech, Philosophy and
As a result, an AI-based measuring system has to Humanities in Informational Society, Vol. 3, Issue 17,
compare a number of such ratios to make a final 2017, pp. 63-82 (in Russian).
decision. This circumstance gives grounds to classify [7]. R. Taymanov, K. Sapozhnikova, S. Prokopchina, What
such measurements as soft measurements [11, 12]. The is immeasurable make measurable with artificial
authors have experience in applying such classification intelligence, Measurement: Sensors, Vol. 18, 2021,
to the automatic translation of animal acoustic signals pp. 1-4.
[8]. K. Sapozhnikova, R. Taymanov, I. Baksheeva,
into the language of human communication [13]. S. Kostromina, D. Gnedykh, I. Danilova, Metrological
It is noteworthy that the dominant definitional approach to measurements of emotions being expected
uncertainty of multidimensional quantities is the in response to acoustic impacts, in Proceedings of the
characteristic feature of some other quantities 18th International Congress of Metrology, Paris,
mentioned above. In the fourth direction, soft France, 19-21 September 2017, pp. 1-7.
measurements are highlighted because they are applied [9]. Artificial Intelligence News Web Portal
here significantly more frequently. (https://artificialintelligence-news.com/).
[10]. Draft standard. Artificial intelligence systems. Quality
assurance. General (in Russian).
[11]. K. Sapozhnikova, R. Taymanov, S. Kostromina,
4. Conclusions Widening borders of measurable is a problem of the
XXIst century, Soft Measurements and Computing,
The emergence, widespread use in many areas of Vol. 37, Issue 12, 2020, pp. 31-44 (in Russian).
human activity, and high improvement rate of AI- [12]. K. Sapozhnikova, A. Pronin, R. Taymanov, Increasing
based systems indicate that human civilization is rising measurement trustworthiness as a necessary part of
to a new stage in its development. The coming changes technology development, Sensors & Transducers,
in the life of society are substantial. Accordingly, the Vol. 251, Issue 4, 2021, pp. 61-69.
main challenge will be to maintain human [13]. K. Sapozhnikova, S. Hussein, R. Taymanov,
Iu. Baksheyeva, Music and growl of a lion: anything in
developmental advantages compared with the systems common? Measurement model optimized with the help
based on AI. of artificial intelligence will answer, Journal of
This problem will be solved if as many people as Physics: Conference Series, Vol. 1379, 2019, pp. 1-6.
possible are capable of continuous self-learning and
active creativity, which requires significant changes in
traditional practices of upbringing and education. It is
a critical point when not to be out of time and avoid
acute social problems is possible.

95
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(026)

Intelligent Sensors Networks for Monitoring and Controlling Complex

Systems under Conditions of Uncertainty

S. V. Prokopchina
Financial University, Russian Federation
E-mail: [email protected]

Summary: Within the framework of the Industry 4.0 concept, intensive development of the processes of intellectualization of
sensor systems is envisaged. Among the most important specific properties of real measuring processes in complex systems
is, first of all, their implementation under conditions of considerable uncertainty. The uncertainty is caused by a priori
incompleteness, inaccuracy, vagueness of information about a complex measuring object and its functioning environment,
which does not allow to build an adequate model of the object before the measurement experiment, to identify and formalize
the influencing factors of the external environment and to develop effective algorithms for the functioning of information and
measurement systems.
The report proposes an approach to the intellectualization of measurement systems in conditions of uncertainty by creating
intelligent sensor networks based on Bayesian intelligent technologies (BIT) and means of their implementation. Typical
modules of such networks are considered, which are integrated sets of various sensors and intelligent measurement information
processing systems. The results of the networks are comprehensive assessments of the state of complex objects and
recommendations for providing of their sustainable functioning. An important part of such systems is the built-in means of a
complete metrological justification of all received solutions. The systems have a hierarchical architecture, according to the
levels of management of complex objects, which has the possibility of self-development based on newly received information.
This is achieved thanks to models and scales with dynamic constraints on which all BIT algorithms are built. The report
provides examples of the use of intelligent sensor networks for monitoring and control of power generation and water supply
systems.

Keywords: Intelligent measurements, Bayesian approach, Sensors network.

1. Introduction technologies developed by the author on the basis of

the regularizing Bayesian approach make it possible to
Within the framework of the Industry 4.0 concept, implement monitoring and management of complex
intensive development of the processes of systems in conditions of significant uncertainty and
intellectualization of sensor systems is envisaged. active dynamic influence of the external environment.
Among the most important specific properties of Methodology, information technologies and applied
real measuring processes in complex systems is, first examples of solving applied problems based on them
of all, their implementation under conditions of are given in the author's works, for example, in [1-3].
considerable uncertainty. The uncertainty is caused by During the implementation of a number of applied
a priori incompleteness, inaccuracy, vagueness of projects related to the creation of intelligent measuring
information about a complex measuring object and its systems for distributed man-made, natural and socio-
functioning environment, which does not allow to economic systems, the concept of intelligent sensor
construct an adequate model of the object before the networks was developed.
measurement experiment, to identify and formalize the Such networks allow for a comprehensive
influencing factors of the external environment and to assessment of the state of distributed systems at any
develop effective algorithms for the functioning of time, to determine the main risks and potentials of both
information and measurement systems. individual sites and the system as a whole.
Other specific properties of complex systems are The intelligent sensor network consists of a central
their dynamism and active relationship with the module for intelligent information processing and
environment, which determines special requirements peripheral modules that collect information on all parts
for methods of creating models, including measuring of the network.
ones. The tasks of monitoring the state of complex As the central module, an intelligent environment
systems and their effective management, especially in built on the Infoanalytic platform is used, which
real-time modes of system operation, determine the receives information flows from peripheral intelligent
need to obtain not individual measurement results, but modules, applied neural networks for image and
assessments, conclusions, and management document processing, statistical databases, expert
recommendations. Such opportunities can be provided assessments and other information. The structure of
by involving artificial intelligence methods in such a network is shown in Fig. 1.
measuring processes. The Bayesian intelligent

96
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 1. Block diagram of a typical module of an intelligent sensor network section.

The experience of such developments made it Q is the generalized algorithm for creating a typical
possible to form a typical peripheral sensor network sensor network module;
module that implements all the above functions for a Q1 is the creation of a conceptual model of an object
separate network section. with dynamic constraints (MDO) for this section of the
It consists of the following subsystems: sensor network;
 integrated sensor sets that measure system Q2 is the Sensor system selection;
parameters; Q3 is the construction of scales with dynamic
 intelligent controllers that implement the functions constraints for each sensor in the intelligent controller;
of intelligent processing of primary measurement Q5 is the implementation of measurements, obtaining
information, consisting in the integration of primary measurement information from sensors;
information from individual sensors in order to Q6 integration of individual measurements of various
obtain estimates, and the coordination of protocols sensors according to the modified Bayes formula;
for transmitting solutions received on this part of Q7 interpretation of integrated information and its
the network to the central module; presentation in the form of an assessment of the state
 subsystems of interfaces and web services in of a network section;
monitoring mode; Q8 is the determination of dynamics and dynamic
 databases of primary measurement information. models, trends in individual parameters and in general
The type of a typical sensor network module is according to the state of the system in this section of
illustrated in Fig. 1. A conceptual model for obtaining the network and trends in the development of
a solution in a peripheral sensor network module based situations;
on Bayesian intelligent technologies (BITS). Q_9 is the assessment of risks and potentials of the
The conceptual model of creating a solution based situation and display by methods of cognitive graphics;
on Bayesian intelligent technologies (BITS) is based Q10 is the interpretation of the current situation;
on the methodology and principles of creating models Q_11 is the generating recommendations to improve
with dynamic constraints based on the regularizing the situation.
Bayesian approach [1-3]. The stages of the complex Convolution of information (in formulas (1) and
solution of the problem of creating an intelligent big (3) this action is indicated by the symbol *) about the
data processing system can be represented by the value of the element x_ij, i=1,...I is performed
following model: according to the modified Bayes formula for discrete
laws of hypothesis distribution in the form of
Q=Q_1*Q_2*Q_3*Q_4*Q_5*Q_6*Q_7*Q_8*Q_9* membership functions (linguistic scale of BII for
Q_10*Q_11 (1), processing non-numeric, qualitative information) or
probability distributions (numerical scale of BII for
where: processing non-numeric, qualitative information). The

97
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

probability of a solution (hypotheses about the state of 2. Conceptual Model for Monitoring the State
an object, the value of a parameter, or the like) is of the Water Supply Network
calculated by the formula:
The conceptual model for monitoring the state of
𝜇 ℎ ℎ | 𝑋 |𝑄 the I-th section of the water supply network based on
| | ° ℎ , , 𝑥 , | 𝑋 |𝑄 Bayesian intelligent technologies (BIT) has the
2 , following form:
∑ , , | , , | | |

Qi=Q_1*Q_2*Q_3*Q_4*Q_5 (3),
where:
ℎ is the integral (for a set of information flows 𝑋 ))
where:
regularizing Bayesian estimation (RBO); Qi is the integral assessment of the condition of the
ℎ is the RBO for X_i data stream; water supply network section;
𝜇 ℎ ℎ | 𝑋 |𝑄 is the posteriori probability of Q_1 is the integral assessment of the condition of the
evaluation ℎ . well on the section of the water supply network;
The general scheme of the system being developed Q_2 is the integral assessment of the condition of a
consists of: a measurement storage server, sensors; a linear section of the water supply network ;
controller; measurement processing and transmission Q_3 is the integral assessment of the condition of the
to the server; a workstation; a neural network for external sections of the water supply network;
generating scripts; a web service; clients; IIS Server Q_4 is the integral assessment of the network status
Windows Infoanalytics; MS SQL; a mathematical based on information from users;
apparatus for data analysis and decision-making; Q_5 is the integral assessment of the network status
system management. based on information from experts.
A typical peripheral module of an intelligent sensor The conceptual model of monitoring the condition
network, as shown in Fig. 1, includes basic measuring of the water supply network as a whole can be written
instruments and an intelligent controller, as well as the as follows:
necessary modules for organizing web services.
Measuring instruments may include both sensors for Q = * i=1,I Qi (4)
measuring physical parameters and virtual measuring
instruments for measuring integral, non-physical and Such methods and technologies are used in the
non-quantitative parameters. This architecture of a implementation of real projects. In particular,
typical module allows you to combine, integrate and examples of applied solutions of an intelligent sensor
process measurement and observation data in network for assessing the state of the water supply
monitoring mode. The use of special scales with network of one of the cities of the Russian Federation
dynamic constraints [1, 2] allows measurements to be are given below.
made under conditions of uncertainty, significant Fig. 2 shows the interpretation of the situation
dynamism of the measured parameters and the active according to the integral factor "Information from
influence of the external environment. metering devices" on 12.11.2021.

Fig. 2. The interpretation of the measuremental situation.

98
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

The results of the work of a typical module in the References

figure are reflected in the form of cognitive graphics.
The color of the circles located to the left of the [1]. Prokopchina S. V., The concept of Bayesian
measured characteristics determines the state of this intellectualization of measurements in the tasks of
parameter. In this example, the green color indicates monitoring complex objects, News of Artificial
that the parameter is within acceptable limits. The Intelligence, No. 3, 1997, pp. 7-56.
yellow color determines the output of the measured [2]. Prokopchina S.V., Fundamentals of scaling theory in
economics, Scientific Library, Moscow, 2021.
parameter value beyond the range of acceptable values. [3]. Prokopchina S. V., Cognitive Bayesian measurement
networks based on the regularizing Bayesian approach,
Soft Measurements and Calculations, 2018, No. 2, pp.
3. Conclusion 56-69.
[4]. Prokopchina S. V., Methodological foundations of
The article presents methodological aspects and scaling in modern Measurement Theory. Classification
principles of construction of the measuring peripheral of measurement scales and their application under
module of the system and the version of the uncertainty based on Bayesian Intelligent
Infointegrator with the possibility of remote intelligent Technologies, Journal of Physics: Conference Series,
processing of various types of information, including 1703, 1, 2020, 012003.
measuring data streams from devices, expert [5]. Prokopchina S. V., New Trends in Measurement
assessments of engineers and technicians, statistical Science. Bayesian Intelligent Measurements, in
data of bypass logs, facts and information from Internet Proceedings of the 5th International Conference on
sources with constant connection of production- Sensors and Electronic Instrumentation Advances
(SEIA’ 2019), Adeje, Spain, 2019, pp. 317-322.
oriented sites. This architecture is implemented for the
first time and allows you to implement a new type of
IIoT, namely, intelligent processing of big data of
industrial production.

99
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(027)

Methods and Technologies of Bayesian Intelligent Measurements

for Human Resources Management in Industry 4.0
S. V. Prokopchina, E. S. Tchernikova
Financial University, Russian Federation
E-mail: [email protected]

Summary: An important place in modern methodologies and technologies of personnel management at the enterprise is
occupied by the direction of innovative development, culture of innovation. For this direction, from the position of
intellectualization of innovative processes and, in particular, motivation and involvement of personnel, non-quantitative
indicators are characteristic, which, following the ISO series standards, should be measurable.
This led to the use of Bayesian intelligent technologies, in particular Bayesian intelligent measurements, for the purpose
of measuring indicators of the state of the company's personnel and their management. The aim of the work is to build a
systematic methodological basis and means of creating a digital environment for the implementation of the principles of the
culture of innovation based on fundamental and best practical psychological work by means of BIT and BII. The
implementation of a number of methodological constructions based on individual modules of the Infoanalytic computer system
is shown, which contributes to the digitalization of this activity, the dissemination of experience and knowledge for the training
and continuous training of specialists, including remotely.

Keywords: Human resources management, Bayesian intelligent measurements.

1. Introduction and adaptation of new practices, processes and

paradigms in order to increase the value of their
The most important part of automation and activities.
intellectualization of production enterprises is the The hypothesis of our research is that such an
personnel of the enterprise. The most advanced environment exists and, by identifying its
production technologies may not ensure its characteristic indicators and measuring them, it is
effectiveness in the absence of qualified, organized and possible to assess the overall state of the innovation
involved in innovative processes personnel. A sphere at the enterprise and formulate the most positive
significant number of factors affecting the state of conditions for creating innovations.
personnel and the impossibility of direct measurement Thus, the aim of the work is to build a systematic
of most of them determine the need to use specialized methodological basis and means of creating a digital
methods of personnel management in modern environment for the implementation of the principles
conditions of the functioning of enterprises. of the culture of innovation based on fundamental and
An important place in modern methodologies and best practical psychological work by means of BIT and
technologies of personnel management at the BII.
enterprise is occupied by the direction of innovative The achievement of such a goal will lead to the
development, culture of innovation. For this direction, transformation of the knowledge and practices of
from the position of intellectualization of innovative innovative companies into an applicable set of tools
processes and, in particular, motivation and that allow the company to build a culture of qualitative
involvement of personnel, non-quantitative indicators change, will make available to managers the entire
are characteristic, which, following the ISO series palette of tools for improving the efficiency of the
standards, should be measurable. company and unlocking its potential
This led to the use of Bayesian intelligent The object of research of this work is intellectual
technologies, in particular Bayesian intelligent technologies of digitalization of innovative
measurements, for the purpose of measuring indicators psychological practices and processes for the
of the state of the company's personnel and their continuous development of personality as a member of
management. various social communities, including labor.
The object of the study is also the specific human
features of the mental regulation of the activity of
2. The Main Terms and Indicators individual and group subjects, depending on the
of the Culture of Innovation natural influence of various factors.
The methodology of the regularizing Bayesian
Cultura of innovation is the sum of codes, norms approach and technologies based on it are proposed as
and habits of employees contributing to the adoption a methodology for creating intelligent systems for

100
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

conditions of information uncertainty. The that affect the organization of labor at the present time
effectiveness of the proposed approach is confirmed by and should be measured:
its methodological principles focused on information 1. The sixth technological order;
processing in conditions of uncertainty 2. Globalization and decentralization of work;
(incompleteness, fuzziness and inaccuracy of 3. New behavior;
psychological information) and the practice of its 4. The growth of creative professions
application to create intelligent systems implemented All these global trends will only intensify in the
for a wide range of tasks related to monitoring and future, but it is innovative companies that have already
management of complex objects, which include the learned how to use them in their work and benefit from
above innovative processes. these total changes.
In the course of this scientific work, the following Technoclade is a set of generally accepted and
results were obtained. applied production technologies and scientific and
1. The terminological basis of the theoretical technological progress
provisions of the culture of innovation and motivation The technological order is not just one change, but
is determined. a whole series of related and complementary changes.
2. A historical review of the existing concepts of The sixth technoclass is primarily associated with the
the implementation of psychological innovations and synthesis of sciences. It is associated with
motivation is carried out. breakthroughs in the field of artificial intelligence and
3. A critical review of methodological approaches robotics. The company's ability to adapt to new
and best practices for the implementation of innovation technological conditions will directly affect its
processes and motivation of individuals or groups of viability. The simplest example associated with the
subjects in accordance with the formulated emergence and spread of the Internet, we saw in the
requirements of methodological and practical fifth technoclass.
relevance. Our education and the structure of social relations
4. The conditions and factors that favor and hinder are still in the fourth, industrial technological order.
the implementation of innovative psychological They are focused on working in a hierarchy,
techniques and motivation of subjects are formulated consistency and efficiency, having an ideal sample and
5. The regularizing Bayesian approach and working according to a template. But the sixth techno-
Bayesian intellectual technologies based on it are layout allows using robotics, the Internet, things and
reasonably chosen for the implementation of the new analytical abilities of computers to replace not
principles of psychological development of subjects only manufacturing professions, but also some office
and communities, including labor. intellectual tasks. We are already seeing an actively
6. The methods of implementation of developing market of bots capable of recognizing and
psychological innovations and motivation based on the solving a client's problem. This means that the
conclusions and methodological decisions made in the company's personnel will have to learn not only to use
work are proposed. new technological capabilities, but also to move from
The practical significance of the work is: purely performing activities to creativity and task
1. Developed practical methods and setting. Innovative business has such experience. He
recommendations for the implementation of the develops creativity in his employees, supports the
principles of the culture of innovation, as well as, as model: "Need - training - development - need - training
part of it, the motivations of subjects that have been - development".
tested on a number of communities. including labor. All this means that companies are turning into
2. Conducted psychological projects and universities and colleges. The performing discipline
experiments, the results of which can be used as inherent in rigid industrial structures is being replaced
effective practices. by creative and intellectual work. Enterprises should
3. Implementation of a number of methodological become a platform for continuous development:
constructions based on individual modules of the develop mentoring competencies, implement training
Infoanalytic computer system, which contributes to the tools and various professional collaborations.
digitalization of this activity, the dissemination of It was possible to identify six integral indicators of
experience and knowledge for the training and the state of the enterprise's innovation sphere, which
continuous training of specialists, including remotely. were broadcast unambiguously and were found in the
The results obtained make it possible to optimize absolute majority of innovative organizations.
the personnel management system in the organization, They include: Transformation, Self-organization,
as well as to predict the nature of the impact of certain Integrity, Diversity, World-centricity, Creation
managerial influences on the level of satisfaction, (entrepreneurship). These indicators together form the
involvement and effectiveness of employees of the basis for the development and implementation of
organization. innovations. They interact and complement each other,
In this study, we turned to the experience of leading creating a catalyst effect for breakthrough solutions,
consulting companies, as well as experts, and based on discoveries, research and technical creativity.
this critical review, we identified four global changes

101
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

3. A Measurement Model based on system that determines the processes of digitalization

Bayesian Intelligent Technologies for should be allocated to the subsystem of conditions for
the rest of the innovation activity.
Personnel Management and Innovative Then the model of the innovation system (1) can be
Culture of the Enterprise in Conditions written as:
of Active Interaction with the External
Environment G G ∗G ∗G (2)

The main methodological basis of this chapter was

where G is the digitalization system.
the work of Professor S. V. Prokopchina and her
Any model of a complex object can be represented
scientific school. The following is the concept of
as a composition or a convolution of its properties. We
S. V. Prokopchina for the formal justification of
denote the enlarged or integral properties of the
innovative processes in conditions of uncertainty.
Within the framework of this methodology, it is innovation system as Q , i 1, I , and the media
advisable to present innovative activity as a complex Q , i 1, J with weights P
system that actively interacts with the external Then the dynamic model of the innovation system
environment. can be written as follows:
The conceptual model of such a system can be
represented as a dynamic model of a real system of G ∗
P Q 3
,
interrelated innovation processes and, according to the
notation introduced above, can be written in the Environment model:
following
∗
G , P Q (4)
G G ∗G ∗G 1
model of digitalization processes:
where G is the model of an innovation system
actively interacting with the external environment, G ∗
P Q 5
,
G is the model of the external environment obtained
on the basis of a priori and incoming information; G Further details of the measured indicators are given
is the model of conditions for the implementation of in [1-3].
innovations; * is the symbol denoting the convolution To measure the above integral indicators, the
of spaces (compacts) of the specified models in which system uses scales with dynamic constraints that allow
they are defined. measuring non-quantitative indicators and making a
In order to solve the problem of identifying convolution of individual indicators included in
influencing factors for the development of innovations equations (1)- (5) and obtaining an overall assessment
in Russia, the conceptual model (1) should be detailed of the state of the innovation environment.
to the level of specific influencing factors. Fig. 1 shows an illustration of the implementation
Based on the methodology of the system approach, of the conceptual model within the innovation sphere
it is possible to build a model of a complex object as a of industrial enterprises for Russia. Due to the
set of its properties and the relationships between them flexibility and self-learning ability, the system can be
in interaction with the set of properties of the external used to organize innovative processes of psychological
environment in the conditions of its study and development of employees. At the same time, it is
operation. possible to measure all the listed indicators and
On the basis of the RBA methodology, the characteristics of innovative and motivational
Infoanalyst computer platform has been developed for processes.
the rapid development of digitalization systems of
activities. Such a system has the properties of a simple
configuration of models - digital images of employees 4. Conclusion
of the organization, the flexibility of measuring,
auditing and generating recommendations in The paper considers an approach to assessing the
conditions of incomplete and inaccurate information. state of personnel and the innovative sphere of an
It is designed in the form of an intellectual workplace enterprise based on Bayesian intellectual
of a personnel management specialist (IRM-Cadres), a measurements. Personnel management is associated
patent for which was obtained by S. V. Prokopchina in with the development of the innovative sphere of the
2004. enterprise and the conditions of its development under
Within the framework of the task, the main the active influence of the external environment. An
attention should be paid to the composition of example of the implementation of the concept in the
environmental factors and conditions of innovation "Infoanalyst" environment is given.
activity. At the same time, the part of the innovation

102
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 1. Implementation of the conceptual model of the innovation sphere in the "Infoanalyst" environment.

References technical and socio-economic problems of the

digital economy, Soft Measurements and
[1]. Prokopchina S. V., System of mathematical Computing, No. 9, 2018, pp. 4-33.
processing of statistical information “Bayesian [3]. Prokopchina S., New trends in measurement
Mathematical Statistics”, in Proceedings of the theory: Bayesian intelligent measurement and its
International Conference on Soft Computing and application in the digital economy, in Proceedings
Measurement, St. Petersburg, 2007, June 25-27, of the CEUR Workshop, 2782, 2020, pp. 80–88.
pp. 35-45.
[2]. Prokopchina S. V., Soft measurements:
methodology and application in scientific,

103
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

(028)

Intelligent Acoustic Monitoring of Underground Communications

V. P. Koryachko 1 , V. G. Sokolov 2 and S. S. Sergeev 3
1
Cad Department of the Supreme Council of the RSRTU, Ryazan, Russia
2
RSRTU, Ryazan, Russia
3
TECHNOAS-SK LLC, Kolomna, Russia
E-mail: [email protected], [email protected], [email protected]

Summary: The article deals with the problem of searching for leaks from pipelines by receiving and processing acoustic
emission signals that occur when the liquid flows at the leakage site. The purpose of the work is to review existing solutions
for acoustic monitoring, determine the main characteristics of acoustic emission signals and assess the possibility of their
recognition in the monitoring process using artificial neural networks.

Keywords: Acoustic emission method, Acoustic leak detector, Correlation leak detector, Acoustic monitoring, Spectral
analysis, Correlation analysis, Artificial neural networks.

1. Introduction the metal in the zones of concentration of stresses and

deformations. It is possible both in contact (with the
A significant part of the city's engineering installation of equipment on the surface of the
communications was often built more than a few years pipeline) and in the non-contact version (using
ago and is characterized by a significant degree of wear portable magnetometer sensors moved by the operator
and tear and high losses due to leaks. Even in relatively near the pipeline).
developed countries, such as Canada, municipal water 4. The method of acoustic emission, based on the
systems lose, on average, up to 13% of drinking water reception and listening to sounds that occur when the
due to leaks associated with damage to liquid flows at the leakage site. Can be implemented
communications. In countries where plumbing systems both with the installation of sensors directly on the
have not changed for decades, the level of losses is pipeline, and with remote listening using a highly
even higher. For example, in the UK this figure is at sensitive microphone.
23 % [1]. In the Russian Federation, the average The acoustic emission method is considered the
leakage in the housing stock in 2004 was estimated at most common and requires less effort and less
20-30% of the total water supply to the population [2]. expensive equipment than previous methods.
In 2010, the number of hidden leaks on the Moscow Unfortunately, the method is also characterized by low
water pipeline was estimated at only 5%, but in noise immunity, because both the leakage signal and
absolute terms it reached 225 thousand cubic meters various kinds of noise (industrial, transport, etc.). ) are
per day [3]! in the audible range, and it is not always possible to
Under these conditions, it is necessary to use distinguish them from each other. It takes a lot of
remote and non-destructive methods of pipeline practical experience and good hearing to determine
inspection in order to identify developing defects as whether a given sound is a leak and what its volume is.
quickly as possible and perform the necessary repairs The functions of the operator can in principle be
in time before a serious accident occurs. The following performed by an intelligent computing system, in
methods are best known: which the role of experience is performed by training
1. Control of pressure and/or flow of liquid in the using a set of reference signals. This approach has
pipeline. The necessary devices, as a rule, are installed proven itself in areas such as image recognition,
on pumping stations, on the inputs to the meters, etc. speech recognition, etc. The most promising direction
and have the ability to transfer data to remote control in intelligent systems at present, of course, are artificial
centers. neural networks (ANNs). The average of the
2. Ultrasonic diagnostics of pipelines and welds, advantages that ANNs have often highlight the
based on the propagation of ultrasound in the metal and possibility of solving difficult to formalize problems
its reflection from various inhomogeneities. Requires for which it is difficult to formulate certain criteria and
direct access to the pipeline and its preliminary algorithms. This article discusses the main types of
preparation (release from insulation and stripping of equipment for acoustic leak detection and evaluates the
sensor installation sites). main characteristics of acoustic emission signals in
3. Magnetic flaw detection based on the order to assess the possibility of using ANNs in this
registration of changes in the magnetic properties of area.

104
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

2. Problem Statement and Review of may be other acoustic vibrations - industrial, transport
Acoustic Monitoring Equipment and natural interference, as well as noise created by the
liquid when interacting with various inhomogeneities
As a result of exposure to adverse factors (flange joints, growths, etc.) inside the pipeline (see
(corrosion, overpressure, extraneous mechanical Fig. 1).
influences, manufacturing defects, etc.), holes and The most common acoustic search for leakage is by
cracks appear in the initially continuous wall of the the signal of the sensor moving on the surface of the
pipeline, through which the leakage of liquid under ground near the pipeline and connected to an acoustic
pressure begins. Leaks are routinely classified by flow receiver in the hands of the operator (sensor D3 and
rate per unit of time: receiver AP in Fig. 1). When the sensor is found
- "very weak" - up to 1 l / min; directly above the leakage site, the signal recorded by
- "weak" - from 1 to 5 l / min; the acoustic receiver will be the greatest. this method
- "average" - from 5 to 15 l / min; is a classic "maximum method", then its error is
- "strong" - more than 15 l / min. relatively large and is ~ 1 m at a depth of penetration
When the liquid jet flows out, acoustic vibrations of the pipe pipeline (1.5 ... 2) m. The appearance of
(acoustic emission) occur, which spread in the soil, common acoustic leak detectors is shown on the
liquid and pipeline walls and can be recorded by Fig. 2.
acoustic sensors. Also in the pipeline and near it there

Fig. 1. On the statement of the problem of pipeline acoustic monitoring.

Fig. 2. Appearance of common acoustic leak detectors.

A smaller error in determining the leakage site is instrumental error of correlation leak detectors is 0.1 m
provided by the correlation method, in which two or less. The appearance of common correlation leak
sensors are used, which are installed, as a rule, on the detectors is shown on Fig. 3.
pipeline on both sides of the presumed leakage Unfortunately, correlation leak detectors also have
(sensors D1 and D2 in Fig. 1). Signals from the sensors drawbacks that can significantly reduce the accuracy
are amplified and transmitted by the UP1 and UP2 of the search for a leakage site, including the
transmitters to the KP correlation receiver, which following:
calculates the correlation function of the two signals (a) Almost always, the correlation function
and determines the time shift between the signals to its calculated from signals from watchers has several
maximum. Knowing the distance between the sensors peaks, the largest of which does not necessarily
and the speed of the call in the walls of the pipeline, it correspond to a real leak. Acoustic interference from
is possible to determine the coordinates of the leakage mechanisms of pumping stations, elevators, valves,
site with high accuracy. The typical value of the etc. can spread in the pipeline. The operator has to

105
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

manually, by the method of subsequent approximation, depending on the material and diameter of the pipe. At
select the frequency band of signal analysis in order to the same time, the speed of sound depends on a number
build up from the crunching signals. of other parameters, especially on the thickness of the
b) The section of the pipeline between the sensors pipe walls. The thickness of the walls of worn pipes
may have bends and taps, due to which the distance can differ from the original several times, as a result of
between the sensors may differ from that entered into which the difference between the real significance of
the correlation receiver. Most regulatory documents the speed of sound and the given one can be up to 30%!
contain a strong recommendation to trace the surveyed To reduce this error, many correlation leak detectors
section of the pipeline using a tracer before starting implement a mode for measuring the speed of the call
work. by solving the inverse problem - determining the delay
c) The source of the greatest error can be the between signals for a source with pre-known
calculated value of the speed of sound in the walls of coordinates.
the pipeline, which, as a rule, is set in tabular form

Fig. 3. Appearance of common correlation leak detectors.

Despite the disadvantages listed above, portable case for transportation and programming and software
acoustic and correlative leak detectors provide control, which allows you to determine the correlation
acceptable results in most cases and are widely used in between the signals of any pair of sensors (see Fig. 4a).
organizations operating heat and water supply The practicals of using Enigma are as follows: during
networks. Less common are more expensive multi- the day, the dutchiks are installed on the pipeline, the
position leak detectors, which include several sensors sensors are programmed to start at night (when the
at once, which can be placed on the pipeline according level of sidely noise is minimal).The next day, the
to a given Schema. The sensors are synchronized with sensors are removed, information is read from them,
each other and simultaneously record acoustic signals, then the sensors are moved to a new section of the
storing them in their own memory. The data can then pipeline. The internal power supply of the sensors
be read on the PC for further processing. ensures operation for 5 years. The sensors themselves
A certain fame in domestic organizations received are water-resistant, can be operated at a depth of up to
a multi-position leak detector Enigma company 10 meters.
Primayer (UK). It consists of 8 self-powered sensors, a

Fig. 4. ‘Enigma’ multi-position leak detector and software.

106
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

The Enigma software allows you to calculate the the road cost 45350 dollars. Across the U.S. as a whole,
correlation functions between the signals of any pair of it is estimated that U.S. utilities spend more than $900
sensors (a total of 28 possible combinations) and million annually to fix water leaks and repair pipe
evaluate the number or absence of leakage in the form ducts.
of the correlation function. Correlation functions with The capabilities of geographically distributed
a minimum background level and a distinct maximum monitoring systems make it possible to implement
are considered to correspond to a leak with a high more complex and efficient algorithms for determining
degree of probability and are marked with a red fill the facts and locations of leakage. Thus, [1] data on the
color (see Fig. 4b). if the background level of the use in Toronto (Canada) of a system of "smart" sensors
correlation function is higher, the fill color will turn and artificial intelligence to determine "anomalous"
yellow (the probability of leakage is average), with a noise in the pipeline, different from the norm and
very high background level, the fill color is blue corresponding to leakage, are given. The server of the
(probably leakage accuracy is small). system analyzes the data received from the sensors and
An even more expensive, but also more effective sends the result to the central control room, which
method of detecting leaks in the present time is the automatically forms a request for repair and sends the
continuous monitoring of communications using repair team.
territorially distributed systems, the sensors of which In general, the above acoustic monitoring tools
are constantly installed in wells, on hydrants, in make it possible to make an introduction that in this
thermal chambers, etc. Reading information from such area, as in many others, there is a transition from
sensors is carried out remotely, as a rule, using traditional "semi-intuitive" methods of work to the use
GPS/GPRS or LoRaWAN channels. The most of intelligent computing systems that can allow:
widespread such systems have received abroad, where - Reduce the routine burden on operators;
with their help they expect to significantly reduce - Reduce the time to find and clarify the place of
losses from water leaks. leakage;
In 2019, Primayer entered into a £40 million ($53 - Reduce the cost of repairing communications.
million) framework agreement with Anglian Water, a To implement intelligent processing algorithms, it
utility in east England. Water utilities in England and is necessary to study the defining characteristics of
Wales are encouraged to reduce losses by at least 15% leakage signals in the time and spectral domain. The
between 2020 and 2025 and halve them by 2050 [4]. obtained characteristics will make it possible to assess
As part of the agreement, about 3500 Enigma3hyQ the hardware characteristics of the complex of
intelligent sensors using multi-point noise correlation processing devices and the necessary methods and
technology will be installed on the Anglian Water algorithms for decision-making.
water supply network. To visualize the locations of the
leak, the location of each sensor is displayed on Google
Maps and Street View. An example of the installation 3. Experimental Studies and Analysis
of Primayer sensors is shown in Fig. 5. of the Characteristics of Acoustic
Emission Signals

For experimental modeling of various leakage

options, a semi-natural stand has been developed and
is used, which is a pipe at several points of which ball
valves are installed to simulate the leakage of
adjustable velicina. At the ends of the pipe there are
areas for installing acoustic sensors. Pressurized water
is supplied to the pipe from the water supply network.
To receive acoustic emission signals, the Iscor-305
correlation leak detector is used, which includes
acoustic sensors, amplifiers-transmitters for
transmitting signals over a radio channel and a
Fig. 5. Acoustic sensor for Primayer geographically
distributed monitoring system. correlation receiver for receiving and processing
signals. The input of an external sound card is
connected to the line output of the receiver, Connected
Since 2010, more than 9400 Mlog Radio acoustic
to a PC. A general diagram of the installation for the
sensors from Itron have been installed on the water
experiment is shown in Fig. 6.
pipelines of Providence Water (USA). Providence
In the experiment, signals were recorded
Water estimated that 11.6 % of all water pumped into
alternately at different leakage values under the
the water supply goes into various kinds of leaks and
following conditions:
does not reach the consumer, which led to annual
- in the absence of leakage (there are only noises -
losses of more than $ 954,000. In addition, the most
transport, industrial, natural);
severe leaks can damage the foundations of buildings
- Leak source #1 is included;
and road covering. For example, to eliminate one of the
- Leak source #2 is included.
pipe breaks under the highway, the state The repair of

107
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

with the maximum value of the spectrum in the low-

frequency region slightly exceeding the maximum
value of the spectrum in the high-frequency region;
2) The spectrum of the signal in the presence of
leakage is more uneven, in it the level of the spectrum
in the low-frequency region is always less than in the
high-frequency, and this difference depends on the
amount of leakage - the greater the difference, the
weaker the leakage.
To assess the change in the spectrum of the signal
Fig. 6. Installation diagram for simulating and recording over time, the spectrograms shown in Fig. 8 are
acoustic emission signals. constructed. It is clearly visible that there is no
noticeable change in the signal spectra over time, and
The spectra of the recorded signals are shown in minor changes in the signal without leakage can be
Fig. 7 and have the following features: considered as a sign of the presence of short-term
1) The spectrum of the non-leakage signal is more interference from sources of transport, industrial or
uniform and has at least two regions with a maximum natural noise. It is also clearly visible that when the
amplitude - low-frequency, with a central frequency of amount of leakage increases, the leakage signal
~200 Hz, and a high-frequency one of about 1 to 4 kHz, spectrum shifts to the low-frequency region.

Fig. 7. Spectra of recorded acoustic emission signals.

108
 2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 8. Spectrograms of acoustic emission signals.

If spectral analysis shows how much the signal is ANNs to determine the fact of the presence or absence
"similar" to a set of sinusoids, then correlation analysis of leakage.
allows you to assess the similarity of the signal with Based on the results obtained, it can be
itself, i.e. to what extent it and the physical process that preliminarily concluded that the marked features of the
generates it are random. The autocorrelation functions spectrum and the autocorrelation function of the
(ACFs) of the recorded acoustic emission signals acoustic emission signal make it possible to determine
computed in MATLAB 2012 are shown in Fig. 9. the fact of the presence or absence of leakage. When
The following features of the ACF obtained can be leakage occurs, the spectrum and ACF of the signal
noted: acquire a characteristic form, which allows us to talk
1) In the presence of leakage, ACF has a more about the feasibility of their recognition using neural
periodic structure, with a shorter period for weaker network algorithms.
leaks, and a longer period for stronger ones;
2) In the presence of a leak, the first side petal of References
the ACF (marked in the figure with a red circle) is
noticeably higher than that of the ACF signal without [1]. https://www.everest.ua/ru/v-kanade-potery-vody-yz-
leakage. Also, the level of the side lobe of the ACF gorodskyh-trub-obnaruzhyvayut-s-pomoshhyu-
changes slightly when the amount of leakage changes. yntellektualnyh-datchykov, 26. 02.2021.
[2]. https://files.stroyinf.ru/Data1/46/46843/
index.htm#i21807, View date 26. 02.2021.
4. Conclusion [3]. http://www.vstmag.ru/ru/archives-all/2010/2010-
4/263-obnaruzhenije-skrytyh, View date 26. 02.2021.
The article discusses the main varieties of [4]. https://www.waterworld.com/technologies/flow-
equipment for acoustic leakage - acoustic, correlation level-pressure-measurement/article/16190942/
and multi-position leak detectors and geographically revolutionizing-remote-leak-identification, View date
26.02.2021
distributed acoustic monitoring systems, presents the [5]. https://www.waterworld.com/home/article/
results of spectral and correlation analysis of various 14070815/how-to-nix-nonrevenue-water-with-
leakage signals and assesses the possibility of using acoustic-leak-detection, View date 2/26/2021.

109
2nd IFSA Winter Conference on Automation, Robotics & Communications for Industry 4.0 (ARCI’ 2022),
2-3 February 2022, Andorra la Vella, Andorra

Fig. 9. Autocorrelation functions of acoustic leakage emission signals recorded in the experiment.

110
View publication stats