10th IFAC Symposium Advances in Control Education

The International Federation of Automatic Control

August 28-30, 2013. Sheffield, UK

An educational software to develop robot mapping and

localization practices using visual information
L. Payá, F. Amorós, L. Fernández, O. Reinoso

Departamento de Ingeniería de Sistemas y Automática

Miguel Hernández University. Avda. de la Universidad s/n. 03202, Elche (Alicante) Spain
(Tel: +34 96 665 88 59; e-mail: [email protected]).

Abstract: In this work, we present a software tool we have developed to be used in a computer vision
and mobile robotics subject whose main objective consists in designing algorithms to control an
autonomous robot. In applications that require the robot to move through an unknown environment, it is
very important to build a model or map of this environment and to estimate the position of the robot in
this map with enough accuracy. Map building and localization are two topics in constant innovation as
new methods are continuously appearing, and some of these methods may be mathematically complex.
Taking this fact into account, we have designed a platform that provides students all the necessary tools
to understand the algorithms and that allows students to configure them to optimize the mapping and
localization processes. We have added some databases, composed of several sets of indoor images,
captured in real environments under realistic lighting conditions, so students will face the problems that
would outcome in a real application. In this paper we present some details of implementation of the
platform and how the students could use it.
Keywords: Robot control, educational software, mobile robots, image matching, robot vision, laboratory
education.

our students learn the main strategies that can be used in map
1. INTRODUCTION
building and localization, using as input information the
When we have to control a mobile robot so that it carries out images captured by an omnidirectional vision system
a task autonomously in an unknown environment, two mounted on the robot. Among these strategies, we focus on
necessary steps in this control process consist in creating a the appearance-based approach. In this approach, no local
database or internal representation (map) of the surroundings feature extraction is carried out to build the map; the
and estimating the relative position of the robot in this information in the scenes is used as a whole and localization
environment. To do it, the robot must use the information is achieved based on the matching of the whole images. This
provided by the sensors it is equipped with and compare it approach has attracted the interest of the researchers due to
with the information previously stored in the map database. its simplicity, robustness and its ability to work well in
Very often, the robot uses the information captured by a unstructured and dynamical environments.
camera mounted on it as input data to solve the map building
To solve the control of the robot using this approach, two
and localization problems. Also, it is usual to work with
phases must be implemented. First, the robot goes through
catadioptric systems that offer omnidirectional images from
the environment to map and it captures some images from
the environment around the robot (Amorós, 2005). These
some positions. A new image is captured when it is different
cameras are a good solution due to the richness of the
enough comparing to the previous one. As images are very
information they provide and their relatively low cost
high dimensional data, in the appearance-based approaches it
comparing to other types of sensors. By these reasons, they
is necessary to compress the information, keeping the
have been studied extensively during the last years in robots
integrity of the scenes, with no feature extraction. Several
control.
techniques can be used to create a descriptor of the image.
The research developed in the topic of map creation and This set of descriptors will compose the map. Second, once
localization using visual information during the last years is the map is build, the robot has to be able to compute its
very large and lots of approaches and new algorithms have localization within it. To do it, the robot captures a new
been developed to date. This way, both computer vision and image, computes its descriptor and compares it with all the
robotics constitute two fields of engineering which are in a descriptors in the database (map). As a result, the current
continuous progress (Siegwart, 2004). position and orientation of the robot could be estimated.

These two topics are studied in a subject of our PhD program, We are aware of the fact that sometimes, students get lost in
in the Miguel Hernandez University (Spain). In this subject, the classroom as the algorithms they study can be

978-3-902823-43-4/2013 © IFAC 174 10.3182/20130828-3-UK-2039.00058

10th IFAC ACE
August 28-30, 2013. Sheffield, UK

mathematically quite complex. Also, hand-on experiments localization is carried out by means of matching this global
are a fundamental part of engineering studies so that students information. If these processes were carried out directly with
can acquire the ability to solve real problems and to use real high resolution images, the computational cost would be
equipment. Training with real equipments is fundamental in unbearable in large environments so, map building and
the learning process. Often, the students focus their attention localization could not be carried out in real time.
in the mathematical tools and they lose the real applications
of the subject. Also, the number of equipments available in
the laboratory to carry out the practical sessions is not usually
very high and some students have to share them so, they
cannot practice at the pace they require and the laboratory is
not usually available whenever they need to practise what
they are studying. These are the reasons why he have
implemented and developed this software tool. We can find
some virtual robotics laboratories in the literature (de
Queiroz, 1998; Ramasundaram, 2005; Jara, 2011).
In the software tool presented in this work, we have included
some databases of images captured in several real indoor
environments, under realistic lighting conditions, so that the
students can test the mapping and localization algorithms. We
have included some of the most important appearance-based (a) (b)
techniques to describe panoramic scenes: Fourier-based
approaches, Principal Components Analysis (PCA)
approaches and gradient-based approaches. So, the student
will be able to test them and he/she can compare in which
situations each of the approaches works better and how they
must be tuned to optimize their performance. Also, we have
included other intermediate images, including noise, rotations
and occlusions to test the localization algorithm. The tool is
fully interactive. Both the mapping and the localization
processes are fully configurable and results are showed
graphically so that students can make comparative studies.
The remainder of the paper is structured as follows; section 2 (c)
outlines the theoretical concepts of the subject to understand
how the application works. In section 3 we present the Fig. 1. (a) Pioneer P3-AT robot, (b) catadioptric system and
software tool, how to use it and its main features and, at last, (c) omnidirectional image captured with the camera and
section 4 presents the conclusions and future work. transformation to panoramic image.

2. APPEARANCE-BASED METHODS IN MOBILE In such applications, different techniques can be used to

ROBOTS CONTROL: MAP BUILDING AND reduce the dimensionality of the data, allowing us to work
LOCALIZATION using a smaller number of features. In this tool we have
included some of the most used compression techniques in
To build the database that we have included in the tool, we robotics applications: PCA (Principal Components Analysis),
have worked with a Pioneer P3-AT robot that is equipped DFT (Discrete Fourier Transform), HOG (Histogram of
with a catadioptric system, consisting on a forward looking Oriented Gradients) and GIST approaches. These methods
camera and a parabolic mirror that provides omnidirectional and the parameters the student can tune in the application are
images from the environment. To work with this information presented in the next subsections.
in an efficient way, the omnidirectional images are
transformed to panoramic images with a size of 64x256 Once the descriptors have been created using one of these
pixels. Fig.1 shows the robot (a), the camera and mirror (b) techniques (the map has been built), the application puts at
and a sample omnidirectional image and a conversion to the disposal of the student several test images, captured in
panoramic format (c). some half-way points among those stored in the map. We
have captured several sets of test images, the first one, at the
To build the databases, we have captured a set of same time we took the training sets and the rest in different
omnidirectional images on a pre-defined 40x40 cm grid in times of the day (including different lighting conditions).
several indoor environments. Fig. 2 includes a bird’s eye Also, the student can introduce different degrees of noise and
view of the grid used to take the images and several examples occlusion. The objective is to compute the position and the
of panoramic images (blue dots). orientation of the robot when it captured the test images. It
must be carried out just comparing the current visual
The appearance-based approach implies storing the visual information with the information in the maps.
information in the map with no local feature extraction and

175
10th IFAC ACE
August 28-30, 2013. Sheffield, UK

2.1 DFT-based techniques cells and then, each cell is represented by means of an
orientation histogram. Each image is represented by the
The Fourier approach we have implemented is based on the histograms of every cell ordered into a vector (Dalal, 2005).
Signature presented in (Menegatti, 2004). This Fourier
Signature presents two interesting properties: the most An omnidirectional image contains the same pixels in a row
relevant information concentrates in the low frequency although the image is rotated, but with a different order. We
components of each row and it presents rotational invariance. can take profit of this characteristic to carry out the
It consists in expanding each row of the panoramic image localization of the robot by means of calculating the
using the DFT. After this process, each image is represented histogram with cells having the same width as the image.
by a vector whose size is m x k1, being m the number of rows This way, we obtain an array of rotational invariant features.
of the panoramic images and k1 the number of Fourier On the other hand, to know the relative orientation between
components retained to build the descriptor. k1 is a two rotated images vertical cells are used with variable width
configurable parameter; the larger is k1, the most information and distance, to get a good resolution in angle estimation.
is retained from the scenes but the higher is the When a new image arrives, first, its histogram of oriented
computational cost. gradients is computed using horizontal and vertical cells with
To compute the position and orientation of the robot for each the same size as when the map was built. So, the time needed
test image, we compute the Fourier Signature and then, we to find the pose of the robot varies depending on both types
compute the Euclidean distance of the power spectrum of the of cells. To find the position of the robot, the horizontal cells
test image with the spectra stored in the map. The best match information is used, whereas to compute the phase it is
is taken as the current position of the robot. On the other necessary to use the information in the vertical cells. In both
hand, the orientation is computed with the shift theorem cases, the information is found by calculating the Euclidean
(Menegatti, 2004). The number of components per row to distance between the histogram of the new image and the
compute the orientation, k2, is also a configurable parameter. stored ones in the map.

2.2 PCA-based techniques 2.4 GIST techniques

Using PCA (Krose, 2004) , a set of images can be represented The descriptors based on GIST try to obtain the essential
in a new subspace where the images are transformed to a information from the image simulating the human perception
feature vector with k3 components, being k3 a tuneable system through its ability to recognise a scene through the
parameter that represents the PCA features which preserve identification of colour or remarkable structures, avoiding the
the most of the variance of the original set of images. PCA representation of specific objects or features (Oliva, 2001;
presents a good compression ratio; however, the resulting Torralba 2003). The descriptor we have implemented in the
features vectors do not present rotational invariance. To solve platform is based on Gabor filtering and it provides frequency
it, we have implemented an algorithm that applies PCA over and orientation information. Each scene is filtered using a
the Fourier Signature of the scene so it constitutes a double bank of Gabor filters with different resolutions and
compression effect and the resulting features present orientations. The scene is divided on cells, using the same
rotational invariance. The PCA map is made up of the matrix principles than in the previous section and the average value
V ∈ R k xM , which contains the k3 main eigenvectors (M is the
3 of each cell is computed.
number of pixels of the image) and the projections of the The location is estimated by calculating the minimum
training images p j ∈ R ; j = 1… N (N is the number of
k x1 3
distance between the horizontal cells’ descriptor of the
images in the map). database and the current image.

To compute the location where the robot took each test Table 1. Tunable parameters in each method
image, we have to project the test image onto the eigenspace.
Once the projection p i ∈ R k3 x1 is obtained, it must be Method Param. Meaning
k1 Number of module components.
compared with all the vectors p j stored in the map. The 1. DFT
k2 Number of phase components.
criterion used is the Euclidean distance. The image of the k1 Number of Fourier module comp.
database that presents minimum value of the distance (1) is 2. PCA k3 Number of PCA components
the corresponding one. k2 Number of Fourier phase comp.
k4 Number of horizontal cells.
∑ (p );
k3
d ij = il
− p jl
2
j = 1… N (1) 3. HOG k5 Number of vertical cells.
l =1 k6 Width of vertical cells.
k7 Number of Gabor Masks (Scale 1)
2.3 HOG techniques k8 Number of Gabor Masks (Scale 2)
4. GIST k9 Number of horizontal cells.
The HOG descriptors are based on the orientation of the k10 Number of vertical cells.
gradient along an image. The scene is divided in a number of k11 Width of vertical cells.

176
10th IFAC ACE
August 28-30, 2013. Sheffield, UK

Fig. 2 Appearance of the graphical interface of the application during an experiment. Part 1: Map creation.

can analyse which is the best descriptor in terms of

3. DESCRIPTION OF THE SOFTWARE TOOL
computational cost and memory requirements and how only a
The Graphical User Interface has been created with the tool good configuration of the parameters will lead to a correct
GUIDE (http://www.mathworks.com/products/matlab) of localization of the robot. We now describe these two steps.
MATLAB. In this section we present a detailed description of
the operations which are available through the interface. Our 3.1 Map building experiments
objective while designing it was to simplify the
understanding of global appearance descriptors in mapping In the first step, the student must create the map. The
and localization tasks. This way, all the parameters of the database shown on fig. 2 (on the left side) is composed of
different techniques are configurable. The distribution of 363 colour panoramic images distributed on a 40x40cm grid
images in the database, the position of the test images and the along 4 different areas. To build the map, the student has to
distance from the test to the map images are represented choose the compression technique to apply and he/she has to
graphically, and we also include the computational cost configure the value of the parameters. There are 5
information of both map building and localization process. At possibilities. As an example, fig. 2 shows the parameters for
last, a tool that allows the student to make a comparison the Fourier Signature and HOG descriptors. The value of
between the techniques under different situations has been every variable can be chosen using pop-up menus. If any
included. Fig. 2 shows the first part of the graphical interface parameter is not changed, the application uses the default
of the application during an experiment. With this part of the values that appear. After selecting the desired parameters, the
interface, the student can test the map building process. The student can generate the map. Once the map is created, a
student can select the desired descriptor and tune its graph is shown, including information about the necessary
parameters. Fig. 3 shows the second part of the graphical time to compute the descriptor of all the images and the
interface, where the student can make localization necessary memory to store this information. With the
experiments with the previously created maps. Both graphical graphical interface shown, the student can decide which
interfaces are quite intuitive and the student is guided through rooms have to be included in the map and, also, he/she can
the map building and localization processes. Also, the carry out a comparison between the time and memory
interface provides all the necessary results so that the student requirements of two descriptors.

177
10th IFAC ACE
August 28-30, 2013. Sheffield, UK

Fig. 3 Appearance of the graphical interface of the application during an experiment. Part 2: Robot localization.

When the student presses the button ‘Show Test Image’, the
3.2 Robot localization experiment application reads the test image and its position in the map.
Then, this position appears in the bird eye’s view of the map
The second step in robot control consists in estimating its (red x-mark) and the test image is shown in the bottom left.
location in the map. With this aim, some sets of test images
have been included in the tool. These images have been At last, when the student presses the button ‘Localize’, the
captured in half-way points between the map images and the application computes the descriptor of the test image and
objective is to estimate the position the robot had when calculates the Euclidean distance with all the map images’
capturing these test images by means of comparing the visual descriptors. The nearest image is returned as the
information. The compression method and parameters used corresponding position of the robot. Next, the application
are the same as in the map building phase. Until the map is estimates the orientation of the robot by computing the phase
successfully created, the ‘Localize’ button is disabled. With lag of the test image comparing to the Nearest Neighbour
the interface, the student can check the computational cost (NN) in the map. Once the localization and orientation have
and precision of each global appearance descriptor depending been computed, the panels of ‘Localization Results’ (fig. 3),
on the value of the parameters. indicate the area where the Nearest Neighbour is, the distance
between it and the test image, and the estimated orientation.
The image test set is composed of 9 positions per area, with Also, the interface shows the time needed to compute these
16 orientations per position (every 22.5º), with a total of 546 data. This is an interesting factor to know if this descriptor
different examples. In the interface, the student can select the could work in a real time robot control application. Under the
area, the number of test image, and its orientation. Also, test image, we include the image of the NN in the bottom left
he/she can test the time of the day when the images were of the window. That way, the interface facilitates the
captured and he/she can add noise or occlusions to the test comparison between both images in order to appreciate
images. Thanks to these features, the experiment will whether the location has been correct and its phase lag.
reproduce some typical situations that could happen in a real
application of an autonomous robot control (fig. 4). On the right side, the application plots the test image distance
with every image of the map. In each element position, a

178
10th IFAC ACE
August 28-30, 2013. Sheffield, UK

coloured circle is displayed. The colour and size of the circles substantially lower than in past years. Thanks to this tool, we
represent the similarity between images. Bigger red dots expect that students are able to design more sophisticated
indicate higher similarity, whereas smaller blue markers algorithms to control the movements of mobile robots and to
correspond with the less similar scenes. The student can set the correct value of the necessary parameters.
appreciate then if there are any other local minimums in the
map that could produce visual aliasing (it commonly happens We are now working in improving this tool from several
in real localization experiments in indoor environments, points of view. First, we would like to include new
where the visual structures tend to repeat). At last, in the localization and mapping approaches to make it more
upper right side, the application shows a recall-precision complete. Also, we would like to integrate this tool with a
graph, which has been calculated taking into account all the remote laboratory. This way, students could teleoperate the
test images. This is a good way to sum up the behaviour of real robot through internet and they could carry out a
the descriptor. probabilistic localization algorithm facing a real situation.

When a different compression method is selected, or any of ACKNOWLEDGEMENTS

the descriptor parameters modified, all the elements included
during the localization routine disappear, and the button This work has been supported by the Spanish government
‘Localize’ is disabled until the map is generated again. through the project DPI2010-15308, ‘Exploración integrada
de entornos mediante robots cooperativos para la creación de
mapas 3D visuales y topológicos que puedan ser usados en
navegación con 6 grados de libertad’.

REFERENCES
Amorós, F., Paya, L., Reinoso O., Fernández, L., and Marin,
J. M. (2010). Visual Map Building and Localization with
an Appearance-based Approach. Proceedings of 7th
Internacional Conference on Informatics, in Control,
Automation and Robotics, ICINCO 2010. 2, 423-426
Dalal, N. and Triggs, B. (2005). Histograms of oriented
Fig. 4. Examples of test images with different occlusion gradients for human detection. Proceedings of IEEE
percentage and with added Gaussian noise with different Conference on Computer Vision and Pattern
variances. Recognition, San Diego, USA.
de Queiroz, L.R.; Bergerman, M.; Campos Machado, R.;
To finish, as fig. 3 shows, the interface also offers the student Bueno, S.S.; Elfes, A. (1998). A robotics and computer
the possibility to compare the performance of the two vision virtual laboratory, Advanced Motion Control,
descriptors used to build the map. The student can carry out a 1998. AMC '98-Coimbra.
comparison between two different descriptors or compare Jara, C., Candelas, F., Puente, F., Torres, F. (2011). Hands-on
how the same descriptor changes its behaviour depending on experiences of undergraduate students in Automatics and
the configuration of the main parameters. Robotics using a virtual and remote laboratory,
Computers & Education, 57(4).
4. CONCLUSIONS Krose, B., Bunschoten, R., Hagen, S., Terwijn, B., and
Vlassis, N. (2004). Household robots look and learn:
In this work, we have presented an educative software tool
environment modelling and localization from an
for the learning of mapping and localization algorithms.
omnidirectional vision system. Robotics Automation
These algorithms constitute two crucial steps during the
Magazine, IEEE, 11(4), 45 – 52.
control of an autonomous mobile robot. The tool has been
Menegatti, E., Maeda, T., and Ishiguro, H. (2004). Image
designed to be used in a PhD subject about computer vision
based memory for robot navigation using properties of
applied to the control of mobile robots.
omnidirectional images. Robotics and Autonomous
Thanks to this tool, the students can put into practise the Systems. 47(4), 251-276.
algorithms studied in the classroom. The practical sessions Oliva, A. and Torralba, A. (2001). Modelling the shape of the
proposed allow the students to get a global view of the scene: a holistic representation of the spatial envelope.
subject and to understand the real applications. The platform Int. Journal of Computer Vision, 42(3), 145-175.
presents several benefits both for the student and the lecturer. Ramasundaram, V., Grunwald, S., Mangeot, A., Comerford,
The student can test all the algorithms through an intuitive N.B., Bliss, C.M. (2005). Development of an
graphical interface and he/she can carry out the experiments environmental virtual field laboratory. Computers &
whenever he/she needs, and he/she can go through the topics Education, 45(1).
at his/her own learning pace, with no timetable restrictions. Siegwart, R., and Nourbakhsh, I.R. (2004). Introduction to
This system empowers the autonomy of the student and autonomous mobile robots. The MIT Press, Cambridge,
allows a fair evaluation system. From the point of view of the Massachusetts.
lecturers, the general degree of knowledge of the students has Torralba, A. (2003). Contextual priming for object detection.
improved and the dropout rate in this subject has been Int. Journal of Computer Vision, 53(2), 169-191.

179