Embedded Visual Control

5HC99

Design and Implementation of a Quadcopter with

Visual Control

By
Stef Louwers (0590864)
Sunil Chokkanathapuram Ramanarayanan (0826874)
Qian Qian (0827493)
GROUP 3

Date: October 16, 2013

Professor: Prof. dr. Henk Corporaal, Mark Wijtvliet, MSc
Contents

1 Introduction 3

2 System Architecture 3
2.1 Local Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Global Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.4 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Software 8
3.1 Local control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Global control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

4 Objectives 9
4.1 Building Quadcopter and flying . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2 Tuning Local control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.3 Altitude hold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.4 Autonomous flight . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.5 360 degrees panorama . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.6 Drift correction in horizontal axis . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.7 Face detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.8 Marker identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

A Who did what? 14

A.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
A.2 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
A.3 Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

B Partlist 15

2
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

1 Introduction

In this project, a quadcopter is designed and implemented to autonomously fly and demon-
strate on-board image processing capabilities. The project is aimed to control the quadcopter
by virtue of two control systems; the local control, and the global control. The local control is
intended for the stabilisation of the quadcopter, so that it is able to fly manually through the
RC receiver and the global control is intended to make the quadcopter to fly autonomously
with the aid of visual control through the camera. These two control loops are implemented
on dedicated processors with appropriate capabilities.

2 System Architecture

In this section, we will discuss the used hardware and system architecture. First we will take
a look at the local control, global control and used sensors, and after that, we will address
communication used between these components.

2.1 Local Control

The local control is powered by an STM32F3Discovery development board (figure 1), powered
by the STM32F303VCT6 processor (a 72 MHz Cortex-M4 MCU), 48 KB RAM and 256 KB
flash memory. This board also contains a gyroscope, accelerometer and a magnetic sensor, all
measuring three axis. This makes it an ideal board for a quadcopter’s local control, because
it contains enough processing power, and has all the basic sensors available.

Figure 1: An STM32F3Discovery.

1 Introduction 3
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

2.2 Global Control

For the global control, we have chosen for the Raspberry PI (figure 2). We chose for this
platform because it was cheap, we already had it available, and the camera module was
expected soon after the start of the project.
Because we wanted two cameras on the quadcopter (one facing to the front, and one facing
down), we decided to add two Raspberry PIs to the quadcopter, as each Raspberry PI only
has the possibility to connect one camera. This also gave us more processing power available,
as each Raspberry PI only has to process one camera.

Figure 2: A Raspberry PI.

2.3 Sensors

The local control board already contained the basic sensors required to fly a quadcopter: a
gyroscope, an accelerometer and a magnetic sensor. To this list, we added a battery monitor,
a barometer and two cameras.

2.3.1 Battery monitor

The battery monitor was a simple voltage divider that connected the battery voltage to one of
the ADC pins of the STM32F3Discovery board. The voltage divider was necessary to convert
the voltage to the 0 – 3V domain.

2.3.2 Ultrasound

Initially the plan was to use the ultrasound sensor (figure 3) for tracking the altitude of the
quadcopter. The HCSR04 sonar driver was requested from Taulabs development team and
merged to our code base. The ultrasound driver implementation could be obtained in Github

2 System Architecture 4
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

link1 . As we further realized, the ultrasound sensor suffered propeller sound interference
which made the sensor values noisy and inaccurate. This happened because there were high
frequency components in the sound waves emitted by the propellers and it fell in the band
of 40 KHz which was the operating frequency of the sound waves used for ultrasound sensor,
hence this interference resulted in changing our design decision to use alternative altitude
tracking sensor such as barometer instead of the ultrasound sensor.

Figure 3: The HCSR04 sensor.

2.3.3 Barometer

In order to facilitate altitude hold, we needed some means of measuring the current altitude
of the quadcopter. Because of the problems we and other teams have experienced with the
ultrasound sensor, we decided to do something different and try a barometer (figure 4).
Compared to an ultrasound sensor, a barometer has less accuracy. Also, a barometer provides
only a absolute height (to sea level). Thus a command like “stay at 1 meter from the ground”
is not possible. Instead, we should say “stay at the current height”. The advantage of the
barometer over the ultrasound sensor is that it has no height limits, and that there are no
false measurements as reported with the other groups.

Figure 4: The BMP085 barometer.

1
https://github.com/scenkov/TauLabs

2 System Architecture 5
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

2.3.4 Cameras

We have chosen for the Raspberry PI cameras, because they are affordable, of relative good
quality, small and light, and they uses few system resources on the Raspberry PI. Because
each Raspberry PI can only interface with one camera, we have placed two Raspberry PIs
and cameras on the quadcopter.
These cameras are able to film in 1080p resolution with 30 fps, and take still images with
2592 x 1944 resolution.

Figure 5: Raspberry PI with camera module.

2.4 Communication

There are many communication-paths, as can be seen in figure 6. In this section, these will
all be discussed.

Figure 6: Overview of different communication protocols.

2 System Architecture 6
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

2.4.1 Local control and global control

The local control on the STM32F3Discovery communicates with the first Raspberry PI using
an USB-cable. They use the UAVTalk2 protocol, and thus communicate high level objects.
Because the UAVTalk protocol is already seamless integrated in the local control (it is also
used for communication with the Ground Control Station), all useful pieces of data are avail-
able as a UAVObject.
This makes it an extremely powerful interface. Global control algorithms have access to all
the information that it available to the local control, and it can influence and override all
aspects of the local control by writing new values to these objects.
The interface is also flexible. Currently it uses USB, but UART or other protocols are also
possible. The UAVObjects on the local control are also accessible on a laptop and on the
other Raspberry PI, using a WiFi or an ethernet connection, thus eliminating the need for a
separated (bluetooth) telemetry module.
A presentation on this topic is also available online on http://youtu.be/uEVFPBQtt0U.

2.4.2 Raspberry PIs

The two Raspberry PIs on the quadcopter need to be able to communicate, and they do this
by a standard ethernet (network) cable. We chose for this option because it is supported
out of the box and widely supported, and it is fast. Secondly, it allows the first Raspberry
PI to share its WiFi connection easily with the second Raspberry PI, allowing easier remote
management.

2.4.3 Laptop

On the Raspberry PI runs a normal Linux installation, and that gives the option to add a
WiFi adaptor so that it can communicate with the outside world using WiFi. We did this,
and even shared the internet connection to the second Raspberry PI, so both could easily
download and install new software.

2.4.4 Barometer

The barometer is connected to the first Raspberry PI using I²C. It is not connected to the
STM32F3Discovery, because there where some problems with the I²C driver in the TauLabs
system, that we where unable to properly debug. The Raspberry PI sends the altitude
information from the barometer to the STM32F3Discovery using the UAVObjects and the
UAVTalk protocol, as discussed in section 2.4.1.

2
http://wiki.openpilot.org/display/Doc/UAVTalk

2 System Architecture 7
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

2.4.5 Camera

The cameras are attached to the Raspberry PIs, and use a flatcable for the connection to a
CSI3 -interface.

3 Software

To get all this hardware flying, we need some software to control everything. We have software
for local and global control, and both will be discussed in this section.

3.1 Local control

For the local control, we chose for the open source project TauLabs4 . This project is a fork
from the OpenPilot project. We choose for this project, because is was already ported to the
hardware we where using (the STM32F3Discovery).
This project provided not only software for the local control, there was also a “Ground
Control Station” (GCS). This program runs on your computer and communicated with the
local control to provide configuration options and means of debugging and monitoring.

3.2 Global control

We have written our own global control program. This program runs on a Raspberry PI, and
is a stripped down version of the GCS provided by TauLabs. This allows it to communicate
with the local control using UAVObjects, as is discussed in section 2.4.1. Our global control
code is also available on GitHub5 .
The global control has a fixed core, and allows functionality to be added by modules. We
did not have enough time to write really interesting modules, but we do have some “proof-
of-concept” modules, which will be discussed next. These plugins show that it is possible to
read and write data from/to the local control.

3.2.1 Recorder plugin

This plugin monitors the “Armed” flag in the “FlightStatus” UAVObject, and as soon as the
quadcopter gets armed, it starts recording video on both Raspberry PIs. On disarming, the
recording stops. This allows us to easily create in-flight footage with the cameras.

3
Camera Serial Interface
4
https://github.com/TauLabs/TauLabs
5
https://github.com/fhp/TauLabs

3 Software 8
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

3.2.2 Barometer plugin

The barometer plugin reads the barometer that is connected to the Raspberry PI using I²C.
It parses the results and writes the resulting altitude information to the “BaroAltitude”
UAVObject that will then be send to the local control.

4 Objectives

Here we will discuss the objectives given in the course, and how we handled them.

4.1 Building Quadcopter and flying

We have build our quadcopter, and made fly. See figure 7 for a photo.

Figure 7: Our quadcopter.

4.2 Tuning Local control

We spend a lot of time on PID tuning, but only in the end we found that some of the settings
where completely wrong; the direction that the motors where spinning was configured wrong,
so yaw corrections resulted in more yaw, crashing the quadcopter. Also one of our speed
controllers was not functioning correctly, which led to an unstable quadcopter.

4.3 Altitude hold

Altitude hold should have been possible with the barometer installed, but there was no time
to actually test this.

4 Objectives 9
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

4.4 Autonomous flight

We did not manage to get the quadcopter fly autonomously, because it was not stable enough
and still needed some manual corrections, and we didn’t have altitude hold working.

4.5 360 degrees panorama

We where unable to implement this, because we didn’t have altitude hold and autonomous
flight.

4.6 Drift correction in horizontal axis

We where unable to implement this, because we didn’t have altitude hold and autonomous
flight.

4.7 Face detection

During our development period of the quadcopter, the Raspberry PI Foundation has released a
camera module (figure 5) which fits specifically into Raspberry PI’s CSI interface. The camera
has a 5 megapixel sensor and supports 1080p, 720p, and 640x480p video. The footprint
dimensions are 25 x 20 x 9 mm. Compared to a usb webcam which is widely used for
personal computers, the Raspberry PI camera has a relatively smaller size and Raspberry
PI has opensource driver for the camera which gives larger throughput in terms of image
processing. So we decided to use the Raspberry PI camera as the front facing camera.
The goal of this face detection task is to make the quadcopter follow and track the movement
of a human face while flying. We intended to use OpenCV6 library as a tool to implement
the face detection program on Raspberry PI, and after processing the image, we will find the
position of the human face relative to the boundary of the image. Based on that position we
can decide whether the quadcopter should fly higher or lower, go to the left or to the right.
Moreover, and estimated distance of the face from the camera can be calculated by measuring
the area of the face within the image taken by the camera. So the quadcopter can track the
face when the face is moving away or towards the quadcopter.
However, when we implement the face detection program, we found out that the API from
OpenCV library which will invoke the camera to take video is working. Because the Raspberry
PI camera uses a different driver model as the conventional usb webcams, the API which is
responsible for invoking the camera was not able to locate the camera.
In the Raspbian system, we where provided with two commands to operate the camera;
raspivid and raspistill. raspivid is a command line application that allows you to capture
video with the camera module, while the application raspistill allows you to capture images.
In order to extract the driver, we looked at the source code of the two applications in the
Raspberry PI github repository7 . We figured out that the Raspberry PI system uses MMAL
6
http://opencv.org/
7
https://github.com/raspberrypi/userland/tree/master/host_applications/linux/apps/raspicam

4 Objectives 10
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

functions to communicate with the camera. And we managed to trim down the the camera
driver and integrated it into the face detection code. By feeding the haar cascade filter with
the image get from the camera, the program is able to detect faces in the image. The output
of the program is the coordinate of the face within the image. Figure 8 shows that the
Raspberry PI can successfully detect faces.

Figure 8: The Raspberry PI has detected faces.

Since the face detection program is based on the haar cascade filter, it will have some draw-
backs. First, it will be able to find multiple faces in the camera scene. This may seem to
be an advantage, but if two faces appear on the scene, then the quadcopter may not know
which one to track, so it may lead to unexpected results. Furthermore, according to the way
that haar filter works, it will only detect a face that is placed vertically straight in the image,
other orientations will not be detected.

4.8 Marker identification

The expected output of our marker detection is that the quadcopter can use the front facing
camera to detect and recognise some specific markers on the wall, so that it can do some
pre-programmed action, that depends on the detected marker.
As an aerial vehicle, a quadcopter is very susceptible to disturbances from the surrounding.
So a quadcopter has to be constantly tuning its position according to the data from the
sensors in order to get its stable position. This has disadvantages for the cameras, because
for the cameras, a constantly moving quadcopter make the image blurred and therefore make
the detection harder. In order to confront this problem, we need to make the markers robust,
so that even for a moving quadcopter, the camera can still recognise the markers with high
detection rate.
What we use is called ArUco8 : a minimal library for Augmented Reality applications based
on OpenCV. It is a open-source marker detection library developed by University of Cordoba.
The library uses markers as shown in figure 9.
Each marker has an internal code given by 5 words of 5 bits each (as shown in figure 10).
The codification employed is a slight modification of the Hamming Code. In total, each word
8
http://www.uco.es/investiga/grupos/ava/node/26

4 Objectives 11
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

Figure 9: ArUco markers

has only 2 bits of information out of the 5 bits employed. The other 3 are employed for
error detection. As a consequence, we can have up to 1024 different ids. With the help
of Hamming coding, the marker is able to tolerate more errors, and the robustness of the
detection is increased.

Figure 10: Marker structure

The main difference with the Hamming Code is that the first bit (parity of bits 3 and 5) is
inverted. So, the id 0 (which in hamming code is 00000) becomes 10000 in our codification.
The idea is to prevent a completely black rectangle from being a valid marker id with the
goal of reducing the likelihood of false positives with objects of the environment.
Since we already have the experience of face detection program, we successfully integrated
the marker detection code with raspberry pi camera driver as well. As a result of the program
it can detect the marker printed on a paper as well as knowing the corresponding id of this
marker (figure 11). Based on the id, the quadcopter can be assigned to do some pre-defined
actions like turn around or go forward, etc.

4 Objectives 12
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

Figure 11: Detected markers

4 Objectives 13
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

A Who did what?

A.1 Hardware

Building the basic quadcopter Stef, Sunil, Qian

Basic PID tuning Stef, Sunil, Qian
More PID tuning Stef
Installing the cameras Stef, Sunil
Installing the ultrasound sensor Stef, Sunil
Installing the battery monitor Stef
Installing the barometer Stef

A.2 Software

UAVTalk on Raspberry PI Stef

Global control Stef
Face recognition Qian
Marker recognition Qian
Sonar integration on local control Sunil

A.3 Report

Ultrasound (section 2.3.2) Sunil

Face detection (section 4.7) Qian
Marker detection (section 4.8) Qian
The other sections Stef

A Who did what? 14

Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

B Partlist

Here follows a list of all the parts and components that we have used while building this
quadcopter.
In the total, all currencies have been converted to euro’s.
HobbyKing #2006549947
1 Extra Large EPP Quadcopter Frame 450mm $27.81 $27.81
1 Turnigy 5000mAh 3S 30C Lipo Pack $40.87 $40.87
2 Slow Fly Electric Prop 1045 SF $3.57 $7.14
Shipping – $7.93
HobbyKing #2006549967
1 Polymax 5.5mm Gold Connectors 10 pairs $5.39 $5.39
2 Slow Fly Electric Prop 1045R SF $3.53 $7.06
4 Turnigy DST-1200 Brushless Bell Motor 1200kv $7.16 $28.64
4 HobbyKing Red Brick 20A ESC $7.18 $28.72
Shipping – $6.01
Deal eXtreme #130402001086499449
2 HC-SR04 Ultrasonic Sensor $3.20 $6.40
1 JY-MCU Bluetooth Serial Port Module $8.20 $8.20
Deal eXtreme #130402001088428719
1 Super Mini Bluetooth 2.0 Adapter Dongle $1.80 $1.80
Dick Best #130506-231645-1685
2 Header 2x28 polig Female Recht Verguld €2.00 €4.00
2 Header 2x11 polig Female €1.20 €2.40
1 Header 15 polig Female €0.90 €0.90
Shipping – €2.25
Farnell #ORP116530
2 Raspberry PI, Model B, 512MB £28.31 £56.62
2 Raspberry PI Camera Board £20.04 £40.08
HobbyKing #2007104426
1 ZIPPY Flightmax 4200mAh 3S1P 30C $32.96 $32.96
Shipping – $6.01
Ebay #434632019023
1 BMP085 Barometric Digital Pressure Sensor $4.09 $4.09
Ebay #874439470014
1 Low Voltage Buzzer For 2s 3s Lipo Battery $1.96 $1.96
Subtotal €299.44

B Partlist 15
Embedded Visual Control S.T.Louwers, S.Chokkanathapuram Ramanarayanan, Q.Qian

Previous page €299.44

Ebay #160991485511
1 STM32F3 Discovery €13.05 €13.05
Shipping – €5.00
HobbyKing #2006549967
4 Turnigy DST-1200 Brushless Bell Motor 1200kv $6.37 $25.48
1 Nylon XT60 Connectors $3.29 $3.29
1 HXT 4mm Gold Connector $4.12 $4.12
Customs – €18.20
Shipping – $9.99
Amazon #304-1140681-4631520
1 TP-Link TL-WN725N Wireless N USB-Adapter €9.72 €9.72
Shipping – €4.69
Ebay #160991485511
1 STM32F3 Discovery £20.98 £20.98
Shipping – £10.99
Total €419.74

B Partlist 16