See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/318689851

ROS-based UAV control using hand gesture recognition

Conference Paper · May 2017

DOI: 10.1109/CCDC.2017.7978402

CITATIONS READS
42 1,502

4 authors, including:

Yangguang Yu Xiangke Wang

National University of Defense Technology National University of Defense Technology
24 PUBLICATIONS 288 CITATIONS 179 PUBLICATIONS 2,329 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Yangguang Yu on 16 September 2018.

The user has requested enhancement of the downloaded file.

 ROS-based UAV Control using hand gesture recognition
Yangguang Yu1 ,Xiangke Wang2 ,Zhiwei Zhong3 and Yongwei Zhang4

Abstract— As the first step towards the natural interaction

between human and multi-UAV system, we present a novel
human-UAV interaction method based on hand gesture recog-
nition. We design different hand gestures corresponding to
different UAV commands and propose a stable algorithm to
distinguish these gestures. Then, considering the fact that the
operation of multi-UAV system may need multiple cooperative (a) Asus Xtion Pro Live camera (b) AR Drone
operators, an algorithm based on position information and color
Fig. 1: experiment hardware
information is designed to identify multiple operators. Thus
multi-user recongnition by hand gestures is achieved. Lastly,
we combine the results of hand gesture recognition of multiple
operators with UAV control under Robot Operation System
(ROS) framework and use AR Drone platform to test the whole
systems performance.

I. INTRODUCTION
Recent years, unmanned robot system, such as UAV, got
a booming development. But it has become a challenge that
how to control them with a good real-time performance. In
this thesis, we intend to find a both quick and reliable method
to achieve that target. The commutation among human gives Fig. 2: main framework of the overall system
us enlightenment. Human can accurately and naturally ex-
press his purpose just using simple hand gestures. Actually,
sending command using hand gesture is a good option both [5]. This approach exploits both data downloaded from the
for human operator and UAV. For human operators, so many web and gestures collected from other users and permits to
instruction sets are not needed and operators can command learn a set of person-specific classifiers. When people make
the UAV in a more natural way, which is very helpful in different gestures, the muscles on hand will have distinct
emergency situations. For UAVs, it only need a camera and changes. Based on this theory, Rui Fukui achieved hand
a processor to understand the command, which is economical gesture recognition using wearable wrist contour measuring
and easily accessible. device [6].
In the past, the problems of gesture and action recognition People have a long desire to achieve natural interaction
have received considerable attention [1]. Many methods are with robots and many efforts have been paid towards this
used to recognize hand gestures, such as histogram of local target. Gaze is used to regulate conversations between a
orientations [2] and a precursor of HOG [3]. Ying Yin humanoid robot and two human participants [7]. Valiallah
proposed a novel method hand recognition algorithm based (Mani) Monajjemi gave an example of creating, modifying
on flattened hierarchical hidden Markov Model [4]. In her and commanding teams of UAVs by an uninstructed human
thesis, she focused on how to smoothly handle two different [8]. Using machine vision techniques, Jawad Nagi gave an
kinds of gestures and determining how and when the system approach which enables human operators to command and
should respond to gestures. Aimed to avoid hundreds of control Parrot drones by giving them directions to move,
training for hand gesture recognition, a transfer learning using simple hand gestures [9]. Onboard depth camera was
approach was proposed by Bellocchio E, Valigi P, et al also utilized to recognize simple hand gestures and many
1 Yangguang Yu is with the College of Mechatronics & Automation,
tasks can be achieved such as UAV person-follow [10].
National University of Defense Technology, Changsha 410072 China
Almost all the literature we read focuses on the interaction
[email protected] between UAV and single operators. But for UAV swarm,
2 Xiangke Wang is with the College of Mechatronics & Automation, Na-
which usually has dozens of UAVs, it may need multiple
tional University of Defense Technology, Changsha 410072 China phone:
+86-13975138479, e-mail: [email protected]
operators to work cooperatively. To solve this issue, operators
1 Zhiwei Zhong is with the College of Mechatronics & Automation, identities should be identified firstly. Therefore, we design a
National University of Defense Technology, Changsha 410072 China simple but effective algorithm to distinguish different oper-
[email protected] ators based on position information and color information.
1 Yongwei Zhang is with the College of Mechatronics & Automa-
tion, National University of Defense Technology, Changsha 410072 China As for hand recognition, we present a novel algorithm to
[email protected] recognize nine different hand gestures corresponding to nine
 Algorithm 1 HAND GESTURE RECOGNITION
Input: Current camera image I
Output: the hand gesture flag set Fg

1: Get position of both hands pl [i] and pr [i]

2: Compute the position error between current frame and
(a) forward (b) backward (c) leftward last frame

3: if The error corresponding to a certain gesture exceeds

a pre-designed threshold then
4: Corresponding hand gesture count number increases
by 1
5: else
(d) rightward (e) ascent (f) descent 6: Set corresponding hand gesture count number zero

7: if The gesture count number reaches a certain level then

8: Set the flag of corresponding hand gesture TRUE
9: else
10: Set the flag of corresponding hand gesture FALSE
11: update all the hand gesture flags
(g) hovering (h) rotating (i) rotating 12: return result
clockwise anticlockwise

Fig. 3: the designed hand gestures

information as well as depth information of every pixels,
which give us much convenience to distinguish different
different UAV commands. At last, we achieve the complete kinds of gestures. In fact, distinguishing different gestures
control loop from hand gesture to UAV control under ROS only using vision images is a complex work. It usually needs
framework and final experiment proves the whole system thousands of lines code as it is often related with a series of
performs well. image processing. But thanks to the Nite library developed
II. APPROACH by PrimeSense, we can save much time and energy for image
processing and get the hand gesture in a more accessible way.
Our approach consists of three main components, namely
By using the functions in the Nite library, we can easily
hand gesture recognition, multi-operator recognition and
getabsolute coordinates of both hands in world frame. But
UAV control. The main framework of the whole system is
as the position of the operator in the image is uncertain,
illustrated in Fig. 2. We employ an Asus Xtion Pro Live
we cannot judge the hand gesture directly by the position of
(see Fig. 1.a) to acquire the image. Firstly, we observe
hands. Therefore, we design a robust algorithm to distinguish
the fact that the operators usually wont move frequently.
designed hand gestures. The steps of algorithm are reported
Therefore, we distinguish operators identities based on their
in Algorithm 1.
position. In order to avoid confusing, we choose to use
a dictionary learning and sparse representation method for B. Multi-operator recognition
identity recognition when operators are close to each other.
We mainly utilize the position variation of hand during the As our final target is to achieve the Human Robot Inter-
process of making gestures to achieve gesture recognition. action (HRI) with multiple-aircraft system, some practical
Finally, we use the AR Drone (see Fig. 1.b) as our platform problems must be considered. One of the problems we faced
and achieve the communication between the hand recognition is that the control for a large number of UAVs may need two
and UAV control under ROS framework. or even more operators. Therefore, as the first step, we want
to achieve distinguishing different operators and recognizing
A. Hand gesture recognition their hand gesture simultaneously.
The basic states of UAV can be divided into nine types, Firstly, we consider the facts that operators won’t move
i.e. moving forward, moving backward, moving leftward, frequently or fiercely and usually hold a distance from each
moving rightward, ascent, descent, rotating clockwise, ro- other when giving commands. Therefore, operators can be
tating anticlockwise and hovering. Therefore we design nine idetified based on their position history. If position error
different hand gestures to represent nine different commands, between current frame and last frame is less than a certain
depicted as Fig. 3. threshold value, we can make a judge that the person detected
To acquire the hand gesture images of operators, we in current frame should be the same as the person detected
choose the Xtion produced by ASUS, a kind of depth camera in last frame. But when operators stand close enough (for
similar to Kinect. Using Xtion, we can acquire the color example, when operators are exchanging their position), this
Algorithm 2 MULTIPLE-COMMANDER RECOGNITION
Input: Current camera image I and image dictionary set D
Output: Identity flag

1: for i=1, 2,. . . ,Nt do

2: Detect the areas which belongs to operator i
3: Cut the detected area into image patch set X
4: Identity f lag[i] = 0
5: for j=1,2,. . . ,Nt do
6: TotalError j = 0
7: for k=1,2,. . . ,Ns do
8: αk = arg min ∥αk ∥1 s.t.∥xk − D j αk ∥2 ≤ ε
αk
9: error jk = ∥xk − D j αk ∥0
10: TotalError j = TotalError j + error jk
11: Identity f lag[i] = arg min TotalError j
j
12: return result

algorithm may won’t work. Then, we consider the fact people

usually wear different clothes and this condition is also very
easy to satisfy. Thus, the vision feature of clothes is utilized
to distinguish different operators. When the operators stand
close enough, the method we use is dictionary learning and
sparse representation. Firstly, as long as the process of recog-
nition start, we record the first frame of image. Let Nt denotes
the number of operators and Ns denotes the number of small
patches that a image is cut into. Then we detect the areas
that belongs to Nt operators and cut corresponding area into
Ns small patches respectively. Then, we use these patches to
form image directory set D = {D1 , D2 , D3 , . . . , DNt }. Fig. 4: Complete algorithm of multiple-operator recognition
Finally, we use these image directories to make a dis-
tinction for the following image frame. The algorithm are
as our experiment platform. Using AR Drone, we test the
reported in Algorithm 2.
whole system’s performance.
But as the dictionary learning method require much com-
putation and computation time, we will still distinguish mul- III. EXPERIMENTAL RESULTS
tiple operators based on position information when operators In this section, we will demonstrate the experiment results
hold a certain distance. At last, the complete algorithm of of our approaches. The goal of our approach is to achieve fol-
multiple-operator recognition is depicted as Fig. 4. lowing functions: (i) the system can recognize the designed
C. UAV control hand gestures with a low mistake rate. (ii) the system can be
operated by multiple operators and distinguish their identity
After hand gesture is successfully recognized, we need to
correctly. (iii) the overall system can allow person to control
send the recognition result to UAV and transfer it to control
UAV using simple hand gestures.
command. The Robot Operation System (ROS) provides a
good interface for message transformation. A. Hand gesture recogniton
In ROS, programs which have different functions can To evaluate the performance of our algorithm for hand
run simultaneously in different nodes. The message between recognition, we invited three volunteers who didnt take any
different nodes can be transferred via a certain topic rec- training at all in advance to test the algorithm. After a
ognized by a unique name. In our work, two nodes runs simple explanation, volunteers made designed hand gestures
simultaneously, namely the hand gesture recognition part and arbitrarily and we made records about what hand gestures
the UAV control part. The recognized type of hand gesture is they made and the success rate. The experiment results are
sent from recognition part to control part via a topic named given in Table I. From the results, we can see the overall
/hand gesture. Then the UAV compute the output of rotors recognition rate is over 90
according to different hand gestures received. The control
structure graph is given in Figure 5. At the final step, we B. Multi-operator recognition
choose AR Drone UAV, which is a small quadcopter UAV Considering there won’t too many operators in practical
equipped with camera, ultrasonic sensors and IMU sensors, situation, we tested our recognition algorithm in two-operator
 Fig. 5: The control structure graph of the whole system

TABLE I: EVALUATION HAND GESTURE RECOGNI- TABLE II: EVALUATION HAND GESTURE RECOGNI-
TION PERFORMANCE TION PERFORMANCE

Commands Person1 Person2 Person3 m0.1¡ m0.1¡

Forward 85% 90% 100% Commands
Person1
Backward 100% 93% 95% Person2
Person3heightForward
Leftward 100% 100% 100% 85%
Rightward 100% 80% 92% 90%
100%heightBackward
Ascent 100% 100% 100% 100%
Descent 90% 97% 100% 93%
95%heightLeftward
Hovering 87% 98% 100% 100%
Rotating 95% 100% 100% 100%
100%heightRightward
clockwise 100%
Rotating anti- 94% 100% 96% 80%
92%heightAscent
clockwise 100%
100%
100%heightDescent
90%
97%
100%heightHovering
scenario. We designed a scenario where two operators are 87%
98%
exchanging their positon and the recognition performance 100%heightRotating clockwise
95%
of system is recorded during this process. The results are 100%
100%heightRotating anticlockwise
depicted in Fig. 6. In Fig. 6.a, two operators stand holding 94%
100%
a distance in the start. In Fig. 6.b, two operators are ap- 96%height
proaching each other. In figure Fig. 6.c, two operators have
exchanged position successfully. In Fig. 6.d, two operators
are departing from each other. During the whole process,
the operators’ identities were distinguished without mistakes.
The results indicate that the recognition algorithm works
well.
was security guard, who can land the UAV in emergency
C. Full system situations. The operators gave gesture commands in front of
To evaluate the overall performance of the whole sys- the Xtion and controlled the AR Drone as shown in Fig.
temwe firstly set some barriers in room as depicted in Fig. 7.b. During the experiment, the AR Drone flied smoothly
7.a and hoped AR Drone can fly through them under the and went through the barrier without much difficulty. We
hand gesture of single commands. Then, in order to verify also tested the security guard land the UAV in emergency
the performance of multiple-operator control, we designed a and the system worked well. The result clearly proves that
scenario where UAV were controlled by two operators. We hand gesture control UAV is an effective and promising HRI
designed one operator was the main operator and the other method.
 [4] Yin Y. Real-time continuous gesture recognition for natural multi-
modal interaction. Massachusetts Institute of Technology, 2014.
[5] Costante G, Bellocchio E, Valigi P. Personalizing vision-based gestural
interfaces for HRI with UAVs: a transfer learning approach, presented
at Intelligent Robots and Systems (IROS 2014), 2014 IEEE/RSJ
International Conference on. IEEE, 2014, pp.3319-3326.
[6] Fukui R, Hayakawa N, Watanabe M, et al. Hand gesture interface
(a) (b) for content browse using wearable wrist contour measuring device,
presented at Intelligent Robots and Systems (IROS) 2015 IEEE/RSJ
International Conference on. IEEE, 2015, pp.1222-1229.
[7] B.Mutlu, T.Shiwa, T.Kanda, H.Ishiguro, and N.Hagita, Footing in
human-robot conversations: how robots might shape participant roles
using gaze cues, in Proc. of Int. Conf. on Human Robot Interaction
(HRI), 2009, pp.6168.
[8] Monajjemi V M, Wawerla J, Vaughan R, Hri in the sky: Creating and
commanding teams of uavs with a vision-mediated gestural interface,
(c) (d) presented at Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ
International Conference on. IEEE, 2013, pp. 617-623.
Fig. 6: Multiple-operator recognition [9] Nagi J, Giusti A, Di Caro G A. Human control of uavs using face
while operators are exchanging position pose estimates and hand gestures, presented at Proceedings of the
2014 ACM/IEEE international conference on Human-robot interaction.
ACM, 2014, pp.252-253.
[10] Naseer T, Sturm J, Cremers D. Followme: Person following and ges-
ture recognition with a quadrocopter, presented at Intelligent Robots
and Systems (IROS), 2013 IEEE/RSJ International Conference on.
IEEE, 2013, pp.624-630.

(a) single operator test (b) multiple operators test

Fig. 7: Experimental scenario

IV. CONCLUSION AND FUTURE WORK

In this paper, we present a novel method for controlling
UAV, namely using the hand gesture. Firstly we solved the
hand recognition problem based on hands position infor-
mation and got good experiment results. In order to pave
the way for controlling swarms in next step, we designed
an algorithm to distinguish multiple operators. Lastly, we
achieve the whole system under the ROS system and got
expected performance. The experiment results indicates that
using hand gesture to control UAV is an accessible and
effective method. We believe this method bears a large
potential for practical applications in the future.
In the future, we may continue our work in the following
domains: i)our experiments were mainly conducted in indoor
environments. In fact, we test the system in outdoor envi-
ronments and the recognition performance is poor. We will
improve the system adaptability for the outdoor environment
in next step. ii) At present, we can just send some simple
operators to UAV using hand gestures, which is somehow
awkward. In the future, we want to achieve a smarter and
more convenient controlling method for multiple UAVs using
hand gestures.

R EFERENCES
[1] J. M. Chaquet, E. J. Carmona, and A. Fernandez-Caballero, A survey
of video datasets for human action and activity recognition, Comput.
Vis. Image Underst., vol. 117, no. 6, 2013, pp. 633659.
[2] W. T. Freeman and M. Roth. Orientation histograms for hand gesture
recognition, in FG, volume 12, 1995, pp. 296-301.
[3] N. Dalal and B. Triggs. Histograms of oriented gradients for human
detection, in CVPR, volume 1, June 2005, pp.886-893,.

View publication stats