Design of Multi-Mode UAV Human-Computer Interaction System

Liang Jie, Cao Jian, and Wang Lei

Abstract—With continuous function expansion of military sometimes touch screen, large UAV GCS got well controlled.
UAV, the rapid increase in the number of airborne sensors, For small UAV GCS, generally control the small
the complexity of UAV operation is increasing. To enhance reconnaissance with UAV, using joystick and following the
future unmanned combat aircraft control and mission visual feedback information to adjust the location, posture
planning efficiency, multi-mode UAV control method will and so on. For front-line combatants the control device is
gradually become mainstream. In this paper a Multi-Mode small, easy to control directly. But with the single controller,
UAV Human-Computer Interaction(HCI) System was combat effectiveness mainly relies on the operator
designed with new technology achievements of virtual reality proficiency of joystick.
and artificial intelligence. Based on the design principle of The multi-mode operation of UAVs were widely carried
usability, maintainability and reliability, according to the out at home and abroad form very beginning, with tow-level
envisaged functions, the overall design of the system was GUI and voice, the TRIPS system has been implemented to
completed. The four modules that divided into the base layer
control multiple UAVs [3]. In 2014, with the development of
and the application layer with hierarchical structure was
virtual reality, artificial intelligence, the new integrative
introduced. The base layer was composed of multi-mode HCI
hardware and software products such as people interaction
device and data recording processing subsystem. Acting as
the system hardware I/O interface, the multi-mode HCI space, immersive helmet display, semantic intelligent voice
device use multi-mode interaction of immersion display, recognition, natural interaction devices are emerging. The
motion capture, eye tracking and voice recognition by means continuous software and hardware's achievements greatly
of intelligent devices such as VR HMD, 3D Sensor, VR facilitate and promote the multi-modal HCI research for
Glove and intelligent speech recognition equipment. And UAV.
according to the actual situation of GCS operation, the above US "Blue Shark Lab" which under the Office of Naval
hardware’s selection was completed. Data recording Research (US ONR) combined display technologies (such as
processing subsystem provides the network and database to virtual and augmented reality technology, 3D visualization
support the underlying. Human factors engineering system) and input devices (including from mobile phones to
evaluation subsystem and UAV simulation training tablet PCs, helmet-mounted displays, and head gesture
subsystem constitute the function application layer, realize control system, etc.), put forward the "communication and
the system functions. The whole system can provide collaboration for the enhanced environment (E2C2)"
multi-mode interaction human factors engineering validation technology, and use it in UAV control [4], as shown in Figure
and evaluation platform, supporting the development of new 1. China also actively explore multi-mode UAV interactive
HCI design, and high-frontal manned UAV operation training control technology, the WJ-600 UAV operating system that
environment for supporting the new CGS ground operator equipped VR helmet was showed in Zhuhai airplane
personalized training. exhibition 2016, as shown in Figure 2.
Index Terms—UAV; Multi-Mode; Ground Control
System(GCS); Human-Computer Interaction(HCI).

I. INTRODUCTION

A t present, the main control interactive devices of UAV

ground control station(GCS) are: joystick, mouse,
keyboard, touch screen etc[1]. Large UAV GCS generally use Fig. 1. US E2C2 UAV Fig. 2. WJ600 control system
PC with a special interface card to connect those devices to application with VR helmet
wireless control box. Using the mouse keyboard and In summary, to enhance future unmanned combat aircraft
control and mission planning efficiency, multi-mode control
Manuscript received August 8, 2017.
method of UAV will gradually become mainstream. As a
Liang Jie is with the Armament Science and Technology Department, result of the extensive use of immersion display, motion
Naval Aeronautical University, Yantai, Shandong, China. (phone: capture, eye tracking, voice recognition and so on, the new
086-0535-6635614; fax: 086-0535-6635614;e-mail: [email protected]) type of HCI will enable the operator to work as closely as
Cao Jian is with the Armament Science and Technology Department,
Naval Aeronautical University, Yantai, Shandong, China. ( e-mail: ddcjd@ possible and improve mission performance, without a
163.com) significant increase in workload.
Wang Lei is with the Aviation Command Department, Naval To this end, combined with the current virtual reality and
Aeronautical University, Yantai, Shandong, China. (e-mail:
[email protected]) artificial intelligence field of new technology achievements,

978-1-5386-3107-2/17/$31.00 ©2017 IEEE

Authorized licensed use limited to: Al-Akhawayn University. Downloaded on February 24,2025 at 16:26:02 UTC from IEEE Xplore. Restrictions apply.
this paper designs a UAV multi-mode HCI System. The programs, truly personalized targeted training will achieve.
system contains uses a series of hardware devices and
C. System Composition
software modules, that can be divided into four major parts:
multi-mode HCI device, data recording processing subsystem, In order to realize the above two main functions, with
human factors engineering evaluation subsystem and UAV modular design ideas, "multi-mode UAV human-computer
simulation training subsystem. The whole system can provide interaction system" was composed of the multi-mode HCI
high realistic UAV operations training and evaluation device, data recording processing subsystem, human factors
environment. engineering evaluation subsystem and UAV simulation
training subsystem. According to the hierarchical structure,
II. OVERALL DESIGN the multi-mode HCI device and the data recording processing
subsystem constitute the base layer, provide the interactive
A. Design Principles hardware and the network database to support the underlying,
The purpose of designing and implementation of bear the bridge connection system software and hardware
multi-mode UAV HCI system is to support the new resources; human factors engineering evaluation subsystem
multi-mode UAV man-machine interface design and and UAV simulation training subsystem constitute the
verification work. In view of the reality of virtual reality and function application layer, realize the system function
artificial intelligence, the related hardware and software described in 1.2, the overall composition structure is shown in
products are listed continuously, the system design of the Figure 3. The components will be described separately below.
overall principles are: maximizing the use of mature
technologies to reduce technological risks; selecting mature 0XOWLPRGH8$9+XPDQ&RPSXWHU
product shelf hardware to improve hardware reliability and ,QWHUDFWLRQ6\VWHP
SSOLF
+XPDQIDFWRUV
reduce maintenance difficulty; the system should have good HQJLQHHULQJHYDOXDWLRQ
8$9VLPXODWLRQWUDLQLQJ DWLRQ
VXEV\VWHP OD\HU
usability, maintainability and reliability; system meets the VXEV\VWHP
EDVH
standardized, modular, general requirements. 0XOWLPRGH+&,GHYLFH
'DWDUHFRUGLQJSURFHVVLQJ OD\HU
VXEV\VWHP
B. Function Design Fig.3. System Composition and Hierarchy
The functions of Multi-Mode UAV Human-Computer
Interaction System include: III. MULTI-MODE HCI DEVICE
a. Providing multi-modal interaction human factors This device, as the system hardware I/O interface, makes
engineering validation and evaluation platform, full use of the current virtual reality and artificial intelligence
supporting the development of new HCI design. new technology and intelligent hardware. Taking into account
Integration of multi-modal interaction means, in essence, is the data size of immersive rendering and body sense capture,
to solve the UAV CGS human factors engineering two independent computers were used to process, and
coordination of "Man-Environment-machine." So, the interactive data was passed by LAN. See Figure 4 for the
primary function of "UAV multi-mode human-computer composition and connection of the whole device.
interaction system," is to provide multi-mode UAV
interactive verification and evaluation platform. Specifically, eye LAN
tracker
through the operator action tracking, eye position capture and
USB
other ergonomic data collection and evaluation, for the layout Intelligent speech
UART recognition device
of the seats and OOP design ,the system can carry out analysis
VR
of content visibility, accessibility and other characteristics. HMD USB
b. Providing high-frontal manned UAV operation WIFI
USB
training environment, supporting the new CGS ground HDMI
USB
operator personalized training.
The traditional training model based on video teaching and
the use of actual equipment training is difficult to meet the Position 3D Sensor VR Glove
tracker camera
teaching and training tasks. The application of virtual reality Fig.4. Composition and Connection of I/O Devices
technology to build the UAV maneuvering simulation
training system can solve the problem. With Head-mounted A. Eye Tracker
display(HMD) equipment, the system can offer the 1:1 UAV The eye tracker uses eye-tracking technology to calculate
ground station scene, 1:1 ground operating table seat layout the position of the eyeball and the direction of the gaze. In
and control content. Based on the realizing of UAV operator principle, eyeball tracking is mainly used to acquire, model
virtual training, the other functions including special disposal and simulate the information of eye movement through image
and pre-mission exercise can be carry out easily. Training processing technology. The eye movement acquisition
with VR can bring the substantial cost reduction and large system device has infrared equipment, image acquisition
extent to training effect. In addition, with smart sensors to equipment or the camera on the general computer and mobile
achieve recognition of trainers and use different training phone, with the support of software algorithm to realize

Authorized licensed use limited to: Al-Akhawayn University. Downloaded on February 24,2025 at 16:26:02 UTC from IEEE Xplore. Restrictions apply.
eye-tracking. After calculation, considering typical CGS data acquisition by VR glove, which has a large number of
space, the collection accuracy should not be greater than 0.5 sensors s to identify the location and gestures. At the meaning
degree, the sampling rate is greater than or equal to 120Hz, time, different tactile feedback can be edited by vibrating
and the left and right eye should be captured separately. motor inside.
The system uses Qi Xin Yi Wei aGlass which satisfied the Noitom Perception Neuron dynamic system was selected.
above sample requirements. Its tracking range reaches the Neuron can support from 3 to 32 nodes, capture the pure arm
limit of human eye rotation (vertical 30 °, level 50 °), can and the fingers movements at the same time (installation of
cover FOV greater than 110 ° that suitable for most VR / AR / sensor as shown in Figure 6), and be perfect suitable of VR
MR equipments. HMD.
B. VR Head-mounted display (HMD)
With HMD and positioning tracker the system can track
displays and data gloves simultaneously in space and provide
operator a complete 3D scene with construction of separate
left and right picture. To meet above functional needs, we
choose HTC Vive. As the one of most mature products, the
parameters of HTC Vive are 110 °FOV, 2160 × 1200 Fig.6. Installation Instructions of Arm Finger
resolution and 90Hz refresh rate. By the way with
E. Intelligent Speech Recognition Device
breakthrough SteamVR ™ tracking technology, the device
has ability to track headset and wireless controller, calculate The intelligent speech recognition device collects and
the exact position at real time, support interactions between preproduces the input voice control instruction[8], uses the
objects in the virtual world, people and environment. speech recognition algorithm and the speech characteristic
database to identify the instruction code of the UAV ground
C. 3D Sensor Camera station, and tests the speech recognition effect.
By using 3D Sensor camera, the system captures the limb The system intends to adopt the SYN7318 Chinese voice
movements of the operator in the three-dimensional space interactive module (using UART interface), which is shown
with optical non-contact measurement, and collects the user in Figure 7.
operation instruction. With the timely dynamic capturing, the SYN7318

image recognition and the skeleton tracking the interaction to

virtual scene that used to support unmanned aerial vehicle Speech
Synthesis Power
$2B1
Audio
ground operator simulation training and CGS human Output $2B3
amplifier $8',2 Speaker

MP3 Player
engineering operation domain analysis and evaluation can be Controlle
r
8$57B7;'
UART
8$57B5;' Interface
realized. Speech
Recognition
Here we choose more mature Kinect 2 as a somatosensory Audio
Input
0,& MIC

interaction device. Kinect is launched by Microsoft, using Speech Wakeup

PrimeSense the infrared space coding technology, can capture Fig.7. Block Diagram of SYN7318
the human skeletal information and location information [5].
At the same time, Microsoft provides a powerful Kinect SDK IV. DATA RECORDING PROCESSING SUBSYSTEM
API to collect human bones point data information, depth
information easily [6].After laboratory verification (see Figure Data recording processing subsystem is mainly used for
5), its indexes meet the normal GCS operating space demand: network communication and database management, its
eye view angles 57 degrees (horizontal) and 43 degrees composition shown in Figure 8.
'DWDUHFRUGLQJ
(vertical), bone tracking distance 1.2 to 5.3 meters . DQGSURFHVVLQJ

''6QHWZRUN
FRPPXQLFDWLRQ 'DWDEDVH
PRGXOH PDQDJHPHQWPRGXOH

,QWHUDFWLYH
7DVNGDWD
GDWD
SURFHVVLQJ
Fig.5. Laboratory Verification of Kinect 2 SURFHVVLQJ
PRGXOH
PRGXOH
D. VR Glove Fig. 8. Data Recording and Processing Subsystem Components
Using Kinect to achieve hand tracking and gesture A. DDS Network Communication Module
recognition had been reported. But the data accuracy is
The high-speed network accessing of multi-mode
difficult to meet the actual needs of the interactive system
[7] intelligent hardware brings new needs of various
.So we carry out operator's hands and gesture capturing and
heterogeneous interaction, high-capacity data transmission

Authorized licensed use limited to: Al-Akhawayn University. Downloaded on February 24,2025 at 16:26:02 UTC from IEEE Xplore. Restrictions apply.
with reliable and stable low-latency, high-resolution The main function of the UAV controller is to transfer control
simulation model distributed computing on multi machine. information to the system. The human factors in controller
The interaction and distribution of high-performance data and design include the identification, size and reaction time of the
the platform reuse capability across heterogeneous platforms controller. Common errors related to the controller are: due to
should be careful studied. DDS network communication the distance between the controllers is too small, the adjacent
module relies on distributed data distribution technology to controller is operated by the wrong operation, the operation
solve the problem of real-time interactive data on distributed sequence of the controller is wrong; the direction of the
data platform, and apply advanced software design mode to operation is reversed; the force is too small or too big.
split the simulated data exchange management engine to Therefore, by capturing the human body and hand
upper interface, core services and underlying protocol then movements operation time ,posture, space etc, the operation
reorganized for proper usage(as shown in Figure 9),to achieve domain analysis module draw an conclusion whether the
the system structure of the orthogonal decomposition and appearance size, the layout of hardware meet the demands of
independent evolution, support distributed heterogeneous convenience, safety, comfort. Based on the virtual reality
platform using a variety of protocols for integration. technology, the above work can also rely on collision
2EMHFWRULHQWHG $VSHFW2ULHQWHG
09&,QWHUIDFH
6WDQGDUG)XQFWLRQ
,QWHUIDFH
detection motivated by captured real training data.
,QWHUIDFH ,QWHUIDFH ,QWHUIDFH

B. Vision Domain Analysis Module

,QWHUIDFH%ULGJLQJ
Depend on somatosensory and eye movement device ,
&RUH6HUYLFHVRI
6LPXODWLRQ
6HUYLFH vision domain analysis module get the in formations such as:
view direction and motion parameters by coordinate
3URWRFRO%ULGJLQJ
transformation, construct fixation point combined with image
processing technology and algorithm of reference map, first
2SHQ03 03, +/$''67&3,3 3URWRFRO fixation point position, the fixation point distribution, fixation
time and watch order and so on. Using these eye movements
0XOWLFRUH6\VWHP &OXVWHU6\VWHP 1HWZRUN6\VWHP
data, the module can judge and analyze the layout of the
+DUGZDUH
+HWHURJHQHRXV
RSHUDWLQJV\VWHP
6HPLSK\VLFDO
3HRSOHLQWKH
ORRS explicit interface, the operator's workload, and the degree of
Fig.9. Hierarchy of heterogeneous simulation platform multi-protocol data watching fatigue, etc[10]. By establishing eye movement as the
interaction software variable, comprehensive evaluation of human machine
B. Database Management module interface is made.
Considering the system's strong scalability and flexibility, C. Speech Instruction Analysis Module
software MySQL was used to manage multiple simulation The intelligent speech recognition device is used to support
and evaluation activities database subsystem, while open the design and performance test of voice control instructions
network access capabilities. Overall, the process can be of UAV GCS command. Verified with a large number of
divided into two sub-modules HCI data and evaluate training samples delay or voice recognition rate as the performance
mission data. Each sub-module is composed of several index, the spatial distribution of microphone array and the
database relational tables and basic data functions that identifiability of limited voice instruction are tested.
support data entry, query, modification, and deletion.
VI. UAV SIMULATION TRAINING SUBSYSTEM
V. HUMAN FACTORS ENGINEERING EVALUATION
Using VR HMD, 3D somatosensory and VR gloves of
SUBSYSTEM
Multi-mode HCI device, the UAV simulation training
With eye trackerˈ3D sensor camera and voice recognition subsystem provide independent left and right eye images to
device, human factors engineering evaluation subsystem construct a complete lifelike scene for operator, capture the
evaluate the data by internal construction of ergonomics three-dimensional space operator's body movements, collect
evaluation system and assessment method, quantify operating user operation instructions and real-time interaction with the
domain, sight and speech interactive control, the composition virtual scene, so as to achieve the purpose of training.
shown as Figure 10. The training subsystem includes virtual console and
Human factors engineering
avionics system, training subject subsystem, UAV flight
evaluation subsystem
simulation module and three-dimensional visual simulation
module, as shown in Figure 11.
8$9VLPXODWLRQ
2SHUDWLRQGRPDLQ 9LVLRQGRPDLQ 6SHHFKLQVWUXFWLRQ WUDLQLQJVXEV\VWHP
DQDO\VLVPRGXOH DQDO\VLVPRGXOH DQDO\VLVPRGXOH
Fig.10. Composition of Human Factors Engineering Evaluation Subsystem YLUWXDOFRQVROHDQG WUDLQLQJVXEMHFW 8$9IOLJKW 'YLVXDO
DYLRQLFVV\VWHP VXEV\VWHP VLPXODWLRQPRGXOH VLPXODWLRQPRGXOH
A. Operation Domain Analysis Module Fig.11. Composition of UAV Simulation Training Subsystem
The operation domain analysis mainly considers whether
the controller design and layout follow the appropriate rules[9].

Authorized licensed use limited to: Al-Akhawayn University. Downloaded on February 24,2025 at 16:26:02 UTC from IEEE Xplore. Restrictions apply.
 A. Virtual Console and Avionics System interactive system based on the latest hardware and software
Virtual Console and Avionics System brings the virtual technology in VR AI. Article embarks from the system
operation scenario for UAV. The key considerations are as overall design, on the basis of functional determine,
follows: a same size(1:1) UAV GCS, floor console and other according to the modularized hierarchical design thinking, the
three-dimensional models should be established by modeling system four subsystems including the multi-mode HCI device,
technology, avionics system internal data processing logic data recording processing subsystem, human factors
should be built by mathematical modeling, and analog GCS engineering evaluation subsystem and UAV simulation
state information display method should be realized using the training subsystem were designed. First of all, considering the
production rules.. design principles and the system functions, this system apply
immersion display, motion capture, eye tracking, voice
B. Training Subject Subsystem recognition and other multi-mode intelligent control to
The subsystem can set various general flight environment, improve mission performance and response time. Then, based
flight program for different characters: for pilot, set UAV on the typical GCS space and operational characteristics, the
flight preparation, ground check, take-off climb, near-field hardware selection is completed. Then, the detailed function
adjustment fly, flight distance, antenna tracking (lost, search, and structure of each subsystem (module) are described. The
capture and automatic tracking) ,aircraft self-destruction and whole system can provide multi-modal interaction human
other training content, etc; for load operator, set the UAV factors engineering validation and evaluation platform,
mission control subjects, including photography, television supporting the development of new HCI design, and
and infrared reconnaissance, target positioning, etc. high-frontal manned UAV operation training environment for
C. UAV Flight Simulation Module supporting the new CGS ground operator personalized
training.
The UAV flight simulation module builds the equations
and control law of the aircraft using the aerodynamic REFERENCES
parameters, airfoil parameters and dynamic parameters of the
[1] Zhou Zhong. A Summary of the Development of Unmanned Aerial
UAV. This module implements kinematics and dynamics, Vehicle Ground Station [[J]. Avionics technologyˈ2010, 4l1 (1): 1-6.
control and sensor simulation at real-time, the CGS virtual [2] Zhang Yugang. Design of Man - machine Interface Design and
scene and aircraft status information (and parts) are actuated Simulation System for Unmanned Aerial Vehicle Ground Control
follow the calculation result, is the core of whole subsystem. System [D]. Northwestern Polytechnical Universityˈ2007
[3] Liu Y Q, Chen S G, Jiang G H, et al. VR simulation system for EVA
D. 3D Visual Simulation Module astronaut training [C] /Proceedings of AIAA Conference on
International Space Station Utilization. Alexander: American Institute
The 3D visual simulation module realizes the functions of of Aeronautics and Astronautics Press, 2010: Article No. 8696
virtual scene display, view point calculation, occlusion [4] US Blue Shark Laboratory VR technology to change the future combat
game [EB/OL] .http://mp.weixin.qq.com/s?__ biz= zIy NTA5Dgy
cutting, image updating and so on. Different from the general MQ%3D%3D&idx=1&mid=401304324 &scene =21&sn=
scene simulation application, GCS operation simulation, 7986ce1896bb4d96c1b870d01876850e .2016-02-23
which belongs to the equipment close-in operation [5] Dong Shiming. The Research on the Interactive Technology of
application, pays more attention to the interaction demand of Augmented Reality Based on Kinect [D]. School of Computer
Engineering and Science Shanghai University, 2014.
the operator's real sense and instruments. From the [6] Yu Tao. Kinect in Action [M]. Beijing, China: China Machine Press,
perspective of simulation, the high accuracy of GCS model, 2012: 28-38.
the complexity of the normal operation of the instrument and [7] Wang Liang. A Study of Dynamic Gesture Tracking Recognition and
Human-Computer Interaction Technology [D] XiDian University,2014
the accuracy of interactive flow control need more computer [8] Gao Xue. Research on the Speech Recognition Technology in Human
hardware and software resources. In particular, the high Computer Interaction [D] North China University of Technology, 2017
resolution dual channel vision output is more demanding for [9] GUO Yuan, SHEN Li-ming, SHI Xin, et al. Study on ergonomics
research in the information age [J]. Chinese Journal of Ergonomics,
the speed and memory capacity of graphics card. The 3d 2016, 22(2): 83–86.
vision simulation module needs to optimize the geometrical [10] Ding Yu Lan. Ergonomics [M]. Beijing, China: Beijing Institute of
model precision, the acceleration of the program and the Technology Pressˈ2011.
control of the view point.

VII. CONCLUSION
As military UAV function and expanding operations,
continuous increase of UAV group control requirements, the
traditional "mouse-keyboard" single mode HCI technology
has become unable to meet the needs of future unmanned war.
On the other hand, the new achievements of virtual
reality(VR), artificial intelligence(AI) technologies bring the
possibility to design and develop the new multi-mode UAV
HCI system.
To this end, this paper designs a new multi-mode UAV

Authorized licensed use limited to: Al-Akhawayn University. Downloaded on February 24,2025 at 16:26:02 UTC from IEEE Xplore. Restrictions apply.