See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/370903296

Evaluating Flight Performance and Eye Movement Patterns Using Virtual

Reality Flight Simulator

Article in Journal of Visualized Experiments · May 2023

DOI: 10.3791/65170

CITATIONS READS

3 397

8 authors, including:

Luoma Ke Zhejun Zhang

Tsinghua University Beijing University of Posts and Telecommunications
5 PUBLICATIONS 12 CITATIONS 7 PUBLICATIONS 7 CITATIONS

SEE PROFILE SEE PROFILE

Jibo He
Nanjing Normal University
100 PUBLICATIONS 1,992 CITATIONS

SEE PROFILE

All content following this page was uploaded by Zhejun Zhang on 13 July 2023.

The user has requested enhancement of the downloaded file.

 Evaluating Flight Performance and Eye Movement
Patterns Using Virtual Reality Flight Simulator
Luoma Ke1, Zhejun Zhang2, Yidan Ma1, Yi Xiao3, Shengjun Wu4, Xiuchao Wang4, Xufeng Liu4, Jibo He1
1 2 3
Psychology Department, School of Social Sciences, Tsinghua University Faculty of Psychology, Beijing Normal University National Key Laboratory of
4
Human Factors Engineering, the China Astronaut Research and Training Center Department of Military Medical Psychology, Air Force Medical University

Corresponding Author
Abstract
Jibo He
[email protected]
Efficient and economical performance evaluation of pilots has become critical to the
aviation industry. With the development of virtual reality (VR) and the combination
Citation
of eye-tracking technology, solutions to meet these needs are becoming a reality.
Ke, L., Zhang, Z., Ma, Y., Xiao, Y., Previous studies have explored VR-based flight simulators, focusing mainly on
Wu, S., Wang, X., Liu, X.,
He, J. Evaluating Flight Performance technology validation and flight training. The current study developed a new VR flight
and Eye Movement Patterns Using
simulator to evaluate pilots' flight performance based on eye movement and flight
Virtual Reality Flight Simulator. J. Vis.
Exp. (195), e65170, doi:10.3791/65170 indicators in a 3D immersive scene. During the experiment, 46 participants were
(2023).
recruited: 23 professional pilots and 23 college students without flight experience.
The experiment results showed significant differences in flight performance between
Date Published
participants with and without flight experience, the former being higher than the
May 19, 2023
latter. In contrast, those with flight experience showed more structured and efficient

DOI eye-movement patterns. These results of the differentiation of flight performance

10.3791/65170 demonstrate the validity of the current VR flight simulator as a flight performance
assessment method. The different eye-movement patterns with flight experience
URL
provide the basis for future flight selection. However, this VR-based flight simulator
jove.com/video/65170 has shortcomings like motion feedback compared to traditional flight simulators. This
flight simulator platform is highly flexible except for the apparent low cost. It can meet
the diverse needs of researchers (e.g., measuring situation awareness, VR sickness,
and workload by adding relevant scales).

Introduction

The European Aviation Safety Agency (2012) categorizes is available for training, from low-level tabletop systems
flight simulators as training facilities, flight and navigation to highly complicated motion-based full flight simulators2 .
program trainers, flight training equipment, and complete The traditional simulator includes a flight dynamics model,
flight simulators1 . To date, a range of flight simulators

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 1 of 20
 a system simulation, a hardware cockpit, an external candidate's capabilities as an unmanned aerial vehicle (UAV)
visualization, and an optional motion simulation3 . pilot based on VR technology. Given that pilot selection is
critical for the aviation industry, it is pressing to develop a new
These traditional flight simulators have some advantages as
VR-based flight simulator focusing on pilot selection, as large-
effective flight training equipment. However, their cost is high
scale pilot selection is susceptible to the cost of the simulator
and environmentally unfriendly, as the drive of each system
and demands in the portability simulator system.
requires substantial electrical energy, especially a full flight
simulator, which requires high temperature and high-pressure Eye movements provide cues for a pilot's performance.
fluid or air pressure, consumes much power and generates a Different studies have found that the eye-scanning mode
lot of noise4 . distinguishes the performance between expert and novice
pilots. By comparing the scanning pattern between experts
However, a simple desktop simulator system is flexible and
and novices, experts' efficient and structural eye movement
low-cost, with lower immersion and fewer interactions than
behavior and the inadequate scanning methods of beginners
a full flight simulator2 . Therefore, it is essential to develop
could be differentiated. Several aviation studies have found
new flight simulators that combine the advantages of desktop
that pilots' eye-scanning strategy highly relates to the
systems and full flight simulators (in other words, the flexibility
level of expertise15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 . According to
of a tabletop simulation and the immersion and interaction
Bellenkes et al.25 , the duration of experts' fixations is shorter,
level close to a full flight simulator).
and the frequency of their fixations on instruments is higher

With the development of computer technology, especially than that of novices. Almost the same conclusion was drawn

virtual reality (VR) technology, a new type of flight simulator by Kasarskis et al.26 , who discovered that expert pilots have

based on emerging VR technology is becoming a reality. more fixations combined with shorter durations than novices,

The VR-based flight simulator is flexible, portable, low- suggested that expert pilots have a better visual mode than

cost, and has fewer space requirements than conventional novices. In another study, Lorenz et al.27 found that experts

flight simulators5 . Researchers have created flight simulators spend more time looking outside the cockpit than novices.

based on VR technology over the past 20 years6 , 7 , 8 , 9 , 10 , 11 ; These results have great practical value in the selection of

however, these VR flight simulators are mainly for flight newcomers.

training, and there are few for pilot selection. Still, with
Flight performance assessment is another critical factor for
cost reduction and technology enhancement, VR-based
pilot selection. However, the following problems exist in pilot
simulators are changing and becoming feasible for personal
flight performance evaluation: conflicting expert opinions,
selection. Some studies have used VR-based simulators for
more selection norms, and a unified selection theory. In the
personal selection in different domains: Schijven et al.12
driving field, Horrey et al.28 compared the absolute value of
selected surgical trainees using a virtual reality simulator.
lane departure from the centerline for different experimental
Huang et al.13 developed a psychology selection instrument
conditions to assess driving performance. Back to the aviation
based on virtual reality technology for air force pilot
domain, the flight quick access recorder (QAR) records all
recruitment. Wojciechowski and Wojtowicz14 assessed a

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 2 of 20
 sorts of pilot manipulation parameters, aircraft parameters, Finally, the present work can lead to an omnibus
environments, and warning information during flight29 . More measurement for pilot selection in the future, from eye
specifically, as the QAR indicators, the pitch angle is the scanning path to objective flying performance data. With the
rotation angle around the left and right axes of the aircraft30 , help of the virtual flight simulator, the cost of flight selection
and the reference line (or the center reference line) is right will be significantly reduced, and the norm of pilots can be
in the middle of the red and green lines28 ; these two flight formed based on extensive data gathering. The work fills a
parameters are used to evaluate the flight performance of gap between conventional and desktop simulators for flight
participants with or without experience in the current study. selection needs.
These QAR data can be used to evaluate flight performance,
yet to our best knowledge, they have seldom been used for
Protocol

personal training and selection in scientific research31 , 32 . All methods described here have been approved by the
Institutional Review Board (IRB) of Tsinghua University, and
Measurements of eye movement patterns can be used
informed consent was obtained from all participants. After
to assess and predict flight performance and guide pilot
completion, all participants were paid $12 (or a gift of equal
training and selection. Gerathewohl33 stated that the eye is
value).
the most important sensory organ of the pilot, processing
80% of the flight information. Pilots must acquire visual
1. Participant selection
information from instruments in the cockpit and integrate
it into a coherent image to manage the flight22 . Further, 1. Recruit participants according to a prior study of power
optimal scanning behavior is essential to accomplish better
analysis using G*Power software35 (see Table of
flight performance15 . However, no affordable flight simulator
Materials) to ensure the participant number meets the
currently integrates an eye tracker to facilitate quantitative
expected sample size given by G*Power, which equals
studies of the relationship between eye movements and flight
21.
performance.
NOTE: The expected sample size given by G*Power

The current study developed a new VR flight simulator to refers to the estimated number of participants needed

assess if participants with flight experience had better flight in a study to achieve a desired level of statistical power

performance than those without flight experience. The VR based on the specified effect size, significance level,

flight simulator integrates eye tracking and a flight dynamics and statistical test used. The expected sample size

system allowing eye-movement pattern analysis and flight is an important consideration in research planning. It

performance evaluation. In particular, it is worth mentioning helps researchers determine the feasibility and cost-

that the VR flight simulator uses a VR eye tracker34 , not effectiveness of their study design and ensures that the

a glass-like or desktop eye tracker, to analyze the area of study has sufficient statistical power to detect meaningful

interest (AOI)-based eye movement without time-consuming effects.

frame counting.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 3 of 20
 2. Ensure none participants have a history of epilepsy, 1. Check the components of the VR, HMD (head-
heart or brain disease, recent endocrine or psychiatric mounted display), and eye-tracking module with the
medication, or severe skin allergies. help of Table 2.

3. Use the Snellen eye chart (use the metric system 2. Ensure that all the flight simulator PC

6/6)36 to confirm that participants' vision is normal or module components meet the following minimum

corrected-to-normal and that they don't have any visual requirements: 3.6G Hz processor, 4G internal

impairments like color blindness or color weakness. memory, 64-bit operating system, and graphics
card. The system will support all flight controllers.
4. Ensure that none of the participants have consumed
alcohol or drugs within the previous 24 h that may affect 3. Check the components of the flight control module

their ability to fly. and ensure the device's parameter configuration is

consistent with Table 3.
5. Ensure the participants have had no less than 6 h of sleep
and are in good mental condition before the experiment. 2. Install the hardware of the flight simulator according to
the layout in Figure 1. Figure 2 shows how the hardware
2. Flight simulator hardware
is connected.

1. Check that all of the flight simulator's hardware is 1. Join the throttle and control panel physically and

complete. According to its function, this hardware is treat them as a unit.

organized into three modules (Table 1) (see Table of 2. Connect the throttle, joystick, and pedal to the flight
Materials). simulator PC module via USB.
NOTE: Researchers need to touch a metal rod before
3. Connect the HMD to the flight simulator PC module
touching the equipment to avoid the risk posed by
via the link box.
electrostatic induction.
4. Connect the base stations and VR controllers to the
HMD via the VR software on the PC.

VR head-mounted display(HMD) 1. Base Station

and eye tracking module 2. VR HMD

Flight Simulator PC module 3. Flight Simulator PC

Flight control module 4. Flight Throttle

5. Flight Joystick

6. Flight Pedal

Table 1: Components of the three modules of flight simulator hardware.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 4 of 20
 Main component Accessories

VR HMD Headset cable (attached)

Face cushion (attached)

Cleaning cloth

Earphone hole cap × 2

Link box Power adapter

DisplayPort cable

USB 3.0 cable

Mounting pad

Controllers (2018) × 2 Power adapters × 2

Lanyards × 2

Micro-USB cables × 2

Base Station 2.0 × 2 Power adapters × 2

Mounting Kit (2 mounts, 4 screws, and 4 wall anchors)

Table 2: List of components of the VR HMD and eye-tracking module.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 5 of 20
 Device Parameter configuration

Flight Joystick Nineteen action buttons

One 8-way "point of view" hat

Several 3D magnetic sensors

One 5-coil spring system

One 16-bit resolution (65536 x 65536 values).

Flight Control Panel Fifteen action buttons

One TRIM wheel

Five programmable LEDs

Flight Throttle Seventeen action buttons

One mouse hat with a push button

One 8-way "point of view" hat

Several 3D magnetic sensors

Two 14-bit resolution

Flight Pedal Tension between 2.5 kg and 5 kg

Angle between 35° and 75°

Table 3: The parameter configuration of the devices of the flight control module.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 6 of 20
 Figure 1: The layout of the VR flight simulator hardware. Please click here to view a larger version of this figure.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 7 of 20
 Figure 2: The connection of flight simulator hardware. (A) Flight control module. The throttle and control panel are
physically joined and treated as a unit. If the term "throttle" is used in this study, it refers to both the throttle and the control
panel. (B) Flight simulator PC module. A computer that meets the requirements outlined in step 2.2. (C) HMD and eye-
tracking module. The software development kits (SDKs) for eye tracking and the 3D engine are kept in sync when installed
on the same computer. Therefore, the eye-tracking functions and the operating system interact and work together. Please
click here to view a larger version of this figure.

about all software used in the experiment is shown in

3. Flight simulator software
Table 4.

1. Ensure all the software (see Table of Materials) has

been installed before the experiment begins. Information

Name Description

VR software A widely used tool for experiencing VR content on the hardware.

VR app store The app store for virtual reality where customers can explore,
create, connect, and experience the content they love and need.

Eye-tracking software Eye-tracking software developed by the research team via Eye-tracking and 3D engine SDKs.

FlySimulator The main program of the flight simulator software, developed by the research team.

Screen recording A free and open source software for video recording and live streaming.
software

Table 4: Information about all software used in the experiment.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 8 of 20
 The eye-tracking program will activate automatically after the
4. Preparation before launching the flight
initial run.
simulator

NOTE: If this is one's first time running the eye-tracking

program, perform the additional steps according to Figure 3.

Figure 3: The additional steps when running the eye-tracking program for the first time. Please click here to view a
larger version of this figure.

1. Ensure that the HMD is turned on and connected to the 1. Confirm that the participants are not using contact
computer. lenses, which causes eye tracking to malfunction.

2. Place the base station diagonally to ensure the HMD 2. Open the eye-tracking calibration program using the
can always be inside the base station's monitoring VR controllers (see Table of Materials).
range. Keep the base station fixed to keep a stable VR 3. Adjust the device height, interpupillary distance
environment. (IDP), and gaze point as the system directs.
3. Set up a standing-only play area according to the 4. Get the participant's eyes to light up each spot for 2
prompts given by the VR software on the computer. s in a clockwise direction to verify the effectiveness
NOTE: The prompt allows a room-scale area or sets up a of the eye-tracking calibration.
standing-only play area. Standing-only mode is generally
5. Connect the tuned flight control module and large screen
used to experience VR scenes that do not require walking
display (i.e., at least a 27 inch monitor) to the same
and can be selected when the user has limited space
computer as the VR HMD. The screen allows the
to move around. Therefore, in this experiment, set up a
experimenter to view what is happening in the VR HMD
standing-only play area.
simultaneously.
4. Set the eye-tracking calibration. The eye-tracking system
must be recalibrated every time the participant is 5. Experimental procedure

switched.
NOTE: The experiment is divided into four steps: "collect
information," "introduce the task and operation," "practice

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 9 of 20
 before the experiment," and "conduct a formal experiment."
The experimental process is summarized in Figure 4.

Figure 4: The flowchart of the experiment. Please click here to view a larger version of this figure.

1. Collect information from participants (15 min). 3. Ask the participants to practice using the VR flight
simulator (15 min).
1. Fill out the payment details form with the payment
information of the participants. 1. Fit the VR HMD to the participants and calibrate
the eye movements. Adjust the device height,
2. Inform the participants to read the informed consent
interpupillary distance (IDP), and gaze point as
document and sign it.
instructed by the system.
2. Introduce the task and simulator operation to the
2. Open the screen recording software.
participants (5 min).
3. Launch the FlySimulator program to assist
1. Explain to the participants the traffic pattern and map
participants with flight training.
(Figure 5).
4. Instruct the participants to use the flight pedal,
2. Instruct participants on how to use the VR flight
joystick, and throttle in concert to steer the aircraft
simulator.
as closely as possible to the reference line of flight
3. Inform the participants that they can leave the
(Figure 5).
experiment if they experience discomfort while
simulating a flight.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 10 of 20
 5. Reset the flight throttle and engine switch buttons 2. Start the FlySimulator program, choose a fit
after exiting the FlySimulator program, and stop perspective, start the engine, let go of the parking
recording the screen after the subject has completed brake, and put the flight throttle to full.
the practice. 3. Exit the FlySimulator program, then reset the flight
4. Conduct formal flight experiments (20 min). throttle and engine switch buttons. Stop recording
the screen.
1. Put on the VR HMD for the participants, and run OBS
Studio to start recording the screen.

Figure 5: The traffic pattern for the VR flight simulator. The pitch angle is the rotation angle around the left and right axes
of the aircraft, and the reference line (or the center reference line) is right in the middle of the red and green lines. Please
click here to view a larger version of this figure.

2. Calculate the AOI difference test using the

6. Data analysis
percentage of dwell time in the AOI zone37 , given
the equipment feature of the VR oculomotor used in
1. Analyze eye movement data.
this study.
1. Divide five area of interest (AOI) zones in this study;
NOTE: The average percentage of dwell time is
for more information about the calculation of AOIs,
the cumulative percentage spent looking within an
see Figure 6, based on the actual panel of the flight
AOI divided by the total fixation time and averaged
cockpit corresponding to the flight cockpit instrument
across participants.
(Figure 7).

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 11 of 20
 spatial distance between the plane and the reference line
2. Analyze flight performance data with custom-developed
during flight, in meters).
programming scripts using Python 3.10. The core
algorithm for performance metrics was adapted from 3. Conduct the statistical analyses.
the QAR analysis method, inspired by the 35 BAE-146 1. Use the Shapiro-Wilk test39 to confirm the data's
aircraft QAR data provided by the National Aeronautics
normality.
and Space Administration (NASA)38 .
2. Use statistical software (see Table of Materials)
NOTE: Flight performance indicators for this study: the
for descriptive statistics, Mann-Whitney U test, and
total flight time (the total length of time from takeoff to
Student's t-test.
landing for each participant, in seconds), pitch angle
1 s before landing (pitch angle of the aircraft 1 s 3. Depict the violin plots to indicate better the shape of

before landing, obtained from raw data, in degrees), the data distribution40 .

mean distance to reference line (mean error of the NOTE: The explanation for the effect size measured

spatial distance between the aircraft and the reference using Cohen's d is as follows: 0.1 is very small, 0.2

line during flight, in meters), and standard deviation of is small, 0.5 is medium, 0.8 is large, 1.2 is very large,

distance to reference line (standard deviation of the and 2.0 is huge41 , 42 . The significance level was set
at p < 0.05.

Figure 6: AOI preprocessing and calculating flow process. Sections 1 to 4 describe how the present study processed the
pilots' eye movement data up to the independent-sample t-test. Please click here to view a larger version of this figure.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 12 of 20
 Figure 7: Schematic diagram of AOI division of flight instrument. The function of instruments: (A) The airspeed indicator
indicates the aircraft's speed relative to the air. (B) The altitude indicator shows the aircraft's pitch and roll altitude. (C) The
vertical speed indicator indicates the aircraft's ascent or descent speed. (D) The altitude indicator indicates the aircraft's
barometric altitude. (E) The engine speed indicator indicates the speed of the aircraft engine. Please click here to view a
larger version of this figure.

indicator (M = 8.56%, SD = 5.45 for experts; M = 5.32%, SD

Representative Results = 4.98 for novices; U = 145.5, p < 0.01), the vertical speed

For the current experiment, 23 experts with flight experience indicator (M = 13.11%, SD = 8.84 for experts; M = 5.33%, SD =

and 23 novices without flight experience were chosen. The 4.11 for novices; U = 127, p < 0.01), and the altitude indicator

participants were between 25 and 58 years of age (experts: M (M = 2.83%, SD = 2.29 for experts; M = 1.34%, SD = 2.22 for

= 32.52 years, SD = 7.28 years; novices: M = 29.57 years, SD novices; U = 159.5, p < 0.05) than novices. The difference in

= 5.74 years). The gender of all participants was male. All the the gaze time at the engine speed indicator was insignificant

novices were recruited from Tsinghua University (students or between experts (M = 0.31%, SD = 0.56) and novices (M =

faculty), and all the experts were from China Eastern Airlines. 0.47%, SD = 1.16, p > 0.05 ). An independent-sample t-test
was performed on the altitude indicator, and the result was
Eye movement
marginally significant [t(44) = 1.843, p = 0.072, Cohen's d =
The eye movement data of the instrument AOIs, airspeed
0.541]; novices (M = 31.03%, SD = 12.2) spent more time
indicator, vertical speed indicator, altitude indicator, and
looking at the altitude indicator than experts (M = 25.15%, SD
engine speed indicator are not normally distributed, according
= 9.23).
to the Shapiro-Wilk test (all p-values <0.05 ). Thus, the Mann-
Whitney U test was used. The results showed that experts As shown in Figure 8, compared with the novices, the experts'

(M = 60.79%, SD = 7.72) paid more time looking at the flying eye movement data were more evenly distributed around the

instruments than novices (M = 53.31%, SD = 12.89; U = 141, mean value in the airspeed indicator (Figure 8A), the altitude

p < 0.01). More precisely, experts looked more at the airspeed

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 13 of 20
 indicator (Figure 8B), the vertical speed indicator (Figure
8C), and the altitude indicator (Figure 8D).

Figure 8: Means and data distribution of percent dwell time. (A) Airspeed indicator. (B) Altitude indicator. (C) Vertical
speed indicator. (D) Altitude indicator. *p < 0.05, **p < 0.01, and ***p < 0.001. Please click here to view a larger version of
this figure.

Flight performance 130, p < 0.01), which indicated a more stable approach and
The data of the pitch angle 1 s before landing, the mean better landing posture control for experts. The mean distance
distance to the reference line, and the standard deviation of to the reference line for experts (M = 176.67 m, SD = 205.52)
the distance to the reference line are not normally distributed, was significantly smaller than that of novices (M = 873.89 m,
according to the Shapiro-Wilk test (all p-values <0.05 ). Thus, SD = 818.43; U = 439, p < 0.001), which represents a minor
the Mann-Whitney U test was used. The pitch angle 1 s before flying deviation. The standard deviation of the distance to the
landing for novices (M = -12.54°, SD = 29.03) was significantly reference line for experts (M = 211.52 m, SD = 225.76) was
smaller than that of experts (M = 3.97°, SD = 24.43; U = also significantly smaller than that of novices (M = 675.78 m,

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 14 of 20
 SD = 589.07; U = 420, p < 0.01), which represents a more As shown in Figure 9, compared with the novices, the experts'
stable and reliable flight performance. These results showed flight performance data was mainly concentrated around the
that experts had a better flying performance than novices. An mean value in the total flight time (Figure 9A), the mean
independent-sample t-test was performed on the total flight distance to the reference line (Figure 9B), the pitch angle 1 s
time, and the result showed a marginally significant difference before landing (Figure 9C), and the standard deviation of the
[t(44) = 1.835, p = 0.076, Cohen's d = 0.541]; experts (M = distance to the reference line (Figure 9D).
759.06 s, SD = 163.58) had a shorter total flight time than
novices (M = 902.32 s, SD = 336.73).

Figure 9: Means and data distribution. (A) The total flight time. (B) Mean distance to reference line. (C) Pitch angle 1 s
before landing. (D) The standard deviation of the distance to the reference line. *p < 0.05, **p < 0.01. Please click here to
view a larger version of this figure.

Supplementary File 1: The script file of airports' Discussion

configuration. Please click here to download this File.
The current study assessed if participants with flight
experience had better flight performance than those without
flight experience in a VR-based flight simulator. More

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 15 of 20
 importantly, it evaluated whether a more optimized eye Second, the propeller aircraft is biased to the right in the
movement pattern could be found in these participants process of flying, so the left flight pedal should be pressed
with better flight performance. The results have significant when starting, and the flight joystick should be kept on the
differences between participants with and without flight left during the flight to offset the right bias keep the aircraft
experience in three key flying QAR indicators: pitch angle 1 flying smoothly. Third, the flight throttle in the takeoff phase
s before landing, the mean distance to the reference line, should be pushed to the maximum, then held at two-thirds
and the standard deviation of the distance to the reference throttle in the level flight phase. Fourth, participants should be
line (Figure 9B-D). These results show that participants with reminded to control the aircraft by matching the flight pedal,
flight experience have less flying deviation and more stable joystick, and throttle to make it fly as close to the guideline
flight performance. At the same time, the results also have of the center reference line as possible. Fifth, whenever one
significant differences between participants with and without exits the FlySimulator program, the experimenter must reset
flight experience in three key instrument AOIs: airspeed the flight launch switch and throttle.
indicator, vertical speed indicator, and altitude indicator
Given researchers' diverse experimental needs, the protocol
(Figure 8A-C). Further, participants with flight experience
can be modified accordingly. First, the flight map can be
spend more time looking at critical flying instruments than
easily changed as a practical scenario by configuring the
those without flight experience.
script file (Supplementary File 1). Second, except for eye-
In summary, flight expertise has an important impact on flight movement data, electroencephalogram (EEG) data can be
performance; experts' flight performance data shows their collected by integrating an EEG with dry electrodes in the
flight track is more concentrated around the center reference HMD, as can heart rate data by a heart rate belt. Third,
line, which means they have better flight performance. In some subjective measures can be applied to the participants'
contrast, novices' flight track deviates significantly from the psychology before and after the experiment by adding some
center reference line. Furthermore, different flight expertise scales and questionnaires, such as the proactive personality
corresponds to different eye-scanning patterns; experts scale43 , the measurement of situation awareness (SA)44 , 45 ,
depend more on key flying instruments (i.e., airspeed the VR sickness questionnaire (VRSQ)46 , and the NASA task
indicator, vertical speed indicator, etc.) and have more gaze load index (TLX)47 , etc. In addition, when the flight simulation
time on the devices. In other words, experts show a more program starts computing the dynamics model data in real-
structural and efficient eye-scanning pattern compared with time, sometimes the aircraft body altitude is not in the correct
novices. These results are consistent with Wickens' research, place, and the experimenter needs to press the grey button
which shows that expert pilots have a better visual mode26 . in the upper right corner of the flight joystick, thus resetting
it. A situation can also occur where the aircraft loses control,
As mentioned earlier, the VR flight simulator can measure
and the experimenter needs to exit for a reset by pressing Alt
flight performance. Still, some critical steps need more
+ F4 on the keyboard.
attention. First, during the demonstration flight session, the
mission should be introduced along with the flight map.

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 16 of 20
 The method also has its limitations. A prominent issue is simulator platform to study hand-eye coordination, fatigue,
that participants get motion sickness when coping with a distraction, mind wandering37 , etc., by designing different
complex flight task in an immersive VR scenario, thus possibly experiments.
withdrawing from the experiment. In addition, compared
with a traditional flight simulator with motion bases4 , the
Disclosures

current study's VR flight simulator is not equipped with a The authors declared no financial disclosure or conflicts of
motion device. It thus cannot produce the actual sensation interest.
of movement. Finally, the flight performance records various
flight parameters, which can be used to evaluate the flight Acknowledgments

performance29 ; only three key indicators are selected in the The authors are incredibly grateful to Mr. Li Yan for his help in
current research. recruiting pilot participants and acknowledge Ms. Bu Lingyun
for her work on drawing pictures. The research was supported
In any case, the VR flight simulator has unique advantages
by the National Natural Science Foundation of China (grant
concerning existing methods. Above all, considering
number T2192931, 72071185), the National Brain Project
conventional flight simulation relies on bulky and expensive
(grant number STI2030-Major Projects2022ZD0208500),
instruments to create a virtual world9 , the current flight
the National Key Laboratory Project of Human Factors
simulator based on VR has a lower cost. Besides, VR
Engineering (grant number SYFD062003), the National Key
technology enables keeping a person in a fully immersive
Laboratory Project of Human Factors Engineering (grant
virtual environment1 , so the VR flight simulator provides more
number 6142222210201), and year 2022 Major Projects
realistic immersion than other methods. More importantly,
of Military Logistic Research Grant and Key Project of Air
this method builds a new platform based on VR technology
Force Equipment Comprehensive Research (grant number
for pilot selection in the future, integrating eye-tracking and
KJ2022A000415).
flight dynamics technology, and thus can provide more
comprehensive screening metrics compared with current
References
methods.
1. Oberhauser, M., Dreyer, D., Braunstingl, R., Koglbauer, I.
The VR flight simulator shows excellent potential as a pilot
What's real about virtual reality flight simulation? Aviation
selection platform. It is well known that current pilot selection
Psychology and Applied Human Factors. 8 (1), 22-34
is expensive and overly dependent on expert opinion. The
(2018).
current VR flight simulator can be used in pilot selection,
2. Oberhauser, M., Dreyer, D. A virtual reality flight
significantly reducing costs and providing more objective data
simulator for human factors engineering. Cognition,
for selection. In addition, the VR simulator can be used
Technology & Work. 19 (2-3), 263-277 (2017).
as a human factor engineering tool that enables ergonomic
assessment of the flight cockpit in the early stages of the 3. Rolfe, J. M., Staples, K. J. Flight Simulation. Cambridge

design process2 . Finally, researchers can use this virtual University Press. Cambridge, New York. (1986).

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 17 of 20
 4. Robinson, A., Mania, K., Perey, P. Flight simulation: 11. Joyce, R. D., Robinson, S. K. The rapidly reconfigurable
Research challenges and user assessments of fidelity. In research cockpit. In AIAA Modeling and Simulation
Proceedings of the 2004 ACM SIGGRAPH International Technologies Conference. Dallas, Texas. June 22-26
Conference on Virtual Reality Continuum and its (2015).
Applications in Industry. Singapore, June 16-18 261-268 12. Schijven, M. P., Jakimowicz, J. J., Carter, F. J.
(2004). How to select aspirant laparoscopic surgical trainees:
5. Moroney, W. F., Moreney, B. W. Flight Simulation. Establishing concurrent validity comparing Xitact
In Handbook of Aviation Human Factors. Lawrence LS500 index performance scores with standardized
Erlbaum Associates. Mahwah, New Jersey. 261-268 psychomotor aptitude test battery scores. The Journal of
(1999). Surgical Research. 121 (1), 112-119 (2004).

6. McCarty, W. D., Sheasby, S., Amburn, P., Stytz, M. R. 13. Huang, P., Zhu, X., Liu, X., Xiao, W., Wu, S. Psychology
Switzer, C. A virtual cockpit for a distributed interactive selecting device for air force pilot recruitment based
simulation. IEEE Computer Graphics and Applications. on virtual reality technology, has industrial personal
14 (1), 49-54 (1994). computer connected with memory, where industrial
control computer is connected with image display device.
7. Dorr, K. U., Schiefel, J., Kubbat, I. Virtual cockpit
CN111920428-A. (2020).
simulation for pilot training. In What is Essential
for Virtual Reality Systems to Meet Military Human 14. Wojciechowski, P., Wojtowicz, K. Simulator sickness
Performance Goals?. RTO human factors and medicine and cybersickness as significant indicators in a primary
panel (HEM) workshop. The Hague, The Netherlands, selection of candidates for FPV drone piloting. In 2022
April 13-15 (2001). IEEE 9th International Workshop on Metrology for
AeroSpace (MetroAeroSpace). Pisa, Italy. June 27-29
8. Bauer, M., Klingauf, U. Virtual-reality as a future training
(2022).
medium for civilian flight procedure training. In AIAA
Modeling and Simulation Technologies Conference and 15. Ziv, G. Gaze behavior and visual attention: A review of
Exhibit. Honoluu, Hawaii. August 18-21 (2008). eye tracking studies in aviation. The International Journal
of Aviation Psychology. 26 (3-4), 75-104 (2016).
9. Yavrucuk, I., Kubali, E., Tarimci, O. A low cost flight
simulator using virtual reality tools. IEEE Aerospace and 16. Lai, M. L. et al. A review of using eye-tracking technology
Electronic Systems Magazine. 26 (4),10-14. (2011). in exploring learning from 2000 to 2012. Educational
Research Review. 10, 90-115 (2013).
10. Aslandere, T., Dreyer, D., Pankratz, F., Schubotz, R.
A generic virtual reality flight simulator.In Virtuelle und 17. Robinski, M., Stein, M. Tracking visual scanning
Erweiterte Realität, 11. Workshop der GI-Fachgruppe techniques in training simulation for helicopter landing.
VR/AR. Shaker Verlag, Aachen. 1-13 (2014). Journal of Eye Movement Research. 6 (2) 1-17 (2013).

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 18 of 20
 18. Yang, J. H., Kennedy, Q., Sullivan, J., Fricker Jr., R. 26. Kasarskis, P., Stehwien, J., Hickox, J., Aretz, A.,
D. Pilot performance: Assessing how scan patterns Wickens, C. Comparison of expert and novice scan
& navigational assessments vary by flight expertise. behaviors during VFR flight. In Proceedings of the
Aviation Space and Environmental Medecine. 84 (2), 11th International Symposium on Aviation Psychology.
116-124 (2013). Columbus, Ohio. January 1-6 (2001).

19. Yu, C. S., Wang, E. M. Y., Li, W. C., Braithwaite, G., 27. Lorenz, B. et al. Performance, situation awareness,
Greaves, M. Pilots' visual scan patterns and attention and visual scanning of pilots receiving onboard taxi
distribution during the pursuit of a dynamic target. navigation support during simulated airport surface
Aerospace Medicine and Human Performance. 87 (1), operation. Human Factors and Aerospace Safety. 6 (2),
40-47 (2016). 135-154 (2006).

20. Haslbeck, A., Zhang, B. I spy with my little eye: Analysis 28. Horrey, W. J., Alexander, A. L., Wickens, C. D.
of airline pilots' gaze patterns in a manual instrument Does workload modulate the effects of in-vehicle
flight scenario. Applied Ergonomics. 63, 62-71 (2017). display location on concurrent driving and side task
performance. In Driving Simulator Conference North
21. Brams, S. et al. Does effective gaze behavior lead
America Proceedings. Dearborn, Michigan. October 8-10
to enhanced performance in a complex error-detection
(2013).
cockpit task? PLoS One. 13 (11), e0207439 (2018).
29. Wang, L., Ren, Y., Sun, H., Dong, C. A landing
22. Peißl, S., Wickens, C. D., Baruah, R. Eye-tracking
operation performance evaluation method and device
measures in aviation: A selective literature review. The
based on flight data. In Engineering Psychology and
International Journal of Aerospace Psychology. 28 (3-4),
Cognitive Ergonomics: Cognition and Design. Springer
98-112 (2018).
International Publishing. Vancouver, British Columbia,
23. Jin, H. et al. Study on how expert and novice pilots
Canada. 297-305 (2017).
can distribute their visual attention to improve flight
30. Wang, L., Ren, Y., Wu, C. Effects of flare operation on
performance. IEEE Access. 9, 44757-44769 (2021).
landing safety: A study based on ANOVA of real flight
24. Lounis, C., Peysakhovich, V., Causse, M. Visual
data. Safety Science. 102, 14-25 (2018).
scanning strategies in the cockpit are modulated by
31. Huang, R., Sun, H., Wu, C., Wang, C., Lu, B. Estimating
pilots' expertise: A flight simulator study. PLoS One. 16
eddy dissipation rate with QAR flight big data. Applied
(2), e0247061 (2021).
Sciences. 9 (23), 5192 (2019).
25. Bellenkes, A. H., Wickens, C. D., Kramer, A. F. Visual
32. Wang, L., Zhang, J., Dong, C., Sun, H., Ren, Y. A method
scanning and pilot expertise: The role of attentional
of applying flight data to evaluate landing operation
flexibility and mental model development. Aviation Space
performance. Ergonomics. 62 (2), 171-180 (2019).
and Environmental Medicine. 68 (7), 569-579 (1997).

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 19 of 20
 33. Gerathewohl, S. J. Leitfaden der Militärischen 43. Bateman, T. S., Crant, J. M. The proactive component
Flugpsychologie. Verlag für Wehrwissenschaften. of organizational behavior: A measure and correlates.
(1987). Journal of Organizational Behavior. 14 (2), 103-118
(1993).
34. Ugwitz, P., Kvarda, O., Juříková, Z., Šašinka, Č., Tamm,
S. Eye-tracking in interactive virtual environments: 44. Endsley, M. R. Measurement of situation awareness in
implementation and evaluation. Applied Sciences. 12 (3), dynamic systems. Human Factors. 37 (1), 65-84 (1995).
1027 (2022). 45. Hunter, D. R. Measuring general aviation pilot judgment
35. Faul, F., Erdfelder, E., Lang, A.-G., Buchner, A. G*Power using a situational judgment technique. The International
3: A flexible statistical power analysis program for the Journal of Aviation Psychology. 13 (4), 373-386 (2003).
social, behavioral, and biomedical sciences. Behavior 46. Kim, H. K., Park, J., Choi, Y., Choe, M. Virtual
Research Methods. 39 (2), 175-191 (2007). reality sickness questionnaire (VRSQ): Motion sickness
36. Boslaugh, S. E. Snellen Chart. (2018). measurement index in a virtual reality environment.
Applied Ergonomics. 69, 66-73 (2018).
37. He, J., Becic, E., Lee, Y.-C., McCarley, J. S. Mind
wandering behind the wheel. Human Factors: The 47. Hart, S. G. NASA Task Load Index (TLX). (1986).
Journal of the Human Factors and Ergonomics Society.
53 (1), 13-21 (2011).

38. Tanveer Alam (2022). GitHub - tanvcodes/qar_analytics:

Scripts for working with publicly available Quick Access
Recorder (QAR) data from a fleet of 35 BAE-146 aircraft.
Retrieved on December 20,2022 from https://github.com/
tanvcodes/qar_analytics. (2022).

39. Shapiro, S. S., Wilk, M. B. An analysis of variance test

for normality (complete samples). Biometrika. 52 (3-4),
591-611 (1965).

40. Hintze, J. L., Nelson, R. D. Violin plots: A box plot-

density trace synergism. The American Statistician. 52
(2), 181-184 (1998).

41. Cohen J. Statistical Power Analysis for the Behavioral

Sciences. (2nd ed.), Erlbaum, Hillsdale, NJ (1988).

42. Sawilowsky, S. S. New effect size rules of thumb. Journal

of Modern Applied Statistical Methods. 8 (2), 26 (2009).

Copyright © 2023 JoVE Journal of Visualized Experiments jove.com May 2023 • 195 • e65170 • Page 20 of 20

View publication stats