Received 26 March 2024, accepted 10 April 2024, date of publication 12 April 2024, date of current version 16 May 2024.

Digital Object Identifier 10.1109/ACCESS.2024.3388194

Architecture and Algorithm Design for Civil

Aviation Data Real-Time Analysis System
YIFENG ZHANG , QI XI , JING WANG, AND SHUHUAI GU
School of Data Science and Engineering, South China Normal University, Shanwei 516600, China
Corresponding author: Qi Xi ([email protected])
This work was supported in part by Shanwei City 2023 Provincial Science and Technology Innovation Strategy Special Project under Grant
2023A011, and in part by the South China Normal University Young Teacher Fund under Grant 22KJ24.

ABSTRACT Currently, the analysis of civil aviation flight data relies primarily on post-flight data. This
involves transmitting onboard data to the servers of the airline and aircraft manufacturer after the aircraft has
landed, and the engines have been shut down, using ground-based cellular stations or wireless hotspots. Real-
time data analysis during flight remains a challenge. With the advent of integrated communication networks
in the aerospace sector, the real-time transmission of onboard data is poised to become a future trend. But
there is a gap in real-time civil aviation data analysis system. This study is based on Kappa distributed
computing architecture, established a real-time aviation data analysis system, which inputs Quick Access
Recorder (QAR) data stream and achieves fast data decoding and multiple real-time analysis algorithms,
including flight trajectory outlier repair, flight phase identification, steady cruising state determination, and
engine lubricating oil quantity monitoring. The results of simulation experiments indicate that the maxi-
mum average latency for individual algorithms is consistently below 250 milliseconds, meeting real-time
performance requirements. And all real-time algorithms have similar performances with or better than
existing excellent post-flight algorithms. Specifically, the flight trajectory outlier repair algorithm can repair
all outliers and cost 0.2ms to process an outlier. The flight phase identification achieved 97.8% accuracy
referring to commercial software’s results. The steady cruising state determination exhibits computational
efficiency approximately five times faster than the naive method. The proposed engine lubricating oil
quantity monitoring method based on the Clustering Optimized Transformer (CO-Transformer) neural
network yields an average mean square error of approximately 0.057 L2 between predicted and actual
values, respectively 7.36%, 66.01% and 66.05% better than common Transformer, LSTM and GRU neural
network, satisfying the airborne leakage monitoring requirements for engine lubrication systems. The system
also exhibits strong load capacity and excellent scalability. Under the employed hardware conditions, the
system can concurrently process several practical real-time QAR data streams for up to 505 civilian planes
in simulated experiments.

INDEX TERMS Distributed data processing, machine learning, aviation data analysis.

I. INTRODUCTION be transmitted, enabling low-level monitoring and processing

Currently, in the field of aviation, the processing of onboard of simple transactions. However, complex algorithms such
sensor data is categorized into two approaches: real-time pro- as machine learning [1] and other data mining methods [2]
cessing using onboard computer equipment and post-flight encounter challenges when deployed in real-time. Post-flight
batch processing. Real-time processing onboard is con- processing, achievable through ground equipment, enables
strained by the performance of onboard devices and through- intricate computations but introduces significant delays. With
put of network. Only a limited amount of essential data can advancement of Air-to-Ground (ATG) network [3], massive
amounts of onboard data can probably be transmitted to the
The associate editor coordinating the review of this manuscript and ground, which holds the potential to implement complex
approving it for publication was Massimo Cafaro . algorithms in real time, enabling to monitor and analyze
2024 The Authors. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
66382 For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/ VOLUME 12, 2024
Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

the operational status and high-level information of aircraft’s whether aircraft is steady in its cruising phase by the informa-
critical components, and enhance the safety and reliability tion of its attitude and environment, there is currently limited
of civil aviation. The real-time transmission of recorded data research on algorithms for assessing the cruising steady state
from onboard sensors is poised to become a new trend. of civil aviation aircraft. Yang [17], in a study on the regional
However, research on real-time analysis systems for avia- aircraft Xinzhou 600, achieved the determination of engine
tion onboard sensor data is still in its nascent stage. Current cruising steady state. Cao et al. [18] proposed a method by
methods are confined to standalone software, providing only calculating a list of judgment conditions to judge steady state
fundamental decoding capabilities, namely, converting raw values in her work. And in Wang’s study [19], his method
binary data records in aviation-specific formats into sensor- of calculating judgement conditions was still common naive
recorded engineering values [4], [5]. These existing systems method based on sliding window. This method is not real-
did not consider streaming architecture and big data pro- time optimized, showing higher time cost in the simulating
cessing, also lack further in-depth analysis or cutting-edge experiment. Presently, the determination of steady cruising
machine learning methods. Thus, they may fall short in state in post-flight analysis employs a naive computational
real-time civil aviation data processing and in-depth anal- method for determining conditions, incurring high compu-
ysis. This study proposes a design of a real-time data tational costs. Therefore, there is a need to design efficient
analysis software system for civil aviation based on the discriminative algorithms for real-time analysis.
Kappa [6] streaming processing architecture, which is one In addressing the issues, this paper proposes three efficient
of the main contributions in this paper, addressing the gap algorithms suitable for real-time aviation data analysis, laying
in real-time civil aviation data analysis system. The system the foundation for subsequent real-time monitoring of critical
is oriented towards real-time Quick Access Recorder (QAR) aircraft components’ performance.
data streams, enabling fast data decoding. Multiple real-time 1. Flight trajectory outlier repair, proposed in this paper,
analysis algorithms have been developed in the system, which concerns the reason why outliers emerge based on the
are also the main contributions and case studies about deploy- characteristic of trajectory data recording format. And
ment of in-depth data analysis on real-time system, including according to the practical aircraft trajectory’s changing
flight trajectory outlier repair, flight phase identification, rate, it utilizes trajectory changing rate to dynamically
steady cruising state determination, and engine lubricating oil detect anomalous trajectory points in real-time and
quantity monitoring. subsequently restores the trajectory values to their orig-
In aviation data analysis, flight trajectory outlier repair, inal true values. The algorithm can promptly identify
flight phase identification, and steady cruising state deter- and accurately rectify trajectory anomalies, thereby
mination are foundational for in-depth analysis such as enhancing system’s precision in tracking aircraft flight
monitoring the performance of critical aircraft components. trajectories.
Existing real-time onboard data processing software sys- 2. Flight phase state identification, a perceptual real-time
tems [4], [5] do not incorporate functionalities such as algorithm, is proposed, which inputs multiple real-time
flight trajectory outlier repair, flight phase identification and parameter data, dynamically assesses the aircraft’s
steady cruising state determination. Methods employed in environmental conditions based on several identifying
post-flight batch processing cannot be directly applied to criteria and calculates the real-time flight phase results.
real-time analysis. For instance, concerning the detection Experimental results demonstrate that, in comparison
of outliers in flight trajectory repair, commonly employed to post-flight identifying method results, the real-time
methods include the Local Outlier Factor (LOF) algorithm one achieves a consistency rate of 97.8%, close to
proposed by Breunig et al. [7], and the Isolation Forest above newly post-flight research’s results. Based on
algorithm utilized by Cheng et. al [8]. Sameer et. al [9] has real-time flight phase values, reliable support can be
applied LOF in anomaly detection in flight data. Regard- offered for further analyses [20].
ing flight phase identification, Hou et al. proposed a flight 3. Steady cruising determination, which needs to deter-
phase division approach based on decision tree model [10]. mine whether aircraft is in steady cruising state.
Zhang et al. [11] employed the Time Interval-based Clus- The algorithm utilizes monotonic queue to compute
tering [12] (TICC) algorithm and Density-Based Spatial the interval extremum of real-time data streams and
Clustering of Applications with Noise [13] (DBSCAN). employs a queue to calculate the interval mean and
Sun et al. [14] and Wang et al. [15] both utilized the DBSCAN sample variance. This algorithm improves the perfor-
algorithm, while Kuzmenko et al. [16] applied Maximum mance of calculating intermediate values and reduces
Posteriori Probability model to construct a model for flight the overall computation latency of the module. In com-
phase identification based on post-flight data. These models parison to the naive method deployed in many recent
both show high accuracy in their works, but all require a studies, our proposed method’s performance is approx-
batch of data for pre-processing or model learning, making it imately five times faster, achieving rapid determination
challenging to deploy them in real-time data analysis systems of steady state during the cruising phase.
for civil aviation. Regarding the determination of steady state Building upon these algorithms, this paper also presents a
during the cruising phase of aircraft, which is to identify machine learning case study, focusing on the aircraft engine

VOLUME 12, 2024 66383

 Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

lubricating oil system, which is a crucial subsystem of air-

craft engines, and its health status significantly influences
the engine’s critical functions. In existing new works [21],
[22], LSTM [23] and GRU [24] neural network models are
commonly used to predict the remaining lubricating oil quan-
tity of a health lubricating system, considering errors occur
FIGURE 1. An example of a WORD with 2 parameter data.
when predicted oil quantity have significant differences with
real values. But there is few research in real time and these
TABLE 1. Example of a one-frame record, where each subframe has wps
2 models did not perform well in our simulating experiment. (word per second) WORD records.
We introduced the Clustering Optimized Transformer (CO-
Transformer) model, which also takes sensor parameters as
input including high-pressure rotor speed of the engine as
input but to predict real-time remaining lubricating oil quan-
tity of the health lubricating system. The CO-Transformer
model integrates multiple Transformer Encoder [25] as sub-
models, utilizing Bootstrap [26] sampling to partition the
training set into sub-training sets. These sub-training sets are
employed to train diverse Transformer Encoder sub-models,
enhancing the overall model diversity. Subsequently, the
DBSCAN clustering algorithm is applied to selectively iden-
tify high-performing sub-models. The retained sub-models’
predicted average values are then chosen as the final out- determination are proposed. The fifth section introduces the
put, thereby improving the overall performance. The test CO-Transformer model utilized for real-time monitoring of
results show that compared to conventional Transformer, aviation engine lubricating oil quantity. The sixth section
LSTM [23], and GRU [24] models, the CO-Transformer covers experimental simulations and result analysis. Finally,
model exhibits improvements of 7.36%, 66.01%, and 66.05% Conclusions are provided in the seventh section.
in mean square error, and 3.49%, 33.25%, and 33.53% in
mean absolute error, respectively. And compared with the II. BACKGROUND INFORMATION
above post-flight work, errors can be checked in real time in This section introduces background information about
our method, which provides probability of real-time lubricat- onboard sensor data logging format, Apache Kafka and
ing system aiding work in the future, dramatically improving Apache Flink big data components, Transformer neural net-
aircraft’s safety. Furthermore, the proposed CO-Transformer work and DBSCAN clustering algorithm.
model is suitable for multi-GPU parallel training and facili-
tates deployment on distributed systems. A. ONBOARD SENSOR DATA LOGGING FORMAT
In this study, the real-time aviation data analysis and Commercial aircraft onboard sensor data is typically recorded
processing software system based on the Kappa streaming using Quick Access Recorders (QAR), specifically adher-
processing architecture was designed and constructed using ing to ARINC 717 and other relevant specifications [28].
the Flink framework [27]. Stream processing operators were In ARINC 717, the fundamental sampling unit is referred to
employed to implement the proposed analysis algorithms. as a subframe. A subframe corresponds to the recorded data of
The effectiveness of each algorithm was validated through various sensors within one second, and four subframes con-
simulated concurrent real-time data transmission across mul- stitute a frame. Each subframe’s recording is further divided
tiple flight routes. The test results indicate that the average into multiple WORDs, and depending on the aircraft configu-
latency for the entire analysis process remains consistently ration, the number of WORDs in a subframe can be 256, 512,
below 250 milliseconds, meeting the real-time requirements or 1024. Each WORD has a length of 12 bits, recording the
for aviation data analysis in actual scenarios. In a hardware sampled values of one or more sensors in binary format.
environment consisting of three dual-CPU 20-core servers, Table 1 illustrates an example of a data frame. Each
the system can handle a maximum real-time onboard sensor recorded WORD consists of 12 bits, and a single WORD
data stream of 79 Mbps (corresponding to 505 civilian planes can store multiple parameter data, as shown in Figure 1 for
sending data concurrently), demonstrating the efficiency of a WORD record instance with two parameter data. Decod-
the proposed software system and various algorithm modules. ing operators utilize the WORD index to retrieve decoding
The second section of this paper introduces some back- configuration information from the parameter table. This
ground information. The third section details the real-time information is used to determine the least significant bit
aviation data analysis and processing software system (LSB) and most significant bit (MSB) where the sensor
based on the Kappa streaming processing architecture. parameters are stored in the WORD. The binary data between
In the fourth section, algorithms for flight trajectory outlier LSB and MSB is extracted, converted from raw binary to
repair, flight phase identification and steady cruising state decimal, multiplied by the conversion factor, and adjusted for

66384 VOLUME 12, 2024

Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

the sign, ultimately restoring the engineering data value of the

sensor record [28].

B. APACHE KAFKA AND APACHE FLINK BIG DATA

COMPONENTS
Apache Kafka [29] is a distributed message streaming buffer
queue known for its high throughput and low latency, pri-
marily designed for buffering real-time data streams. In this
study, QAR data packets received by the Kafka buffer receiver
are utilized. Kafka clients are categorized as producers and
consumers. Producers send data to Kafka topics in the form
of messages, where each topic may encompass multiple mes-
sage queues, facilitating parallel data reception. Consumers
subscribed to relevant topics can consume messages from the
queues, extracting the data within the messages. Messages
primarily consist of a message header, a key, and a value.
Kafka provides data persistence and ensures consistency for
the messages. FIGURE 2. Overall architecture of civil aviation real-time data processing
and analysis system.
Apache Flink [27] is a high-performance distributed stream
data processing system that natively supports the processing
of time-series data and handles event data at millisecond
series and finds wide applications in computer vision [35],
level. Flink demonstrates outstanding performance in stream-
natural language processing [36], and various other domains
ing event processing, fault tolerance, snapshot creation, and
of time-series data processing [37], [38], [39]. In comparison
state persistence [30]. In this study, algorithm modules are
to recurrent neural networks, this feature allows Transformers
deployed using operators from the Flink DataStream APIs,
to better capture long-range dependencies and information
and Flink’s built-in state variables are employed to store inter-
over extended periods. Moreover, the attention mechanism
mediate values in computations. The state variables primarily
is structurally more suitable for parallel computation. In this
include ValueState, ListState, and MapState, utilized for stor-
study, the Encoder component of the Transformer is utilized
ing values, lists, and key-value tables, respectively. Flink
as a sub-model in the CO-Transformer.
provides data persistence and strong consistency support for
these state variables.
E. DBSCAN CLUSTERING ALGORITHM
DBSCAN (Density-Based Spatial Clustering of Applications
C. KAPPA SOFTWARE ARCHITECTURE
with Noise) is a density-based spatial clustering algorithm
The Kappa architecture [6], proposed by Jay Kreps, is a that partitions regions with sufficient density into clusters.
kind of software architecture designed for streaming data, It can discover clusters of arbitrary shapes in spatial databases
comprising layers for data sources, stream data process- containing noise. The algorithm involves two hyperparam-
ing, and services. Diverging from Lambda architecture, the eters, namely the maximum radius and minimum classified
Kappa architecture discards the batch processing layer, using points, which can provide excellent adaptability [12].
only the stream processing layer as the core data processing
component. This avoids issues such as redundant develop- III. SOFTWARE SYSTEM ARCHITECTURE DESIGN
ment and code synchronization associated with maintaining This paper establishes a real-time data stream analysis system
both batch and stream processing in the Lambda archi- for civil aviation based on Kappa architecture, as illustrated
tecture [31]. With stream data processing at its core, the in Figure 2. The system comprises layers for data reception,
Kappa architecture has found applications in industries such data buffer, data processing and data storage. Real-time avia-
as telecommunications [32], electricity [33], and convenient tion data stream, transmitted via the User Datagram Protocol
transportation [34], where Kafka and Flink are employed (UDP), enters through the reception layer, and undergoes
to build real-time data analysis systems. However, in the buffering through multiple Kafka queues. The parallel data
domain of real-time aviation data processing, there is a processing layer is implemented using Flink, which reads
lack of research on software systems adopting the Kappa data from Kafka queues, executes algorithms for decoding
architecture. and analysis, and stores the results of decoding and analysis
in InfluxDB [40]. A dual-layer mapping table optimization
D. TRANSFORMER NEURAL NETWORK is introduced at data processing layer, significantly reducing
The Transformer neural network incorporates an attention the latency in the computation process of sliding windows for
mechanism, compared with common neural networks [36]. multidimensional time-series data. The system also exhibits
This mechanism enables it to handle data correlations in time excellent scalability, allowing for migration and extension of

VOLUME 12, 2024 66385

 Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

various algorithm modules in the Flink pipeline. This capa-

bility facilitates in-depth real-time analysis of other critical
aircraft systems such as fuel, hydraulic, and avionics.

A. DATA RECEPTION AND BUFFER LAYER

Quick Access Recorder (QAR) data packets are generated
by an airborne terminal and can be transmitted via various
mediums such as satellites, Starlink [41], and Air-to-Ground
(ATG) Networks. The advent of the 5th generation-based
(5G-based) practical ATG network, which is rapidly becom-
ing the standard communication technology in aviation [42],
enables direct wireless communication between the aircraft’s
system and ground stations dispersed along flight routes.
Data packets are dispatched from the airborne terminal
to ground stations in UDP protocol and subsequently for-
warded to clusters for further processing and analysis. On the
receiving end, servers acting as Kafka producers concurrently
receive packets from various aircraft, convert these data pack-
ets into Kafka messages, and buffer them into Kafka queues
based on the flight number. Packets from each flight can FIGURE 3. Schematics of ideal sliding window and sliding window
optimized by dual-layer mapping table.
be distributed across multiple queues, with the data in each
queue being processed by a distinct decoding operator.

B. DATA PROCESSING LAYER C. SLIDDING WINDOW DESIGN BASED ON

The data processing layer is implemented by Flink. Fast OPTIMIAZTION OF DUAL-LAYER MAPPING TABLE
decoding, flight trajectory outlier repair, flight phase iden- In Flink, the time interval for processing data within each
tification, steady cruising stats determination and engine sliding window and the time interval for sending QAR data
lubricating oil quantity monitoring algorithms were encap- packets are fixed. If the times when the software system starts
sulated into algorithmic modules, deployed to the Flink job and when QAR data start to be sent differ, it may result in
platform. Each module can consist of multiple parallel pro- a time gap between the entry of all data packets into the
cessing operators, effectively leveraging the computational corresponding window and the triggering of data processing
resources of distributed processing platforms to achieve high for that window, leading to significant latency, shown in the
throughput. left of Figure 3.
For the fast decoding module, multiple operators within This paper designs an optimized sliding window mecha-
the module concurrently read QAR data packets from Kafka nism based on dual-layer mapping table, which is built with
message queues. Specialized parameter configuration tables Flink’s table-structured state variables (MapState) and a hash
are employed for decoding, and these tables are cached in table. It replaces the native sliding window implementation
Redis. The decoded results are written into the InfluxDB in Flink, aiming to reduce the processing latency of multidi-
database. Simultaneously, leveraging the Flink data buffering mensional time-series data sliding windows. As the QAR data
and transmission mechanism, the decoded results are trans- stream packet transmission frequency is fixed, the amount of
mitted to subsequent operators for further in-depth analysis. data records for a parameter within a specific time interval
In accordance with algorithmic requirements, flight trajec- can be used to infer the length of that interval. Hence, this
tory outlier repair, flight phase identification, steady cruising proposed sliding window utilizes the data amount within the
stats determination and engine lubricating oil quantity mon- window as the criterion for data collection completion and
itoring algorithm modules select relevant parameters from triggering data processing. It employs a dual-layer mapping
the post-decoding data, using the aircraft flight number as table to rapidly cache sensor data within the window and
the partition key to divide the data stream into multiple sub- support real-time computation. The architecture of dual-layer
streams. In each module, a separate operator is deployed to mapping tables is shown in Figure 3. The first layer mapping
each sub-stream to achieve processing different aircraft data table is employed to manage the timeliness of data within
in parallel. Each operator utilizes Flink state variables to store the window, implemented using Flink MapState. The keys of
intermediate values, fully leveraging the Flink user-state vari- MapState are set as the timestamps of parameter data records
able state management mechanism to enhance computational generated. When the window receives new parameter data,
efficiency. This approach capitalizes on the advantages of if there exists a key in the first-layer mapping table that is
Flink state consistency management, fault handling recovery, equal to the timestamp of the data record, the data record is
and efficient storage and access [30]. inserted into the corresponding second-layer mapping table

66386 VOLUME 12, 2024

Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

Algorithm 1 Flight trajectory outliers repair

Data: i-th high part data point fi , i-th low part data point ci
Result: value of i-th trajectory point di
1 for i ∈ {2, . . . , n} do
2 /∗ The preset threshold for anomaly detection is
denoted as m. ∗/
3 d i ← ci + f i ;
4 if ( di − di−1 > m) and ( di−1 − di−2 < m) then
5 ci ← ci−1 ;
6 di ← ci + fi ;
7 end
8 writeAPI(fi , ci , di );
9 end

ARINC 717 records is periodically divided into multiple

subframes, with each subframe containing several WORDs.
FIGURE 4. Improved index-based configuration table structure. To rapidly convert the original WORD into sensor recording
values, a decoding configuration table is devised. This decod-
ing configuration table is stored in a Redis cache and consists
using this timestamp. If the timestamp of the data record of multiple units that store decoding information for sensor
is greater than all keys in the first-layer mapping table, the parameters. Each decoding information unit encompasses the
window’s end timestamp is updated, a new second-layer complete decoding information for a WORD across different
mapping table is allocated for the new timestamp, and expired subframes.
second-layer mapping tables are removed. The second-layer To meet the real-time aviation data processing require-
mapping table is implemented using a hash table, where keys ments for low latency, this paper employs a rapid decoding
are parameter names of the parameter data record, and values algorithm. The algorithm adds an index to each parameter
are parameter values. By leveraging the characteristics of the configuration unit in the original configuration table based
hash table, subsequent data analysis operators can quickly on the subframe number and WORD number. The archi-
retrieve the required parameter values based on the parameter tecture of configuration table is shown in Figure 4. In the
names. decoding process, the algorithm utilizes the subframe number
The proposed sliding window function optimized by and WORD number of the received data packet to obtain
dual-layer mapping table is encapsulated into a generic the storage location of the corresponding decoding informa-
pre-processing module for input. Window length and sliding tion configuration unit through an index table. Subsequently,
step are fixable configurations in the module. In engine lubri- it decodes the complete parameter information of this WORD
cating oil quantity monitoring module, experimental results based on the configuration information. This approach avoids
show that the proposed method can effectively reduce the the traditional decoding algorithm’s exhaustive table traversal
processing delay and provide support for multi-dimensional process, thereby enhancing decoding speed. Details about the
time series data preprocessing. decoding algorithm are shown in [43].

D. DATA STORAGE LAYER B. FLIGHT TRAJECTORY OUTLIER REPAIR

InfluxDB database [39] are utilized as a component of the For parameters such as latitude and longitude that represent
data storage layer, which is a database dedicated to the high- the flight trajectory and require higher precision, the 12-
performance read, write, storage, and real-time analysis of bit WORD records are insufficient to meet the precision
massive time-series data. Data storage layer is used to store requirements. In engineering practice, a common approach is
the decoded engineering data and analysis results from vari- to split a single parameter into two WORD records (high part
ous algorithm modules. and low part). For real-time streaming data transmission, the
high and low parts of a single parameter may be distributed
IV. ALGORITHM DESIGN across different data packets and transmitted to the ground
This section introduces fast QAR data packet decoding, flight at different subframes. When the low part WORD has been
trajectory outlier repair, flight phase identification and steady transmitted, numerical carry or borrow between the high part
cruising determination real-time algorithms. and low part WORDs may occur before high part WORD
is transmitted, which can lead to a mismatch in the high
A. FAST QAR DATA PACKET DECODING ALGORITHM and low part of the data and resulting in abnormal flight
The fast QAR data packet decoding algorithm transforms trajectory [44].
aircraft raw data records, which are in a specific format Currently, there is a lack of methods for real-time air-
(ARINC 717 is used in this paper), into sensor engineering craft data trajectory anomaly correction. This paper pro-
data records for subsequent data analysis. The time axis in poses a real-time trajectory anomaly correction algorithm

VOLUME 12, 2024 66387

 Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

FIGURE 5. Workflow of flight phase identification.

(Algorithm 1). The algorithm sets a maximum discriminative 2. After the taxi-out phase, Wang et al. [15] utilized
threshold m for the latitude and longitude changing rate, the data from the lift devices (slats and flaps) to
which is represented as the difference between the current partition the flight phases. In this study, we further
and previous values. When the absolute value of difference incorporate radio altitude and relative barometric alti-
exceeds the threshold m, indicating a mismatch between high tude (H = ALT-ALTg ) to distinguish the takeoff,
and low parts, a trajectory anomaly occurs. According to the initial climb, and climb phases. This helps elimi-
ARINC 717 data format, both latitude and longitude low part nate the impact of variations in lift device models
data will experience numerical carryover when growing to and pilot-induced delays in operating the lift devices,
0.7032832◦ . Therefore, the threshold m needs to be no higher enhancing the algorithm’s overall applicability.
than 0.7032832◦ . At the same time, to prevent misjudgment 3. Upon entering the stratosphere and setting the route, the
of correct data, m must be greater than the maximum correct algorithm employs a 5-second average rate of change
difference. Considering that the maximum flight speed of of barometric altitude to determine the aircraft’s verti-
civil aviation aircraft does not exceed 1000 kilometers per cal motion attitude, thus distinguishing between climb,
hour, this paper sets m = 0.5◦ . In Algorithm 1, if the absolute cruise, and descent phases. The partitioning condition
difference between current trajectory value point and the is expressed as (1), where the threshold value m can be
previous is over the threshold, the low part value ci will be adjusted based on the aircraft model [15].
restore to ci−1 at which numerical carry or borrow does not 4. As the aircraft approaches the destination, the
occur. Subsequently, the high and low part data are combined algorithm employs vertical motion attitude, slat and
to obtain the complete data value di , which is then written to flap status, and radio altitude data to partition the initial
the database. approach, approach, and final approach phases. When
the radio altitude reaches zero, indicating landing, the
C. FLIGHT PHASE IDENTIFICATION algorithm utilizes ground speed (GS) to determine entry
This paper proposes a perception-based real-time flight phase into the flare phase.
identification method. It utilizes the flight phase at the
previous and current parameter data, employing multiple

 dALT < −m,
 DESCENT
identifying criteria to perceive the aircraft’s flight phase
−m ≤ dALT ≤ m, CRUISE (1)
and environmental conditions. This process calculates the
m < dALT ,

CLIMB

real-time flight phase at the current moment. The specific
identifying workflow is illustrated in Figure 5. The thresholds
in conditions are determined by the specific dataset. D. STEADY CRUISING STATE DETERMINATION
The algorithm dynamically divides the aircraft’s flight This subsection will introduce the algorithm for determining
process into 14 flight phases using six parameters listed in the steady state during the cruise phase. Steady state during
Table 2. The main process is outlined as follows: the cruise phase refers to the condition where commercial
1. In the initial phase of engine stopped, the algorithm aircraft maintain stable flights by controlling factors such
utilizes the high-pressure rotor speed of the engine as a as speed, altitude, attitude, and considering external factors
criterion to determine the aircraft’s engine-started and such as weather conditions and turbulence. The determination
taxi-out phases. of steady cruising state plays a crucial role in aviation data

66388 VOLUME 12, 2024

Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

TABLE 2. Parameters used in flight phase identification. Algorithm 2 Computing interval maximum/minimum based
on sliding window monotonic queue
Data: x is a data record, q is a double-ended queue for data
buffering, where h is the head element, t is the last
element of the queue.
Require: Maximum/Minimum value of the required interval
1 /∗ Assume T (·), V (·) is to get the timestamp and value
of a data record respectively ∗/
2 h ← q. headElement();
3 while h ̸= null and T (h) < T (x) − M do/∗ window
length M ∗/
TABLE 3. Parameters used in flight phase identification. 4 q.popHead();
5 h ← q. headElement();
6 end
7 t ← q. tailElement();
8 while t ̸= null and V (t) ≤ V (x) do/∗ For minimum value,
V (t) ≥ V (x). For maximum value V (t) ≤ V (x) ∗/
9 q.popHead();
10 h ← q. headElement();
11 end
12 q.pushAsTail(x);
13 h ← q. headElement();
14 return h

analysis. For instance, analyzing fuel consumption during length is M , the overall enqueue and dequeue operations
the steady cruising state provides insights into the aircraft’s are both performed N times since each timestamp’s record
performance degradation, guiding maintenance efforts [45], enters and exits the queue only once. The query operations
[46]. Additionally, analyzing flight attitudes during the steady are performed N -M + 1 times. The time complexity for all
state of the cruise phase helps detect attitude anomalies, guid- cruising data is O(N ) and the amortized one is O(1). The
ing adjustments to aircraft control surfaces [47]. Deploying a minimum value is similar with above.
real-time algorithm for determining steady state during the For the calculation of the window interval mean and sample
cruise phase can provide support for further data analysis. variance, this paper introduces a fast algorithm that utilizes
The criteria for determining the steady state during the queues to update real-time prefix sum [49] and prefix square
cruise phase include maintaining stability in various param- sum [49] for mean and variance computation, reducing the
eters such as aircraft altitude, flight speed, aircraft attitude, computational complexity. For a parameter of time-series
and atmospheric static temperature over a specified time data, establish a queue q to buffer parameter data, and set
interval (Table 3). The mean(∗), std(∗), min(∗), and max(∗) up two state variables s and ss to record the total sum and
in Table 3 represent the mean, standard deviation, minimum, total square sum of elements in the queue, respectively. Based
and maximum values of the parameters over the time interval, on real-time data stream, elements are dynamically popped
respectively. out and pushed into the queue q to ensure a constant queue
This paper proposes a fast real-time determination method length of M , where M is the interval length of sliding window,
for steady cruising state, improving the calculation process and simultaneously update the values of state variables s and
for interval extrema, mean, and sample variance. For the ss. By combining state variables s, ss, and the queue length
maximum and minimum interval value, a monotonic queue L(q), interval mean and interval sample variance within a
algorithm [48] is employed to reduce the number of redundant window can be calculated. Refer to Algorithm 3 for the
calculations, as detailed in Algorithm 2. Taking the example detailed algorithmic process. For a parameter with a total of
of calculating the maximum value within a window interval N data records in the cruise phase, there are N -M + 1 data
for a specific parameter, establish a queue for that parameter windows. Each data record is pushed into and popped out of
and ensure that the elements in the queue are always arranged the queue only once. For each window, state variables s and
in descending order. When new data for the parameter is ss are recalculated only when queue elements are updated,
received at a new timestamp, update the window interval first and interval mean, and sample variance are calculated only
and remove outdated data from the queue. Then, compare the when queue elements are updated. Therefore, the total time
new data with the elements in the queue starting from the tail complexity for calculating all windows of a single parameter
until encountering an element greater than the new data. Stop is O(N ), amortized to O(1) per timestamp.
the comparison, remove elements smaller than or equal to the
new value from the queue, and finally add the new data to V. CO-TRANSFORMER
the tail of the queue. Assuming a parameter has a total data This section introduces the CO-Transformer model, which
count of N within the cruising phase, and the window interval is used for real-time monitoring of engine lubricating oil

VOLUME 12, 2024 66389

 Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

Algorithm 3 Computing interval mean and sample variance TABLE 4. Symbols that CO-Transformer utilize and their meanings.
base on sliding window queue
Data: x is a data record, q is a double-ended queue for data
buffering, where h is the head element, t is the last
element of the queue.
Require: Interval mean µ, interval sample variance σ .
1 /∗ Assume L (·), is to get the length of queue ∗/
2 h ← q. headElement();
3 while h ̸= null and T (h) < T (x) − M do/∗ window length M
∗/
4 s ← s − V (h);
5 ss ← ss − [V (h)]2 ;
6 q. popHead();
7 h ← q. headElement();
8 end
9 q.pushAsTail(x);
10 s ← s + V (x);
11 ss ← ss − [V (x)]2 ;
12 µ ← p s/L(q);
13 σ ← [ss − s2 /L(q)]/[L (q) − 1]
14 return µ, σ
where WQ , WK , WV are weight matrices learned during the
training process. And X is input data matrix. The parameters
quantity. The lubrication system of an aviation engine plays the data set utilizes are shown in Table 4. The attention unit
a crucial role, and a malfunction can lead to engine shut- can horizontally expand into a multi-head attention mecha-
down, significantly impacting flight safety [50] Currently, nism, and its output is the weighted sum of the outputs from
fault monitoring of the lubrication system relies largely on multiple attention units (4).
experience and qualitative analysis, lacking precision and
quantification [51]. There is still a lack of real-time cal-
MutiHeadAttention (Q, K , V )
culation methods for the baseline value of lubricating oil
quantity during flight. This paper proposes a CO-Transformer = concati∈heads (Attention (Qi , Ki , Vi )) WO (4)
model, using clustering algorithm to optimize Transformer.
The model utilizes 15 parameters (Table 4) including engine where heads number is a set of heads and WV is a weight
high-pressure rotor speed, aircraft altitude, flight speed, and matrix. The output from one Encoder is further processed
attitude as inputs, with the baseline value of lubricating through an activation layer, average pooling layer, and fully
oil quantity as the output. When a significant deviation is connected layer to obtain the final output, which represents
observed between the monitored actual oil quantity during the predicted baseline value of lubricating oil quantity for
flight and the calculated baseline value, predicting lubricant an individual sub-model. The loss function value l between
leakage, an alarm is triggered. This prompts both the flight the predicted values from the Encoder and the true values is
crew and ground maintenance personnel to take necessary utilized to update sub-model’s parameters. Refer to Figure 6
measures to enhance flight safety. for an illustrative diagram of the sub-model computation,
The CO-Transformer model is based on the Bagging con- where y represents the output baseline value of lubricating
cept in ensemble learning within the context of a random oil quantity.
forest [52]. It integrates multiple Transformer Encoder as sub- To introduce diversity to sub-models, Bootstrap sam-
models, utilizes the Bootstrap method to generate training pling [26] is employed to partition the training data into
datasets for these sub-models, and employs the DBSCAN distinct training subsets. These subsets are independently
clustering algorithm to evaluate and select sub-models that utilized for training each sub-model, enhancing the overall
exhibit superior performance. The Transformer Encoder model’s generalization capability, and mitigating overfitting.
sub-models use an attention mechanism to depict the signifi- The CO-Transformer model employs DBSCAN clustering
cance of past temporal data for predicting values in sequence algorithm to filter out sub-models with superior performance.
data, and its calculation formula (2) is expressed as follows. Model selection utilizes the sub-model loss function value (l
in Figure 6) as the clustering metric. After z-score normal-
QK T
 
Attention (Qi , Ki , Vi ) = softmax √ V (2) ization [53], the loss values are inputted into the DBSCAN
dk clustering model, which clusters the sub-models into multi-
where Q, K and V are all vectors and dk are the dimension ple model clusters based on the density of the metric. The
of K . Their calculation formula (Formula 3) is expressed as Mean Square Error (MSE) is used as the loss function for
follows. filtering. And compared to Mean Absolute Error (MAE), the
differences in normalized MSE are more pronounced among
Q = WQ X, K = WK X, V = WV X (3) the sub-models. The formulas of MSE and MAE are shown

66390 VOLUME 12, 2024

Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

FIGURE 6. The architecture of CO-Transformer model.

in (5) and (6). VI. SIMULATION TEST AND RESULT ANALYSIS

In this section, the accuracy and timeliness of each algorithm
MSE module were tested using simulated data streams. Further-
m
1X
2
more, the overall load performance of the real-time aviation
= yi − ŷi yi is true value, ŷi is predicted value data analysis system was comprehensively evaluated through
m
i=1
testing and analysis.
(5)
MAE A. EXPERIMENT ENVIROMENT
m
1X
In the experiments, three servers were utilized, each equipped
= yi − ŷi yi is true value, ŷi is predicted value with two 20-core Intel Xeon Silver 4210 processors. The base
m
i=1
frequency was 2.20 GHz, with a maximum turbo frequency
(6) of 3.20 GHz. A big data infrastructure environment was
According to the number of clusters in DBSCAN cluster- established on these three servers, where the Flink cluster
ing, the algorithm selects sub-models from clusters with operated in Yarn mode across all servers. The parallelism
better performance in a certain proportion. The selected sub- of the algorithm modules was configured to the maximum
models’ output results are then combined to obtain the final parallelism, based on the maximum allocatable resources in
output, and the number of retained clusters can be set inde- Yarn. The Kafka cluster ran on the three servers, while the
pendently. In Figure 6, there are m sub-models final remain, InfluxDB database and Redis service operated in standalone
which are c1 , c2 . . . cm th sub-model. mode on a single server.
During the training process, the individual training of each
sub-model in CO-Transformer allows for efficient utiliza- B. SIMULATING DATA STREAM GENERATION
tion of GPU resources within the cluster. In the deployment This study leverages post-flight Quick Access Recorder
phase, based on the Flink architecture, the selected Encoder (QAR) data from real flights to simulate real-time QAR data
sub-models can be deployed to multiple operators (Figure 6), streams, transmitted under UDP protocol. A multi-threaded
enabling parallel computation among sub-models, and ensur- packet transmission program was developed using the Go
ing timely processing. According to experimental results, the language to simulate the concurrent transmission of real-time
proposed model accurately predicts the oil quantity when the data from multiple aircraft. The total size of simulated data
engine lubrication system operates normally. Computation amounts to 337MB, divided into 16913 subframes. For air-
latency meets the requirements of real-time monitoring sce- craft sensor parameters sampled at a frequency of 1Hz, there
narios, providing a reliable solution for real-time detection of are an average of 5637 data records per flight. Regarding
engine oil leakage. the engine lubricant monitoring model, a total of 200 real

VOLUME 12, 2024 66391

 Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

FIGURE 7. Comparison between repaired and unrepaired flight

trajectories: (a) Repaired flight trajectories; (b) Unrepaired flight
trajectories with anomalies (outliers) marked by red squares.
FIGURE 9. Comparison of flight phase results between proposed method
and commercial post-flight data analysis software in all flights.

LOF’s time cost (11ms/outlier) is much worse than ours

(0.2ms/outlier) in the experimental hardware environment
upon eliminating the influence caused by I/O of Flink state
variables and intermedia variable persistence.

D. FLIGHT PHASE IDENTIFICATION EXPERIMENT

In the experiment of flight phase identification, the simulated
data stream generated in Section VI-B was employed as input
to verify the accuracy of real-time algorithmic division. The
flight phases are compared with the ones from mainstream
commercial post-flight analysis software, which serves as the
real values. Figure 8 provides two examples of comparison
FIGURE 8. Comparison of results of real-time flight phase identification of the consistency in phase results for two flight routes, con-
method and commercial post-flight data analysis software. taining 8156 and 8576 flight records respectively, which data
consistencies in identification are 98.32% and 98.16% for the
two flight routes. And the histogram of data consistency in the
flight data sets were utilized. Among these, 160 flights were whole dataset is shown in Figure 9. The average consistency
used for model training, 20 flights for generating DBSCAN of all data records in the dataset is around 97.8%, approaching
evaluation metrics, and the rest 20 flights for testing models to Wang’s result (99.7%) using complex pre-process works
performance. The model’s real-time deployment capability and multiple non-real-time algorithms [15], near Hou’s result
was further validated through real-time simulated QAR data. (98.2%) using decision tree to identify only 2 phases, landing
and takeoff phases, in post-flight dataset [10] and higher than
C. FLIGHT TRAJECTORY OUTLIER REPAIR EXPERIMENT Zhang’s result (97.4%) [11].
In the experiment for repairing flight trajectory outliers, the Based on experimental results and comparison with main-
simulated data stream generated in Section VI-B was uti- stream commercial post-flight analysis software and others’
lized as input. The repair results are illustrated in Figure 7, research, it indicates that the real-time flight phase identi-
where three trajectories of different colors correspond to three fication results align with the general identification rules in
parallel simulated flight routes transmitting real-time QAR the civil aviation domain. And its accuracy is close to other
data. The left panel depicts the repaired trajectories, while post-flight algorithms, which applied much pre-process and
the right panel shows the trajectories before repair. A total correcting works on post-flight data than real-time algorithm.
of 12 trajectory anomalies(outliers) occurred across the three Hence, the algorithm provides accurate flight phase infor-
routes before repair, with the anomalous points marked mation for ground monitoring of aircraft status and supports
by red squares in Figure 7(b). By comparing Figure 7(a) subsequent in-depth analysis.
and Figure 7(b), it is evident that all trajectory anomalies
were accurately repaired. The experimental results demon- E. STEADY CRUISING STATE DETERMINATION
strate that the flight trajectory outlier repair algorithm can EXPERIMENT
effectively capture and rectify anomalies in latitude and lon- The experiment utilized simulated data streams as input,
gitude caused by asynchronous carry-over of high and low comparing the computational time between the naive
parts in QAR recordings. Compared with Sameer’s work non-optimized calculation method and the queue-based
(LOF method) [9], both can accurately correct outliers. But improved method. Following aviation domain expertise,

66392 VOLUME 12, 2024

Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

TABLE 5. Results of different models under various metrics.

TABLE 6. The results of the two types of latency for each module.
FIGURE 10. Comparison of results of real-time flight phase identification
method and commercial post-flight data analysis software.

window size M was set to 180 seconds. Within each window,

calculations were performed for the maximum, minimum and
mean of pressure altitude, as well as the variance of seven
parameters such as Mach number and roll angle. Compared
with proposed Cao’s work [18] and Wang’s work [19], which
used naive method (namely sliding window), the computa- All models are restricted to a maximum of 25 training epochs,
tional time cost of our queue-based improved method was where all models’ metric values will converge. The details
significantly lower than the naive method, as illustrated in of the training process are shown in Figure 11. And detail
Figure 10. When the cumulative data processing reached parameter numbers of the whole network in models are shown
3300 records, the improved method exhibited a computation in Table 5.
speed five times faster than the naive method. From this Figure 12 illustrates the comparison between lubricating
comparison, queue-based improved method obviously can oil prediction values and actual values under different models
boost the speed of steady cruising state determination. for a specific test flight segment. It can be observed from the
The experimental results indicate that the improved cruise graph that the CO-Transformer model closely aligns with the
phase steady-state determination algorithm based on the actual results. And Figure 11 displays all models’ training
queue enhancement method has a significant advantage in process. It seems that CO-Transformer has a faster conver-
computational efficiency. It can effectively enhance data gence speed and predictive performance than others. And
processing efficiency, laying the foundation for subsequent Table 5 provides metric values and parameter numbers for
analyses such as fuel consumption monitoring during cruise each model. The CO-Transformer model has an average mean
steady-state and monitoring of aircraft control surface status. squared error of 0.057 L2 , representing improvements of
7.36%, 66.01%, and 66.05% compared to the regular Trans-
F. EXPERIMENT ON THE ENGINE LUBRICATING OIL former, LSTM, and GRU models, respectively. The average
QUANTITY MONITORING MODEL absolute error is 0.178 L, indicating enhancements of 3.49%,
All neural network models were implemented by Tensor- 33.25%, and 33.53% compared to the same models. The
flow 2.10 framework. The optimizer used in training was experimental results demonstrate that the CO-Transformer
Adam (adaptive moment estimation). The training dataset model can more accurately predict aviation engine lubricat-
mentioned in Section VI-B was used. For model’s param- ing oil levels, providing a basis for monitoring faults such as
eters, the key dimension and number of attention heads lubricating oil leakage.
in CO-Transformer’s sub-models and normal Transformer In terms of computational efficiency, this paper deploys
were set to 168 and 16, respectively, which yielded good the CO-Transformer model using Flink’s general process-
model performance and made each training epoch of Trans- ing operator, where each sub-model is implemented as an
former cost around 40s. And initial sub-model number independent operator. The parallel execution of sub-models
and remained sub-model number was 9 and 3 respectively enhances computation speed. Test results obtained using sim-
in CO-Transformer. The DBSCAN clustering algorithm is ulated real-time data from Section VI-B demonstrate that the
implemented using the sklearn library, with a minimum clus- CO-Transformer model that predicts a lubricating oil value
ter size set to 1 to ensure each sub-model can be assigned costs within 50ms, meeting the real-time monitoring latency
to a cluster. The maximum radius and the number of clus- requirements for engine monitoring.
ters to retain are set to 0.25 and 3, respectively. Commonly
used models including LSTM [23], GRU [24] and Trans- G. EXPERIMENT ON SYSTEM LATENCY AND LOAD
former [25] are tested for comparison. And the time cost PERFORMANCE
of LSTM and GRU in each training epoch is also limited In this section, using the simulated data flow presented in
to around 40s by adjusting their parameters, to show model Section VI.B, we conducted tests on the latency performance
performance differences under the similar training time cost. of the aviation real-time data analysis system based on the

VOLUME 12, 2024 66393

 Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

FIGURE 11. Metric values of different models in training dataset and validating dataset during the whole training process.

FIGURE 12. Comparison of results of real-time flight phase identification method and commercial post-flight data analysis software.

TABLE 7. As the average throughput increases, the average cluster CPU occupancy and the average type 1 latency of each module.

Kappa architecture and various algorithm modules. We ana- packets into the system and ends with each module complet-
lyzed the system’s load capacity by recording two types of ing the calculation and outputting the result. We calculated
delays. The first type of latency starts with the entry of data the average latency of each algorithm module, including

66394 VOLUME 12, 2024

Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

queue delay, algorithm processing delay, and inter-module TABLE 8. The regression formulas for the first type of average latency in
each module with the system throughput as the independent variable.
data interaction delay. The second type of latency starts with
the module receiving data and ends with the module com-
pleting the calculation and outputting the result, measuring
the module’s inherent processing delay. The overall system
latency is the maximum latency among all modules. The
latency test results are shown in Table 6, where both types
of latency for each module are below 250 milliseconds. This
is less than the QAR data packet sending interval (1 sec-
ond), ensuring that there is no data packet accumulation and
meeting the real-time aviation data processing requirements.
And compared with Type 1 latency of decoding module in
Qin’s work (71.45ms) [4], this system has significantly higher
performance (12ms) in the experimental hardware environ- data analysis and processing scenarios for multiple flights in
ment, which enables the system to leave more time for other actual flight routes.
complex data analysis algorithms.
For the system’s load testing, the average throughput of VII. CONCLUSION
QAR data for a single flight in the simulated data stream is This paper designs and implements a real-time aviation data
164 kbps. In the tests, we increased the parallel flights send- processing system based on the Kappa streaming processing
ing real-time data streams to simulate a high-load data stream architecture. Multiple real-time analysis algorithms, includ-
environment. We then recorded and analyzed the average ing flight trajectory outlier repair, flight phase identification,
CPU usage across all servers in the cluster and the first type of steady cruising state determination, and engine lubricant oil
average delay for each algorithm module. The experimental quantity monitoring, have been developed. This provides a
results are presented in Table 7, where the average CPU novel data analysis solution for the future real-time transmis-
utilization of the cluster shows a linear increase with the rising sion scenarios of onboard sensors in civil aviation. Simulation
throughput. Based on the experimental results, a regression test results demonstrate that the maximum average delay for
formula for the average CPU utilization as a function of the individual algorithms is consistently below 250 milliseconds.
average throughput is derived as L(x) =0.701x-11.85, where The algorithms can run in parallel, meeting real-time analy-
x is the system’s average throughput, and L(x) is the CPU sis requirements. Specifically, the improved steady cruising
average utilization. For each algorithm module, the regres- state determining algorithm exhibits computation efficiency
sion formula between its first type of latency and the system’s five times faster than the naive non-optimized method. The
average throughput can be obtained, as shown in Table 8, CO-Transformer neural network-based lubricant oil quantity
where Fi (x) represents the first type of delay for i-th algorithm monitoring method achieved an average mean square error of
module. approximately 0.057 L2 between predicted and actual values,
According to the system performance formula, under the showing significant performance improvement compared to
constraints of CPU utilization and system processing delay, existing methods and meeting the requirements for moni-
a theoretical model maximizing the system throughput can be toring lubricant system airborne leaks. The system shows
established, as shown in (7), where the optimization objective strong load capacity and good scalability, providing a solution
is the system throughput. The constraints Fi (x) <1000 rep- to the gap in real-time civil aviation data analysis system,
resent that the first type of delay for i-th algorithm module enabling further research on real-time analysis of aircraft
is less than the real-time data packet sending interval (1000 fuel efficiency and critical systems such as control surfaces,
milliseconds), and the constraint L(x) <100% indicates that hydraulics, and avionics.
the CPU average utilization does not exceed 100%. But the system still has shortcomings that need concerns
in the future. According to Kumari et al. work [54], [55],
max x which surveyed existing literature about multimedia big data
( (MMBD), computing security, and other challenges about big
Fi (x) < 1000, i = 1, 2, . . . , 5 data processing in IoT. Encrypted data transfer and private
s.t. (7)
L (x) < 100% computing also deserves more contributions in this system,
also in the field of aviation big data, which are the most
By employing the linear programming method to solve the important cyber security aspects in your solution. Moreover,
optimization problem, the maximum system throughput can fog computing (or edge computing) in industry is also vital
be obtained. Under the hardware conditions used in the to be concerned about, especially in the new 5G environ-
experiment, the maximum throughput is 79 Mbps, corre- ment [56], [57]. This technology can be introduced to reduce
sponding to 505 civilian planes concurrently send real-time the load of center cluster and throughput of ground net-
data streams. The analysis results indicate a strong system work, like data compression and pre-processing work can be
load capacity, capable of meeting high-concurrency real-time deployed in the ground station or edge servers.

VOLUME 12, 2024 66395

 Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

ACKNOWLEDGMENT [21] M. Ma, T. Wang, and L. Wang, ‘‘Research on monitoring method of

The authors would like to thank an editor and reviewers for lubricating oil based on deep learning,’’ J. Propuls. Technol., vol. 41,
no. 5, pp. 1159–1167, 2020, doi: 10.13675/j.cnki.tjjs.190240.
their time and suggestions. [22] P. Gao, ‘‘Research on intelligent prediction of civil aviation engine oil
quantity,’’ M.S. thesis, College Aeronaut. Eng., Civil Aviation Univ. China,
REFERENCES Tianjin, China, 2021, doi: 10.27627/d.cnki.gzmhy.2021.000299.
[23] Y. Yu, X. Si, C. Hu, and J. Zhang, ‘‘A review of recurrent neural networks:
[1] R. Wang, J. Wu, C. Liu, and H. Yang, ‘‘Fault cause identification method
LSTM cells and network architectures,’’ Neural Comput., vol. 31, no. 7,
for aircraft equipment based on maintenance log,’’ J. Softw., vol. 30, no. 5,
pp. 1235–1270, Jul. 2019.
pp. 1375–1385, 2019.
[24] S. Nosouhian, F. Nosouhian, and A. K. Khoshouei, ‘‘A review of
[2] B. Li, J. Huang, X. Wang, Y. Shi, M. Lv, and M. Hu, ‘‘Overview of
recurrent neural network architecture for sequence learning: Compar-
application in data mining techniques to QAR data ansys,’’ in Proc. 2nd
ison between LSTM and GRU,’’ Preprints, pp. 1–7, Jul. 2021, doi:
Int. Conf. Big Data Eng. Educ. (BDEE), Chengdu, China, Aug. 2022,
10.20944/preprints202107.0252.v1.
pp. 13–21.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
[3] N.-N. Dao, Q.-V. Pham, D.-T. Do, and S. Dustdar, ‘‘The sky is the edge—
Ł. Kaiser, and I. Polosukhin, ‘‘Attention is all you need,’’ in Advances
Toward mobile coverage from the sky,’’ IEEE Internet Comput., vol. 25,
in Neural Information Processing Systems. Red Hook, NY, USA: Curran
no. 2, pp. 101–108, Mar. 2021.
Associates, 2017.
[4] H. Qin, X. Kong, and P. Shu, ‘‘Real-time downloading and analysis of QAR
[26] T. Hesterberg, ‘‘Bootstrap,’’ WIREs Comput. Statist., vol. 3, no. 6,
data using air-to-ground wireless communication,’’ in Proc. IEEE 1st Int.
pp. 497–526, Nov. 2011, doi: 10.1002/wics.182.
Conf. Civil Aviation Saf. Inf. Technol. (ICCASIT), Oct. 2019, pp. 519–524.
[27] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and
[5] S. Liu, M. Xiao, L. Li, Z. Li, and C. Zhang, ‘‘Design of real-time flight
K. Tzoumas, ‘‘Apache FlinkTM: Stream and batch processing in a single
data analysis system based on the ATG network,’’ in Proc. IEEE 4th Int.
engine,’’ Bull. IEEE Comput. Soc. Tech. Committee Data Eng., vol. 36,
Conf. Civil Aviation Saf. Inf. Technol. (ICCASIT), Oct. 2022, pp. 574–578.
no. 4, 2015.
[6] J. Lin, ‘‘The lambda and the kappa,’’ IEEE Internet Comput., vol. 21, no. 5,
[28] F. Schwaiger and F. Holzapfel, ‘‘Fast decoding of ARINC 717 flight
pp. 60–66, Sep. 2017, doi: 10.1109/MIC.2017.3481351.
data recordings,’’ in Proc. AIAA Scitech Forum, Jan. 2021, pp. 1–13, doi:
[7] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, ‘‘LOF: Identifying
10.2514/6.2021-1982.
density-based local outliers,’’ in Proc. ACM SIGMOD Int. Conf. Man-
[29] J. Kreps, N. Narkhede, and J. Rao, ‘‘Kafka: A distributed messaging system
age. Data. New York, NY, USA: Association for Computing Machinery,
for log processing,’’ in Proc. NetDB, 2011, pp. 1–7.
May 2000, pp. 93–104, doi: 10.1145/342009.335388.
[8] Z. Cheng, C. Zou, and J. Dong, ‘‘Outlier detection using isolation forest and [30] P. Carbone, S. Ewen, G. Fóra, S. Haridi, S. Richter, and K. Tzoumas, ‘‘State
local outlier factor,’’ in Proc. Conf. Res. Adapt. Convergent Syst. New York, management in apache Flink® : Consistent stateful distributed stream
NY, USA: Association for Computing Machinery, Sep. 2019, pp. 161–168, processing,’’ Proc. VLDB Endowment, vol. 10, no. 12, pp. 1718–1729,
doi: 10.1145/3338840.3355641. Aug. 2017, doi: 10.14778/3137765.3137777.
[9] S. K. Jasra, G. Valentino, A. Muscat, and R. Camilleri, ‘‘Hybrid machine [31] G. K. Kalipe and R. K. Behera, ‘‘Big data architectures: A detailed
learning—Statistical method for anomaly detection in flight data,’’ Appl. and application oriented review,’’ Int. J. Innov. Technol. Explor., vol. 8,
Sci., vol. 12, no. 20, p. 10261, Oct. 2022, doi: 10.3390/app122010261. pp. 2182–2190, Oct. 2019.
[10] X. Hou, D. Du, Y. Wang, Y. Song, and J. Qian, ‘‘Decision tree model- [32] N. Seyvet and. V. I. Mulas. Applying the Kappa Architecture in the Telco
based flight phase classification method,’’ in Proc. Int. Conf. Comput. Appl. Industry. O’Reilly Media. Accessed: Jan. 16, 2024. [Online]. Available:
Technol. (CCAT), Guiyang, China, Sep. 2023, pp. 13–16. https://www.oreilly.com/content/applying-the-kappa-architecture-in-the-
[11] Q. Zhang, J. H. Mott, M. E. Johnson, and J. A. Springer, ‘‘Development telco-industry/
of a reliable method for general aviation flight phase identification,’’ IEEE [33] P. Lipcák, M. Macak, and B. Rossi, ‘‘Big data platform for smart
Trans. Intell. Transp. Syst., vol. 23, no. 8, pp. 11729–11738, Aug. 2022. grids power consumption anomaly detection,’’ in Proc. Federated
[12] D. Hallac, S. Vare, S. Boyd, and J. Leskovec, ‘‘Toeplitz inverse covariance- Conf. Comput. Sci. Inf. Syst. (FedCSIS), Sep. 2019, pp. 771–780, doi:
based clustering of multivariate time series data,’’ in Proc. 23rd Int. 10.15439/2019F210.
Joint Conf. Artif. Intell. New York, NY, USA: Association for Computing [34] Y. Fu and C. Soman, ‘‘Real-time data infrastructure at Uber,’’ in Proc. Int.
Machinery, Jul. 2018, pp. 215–223. Conf. Manage. Data. New York, NY, USA: Assoc. Comput. Machinery,
[13] E. Schubert, J. Sander, M. Ester, H. P. Kriegel, and X. Xu, ‘‘DBSCAN Jun. 2021, pp. 2503–2516.
revisited, revisited: Why and how you should (still) use DBSCAN,’’ ACM [35] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo,
Trans. Database Syst., no. 3, pp. 19:1–19:21, 2017. ‘‘Swin Transformer: Hierarchical vision transformer using shifted win-
[14] J. Sun, J. Ellerbroek, and J. Hoekstra, ‘‘Large-scale flight phase identifi- dows,’’ in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021,
cation from ads-b data using machine learning methods,’’ in Proc. 7th Int. pp. 9992–10002. Accessed: Jan. 16, 2024.
Conf. Res. Air Transp., Philadelphia, PA, USA, 2016, pp. 1–8. [36] F. A. Acheampong, H. Nunoo-Mensah, and W. Chen, ‘‘Transformer
[15] B. Wang, Y. Zhang, H. Xie, and J. Li, ‘‘A flight phase identification method models for text-based emotion detection: A review of BERT-based
based on airborne data of civil aircraft,’’ J. Traffic Transp. Eng., vol. 22, approaches,’’ Artif. Intell. Rev., vol. 54, no. 8, pp. 5789–5829, Dec. 2021.
no. 1, pp. 216–228, 2022. [37] N. Wu, B. Green, X. Ben, and S. O’Banion, ‘‘Deep transformer mod-
[16] N. Kuzmenko, I. Ostroumov, Y. Bezkorovainyi, Y. Averyanova, V. Larin, els for time series forecasting: The influenza prevalence case,’’ 2020,
O. Sushchenko, M. Zaliskyi, and O. Solomentsev, ‘‘Airplane flight phase arXiv:2001.08317.
identification using maximum posterior probability method,’’ in Proc. [38] L. Cai, K. Janowicz, G. Mai, B. Yan, and R. Zhu, ‘‘Traffic transformer:
IEEE 3rd Int. Conf. Syst. Anal. Intell. Comput. (SAIC), Oct. 2022, pp. 1–5, Capturing the continuity and periodicity of time series for traffic forecast-
doi: 10.1109/SAIC57818.2022.9922913. ing,’’ Trans. GIS, vol. 24, no. 3, pp. 736–755, Jun. 2020.
[17] X. Yang, ‘‘Real-time status monitoring and visualization of Modern Ark [39] R. M. Farsani and E. Pazouki, ‘‘A transformer self-attention model for time
600 aircraft,’’ M.S. thesis, School Air Traffic Manag., Civil Aviation Flight series forecasting,’’ J. Electr. Comput. Eng. Innov. (JECEI), vol. 9, no. 1,
Univ. China, Deyang, China, 2016. pp. 1–10, 2020, doi: 10.22061/jecei.2020.7426.391.
[18] H. Cao, L. Xu, and Y. Li, ‘‘Research on the effect of performance degrada- [40] S. N. Zehra, ‘‘Time series databases and InfluxDB,’’ Studienarbeit, Uni-
tion on pollutant emission of civil aircraft in cruise phase,’’ Acta Scientiae versité Libre de Bruxelle, Brussels, Belgium, Tech. Rep., Dec. 2017.
Circumstantiae, vol. 42, no. 4, pp. 341–352, 2022. [41] J. Sekera and A. Novák, ‘‘The future of data communication in Aviation
[19] L. Wang, ‘‘Research on the selection of performance monitoring parame- 4.0 environment,’’ INCAS Bull., vol. 13, no. 3, pp. 165–178, Sep. 2021.
ters and optimization of algorithms for domestic civil aircraft,’’ M.S. thesis, [42] X. Lin, A. Furuskär, O. Liberg, and S. Euler, ‘‘Sky high 5G: New radio
College Flight Technol., Civil Aviation Flight Univ. China, Deyang, China, for air-to-ground communications,’’ in 5G and Beyond: Fundamentals and
2022, doi: 10.27722/d.cnki.gzgmh.2021.000153. Standards, X. Lin and N. Lee, Eds. Cham, Switzerland: Springer, 2021,
[20] Z. Wang and S. Fu, ‘‘An analysis of pilot’s physiological reactions in differ- pp. 503–515.
ent flight phases,’’ in Engineering Psychology and Cognitive Ergonomics [43] C. Ma, T. Qiao, X. Wang, Q. Xi, J. Wang, and Y. Zhang, ‘‘Decoding the
(Lecture Notes in Computer Science), D. Harris, Ed. Cham, Switzerland: real time QAR data with cached configuration structure,’’ Comput. Meas.
Springer, 2014, pp. 94–103. Control, vol. 31, no. 3, pp. 255–261, 2023.

66396 VOLUME 12, 2024

Y. Zhang et al.: Architecture and Algorithm Design for Civil Aviation Data Real-Time Analysis System

[44] J. Kim and J. Lyou, ‘‘Enhanced QAR flight data encoding and decod- QI XI received the bachelor’s and Ph.D. degrees in
ing algorithm for civil aircraft,’’ in Proc. SICE-ICASE Int. Joint Conf., information and communication engineering from
Oct. 2006, pp. 5169–5173. Shanghai Jiao Tong University, in 2017.
[45] J. L. Speyer, ‘‘Nonoptimality of the steady-state cruise for aircraft,’’ AIAA He used to be the Key Engineer of the Data
J., vol. 14, no. 11, pp. 1604–1610, Nov. 1976. Analysis Team, Commercial Aircraft Corporation
[46] K. N. Amrutha, Y. K. Bharath, and J. Jayanthi, ‘‘Aircraft engine fuel flow of China, Ltd., Shanghai, China. Currently, he is
parameter prediction and health monitoring system,’’ in Proc. 4th Int. Conf. the Deputy Director of the Industrial Big Data
Recent Trends Electron., Inf., Commun. Technol. (RTEICT), May 2019, Laboratory, School of Data Science and Engineer-
pp. 39–44, doi: 10.1109/RTEICT46194.2019.9016703.
ing, South China Normal University. His research
[47] G. Zhang, Z. Chen, and M. Xu, ‘‘High-speed aircraft position and
interests include industrial internet, industrial data
attitude control using reinforcement learning,’’ in Proc. 2nd Int. Conf.
Robot., Artif. Intell. Intell. Control (RAIIC), Aug. 2023, pp. 156–161, doi: analytics, and 5G communications.
10.1109/raiic59453.2023.10280846.
[48] C. E. Leiserson, R. L. Rivest, T. Cormen, and C. Stein, Introduction to
Algorithms, vol. 3. Cambridge, MA, USA: MIT Press, 1994.
[49] G. E. Blelloch, ‘‘Prefix sums and their applications,’’ School Comput.
Sci., Carnegie Mellon Univ., Pittsburgh, PA, USA, Tech. Rep., Nov. 1990.
[Online]. Available: http://shelf2.library.cmu.edu/Tech/23445461.pdf
[50] G. Pierre, ‘‘Monitoring of the lubrication system of an aircraft engine
through a prognostic and health monitoring approach,’’ M.S. thesis, School
Ind. Eng. Manag., KTH, 2015.
[51] E. D. Lledo, ‘‘Diagnostic et pronostic de défaillances dans des composants
d’un moteur d’avion,’’ Ph.D. thesis, Toulouse III, 2008. Accessed: Jan. 18,
2024. [Online]. Available: https://www.theses.fr/2008TOU30109
[52] G. Biau and E. Scornet, ‘‘A random forest guided tour,’’ TEST, vol. 25,
no. 2, pp. 197–227, Jun. 2016, doi: 10.1007/s11749-016-0481-7. JING WANG received the Ph.D. degree in engi-
[53] V. N. G. Raju, K. P. Lakshmi, V. M. Jain, A. Kalidindi, and neering from Shanghai Jiao Tong University,
V. Padma, ‘‘Study the influence of normalization/transformation process in 2016.
on the accuracy of supervised classification,’’ in Proc. 3rd Int. Conf. She is currently an Assistant Researcher with
Smart Syst. Inventive Technol. (ICSSIT), Aug. 2020, pp. 729–735, doi: South China Normal University. Her research
10.1109/ICSSIT48917.2020.9214160.
interests include system reliability and engineering
[54] A. Kumari, S. Tanwar, S. Tyagi, N. Kumar, M. Maasberg, and
optimization.
K.-K.-R. Choo, ‘‘Multimedia big data computing and Internet of Things
applications: A taxonomy and process model,’’ J. Netw. Comput. Appl.,
vol. 124, pp. 169–195, Dec. 2018.
[55] A. Kumari, S. Tanwar, S. Tyagi, and N. Kumar, ‘‘Verification and vali-
dation techniques for streaming big data analytics in Internet of Things
environment,’’ IET Netw., vol. 8, no. 3, pp. 155–163, May 2019.
[56] A. Kumari, S. Tanwar, S. Tyagi, and N. Kumar, ‘‘Fog computing for
Healthcare 4.0 environment: Opportunities and challenges,’’ Comput.
Electr. Eng., vol. 72, pp. 1–13, Nov. 2018.
[57] A. Kumari, S. Tanwar, S. Tyagi, N. Kumar, M. S. Obaidat, and
J. J. P. C. Rodrigues, ‘‘Fog computing for smart grid systems in the 5G
environment: Challenges and solutions,’’ IEEE Wireless Commun., vol. 26,
no. 3, pp. 47–53, Jun. 2019.

YIFENG ZHANG was born in Guangdong, China, SHUHUAI GU was born in Guangdong, China,
in 2002. He is currently pursuing the B.S. degree in 2003. He is currently pursuing the B.S. degree
in data science and engineering with South China in data science and engineering from South China
Normal University. Normal University.
His research interests include big data applica- His research interests include system reliability
tions, distributed data processing, machine learn- and engineering optimization, machine learning,
ing, and aviation data analysis. and aviation data analysis.

VOLUME 12, 2024 66397