ICRAT 2024 Nanyang Technological University, Singapore

Distributionally Robust Ground Delay Programs with

Learning-Driven Airport Capacity Predictions
Haochen Wu, Max Z. Li Xinting Zhu, Shuchang Li, Ying Zhou, Lishuai Li
University of Michigan City University of Hong Kong
Ann Arbor, MI, USA Kowloon Tong, Hong Kong
{haocwu, maxzli}@umich.edu {xt.zhu, shuchanli2-c, y.zhou, lishuai.li}@cityu.edu.hk
arXiv:2402.11415v1 [math.OC] 18 Feb 2024

Abstract—Strategic Traffic Management Initiatives (TMIs) such operational (e.g., runway availability and traffic volume [3])
as Ground Delay Programs (GDPs) play a crucial role in sources. Although a large variety of prior work focuses on, e.g.,
mitigating operational costs associated with demand-capacity airport runway configuration prediction [4], weather impact
imbalances. However, GDPs can only be planned (e.g., duration,
delay assignments) with confidence if the future capacities at predictions [2], and TMI implementation predictions [5], all
constrained resources (i.e., airports) are predictable. In reality, predictions result in a probability distribution over potential
such future capacities are uncertain, and predictive models may outcomes. If such predictions were to be incorrect (e.g., due
provide forecasts that are vulnerable to errors and distribution to distribution shifts or misspecification), the resultant GDP
shifts. Motivated by the goal of planning optimal GDPs that are implementation may be sub-optimal.
distributionally robust against airport capacity prediction errors,
we study a fully integrated learning-driven optimization frame- A. Motivation and research problem
work. We design a deep learning-based prediction model capable
of forecasting arrival and departure capacity distributions across In this work, we are motivated by the goal of leveraging
a network of airports. We then integrate the forecasts into a advancements in machine learning (ML)-based predictions
distributionally robust formulation of the multi-airport ground for airport capacities while adopting a cautiously optimistic
holding problem (DR -MAGHP). We show how DR -MAGHP can
outperform stochastic optimization when distribution shifts occur, approach towards prescribing GDP solutions. The former ac-
and conclude with future research directions to improve both the knowledges the adoption of ML models which enables data-
learning and optimization stages. driven predictive models for aviation (a key tenant in the
Keywords—Air traffic management; Ground Delay Programs Federal Aviation Administration’s Information-Centric NAS
(GDPs); Airport capacity prediction; Distributionally robust opti- vision [6]). Moreover, the latter acknowledges the need to
mization make robust decisions, understanding that such probabilistic
predictions may be incorrect.
I. I NTRODUCTION ML has continued to see significant developments and
Congestion in air transportation systems results from applications in the field of aviation, particularly concerning
demand-capacity imbalances, often stemming from airport airport and airspace operations. By using historical and real-
capacity reductions. Within the US National Airspace System time data, ML models can help airlines and airport operators
(NAS), the strategic implementation of Traffic Management anticipate and mitigate flight delays. ML is aiding in the
Initiatives (TMIs) seeks to reduce the operational costs of such development of more advanced ATM systems, which can
imbalances. A prominent example of TMIs is Ground Delay predict traffic flows and flight delays, optimize flight paths,
Programs (GDPs), which aims to strategically delay flights on and improve overall airspace efficiency. The European Union’s
the ground at the origin airport and mitigate costly airborne SESAR (Single European Sky ATM Research) and the United
delays. The optimal implementation of GDPs is the purview States’ NextGen programs have been working on integrating
of airport ground holding optimization problems, or GHPs. ML to enhance ATM. However, most existing ML methods
Airport capacities at future time periods play a significant can only generate point estimations for prediction, which do
role in GDP implementations. If such capacities are known, the not fully characterize the uncertainties brought by the dynamic
optimal delay allocation decisions can be found by solving and complex nature of air transportation systems.
GHPs [1]. However, in practice, it is extremely difficult for Existing methods for optimizing GDPs include the de-
traffic management decision-makers to ascertain future airport terministic multi-airport ground holding problem (MAGHP)
capacities (i.e., Airport Arrival Rates and Airport Departure and the stochastic MAGHP; the former assumes deterministic
Rates) due to myriad uncertainties. Such uncertainties stem airport capacities, while the latter relaxes this assumption.
from environmental (e.g., convective weather forecasts [2]) and The feasibility of implementing ML models in real-world

1 1
H. Wu was partially supported by a Departmental Fellowship from the University of
Michigan. X. Zhu, S. Li, and Y. Zhou were partially supported by Hong Kong Innovation
and Technology Commission Innovation and Technology Fund (GHP/145/20) and City
University of Hong Kong Internal Fund (PJ9678283).
ICRAT 2024 Nanyang Technological University, Singapore

settings is compromised when the downstream learning-driven recently, data-driven approaches such as [19]. Other related
optimization model is highly sensitive to predictions from the work focused on predictions of, e.g., traffic flow [20]. Few
ML model. This sensitivity leads to decision-making based works have been conducted on the prediction of real-time
on potentially inaccurate information (e.g., inaccurate airport airport capacities [21], and nascent work has been done on
capacity predictions). Consequently, it is critical for GDP distribution predictions for flight delays [22].
solutions to acknowledge and address uncertainties in airport
capacity prediction models. II. C ONTRIBUTIONS
To account for distributional uncertainties in airport capacity In this work, our contributions are as follows:
predictions, we investigate a distributionally robust version of
the MAGHP, termed the distributionally robust multi-airport 1) We develop an ML framework for providing distribu-
ground holding problem (DR -MAGHP). Distributionally ro- tional predictions of the airport capacity, rather than a
bust optimization (DRO) aims to identify the optimal solution single value.
by considering the worst-case (with respect to the objective 2) We formulate and solve the DR -MAGHP given input
function) distribution within a predefined set of distributions. capacity distributions from the ML model.
This predefined set is known as the ambiguity set, and is 3) We conduct a sensitivity analysis to evaluate the perfor-
constructed based on the predicted airport capacity distribution mance of DR -MAGHP under varying levels of airport
from the upstream ML model. In Section III we will rigorously capacity overestimation.
restate relevant terms above.
III. M ETHODOLOGY
B. Background and prior works We first introduce our airport capacity distribution prediction
The deterministic MAGHP, stochastic MAGHP, and other model. The predicted distributions serve as inputs to the DR -
MAGHP variations have been well studied [1]. Stochastic MAGHP. Figure 1 provides an overview of our combined
implementations include using two-stage stochastic program- learning-driven optimization framework.
ming and chance-constrained programming [7, 8]. Moreover,
A. Airport capacity distribution prediction
[9] develops a data-driven control framework that not only
minimizes the total cost but redistributes delays spatially, We develop a deep learning model for airport capacity distri-
potentially improving operational recovery capabilities. [10] bution prediction. The model can forecast arrival and departure
and [11] also take equity, fairness, and passenger-centric capacity distributions across a 12-hour prediction horizon, dis-
considerations into account. cretized in 15-minute intervals. A separate prediction model is
In the operations research literature, DRO is emerging as built for each airport to account for airport-specific differences
a rigorous way to combine elements of stochastic and robust [23]. Furthermore, for each airport, separate and independent
optimization approaches. [12] proposes moment-based DRO, models are built for arrival and departure capacity predictions.
constructing the ambiguity set using the first and second mo- 1) Deriving actual capacities from throughput: In order to
ments of the distribution. Alternatively, [13] builds a Wasser- train and validate the prediction model, we need observations
stein distance-based ambiguity set: This approach considers all of the actual, true airport capacity. A unique challenge for
distributions within a Wassertein ball of radius ϵ, and proposes this prediction problem is that we do not directly observe the
a convex reformulation technique to recover a tractable form of actual capacity value from historical data [24]. In this study, we
the original DRO problem. [14] studies the Wassertein ambi- derive estimates of the actual capacity from historical records
guity set-based distributionally robust mixed-integer program of airport arrival and departure throughput. The throughput at
and solves it using dual decomposition methods. time t (observable) typically are underestimations of actual
Airport capacity can be defined as the maximum sustainable capacity at time t (unobservable, to be estimated), particularly
throughput for arriving (the Airport Arrival Rate, or AAR) and during off-peak periods. To address this, we use two rules
departing (the Airport Departure Rate, or ADR) flights [15]. as defined in (1) to filter out time periods with low-volume
In contrast to declared capacities obtained from theoretical of flight operations and select peak operational times when
analyses or statistical approaches [16], real-time capacity is throughput generally reaches its maximum to estimate the
dynamic and challenging to predict in advance. Real-time actual airport capacity:
airport capacities are influenced by several interconnected \ t = throughputt ⇐⇒ (Rule 1) ∨ (Rule 2),
capacity (1)
operational and environmental factors [17]. As the prediction
time horizon increases, forecast uncertainty grows as well, where Rule 1 is given by demandt ≥ throughputt + 3 and
rendering accurate long-term predictions difficult [18]. A sam- Rule 2 by (avg. delayt > 30) ∧ (no. of delayed flightst > 1).
pling of previous work include analytical approaches such as Only data from time periods satisfying (1) are used to train
the Integrated Airport Capacity Model (IACM) [15], and more and validate the airport capacity prediction model.

2 2
ICRAT 2024 Nanyang Technological University, Singapore

Fig. 1: Learning-driven airport capacity distribution prediction and distributionally robust GDP optimization framework.

2) Probabilistic capacity prediction: Our model includes

input feature engineering, a multilayer perceptron (MLP)-
based prediction model, and output feature engineering. De-
z
note by ξb{a,g},t the predicted airport arrival (a) or departure (g)
capacity distribution at time t for airport z. We seek to learn a
mapping Ψz{a,g} from inputs Xtz to capacity outputs ξb{a,g},t
z
, or
z
explicitly, ξb{a,g},t = Ψz{a,g} (Xtz ). On the model output, the
z (a) Gaussian (b) Gamma
airport capacity ξb{a,g},t , we introduce a feature engineering
z
technique for distributional prediction: ξb{a,g},t is represented
as a set of discrete value, encoded via a one-hot encoding
method. This process converts scalar time-indexed capacity
values into high-dimensional binary vector representations. For
example, as shown in Figure 1, a capacity of 2 is encoded into a
sparse vector as [0 0 1 0 0 0]t , where the length of this vector
is 6, corresponding to the range of capacity values observed
at this airport historically. (c) Erlang (d) Lognorm
Due to data limitations (e.g., historical runway configu- Fig. 2: Distributions in the Wasserstein ambiguity set.
rations could not be obtained), we only use meteorological
conditions for the model input Xt . Weather data are converted
and vectorized, with the following seven features: ceiling, B. Distributionally robust MAGHP (DR -MAGHP)
visibility, vertically integrated liquid (VIL), temperature, dew Recall that we seek to make optimal ground delay allocation
points, wind direction, and surface wind speeds. The selection decisions that are robust with respect to the predicted airport
of these features is based on their studied impacts on airport capacity distribution (output of Section III-A). To do so, we
capacity (e.g., [25, 26]). will solve the MAGHP under a “worst-case” capacity distribu-
For the prediction models Ψz{a,g} , we use a 3-layer MLP tion, located within the Wasserstein ambiguity set centered at
whose architecture details are given in Table I. The output the predicted distribution. Figure 2 shows a numerical example
layer of the MLP reflects the range of airport capacity classes, of different distributions—and distribution families—which lie
similar to the multiclass classification problem setting. within the ambiguity set with radius ϵ = 0.005 centered at
a Gaussian distribution N (20, 3). Formally, let M (Ξ) be the
space of all probability distributions Q with support Ξ. The
No. of layers No. of neurons Activation Output Loss
3 [17, 32, M AX i + 1] ReLU Softmax Cross-entropy
Wasserstein distance dw : M (Ξ) × M (Ξ) → R≥0 is the min-
imum transportation cost between distributions Q1 ∈ M (Ξ)
TABLE I: 3-layer MLP architecture for airport capacity dis- and Q2 ∈ M (Ξ) [13], and is given explicitly as:
tribution prediction. M AX i indicates the historical maximum Z
capacity of airport i, with indexing normalized.
dw (Q1 , Q2 ) = inf ∥ξ1 − ξ2 ∥2 Π(dξ1 , dξ2 ), (2)
Π∈DΠ (ξ1 ,ξ2 ) Ξ2

3 3
ICRAT 2024 Nanyang Technological University, Singapore

where Π is a joint distribution of random variables ξ1 this, the full DR -MAGHP can be written as follows:
and ξ2 with marginals Q1 and Q2 , respectively. We denote 
DΠ (ξ1 , ξ2 ) as the set of all joint distributions on ξ1 and ξ2
X h  i
min (Cg gf + Ca af ) + max Ep Q u, ξ (g)
with nmarginals Q1 and o Q2 . Let Z be the set of all airports u,v 
f ∈F
p∈Pϵg (P
b(g) )
and ξb1 , ξb2 , . . . , ξb|Z| be the set of |Z| airport capacities )
h  i
with
 the corresponding estimated probabilities of occurrence + max Ep Q v, ξ (a) (4a)
pb1 , pb2 , . . . , pb|Z| (i.e., this is precisely the predicted capacity p∈Pϵa (Pb(a) )

distribution’s probability mass function, or PMF). The ambi- X

s. t. uf,t = 1, ∀f ∈ F, (4b)
guity set centered around a predicted   capacity distribution P
b
t∈Tfd
with radius ϵ > 0, denoted as Pϵ P , is given byb X
vf,t = 1, ∀f ∈ F, (4c)
t∈Tfa

  n   o gf ′ + af ′ − sf ′ ≤ af , ∀(f, f ′ ) ∈ C, (4d)
Pϵ Pb := Q ∈ M (Ξ) : dw Pb, Q ≤ ϵ . (3)


where Q u, ξ (g) is:

Note that to differentiate between arrival and departure capac-  

(g)
XX
ities, we will use subscripts and superscripts with a and g, min Cg yz,t ξ (g) (5a)
y (g)
respectively, when we write out the full DR -MAGHP model. z∈Z t∈T
   
We refer readers to [27] for additional technical details on (g)
X
s. t. uf,t ≤ Dz,t ξ (g) + yz,t ξ (g) , (5b)
specifying the ambiguity set. f :zfg =z
We denote the set of flights by F , Dz (t) and Rz (t) as  
∀ z, t, ξ (g) ∈ Z T Ξ(g) ,
the departure and arrival capacity of airport z ∈ Z at time    
t, respectively, and C = F × F as the set of all flight pairs (g)
yz,t ξ (g) ≥ 0, ∀ z, t, ξ (g) ∈ Z T Ξ(g) , (5c)
connected by the same aircraft (or tail), where (f, f ′ ) ∈ C
denotes the preceding flight f and successive flight f ′ . The


decision variables uf,t and vf,t are binary variables where and Q v, ξ (a) is analogous to Q u, ξ (g) but for the arrivals
uf,t equals one if flight f will departure at time t; similarly, end. Note that we use different ambiguity set sizes between
vf,t equals one if flight f will land at time t. zfg and zfa arrival and departure capacity distributions.
represent flights scheduled to depart or land at airport z, The objective function (4a) includes two inner maximiza-
respectively. df is the scheduled departure time of f and rf is tion problems, which seek the worst-case distribution within
the scheduled arrival time of f . Tfd is the set of available time each ambiguity set that maximizes the expected second stage
periods for f to take off and Tfr is the set of available time cost. Constraints (4b)-(4d) are the first stage assignment and
periods for f to land. gf , af are ground holding
P delay and coupling constraints inherited from the standard, deterministic
airborne delay, respectively, where gf = t∈T d tuf,t − df MAGHP. The first constraint in the second stage minimization
f
problem ensures that, even if the airport capacity is reduced,
P
and af = t∈T a tvf,t − rf − gf . These definitions follow the
f
standard formulation of the MAGHP [1]. the number of departing (or arriving) flights at time t does
not exceed the realized airport capacity, plus the total number
We incorporate predicted airport departure and arrival ca- of extra flights allowed to depart (or arrive) at time t, across
pacity distributions from Section III-A, along with robustness all airports. When there is a drop in airport capacity, the two-
guarantees, through a two-stage formulation. In the two-stage stage model will optimally adjust delay allocations based on
setting, in addition to the first stage decision variables uf,t the weighting between airborne and ground delays (typically
(g)
and vf,t , we introduce second stage decision variables yz,t airborne delays are more expensive [28]).
(a) (g)
and yz,t . Decision variables yz,t denote the additional number
of flights entering the departure queue at time period t, while C. Scenario reduction and DR -MAGHP reformulation
(a)
yz,t represents the additional number of flights entering the Recall that we are making decisions across a prediction
arrival queue at the same time period. The second stage of the horizon of 12 hours, subdivided into 48 time periods. From
model focuses on reallocating the actual departure and arrival the airport capacity distribution prediction models (Section
times of flights once the actual airport capacity distributions III-A), we are given new predicted distributions of the arrival
are realized. The objective of the second stage model is to and departure capacities (and hence, associated Wasserstein
minimize the number of flights joining either queues. With ambiguity sets) at each time, across all airports.

4 4
ICRAT 2024 Nanyang Technological University, Singapore

This approach results in a substantial increase in the number details in Table I; hyperparameter turning is done via grid
of scenarios, leading to poor scalability and computational search, with a resultant learning rate of 0.0001, 300 epochs,
intractability. To reduce the number of scenarios, we focus and a batch size of 16.
on clustering time periods together with similar capacity We use three metrics—Root Mean Squared Error (RMSE),
distributions. We utilize the Wasserstein distance to quantify Coverage Rate (CR), and Average Confidence Interval Length
the pairwise similarity between capacity distributions of two (ACIL)—to evaluate model performance. Briefly, RMSE mea-
consecutive time periods. Highly similar time periods are sures model performance with respect to a single, likeliest pre-
grouped together, whereas a distinct time period (marked by a dicted capacity, ignoring the rest of the predicted distribution.
significant change in predicted airport capacity distributions) is In contrast, CR and ACIL incorporates differences between
demarcated when the pairwise similarity exceeds a predefined the predicted and actual capacity distributions through the use
threshold. The representative capacity distribution for a group of confidence intervals. The deep learning model attempts to
is its average (centroid) capacity distribution. predict airport capacity distributions that minimize the RMSE,
After scenario reduction, we reformulate the DR -MAGHP increase the CR, and reduce the spread of the ACIL.
in (4a)-(4d) by converting the inner second-stage maximization 3) Capacity distribution prediction results: Table II summa-
problem into a minimization problem, resulting in a semi- rizes the model performance across the test set for the US Core
infinite program. We then apply discretization techniques to 30 airports on December 31, 2019. A salient future research
address the continuous supports of this semi-infinite program, direction would be to tune specific ambiguity sets within the
resulting in a deterministic equivalent form for the DR - DR -MAGHP in response to the prediction performance at
MAGHP. Due to page limitations, we omit the technical individual airports. Furthermore, these performance metrics
details and refer readers to [27]. can guide future prediction model enhancements (e.g., using
refined prediction models for airports with large RMSEs such
IV. E XPERIMENTAL RESULTS AND DISCUSSION as IAD and CLT, or low CR with a large ACIL).
A. Capacity distribution prediction We select Los Angeles International (LAX) Airport as a
1) Data description: We obtained airport throughput data case study to visualize the prediction results. Figure 3 plots the
from the US Department of Transportation’s Bureau of Trans- actual estimated departure capacity (red dots) and the predicted
portation Statistics (BTS), and weather data from the US departure capacity distribution for LAX across December 31,
National Oceanic and Atmospheric Administration’s High- 2019. At 18:00 local time, we observe a rapid decline in
Resolution Rapid Refresh (HRRR) database. BTS provides the predicted capacity, which aligns with the estimated actual
detailed information for each flight, including the scheduled capacity: During this time, LAX experienced a sudden change
departure and arrival times, actual departure and arrival times, in the wind direction—this most likely triggered a change in
delay duration, cancellation status, among others. Using these the airport runway configuration. Furthermore, the predicted
data points and the procedure described in Section III-A1, capacity distributions appear to be relatively stable, which
we estimate the actual airport capacity at each airport of reinforces the validity of the scenario reduction procedure
interest. In this study, we examine the FAA Core 30 airports, described in Section III-C.
but both the prediction and optimization frameworks can be
easily extended to a larger network of airports. HRRR provides
weather data on a 3 km × 3 km grid covering all 50 US states,
with a forecast horizon of up to 23 hours from the current hour.
We collect data for the entire year of 2019. Each day is divided
into 96 quarter-hour intervals, resulting in 35,040 time periods
in total. We use 2019 weather data from January 1-December
30 to train the capacity distribution prediction models. We
reserve December 31, 2019 as the day across which we will
forecast airport capacity distributions that will serve as inputs
for the DR -MAGHP model.
2) Prediction experimental setup and results: We desig- Fig. 3: Predicted departure capacity distributions for LAX on
nate December 31, 2019 as the test set to evaluate model December 31, 2019. Blue area from 9:00-21:00 is the 12-hour
performance. For training-validation sets, we randomly split solution window of the DR -MAGHP.
the remaining data into a standard ratio of 80:20. We then
normalize all numeric features on the training set using min- B. DR -MAGHP optimization
max normalization. We use the same normalization procedure 1) Experiment setup: To ensure consistency with the ca-
for the test set as well. We give the 3-layer MLP architecture pacity distribution prediction procedure, we also use BTS

5 5
ICRAT 2024 Nanyang Technological University, Singapore

Airport Arrival Departure

RMSE CR (%) ACIL (µ ± σ) RMSE CR (%) ACIL (µ ± σ)
ATL 5.00 100 25.95 ± 0.92 5.43 100 26.46 ± 0.84
BOS 3.21 91 10.18 ± 0.39 2.42 100 11.83 ± 0.37
BWI 1.78 100 7.50 ± 0.50 1.73 100 9.50 ± 0.50
CLT 6.41 100 22.53 ± 0.75 11.61 93 24.00 ± 0.00
DCA 2.72 100 10.20 ± 0.40 2.65 100 11.14 ± 0.35
DEN 7.39 100 23.39 ± 0.89 9.09 86 26.86 ± 1.12
DFW 8.87 96 25.91 ± 1.56 6.82 100 28.47 ± 0.70
DTW 6.55 100 19.85 ± 1.66 8.85 91 22.36 ± 0.88
EWR 2.59 94 10.35 ± 0.48 3.33 95 12.00 ± 0.00
FLL 3.69 91 8.36 ± 1.23 3.12 100 8.42 ± 0.64
HNL 2.05 100 8.00 ± 0.94 3.11 100 9.00 ± 0.00
IAD 11.12 100 18.33 ± 0.47 7.31 100 21.20 ± 2.71
IAH 6.77 100 19.87 ± 0.81 9.59 100 22.73 ± 0.45
JFK 2.38 100 9.00 ± 0.00 3.44 100 13.57 ± 1.68
LAS 2.75 100 10.57 ± 0.62 2.69 100 11.14 ± 0.64
LAX 2.77 100 15.05 ± 0.86 3.42 95 15.77 ± 0.42
LGA 3.21 100 11.33 ± 0.47 2.87 100 12.29 ± 0.45
Fig. 4: In-sample performance of SP -MAGHP and DR -
MCO 3.13 86 9.29 ± 0.96 2.06 100 10.58 ± 0.64 MAGHP based on predicted capacity distributions
MDW 1.00 100 6.25 ± 0.43 2.68 88 7.88 ± 0.32
MEM 0.71 100 7.00 ± 0.00 2.00 100 8.00 ± 0.00
MIA 3.52 92 10.25 ± 0.83 1.47 100 12.33 ± 1.60
MSP 4.11 100 17.44 ± 0.50 5.59 100 21.00 ± 0.00 grow (i.e., increasing radii ϵ(a) , ϵ(g) ). When ϵ(a) = ϵ(g) = 0,
ORD 3.73 100 25.48 ± 0.50 6.46 97 28.34 ± 1.59
PHL 2.26 100 13.09 ± 0.51 5.67 80 13.70 ± 0.90
the ambiguity sets are singletons containing only the pre-
PHX 3.70 100 14.84 ± 1.09 4.09 94 14.59 ± 0.49 dicted distributions. Hence, the in-sample performance of SP -
SAN 1.80 100 8.11 ± 0.99 2.85 100 11.80 ± 0.40
SEA 2.59 100 13.19 ± 0.39 2.35 100 14.94 ± 0.23 MAGHP is identical to that of DR -MAGHP when the ambi-
SFO 3.38 93 11.93 ± 0.59 3.48 93 13.14 ± 0.35
SLC 2.81 100 15.78 ± 1.47 3.35 100 20.75 ± 0.83 guity set contains only the predicted distribution, as expected.
TPA 1.63 100 6.00 ± 1.15 2.52 100 10.00 ± 1.63 As the ambiguity sets grow, DR -MAGHP tends to generate
TABLE II: Capacity distribution prediction performance for more conservative ground holding policies, which increases
FAA 30 core airports on Dec 31, 2019. RMSE and ACIL are delay costs when compared to SP -MAGHP.
given in aircraft per 15 minutes.
Algorithm 1 Capacity Resampling Algorithm
data from December 31, 2019 to construct flight schedules. 1: Input: Predicted PMF {b p(1) , pb(2) , . . . , pb(|Z|) } of each air-
The solve horizon for the DR -MAGHP extends over twelve port’s capacity {ξb(1) , ξb(2) , . . . , ξb(|Z|) } with mean values
hours (09:00:00 to 21:00:00 local time), and is divided into {µ(1) , µ(2) , . . . , µ|Z| }, capacity reduction level r for all
forty-eight time periods (|T | = 48), each representing fifteen airports and a maximum variability rate of probability δ.
minutes. We optimize over 10,869 flights associated with the 2: for each airport z do
FAA Core 30 airports (|Z| = 30). We set realistic minimum 3: Target mean µ∗ = µ b(z) (1 − r)
turnaround times (45 minutes), and follow standard MAGHP 4: Update weights for each support p(z) :
setups such as an additional time period to absorb excess
|ξb(z) |
flights. (z)
X (z)
p = arg min pi ξbi
2) Results and comparisons: We define the outputs from p
i=1
the upstream capacity distribution prediction procedure as
|ξb(z) | |ξb(z) |
predicted capacity distributions. To test the performance of X (z)
X
the DR-MAGHP when predicted capacity distributions are s.t. pi ξbi ≥ µ∗ , pi = 1
i=1 i=1
not true capacity distributions, we define the realized capacity (z) (z) (z) (z)
distributions as testing distributions. Testing distributions of pi − pbi ≤ δb
pi , −δb
pi ≤ pi − pbi , ∀i ∈ N
interest are the cases where realized capacity distributions are (6)
shifted to the left, i.e., the realized probabilistic capacities 5: end for
are lower than anticipated. We also develop the stochastic 6: Draw i.i.d samples ξ¯(z) ∼ p(z) , z = 1, 2, . . . , |Z|
programming based MAGHP (SP -MAGHP), which is the 7: Output: Reduced testing capacities for all airports
MAGHP model generating ground holding policies based on {ξ¯(1) , ξ¯(2) , . . . , ξ¯(|Z|) }
predicted capacity distributions, to compare its delay costs with
those of the DR-MAGHP when predicted capacity distribu- We provide a more operationally interpretable plot in Figure
tions are not accurate. We use terms in-sample performance 5, which depicts how GDP-induced delays change between
(ϕIS ) and out-of-sample performance (ϕOS ) to refer to the the SP -MAGHP and DR -MAGHP solutions. Each directional
costs (i.e., optimal value in expectation) of SP -MAGHP and edge represents the percentage of flights delayed across all
DR -MAGHP when evaluated on predicted distributions and direct flights from one airport node to another. Comparing
testing distributions respectively. Figures 5a and 5c reveals that larger ambiguity sets lead to
Figure 4 plots the in-sample performance of DR -MAGHP more assigned delays across the board. We emphasize that
compared to SP -MAGHP as the size of the ambiguity sets these comparisons and interpretations must be made on a day-

6 6
ICRAT 2024 Nanyang Technological University, Singapore

Reduction ϕsp
OS ϕdr
OS ϵ∗ % dec.
10% 331469.45 331443.65 0.01 0.007
20% 395454.05 395420.20 0.01 0.009
30% 473078.15 473068.50 0.02 0.002
40% 566420.20 566335.80 0.02 0.015
50% 673100.10 672943.65 0.04 0.023
60% 792804.80 792645.95 0.02 0.020
70% 935837.35 935473.75 0.04 0.038
80% 1105686.30 1104923.10 0.09 0.069
90% 1278823.30 1277608.70 0.09 0.095
100% 1454765.00 1453077.00 0.09 0.116
(a) Percent increase in flights subject (b) Percent decrease in flights subject
TABLE III: Sensitivity analysis results.

(c) Percent increase in flights subject (d) Percent decrease in flights subject

Fig. 5: Percent increase in delayed flights under DR -MAGHP (a) 10% reduction in capacities. (b) 50% reduction in capacities.
versus SP -MAGHP with (a) ϵ = 0.1, (c) ϵ = 0.5; Percent
Fig. 6: Out-of-sample performances of SP -MAGHP and DR -
decrease in delayed flights under DR -MAGHP versus SP -
MAGHP S for two levels of capacity reductions.
MAGHP with (b) ϵ = 0.1, (d) ϵ = 0.5.

by-day basis, just as unique GDP policies must be developed away from predicted ones, larger ambiguity sets are needed. It
for each encountered NAS scenario. is important to note that the out-of-sample performances of SP -
3) Sensitivity to distribution shifts: To evaluate how vari- MAGHP and DR -MAGHP, when radii are set to zero, might
ations in the discrepancies between predicted and realized not align, even if their in-sample performances appear identical
capacity distributions influence delay costs, and to determine under the same conditions. This discrepancy arises because
the effectiveness of DR -MAGHP in mitigating these effects, the specific delay assignment policies implemented by SP -
we perform a sensitivity analysis. This analysis generates MAGHP and DR -MAGHP based on predicted distributions
testing distributions at various levels of capacity reductions to may vary. These differences can lead to identical in-sample
compare the out-of-sample performance of SP -MAGHP and performance, yet diverge in out-of-sample performance.
DR -MAGHP. Table III gives the out-of-sample performances (e.g., delay
To sample from reduced capacity distributions at various assignment costs for SP -MAGHP and DR -MAGHP, respec-
reduction levels, we introduce a linear program in (6) that tively) for different realized capacity distributions with reduced
performs valid adjustments to the PMFs to minimize the capacities. The best ambiguity set size ϵ∗ is also given.
deviation between the current mean value and the targeted Although the cost reductions may seem modest when utilizing
(reduced) mean value. We also introduce a parameter δ for the DR -MAGHP, we note that this was across an arbitrary day
maximum variability rate to ensure a more uniform distribution (December 31, 2019) in the NAS. A full analysis of the
of probability mass. The details of this procedure are outlined advantages gained by robustifying ground holding decisions
in Algorithm 1. We then draw 100 samples from the reduced with ML-driven airport capacity prediction inputs will require
capacity distributions, and evaluate the out-of-sample perfor- performing this experiment across a longer time horizon.
P|ξ̄| ¯ ¯ ¯
mance ϕOS (x̄) = i=1 ϕd OS (x̄, ξi )/|ξ|, where ϕOS (x̄, ξi ) is
d The goal of this paper is to provide the foundations for
the objective function value of the derived ground holding robust, learning-driven optimization approaches to strategic
policy x̄ with reduced capacity sample ξ¯i . ATM considering uncertain airport capacities. Should users
We examine erroneous predictions where realized capacities trust predictive models, then the robustness aspects can (and
are reduced between 10-100% (note that a GDP with 0 AAR should, to avoid overly-conservative solutions) be dialed down.
is essentially a Ground Stop). When the realized capacities are On the other hand, should users not trust prediction model
only slightly lower than forecast (i.e., 10-20% reductions), only outputs (e.g., current conditions are particularly unstable, or the
a small ambiguity set (i.e., small ϵ) is needed to modestly out- prediction horizon is long), distributionally robust approaches
perform SP -MAGHP. As the realized capacities move farther can be considered.

7 7
ICRAT 2024 Nanyang Technological University, Singapore

V. C ONCLUDING REMARKS [11] A. Jacquillat, “Predictive and prescriptive analytics toward

passenger-centric ground delay programs,” Transportation Sci-
In this paper, we introduce a framework that merges an
ence, vol. 56, no. 2, pp. 265–298, 2022.
upstream airport capacity prediction model with a downstream [12] E. Delage and Y. Ye, “Distributionally robust optimization under
distributionally robust ground delay program optimization moment uncertainty with application to data-driven problems,”
model. The output of this learning-driven optimization frame- Operations Research, vol. 58, no. 3, pp. 595–612, 2010.
work is a policy for assigning ground and airborne delays to [13] P. M. Esfahani and D. Kuhn, “Data-driven distributionally
robust optimization using the wasserstein metric: Perfor-
a set of flights, robustified against distributional uncertainties
mance guarantees and tractable reformulations,” arXiv preprint
in the predicted capacities. We use a neural network approach arXiv:1505.05116, 2015.
(multilayer perceptron model) to produce distributional pre- [14] K. Kim, “Dual decomposition of two-stage distributionally
dictions of future airport capacities: The prediction is not a robust mixed-integer programming under the wasserstein am-
single capacity, but instead a set of different capacity scenarios, biguity set,” Preprint manuscript, 2020.
[15] R. Kicinger, J. Krozel, M. Steiner, and J. Pinto, “Airport capacity
each with a probability of occurrence. These distributional
prediction integrating ensemble weather forecasts,” in Infotech
predictions are fed into the constraints of DR -MAGHP, where Aerospace 2012, 2012, p. 2493.
we explore the impacts of ambiguity set sizes and prediction [16] E. P. Gilbo, “Airport capacity: Representation, estimation, opti-
errors (in the form of distribution shifts). Future work includes mization,” IEEE Transactions on Control Systems Technology,
expanding the prediction model inputs to include additional vol. 1, no. 3, pp. 144–154, 1993.
[17] S. Choi and Y. J. Kim, “Artificial neural network models for
operational factors (e.g., runway configurations), improving the
airport capacity prediction,” Journal of Air Transport Manage-
estimation method for deriving “true” airport capacities, and ment, vol. 97, p. 102146, 2021.
determining optimal ambiguity set sizes without exhaustively [18] S.-L. Tien, C. Taylor, E. Vargo, and C. Wanke, “Using ensemble
searching through a large range of ϵ. Additionally, methodolog- weather forecasts for predicting airport arrival capacity,” Journal
ical improvements will be needed to consider non-independent of Air Transportation, vol. 26, no. 3, pp. 123–132, 2018.
[19] J. Cox and M. J. Kochenderfer, “Probabilistic airport acceptance
arrival and departure processes both in the prediction and the
rate prediction,” in AIAA Modeling and Simulation Technologies
optimization steps. Conference, 2016, p. 0165.
R EFERENCES [20] M. C. R. Murca and R. J. Hansman, “Identification, character-
ization, and prediction of traffic flow patterns in multi-airport
[1] P. B. Vranas, D. J. Bertsimas, and A. R. Odoni, “The multi- systems,” IEEE Transactions on Intelligent Transportation Sys-
airport ground-holding problem in air traffic control,” Opera- tems, vol. 20, no. 5, pp. 1683–1696, 2018.
tions Research, vol. 42, no. 2, pp. 249–261, 1994. [21] Y. Wang and Y. Zhang, “Prediction of runway configurations and
[2] R. Kicinger, J.-T. Chen, M. Steiner, and J. Pinto, “Airport ca- airport acceptance rates for multi-airport system using gridded
pacity prediction with explicit consideration of weather forecast weather forecast,” Transportation Research Part C: Emerging
uncertainty,” Journal of Air Transportation, vol. 24, no. 1, pp. Technologies, vol. 125, p. 103049, 2021.
18–28, 2016. [22] Z. Wang, C. Liao, X. Hang, L. Li, D. Delahaye, and M. Hansen,
[3] D. Tittle, P. McCarthy, and Y. Xiao, “Airport runway capacity “Distribution prediction of strategic flight delays via machine
and economic development: A panel data analysis of metropoli- learning methods,” Sustainability, vol. 14, no. 22, 2022.
tan statistical areas,” Economic Development Quarterly, vol. 27, [23] Y. Liu, Y. Liu, M. Hansen, A. Pozdnukhov, and D. Zhang,
no. 3, pp. 230–239, 2013. “Using machine learning to analyze air traffic management
[4] J. Avery and H. Balakrishnan, “Predicting airport runway con- actions: Ground delay program case study,” Transportation
figuration: A discrete-choice modeling approach,” 2015. Research Part E: Logistics and Transportation Review, vol. 131,
[5] K. Vlachou, R. Sharma, and F. Wieland, “Simultaneous traffic pp. 80–95, 2019.
management initiatives: The double delay problem,” in 2019 [24] V. Ramanujam and H. Balakrishnan, “Estimation of arrival-
IEEE/AIAA 38th DASC. IEEE, 2019, pp. 1–6. departure capacity tradeoffs in multi-airport systems,” in Pro-
[6] Federal Aviation Administration (FAA), “Initial Concept of ceedings of the 48th IEEE Conference on Decision and Control
Operations for an Info-Centric National Airspace System,” 2022, (CDC). IEEE, 2009, pp. 2534–2540.
accessed: Feb 2024. [25] S. Allan, J. Beesley, J. Evans, and S. Gaddy, “Analysis of delay
[7] M. O. Ball, R. Hoffman, A. R. Odoni, and R. Rifkin, “A causality at newark international airport,” in 4th USA/Europe
stochastic integer program with dual network structure and Air Traffic Management R&D Seminar, 2001, pp. 1–11.
its application to the ground-holding problem,” Operations [26] Z. Renhe, Q. Li, and R. Zhang, “Meteorological conditions for
research, vol. 51, no. 1, pp. 167–171, 2003. the persistent severe fog and haze event over eastern china in
[8] J. Chen and D. Sun, “Stochastic ground-delay-program planning january 2013,” Science China Earth Sciences, vol. 57, pp. 26–
in a metroplex,” Journal of Guidance, Control, and Dynamics, 35, 2014.
vol. 41, no. 1, pp. 231–239, 2018. [27] H. Wu and M. Z. Li, “Distributionally robust airport ground
[9] C. Chin, M. Z. Li, K. Gopalakrishnan, and H. Balakrishnan, holding problem under wasserstein ambiguity sets,” arXiv
“Airport ground holding with hierarchical control objectives,” preprint arXiv:2306.09836, 2023.
in USA-Europe ATM R&D Seminar, 2021. [28] Airports Council International – North America , “Aircraft
[10] C. N. Glover and M. O. Ball, “Stochastic optimization models Operating and Delay Cost per Enplanement,” 2014, accessed:
for ground delay program planning with equity–efficiency trade- Feb 2024.
offs,” Transportation Research Part C: Emerging Technologies,
vol. 33, pp. 196–202, 2013.

8 8