3rd URSI AT-AP-RASC, Gran Canaria, 29 May – 3 June 2022

Machine Learning Ensemble Approach for Ionosphere and Space Weather Forecasting with
Uncertainty Quantification

Randa Natras*(1) , Benedikt Soja(2) , and Michael Schmidt(1)

(1) Deutsches Geodätisches Forschungsinstitut der Technisches Universität München (DGFI-TUM), Department of Aerospace
and Geodesy, Technical University of Munich, Munich, 80333, Germany
(2) Institute of Geodesy and Photogrammetry, ETH Zurich, Zurich, 8093, Switzerland

Abstract data [1]. Previous work on VTEC forecasting mainly in-

cluded deep learning methods such as Feed-forward Neural
This paper presents a novel Machine Learning (ML) ap- Networks [2] and Long Short-Term Memory (LSTM) [3].
proach to ionospheric forecasting, including forecasting the Deep learning proved to be more accurate for ionosphere
space weather impact on the ionosphere. It exploits a data- modeling than traditional linear approaches of Empirical
driven approach in which the models learn underlying pro- Orthogonal Functions (EOF) [2] and AutoRegressive Inte-
cesses and relationships from data describing solar activity, grated Moving Average (ARIMA) [3]. Only a few studies
solar wind, interplanetary and Earth’s magnetic fields, and so far have used other learning algorithms such as XGBoost
the ionosphere. We applied a multi-model and multi-data [4]. However, the uncertainty of the forecasts has not yet
ensemble forecasting approach using diverse models of dif- been provided. The novelties of this study are primarily the
ferent learning algorithms with different training datasets to application of ensemble modeling and computation of the
generate 1-day VTEC forecasts. This approach improved forecast uncertainties.
forecasting accuracy compared to a single-model-based ap-
proach. In addition, the forecast uncertainty of the super- 2 Methodology
ensemble model was assessed by estimating an ensemble
spread. The results show potential for forecasting VTEC in
In this study, the ML model for VTEC forecasting is based
different ionospheric regions during quiet and storm peri-
on supervised learning, i.e. it learns from the past experi-
ods while quantifying their uncertainties.
ence (represented as the training data) with respect to the
task of the VTEC forecast and the performance measure as
1 Introduction root mean square error (RMSE) and correlation coefficient
(Corr.). Supervised learning can be seen as a function esti-
In order to model the Vertical Total Electron Content mation or predictive learning problem [5]. Using a training
(VTEC) and space weather effects in the ionosphere, a com- sample of the input xi (predictors, features, or the indepen-
plex chain of physical, dynamic processes between the Sun, dent variables) and the output yi (response or the dependent
the interplanetary space, the interplanetary (IMF) and the variable), for each of the N observations (i = 1, 2, . . . , N),
Earth’s (EMF) magnetic fields, and the ionosphere must the goal is to find an approximation F̂(xi ) of the function
be taken into account. However, we have a limited un- F(xi ) that maps inputs to the output (Figure 1). The ML
derstanding of the underlying space weather processes to model parameters and hyperparameters are optimized using
accurately model them using traditional methods. On the cross-validation during the learning phase. When the itera-
other hand, there is an urgent need to develop an advanced tive process of feature selection, model tuning, and training
forecasting system to mitigate possible catastrophic failures is completed, the model can be tested on previously un-
of space- and ground-based technological systems, includ- seen data to forecast VTEC. Publication [6] discusses the
ing GNSS (Global Navigation Positioning System), caused ML workflow for space weather forecasting in more de-
by severe space weather events. Since numerous data from tail, from problem formulation, feature engineering, learn-
satellites and observatories that monitor space weather are ing algorithms to model training, evaluation, and deploy-
available nowadays, they can be exploited for improving ment with challenges and open issues.
space weather forecasting. Machine Learning (ML), a sub-
set of artificial intelligence, is one of the most rapidly grow- Solar activity, solar wind, IMF, and EMF data
ing areas today and most promising recent technologies that were downloaded from NASA/GSFC OMNI-Web
provides new capabilities to learn directly from data. This (https://omniweb.gsfc.nasa.gov/ow.html). The VTEC
approach can help us to discover the hidden relationships values for high-latitude (10E 70N), mid-latitude
within the data, deepen our physical understanding and, ap- (10E 40N), and low-latitude (10E 10N) regions
proximate nonlinear functions in order to describe the un- were extracted from the Global Ionosphere Map
derlying space weather processes based on the provided (GIM) of CODE (IGS AC at the University of Bern)

Authorized licensed use limited to: SRM Institute of Science and Technology Kattankulathur. Downloaded on January 28,2025 at 08:37:12 UTC from IEEE Xplore. Restrictions apply.
 crease for the correct ones. The final outcome represents the
weighted estimations from all iterations. Gradient boost-
ing offers a generalization of boosting to arbitrary differen-
tiable objective functions, where a tree is fitted to the gradi-
ent. XGBoost (eXtreme Gradient Boosting) [9] is an opti-
mized gradient boosting algorithm that uses regularization
to avoid overfitting. A detailed explanation of the meth-
ods and data sets can be found in [10]. These three learn-
ing algorithms were individually trained on the three data
sets, i.e. nine models were developed for each ionospheric
region, resulting in a total of 27 models. The developed
models were then combined into a super-ensemble model,
which provides average forecasts of all models within the
ensemble (Figure 2). Ensemble modeling is a standard ap-
proach for terrestrial weather forecasting. The standard de-
viation of ensemble members with respect to the ensemble
mean, known as ensemble spread, provides an estimate of
the forecasting uncertainties. A large (small) spread indi-
cates a low (high) confidence in the forecast.

Figure 1. Workflow of the ML model development for

VTEC forecasting consisting of a learning/training phase
and a testing phase.

(https://cddis.nasa.gov/archive/gnss/products/ionex). In
order to model the VTEC temporal and seasonal depen-
dencies, hour of day and day of year (DOY) were added as
features to the model. Exponential moving average (EMA)
and time derivatives of VTEC are additional features. The
exponential moving average gives higher weights on the
latest data values, and is, therefore, more sensitive to recent
values and recent value changes than a simple moving
average. The training data cover the period from January
2015 to December 2016, while the testing is performed for
the subsequent year 2017. Three data sets are prepared:
1) Original data (D1); 2) Daily differences for both inputs Figure 2. Overview of the developed ML models, trained
and outputs (D2); 3) Input data combining the first and on different data sets using tree-based learning algorithms.
the second data sets, while the output corresponds to the Afterward, they are combined into a super-ensemble model
original VTEC (D3). that provides the VTEC forecast and its uncertainty.

Learning algorithms from ensemble learning were selected

for the study. Ensemble learning combines several simple 3 Results
models (usually based on a Decision Tree) to produce a ro-
bust model that can generalize better than a single model. An overview of solar activity, solar wind speed, and con-
We used popular ensemble learning methods of bagging ditions in IMF and EMF for September 6 - 13, 2017 is
(Random Forest) and boosting (AdaBoost and XGBoost). presented in Figure 3. Two minimum Dst values can be
Random Forest [7] builds a large collection of de-correlated observed: at the midnight and in the evening of Septem-
trees and then averages them to form a final outcome. In ber 8 (UTC), representing the main phases of geomagnetic
the boosting method, the trees are grown sequentially using storms. Most of the storms happened at September 8. The
the information from previously grown trees with a mod- recovery phase followed from September 9 to 11, to less
ified version of the training data. In AdaBoost (Adaptive stormy conditions on September 12 to 13. On September
Boosting) [8], the data are modified by applying weights 6, the day before the initial phase of the first storm, solar
to each of the training examples (xi , yi ). In each succes- activity was increased, while EMF conditions were mostly
sive iteration, the weights increase for wrongly estimated quiet. Data from September 6 were used to forecast VTEC
observations in the previous iteration, while the weights de- during the initial storm phase on September 7 in Figure 4.

Authorized licensed use limited to: SRM Institute of Science and Technology Kattankulathur. Downloaded on January 28,2025 at 08:37:12 UTC from IEEE Xplore. Restrictions apply.
 timated, and on September 9 (beginning of the recovery
phase), when the VTEC forecast is overestimated. In the
main storm phase, GIM VTEC is mostly within the ensem-
ble spread. The ensemble mean is slightly overestimated
during the main phase. During the recovery phase (Septem-
ber 10 - 13), the mid-latitude GIM VTEC is within the
ensemble spread and corresponds to the ensemble mean.
For the low-latitude region, the differences between the en-
semble mean and GIM VTEC are larger during daytime on
September 8 and 9. An overview of the RMSE of ML mod-
els within the ensemble, the ensemble mean and the mean
variance (2σ ) is provided in Table 1. The super-ensemble
model provides the optimal results.

Table 1. The RMSE of the ML models Random For-

est (RF), AdaBoost (AB), and XGBoost (XGB) for three
datasets, as well as, super-ensemble (SE) and mean vari-
Figure 3. Solar radio flux F10.7, solar wind speed Vsw,
ance (2σ ) for high-latitude, mid-latitude and low-latitude
IMF Bz, and EMF Dst and Kp data for September 6 - 13,
VTEC 1-day forecasts from September 7 to 13, 2017.
2017. Quiet (green): Kp < 3, moderate (yellow): 3 ≤ Kp
< 4, active (orange): 4 ≤ Kp < 5, storm (red): Kp ≥ 5. RF AB XGB SE 2σ
D1 1.62 1.67 1.78
10E 70N D2 1.75 1.69 1.66 1.63 0.90
The VTEC forecasts of different ML models with the en-
D3 1.66 1.76 1.77
semble mean µ and a probabilistic forecast in terms of ±2
times the standard deviations σ can be seen in Figure 4. D1 3.08 3.05 3.00
The period September 7 to 13, 2017, covers the initial, 10E 40N D2 2.82 2.61 2.61 2.80 1.28
main, and recovery phase of the severe geomagnetic storms. D3 2.96 3.22 2.80
The forecast uncertainty is largest during the main storms D1 3.14 3.38 3.20
phases (September 8). Afterwards, the ensemble spread 10E 10N D2 3.31 3.27 3.22 3.13 1.60
gradually narrows from September 9 onwards. The GIM D3 3.20 3.30 3.26

The most influential input variables (features) for the VTEC

forecast are estimated using Shapley values (Figure 5).
SHAP (SHapley Additive exPlanations) [11] is a widely

Figure 5. The ten most important features for mid-latitude

VTEC 1-day forecasts from the XGBoost model estimated
using SHAP for the third test data set (year 2017). "EMA"
Figure 4. 1-day VTEC forecast of the Super-ensemble stands for the exponential moving average over the previous
model (ensemble mean µ) and ensemble spread (µ ± 2σ ) 30 days ("30d") and "Diff. 24h" for daily differences.
for September 7-13, 2017.
used approach from cooperative game theory to interpret
VTEC at high-latitude is mainly within or at the edge of the output of ML models. Features are sorted by impor-
the ensemble spread. The mid-latitude GIM VTEC shows tance, starting with the most important ones at the top. A
greater differences to the ensemble mean on September 7 standard violin plot illustrates the distribution of the im-
(initial storm phase), when the VTEC forecast is underes- portance for each feature. Red stands for large values of a

Authorized licensed use limited to: SRM Institute of Science and Technology Kattankulathur. Downloaded on January 28,2025 at 08:37:12 UTC from IEEE Xplore. Restrictions apply.
 feature, while blue stands for small ones. The x-axis corre- vier, Amsterdam, Netherlands, 22, May 2018, ISBN:
sponds to Shapley values that characterize the average im- 9780128117880.
pact on the model output. Positive Shapley values mean
that a feature increases the VTEC forecast, while negative [2] J. C. Uwamahoro, J. B. Habarulema “Modelling to-
values represent a feature impact that decreases the VTEC tal electron content during geomagnetic storm condi-
forecast. The larger the absolute Shapley values, the higher tions using empirical orthogonal functions and neu-
impact of a feature on the VTEC forecast. The most impor- ral networks,” Journal of Geophysical Research:
tant variables for the mid-latitude VTEC forecast are the Space Physics, 120, 3, December 2015, pp. 794–795,
current VTEC, hour of day, F10.7 solar activity index, ex- doi:10.1002/2015JA021961.
ponentially moving VTEC average and VTEC daily differ- [3] R. Tang, F. Zeng, Z. Chen “The comparison of pre-
ence. Other important variables are DOY, geomagnetic in- dicting storm-time ionospheric tec by three meth-
dices Kp and Dst, sunspot number and daily Dst difference. ods: Arima, lstm, and seq2seq,” Atmosphere, 11, 25,
March 2020, doi:10.3390/atmos11040316.
4 Conclusion
[4] A. V. Zhukov, Y. V. Yasyukevich, A. E. Bykov “Cor-
Today’s society relies on space- and ground-based techno- rection to: Gimli: Global ionospheric total electron
logical infrastructures, which are not adequately protected content model based on machine learning,” GPS So-
against space weather. Therefore, accurate forecasting and lutions, 25, 21, 22, November 2020, pp. 794–795,
early-warning systems of space weather events are urgently doi:10.1007/s10291-020-01063-1.
needed. This paper presents a novel model for forecasting
[5] J. H. Friedman “Greedy function approximation: A
VTEC and space weather effects in the ionosphere based
gradient boosting machine,” The Annals of Statis-
on ML and ensemble modeling. To get the most out of
tics, 29, 5, 1, October 2001, pp. 1189–1232,
the ML model, it is necessary to prepare relevant data and
doi:10.1214/aos/1013203451.
useful features that enhance learning, particularly of space
weather events. Also, it is critical to quantify the uncer- [6] R. Natras, M. Schmidt “Machine Learning Model
tainty of weather forecasts, especially when forecasting ex- Development for Space Weather Forecasting in the
treme weather events. In terrestrial weather forecasting, Ionosphere,” Proceedings of the CIKM 2021 Work-
this is typically achieved with ensemble forecasting sys- shops: 1st Workshop on Complex Data Challenges
tems. Following that approach, ML models were devel- in Earth Observation (CDCEO’21), CEUR-WS.org,
oped using different learning algorithms and different data 3052, 12, December 2021, http://ceur-ws.org/Vol-
sets and then combined via ensemble modeling to produce 3052/short10.pdf
a super-ensemble model with more reliable results that can
quantify their uncertainties. During the severe geomagnetic [7] L. Breiman “Random forests,” Machine Learn-
storm, the forecast of the super-ensemble ML model is less ing, 45, 1, 1, October 2001, pp. 5–32,
confident than after the storm. Most of the time, the mean doi:10.1023/A:1010933404324.
and the spread of the ensemble corresponds to the refer-
[8] Y. Freund, R. E. Schapire “A decision-theoretic
ence VTEC values. The largest differences occur during
generalization of on-line learning and an applica-
the initial storm phase and at the beginning of the recovery
tion to boosting,” Journal of Computer and Sys-
phase. The changes in the interplanetary and geomagnetic
tem Sciences, 55, 1, 1, August 1997, pp. 119–139,
fields were short-term and insufficient information on these
doi:10.1006/jcss.1997.1504.
changes was provided in the input data from a day before.
However, during the main storm phase, the reference VTEC [9] T. Chen, C. Guestrin “Xgboost: A scalable tree
does not deviate much from the ensemble spread. After the boosting system,” Proceedings of the 22nd ACM
storm, the ensemble spread is narrower and the ensemble SIGKDD international conference on knowledge dis-
mean agrees with the reference VTEC. covery and data mining, 1, August 2016, pp. 785–794,
doi:10.1145/2939672.2939785.
5 Acknowledgments
[10] R. Natras, B. Soja, M. Schmidt “Ensemble Machine
Learning of Random Forest, AdaBoost and XGBoost
We acknowledge the use of NASA/GSFC OMNIWeb ser-
for Forecasting Vertical Total Electron Content of the
vice for OMNI data, and the University of Bern for GIM
Ionosphere,” J Geod, submitted 13, December 2021
data. This study was funded by Research Grants - Doctoral
(under review).
Programmes in Germany from the German Academic Ex-
change Service (DAAD). [11] S. M. Lundberg, S. Lee “A Unified Approach to
Interpreting Model Predictions,” Advances in Neural
References Information Processing Systems, 30, 2017, pp. 4765-
4774, http://papers.nips.cc/paper/7062-a-unified-
[1] E. Camporeale, S. Wing, and J. Johnson, “Ma- approach-to-interpreting-model-predictions.pdf.
chine learning techniques for space weather,” Else-

Authorized licensed use limited to: SRM Institute of Science and Technology Kattankulathur. Downloaded on January 28,2025 at 08:37:12 UTC from IEEE Xplore. Restrictions apply.