Modulation Classifier Based on Deep Learning for

Beyond 5G Communications

Osman Kaya1, Muhammet Ali Karabulut2, A. F. M. Shahen Shah1, Haci Ilhan1

1
Department of Electronics and Communication Engineering, Yildiz Technical University, Istanbul-34220, Turkey.
2
Department of Electrical and Electronics Engineering, Kafkas University, Kars, Turkey.
[email protected], [email protected], [email protected], [email protected]
2024 47th International Conference on Telecommunications and Signal Processing (TSP) | 979-8-3503-6559-7/24/$31.00 ©2024 IEEE | DOI: 10.1109/TSP63128.2024.10605950

Abstract— In order to stay under a certain block error rate, a classification method [4], and the use of deep learning in
fourth generation (4G) or fifth generation (5G) base station AMC has increased rapidly in recent years. For digital
chooses the proper modulation and coding scheme based on the modulation classification, a feature-based method has been
channel condition. Beyond 5G (B5G) communications will proposed, which aims to extract the constellation shape from
enable the receiver to receive a signal from a wider range of
sources in the future. Consequently, a real-time system is
the received information after the noise and channel
required to recognize and categorize the kind of modulation on distortions [4]. In a study for amplitude and phase
the receiver's end. Automatic Modulation Classification modulation classification [5], an ML classifier was first
(AMC) involves identifying the modulation scheme used in a used. Then, to alleviate the high computational complexity
signal received by communication systems. Recently, deep of ML, a minimum distance (MD) classifier was proposed
learning, a method in the field of machine learning, has and their performances were compared. The local binary
received considerable attention due to its remarkable ability to pattern was used as a feature in a study to classify phase
categorize complex data patterns. This research investigates shift keying (2/4/8-QPSK), quadrature amplitude
the applications of automatic modulation classification using modulation (16/64-QAM), and amplitude shift keying (4-
Convolutional Neural Networks (CNNs) as a deep learning
method in both civilian and military domains. In this study, the
ASK) modulations [6].
effectiveness of this deep learning methodology in modulation The extracted features were passed through a single
classification is evaluated using the RadioML.2016a dataset. hidden-layer feedforward classifier, and the proposed
The performance of the developed deep neural network method was claimed to be faster than likelihood-based
structure is analyzed using different hyperparameters and data methods. Studies on modulation classification using deep
formats. learning include architectures such as Convolutional Neural
Networks (CNN), Recurrent Neural Networks (RNN),
Keywords—deep learning, modulation, classification Long-Short Term Memory (LSTM), as well as hybrid
I. INTRODUCTION structures combining these classic deep network
architectures. İn a study using the RadioML.2016a dataset
A 4G or 5G base station chooses the suitable modulation was achieved performance up to the 92% range with four
and coding method based on the channel condition, with the consecutive 32x1x5 CNN layers [7]. Another study using
goal of not surpassing a certain block error rate. In the the RadioML2018.A dataset used eight 1D CNN filters of
context of 5G and unmanned aerial vehicle (UAV) sizes (3x1), (1x3), and (1x1) and achieved performance up
communications, the receiver encounters a phenomenon to 93% [8]. In another study, two CNN layers of sizes
known as multipath fading. This occurs when the receiver (64x1x3) and (16x2x3) were followed by a dense layer of
receives a signal from numerous sources and directions, 128 units, and three dropout values (0, 0.5, and 0.6) were
resulting in difficulties in accurately identifying the signals. examined, showing that dropout values of 0.5 and 0.6 had
In the future of beyond 5G (B5G) communications, the use similar performance and were better than 0 [9]. This study
of multiple-input and multiple-output (MIMO) technology used the RadioML2016.10b dataset (which has more
will be significantly expanded. This will result in the examples than RadioML2016.10a). Principal Component
receiver being able to receive signals from a greater number Analysis (PCA) was applied to the raw data and used as
of diverse sources. Hence, there is a need for a real-time input to the network. In addition, raw data was used directly
system that can accurately detect and categorize the as input and their performance was compared. The
modulation type on the receiver's end [1-2]. Signal detection performance of CNN and CLDNN (Convolutional Long
is the first and most important step in the reception and Short-Term Deep Neural Network CNN+LSTM) was
decoding of communication signals. In the context of evaluated. While the CNN network had a model with sizes
Automatic Modulation Classification (AMC), high (256x1x3), (256x2x3), (80x1x3), and a dense layer (128),
accuracy, low computational complexity, and the ability to the CLDNN network, unlike the CNN, had an LSTM layer
serve multiple purposes are expected from the structure. In before the dense layer. The results showed that working
the literature, studies in this area are classified into two main with raw data without PCA had higher performance. In
types: likelihood-based and feature-based, referred to as addition, while the CNN network achieved about 91%
traditional methods [3]. In contrast, the deep learning accuracy, the performance of the CLDNN network remained
approach has emerged as an advanced modulation in the 87% range [10]. RadioML dataset and modified

979-8-3503-6559-7/24/$31.00 ©2024 IEEE 336 TSP 2024

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on May 09,2025 at 11:01:03 UTC from IEEE Xplore. Restrictions apply.
 TABLE I. CHARACTERISTICS OF THE DATA SETS USED IN THE STUDY
SNR
Modulation Sample Data
Data Set Range
Types Size Set Size
(dB)
BPSK, QPSK,
RadioML CPFSK, GFSK,
2x128 140000 20:2:18
2016.a 4-PAM, AM-
DSB, QAM64
Fig 1. Block diagram of modulation classification in
communication system TABLE II. STRUCTURE FEATURES USED IN THE STUDY

RadioML. (In the modified RadioML dataset, the sample Layer Number of
Characteristics Filters
length is not 128, but varies between 128 and 512). Using
amplitude and phase data as input, but not IQ data, the Deep Conv2D 256
Neural Conv2D 64
performance of the 2-layer LSTM structure was measured
Network Dense 256
and compared with CNN with (256x1x3), (80x1x3) layers. Structure
The performance of the LSTM model is about 90%, the
performance of the CNN is about 80% [11]. The In the second section, the considered methodology will
Radio.ML2016a dataset was used. The effect of different be discussed. In the third section, the considered simulation
number of Gated Recurrent Unit (GRU) layers on the results section will be discussed. In the fourth section, the
performance was investigated and it was found that using considered conclusion and future research directions"
more than two layers does not provide a significant change section will be discussed.
in performance. The results of the GRU network are
compared with CNN and RNN networks, and it is shown II. METHODOLOGY
that the effect of GRU on the results is not significant. The
The RadioML2016.a dataset was created for the purpose
performance of single-layer GRU was 81%, while the
of developing and evaluating machine learning algorithms
performance of 2-layer GRU was 90%. The CNN structure
for radio signal modulation classification [15-18].
it is compared to contains 2 (50x1x8) CNN layers. Since
Specifically designed to provide a realistic representation of
this CNN model has fewer layers, it achieves lower
radio signals commonly used in wireless communication
performance than GRU (about 80%) [12]. The
systems, this dataset, like many others in wireless
RadioML.2018b dataset was used. (The sample length is
communication and signal processing, was generated using
1024, not 128, and contains 24 different modulation types.)
GNU Radio software. In the context of this study, the first
The CNN structure used contains 3 CNN layers
part focused on the BPSK, QPSK, 64QAM, BFSK, CPFSK,
(256x2x130), (168,2,133), (80x1x135) and 2 dense layers
PAM4, and AM-DSB modulations from the RadioML
with (256) and (128) units.
2016.10a dataset. Each modulation type contains 1000
The CNN performance is analyzed for different batch
examples for each SNR value in the range -20 to 18 dB (in 2
sizes (200, 400, 600, 800, 950), different dropout values
dB increments). Each example is in IQ format and contains
(0.1, 0.3, 0.5, 0.7 0.9) and different gradient descent
128 sample points. Thus, for each modulation type at each
algorithms (sgd, adagrad, adamax, nadam, adadelta,
SNR value, a vector of size 2x128x1000 is created. With
rmsprop, adam), and a maximum performance of 81% is
seven modulations, 20 different SNR values for each
obtained with dropout = 0.5, batch size = 0.5 and Adam
modulation, and 1000 examples for each SNR value, the
combination [13]. RML2016.a and RML2016.04c (smaller
labeled data set used in the study has a total of 140,000
than RML2016.a) datasets were used. The CNN structure
examples. Table 1 summarizes the characteristics of these
consists of 64x2x3, 16x2x3 CNN and 128 units of Dense,
datasets. The RadioML2016.a dataset includes labels for
while the CNN-LSTM structure consists of 128x1x3,
modulations. The sets of labels were generated using binary
32x2x3 CNN and 128 units of LSTM. In the RML2016.04c
representation of categorical variables (one-hot encoding).
dataset, the CNN_LSTM performance is approximately
In this representation, the label for each modulation type is a
95% and the CNN performance is approximately 91%; in
binary number with a length equal to the number of
the RML2016.a dataset, the CNN_LSTM performance is
classification categories, and for each modulation type, only
approximately 83% and the CNN performance is
one digit in this number is 1, while the others are 0. As there
approximately 76% [14]. The integration of AMC into a
are 7 different modulations in the RadioML2016.a dataset,
communication system is shown in Figure 1, which
the label length for modulations is 7. For example, the label
illustrates how AMC becomes part of the communication
vector for BPSK is [1 0 0 0 0 0 0], and for QPSK, it is [0 1 0
system.
0 0 0 0].
The following are the contributions of this article to the
literature:
x This study demonstrated the performance of
different data formats(e.g., Amplitude-Phase, I-Q)
and hyperparameters(e.g., Batchsize, Optimizers)
on the modulation classification problem.

337
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on May 09,2025 at 11:01:03 UTC from IEEE Xplore. Restrictions apply.
 In this study, we investigate the modulation classification
performance of signals from the RadioML2016.a dataset
using a deep neural network structure. The results are
compared using different hyperparameters and data formats.
The characteristics of the deep neural network structure used
in the study are shown in Table II.
III. SIMULATION RESULTS
In this section of the study, the results obtained for the
RadioML.2016a dataset are presented, and the performance
of the deep neural network structure created in Figure 2 is
analyzed against the I-Q and amplitude-phase data formats.
With the I-Q data format, the success rate reached about 90
percent. With the amplitude-phase data format, the success
rate remained around 60 percent.
In Figure 3, the performance level of RMSprop, Adam,
Fig 2. Performance analysis for different data formats
SGD, Nadam, Ftrl, Adagrad optimizers with the created deep
neural network structure is examined. It can be seen that the
performance level of RMSprop, Adam, Nadam optimizers
reaches about 90 percent.
Figure 4 shows the effect of varying the batchsize
hyperparameter on the performance level. It can be seen that
the performance level of different batchsize values in the
modulation classification problem is close to each other
(about 90 percent).

IV. CONCLUSION AND FUTURE RESEARCH DIREECTIONS

In this study, the effectiveness of this deep learning
methodology in modulation classification is evaluated for
beyond 5G communications. Using data from the
RadioML.2016a dataset, the performance of the deep neural
network structure on the classification problem of
Fig 3. Performance analysis of different optimizers
modulation data with different hyperparameters and data
structures is investigated. The results show that the
performance is directly related to the signal-to-noise ratio
(SNR) level, and the increase in SNR corresponds to better
performance. It is also found that the I-Q data format
provides a high level of performance compared to the
amplitude-phase data format. Among the optimizers studied,
the RMSprop, Adam, and Nadam optimizers provide a high
level of performance. It is observed that increasing the SNR
value does not significantly affect the performance at the
batchsize values examined.
In future studies, AMC will be a valuable tool for
military applications. It has the potential to identify potential
threats and improve communications reliability. In future
research, AMC will remain an open method to be developed
for effective use in these areas. AMC can be used to classify
signals from military technologies such as drones or
unmanned aerial vehicles (UAVs). This is a critical step in
detecting and neutralizing enemy UAVs. In addition, AMC
can be more effective when combined with multiple antenna
technologies. These technologies, which use multiple
antennas, can determine the direction and strength of the Fig 4. Performance analysis of different batchsize values
signal, leading to more accurate classification results. As a
result, future studies hold promise for military
communications and electronic warfare applications as AMC
technologies continue to evolve and improve.

338
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on May 09,2025 at 11:01:03 UTC from IEEE Xplore. Restrictions apply.
 ACKNOWLEDGMENT [18] J. Jang, J. Pyo, Y. -I. Yoon and J. Choi, "Meta-Transformer: A
Meta-Learning Framework for Scalable Automatic Modulation
The Scientific and Technological Research Council of Classification," in IEEE Access, vol. 12, pp. 9267-9276, 2024
Turkey (TUBITAK) is funding this work through Project
123E106.
REFERENCES
[1] A. P. Hermawan, R. R. Ginanjar, D. -S. Kim and J. -M. Lee,
"CNN-Based Automatic Modulation Classification for Beyond
5G Communications," in IEEE Communications Letters, vol. 24,
no. 5, pp. 1038-1041, May 2020.
[2] A. F. M. S. Shah, " A Survey From 1G to 5G Including the
Advent of 6G: Architectures, Multiple Access Techniques, and
Emerging Technologies," in Proc. of IEEE 12th Annual
Computing and Communication Workshop and Conference
(CCWC), Las Vegas, NV, USA, 2022, pp. 1117-1123.
[3] D. Zhang et al., “Automatic Modulation Classification Based on
Deep Learning for Unmanned Aerial Vehicles,” Sensors, vol. 18,
no. 3, p. 924, Mar. 2018.
[4] B. G. Mobasseri, “Digital modulation classification using
constellation shape,” Signal Processing, vol. 80, pp. 251-277,
2000.
[5] M. D. Wong and A. K. Nandi, “Semi-blind algorithms for
automatic classification of digital modulation schemes,” Digital
Signal Processing, vol. 18, pp. 209-227, 2008.
[6] A. Güner, Ö. F. Alçin, and A. Şengür, “Automatic digital
modulation classification using extreme learning machine with
local binary pattern histogram features,” Measurement, vol. 145,
pp. 214-225, 2019.
[7] L. Huang, Y. Zhang, W. Pan, J. Chen, L. P. Qian, and Y. Wu,
‘‘Visualizing deep learning-based radio modulation classifier,’’
IEEE Trans. Cognit. Commun. Netw., vol. 7, no. 1, pp. 47–58,
2021
[8] T. Huynh-The, C. Hua, Q. Pham, and D. Kim, ‘‘MCNet: An
efficient CNN architecture for robust automatic modulation
classification,’’ IEEE Commun. Lett., vol. 24, no. 4, pp. 811–815,
2020.
[9] T. J. O’Shea, J. Corgan, and T. C. Clancy, “Convolutional Radio
Modulation Recognition Networks,” Engineering Applications of
Neural Networks, pp. 213–226, 2016,
[10] S. Ramjee, S. Ju, D. Yang, X. Liu, A. E. Gamal, and Y. C. Eldar,
“Fast Deep Learning for Automatic Modulation Classification,”
arXiv:1901.05850 [cs, eess, stat], Jan. 2019.
[11] S. Rajendran, W. Meert, D. Giustiniano, V. Lenders and S. Pollin,
"Deep Learning Models for Wireless Signal Classification With
Distributed Low-Cost Spectrum Sensors," in IEEE Transactions
on Cognitive Communications and Networking, vol. 4, no. 3, pp.
433-445, Sept. 2018, doi: 10.1109/TCCN.2018.2835460.
[12] D. Hong, Z. Zhang and X. Xu, "Automatic modulation
classification using recurrent neural networks," 2017 3rd IEEE
International Conference on Computer and Communications
(ICCC), Chengdu, China, 2017, pp. 695-700.
[13] K. Ma, Y. Zhou and J. Chen, "CNN-Based Automatic
Modulation Recognition of Wireless Signal," IEEE 3rd
International Conference on Information Systems and Computer
Aided Education (ICISCAE), Dalian, China, 2020, pp. 654-659.
[14] Y. Wu, X. Li and J. Fang, "A Deep Learning Approach for
Modulation Recognition via Exploiting Temporal Correlations,"
2018 IEEE 19th International Workshop on Signal Processing
Advances in Wireless Communications (SPAWC), Kalamata,
Greece, 2018, pp. 1-5, doi: 10.1109/SPAWC.2018.8445938.
[15] https://github.com/radioML/examples/blob/master/modulation_re
cognition/RML2016.10a_VTCNN2_example.ipynb
[16] L. Guo, Y. Wang, Y. Liu, Y. Lin, H. Zhao and G. Gui, "Ultralight
Convolutional Neural Network for Automatic Modulation
Classification in Internet of Unmanned Aerial Vehicles," in IEEE
Internet of Things Journal, vol. 11, no. 11, pp. 20831-20839,2023
[17] Z. Elkhatib, F. Kamalov, S. Moussa, A. B. Mnaouer, M. C. E.
Yagoub and H. Yanikomeroglu, "Radio Modulation
Classification Optimization Using Combinatorial Deep Learning
Technique," in IEEE Access, vol. 12, pp. 17552-17570, 2024

339
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on May 09,2025 at 11:01:03 UTC from IEEE Xplore. Restrictions apply.