ACCEPTED FROM OPEN CALL

Semantic Communications: Overview,

Open Issues, and Future Research Directions
Xuewen Luo, Hsiao-Hwa Chen, and Qing Guo

Abstract On the other hand, the existing communication

technologies have nearly approached the Shan-
With the deployment of the fifth generation (5G) non physical-layer capacity limit. The goals and
in many countries, people start to think about what services provided by B5G prompt researchers to
the next-generation of wireless communications will think about what the next-generation of wireless
be. The current communication technologies are communications will be. A pioneering work done
already approaching the Shannon physical capac- by Weaver [1, 2] revealed that communications
ity limit with advanced encoding (decoding) and can be categorized into three levels, as shown in
modulation techniques. On the other hand, artifi- Fig. 1. The lowest level is the technical level, which
cial intelligence (AI) plays an increasingly important is defined by Shannon’s classical information theo-
role in the evolution from traditional communication ry and focuses on how to transmit symbols (bits)
technologies to the future. Semantic communication accurately and effectively from a transmitter to a
is one of the emerging communication paradigms, receiver. On the middle level, that is, the semantic
which works based on its innovative “seman- level, semantic information of the data is extracted
tic-meaning passing” concept. The core of semantic and transmitted via a semantic channel, whereas
communication is to extract the “meanings” of sent the upper level, that is, the effectiveness level, is
information at a transmitter, and with the help of a responsible for providing the needed communica-
matched knowledge base (KB) between a transmit- tion efficiency on the lower two levels.
ter and a receiver, the semantic information can be Benefitting from the advancements in micro-
“interpreted” successfully at a receiver. Therefore, electronics and AI technologies, deep learning
semantic communication essentially is a communi- (DL) and end-to-end (E2E) communication technol-
cation scheme based largely on AI. In this article, ogies emerged recently to play an important role
an overview of the latest deep learning (DL) and in the transformation of traditional communication
end-to-end (E2E) communication based semantic technologies to the future. Semantic communica-
communications will be given and open issues that tion was proposed as an intelligent communication
need to be tackled will be discussed explicitly. scheme, which concerns the meaning of trans-
mitted messages rather than accurate bit stream
Introduction transmission [3]. For instance, in a natural language
From the first generation (1G) to the fifth gener- system, if a source sends a message that “Bob’s
ation (5G), the goals of communication systems automobile was parked there,” its destination may
have changed drastically from analog audio signal receives “Bob’s car was parked there” in a seman-
transmission to high speed and low-latency multi- tic communication system, but “Bab’s autmkobile
media services. Especially in 5G, various advanced was pbrked there” in a traditional communication
wireless communication technologies have been system. In this example, semantic communications
used, such as non-orthogonal multiple access concern the meaning behind the transmitted sym-
(NOMA), massive multiple-input multiple-output bols (bits). Even though the word phrases inter-
(MIMO), millimeter wave communications, and preted at the receiver have been changed a little
so on. Despite the fact that 5G can meet most bit, the receiver can still understand it. However, in
requirements of different services with a low-laten- a traditional communication system, the received
cy and a high data rate, the existing technologies message is confusing because the transmitted
may not be able to support many intelligent appli- symbols (bits) have been distorted due to chan-
cations in beyond-5G (B5G) communications. The nel noise and interference. Therefore, although
services in B5G networks, such as connected living, syntactic mis-matches may exist in semantic com-
brain-to-computer interaction, virtual reality (VR), munication systems, there are no semantic errors.
augmented reality (AR), and mixed-reality (MR), Moreover, it also suggests that when bandwidth
will be supported to enrich our future intelligent is limited or signal-to-noise ratio (SNR) is relatively
life. The technical requirements for these services low, a semantic communication system may still
are much higher than 5G networks, such as 1 ∼ 10 perform well and likely consume less energy.
Gb/s/m3 traffic density, 1 Tb/s uplink and down- It should be noted that semantic communica-
link data rate, 0.1 ms latency, and so on [1]. In this tion is not a security communication scheme, but
context, 5G serves as only a transitional platform an intelligent way to exchange information. The
from traditional communications to futuristic artifi- biggest difference between semantic communica-
cial intelligence (AI) communications. tion and encryption-based security communication
Digital Object Identifier:
10.1109/MWC.101.2100269 Xuewen Luo and Qing Guo are with Harbin Institute of Technology; Hsiao-Hwa Chen is with the National Cheng Kung University.

210 1536-1284/22/$25.00 © 2022 IEEE IEEE Wireless Communications • February 2022

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 Perception Perception Traditional commu-

Effectiveness level
Level C: Source Environment Destination nication systems aim
to offer a high data
Knowledge sharing Destination transmission rate and a
Source KB
KB low symbol (bit) error
rate. However, the
basic idea of semantic
Semantic Features Semantic Features Semantic
communications is to
Semantic level

transmitter channel receiver

Level B:

extract the “meanings”

Semantic or “features” of sent
noise
Semantic feedback information from a
source, and “interpret”
Syntactic feedback the semantic informa-
tion at a destination.
Technical level

Source Channel Wireless Channel Source

Level A:

Modulation Demodulation
encoding encoding channel decoding decoding
Bits Bits
Channel
Interference
noise

FIGURE 1. Three-layer model in semantic communications.

lies in their goals. Semantic communication works communications. An overview of semantic com-
on the “semantic meaning passing” concept, but munication systems will be given, followed by use
the goal of encryption-based security communi- cases and open issues on semantic communica-
cation is to prevent unauthorized decoding at an tions. Finally, the conclusions will be given at the
attacker or an eavesdropper. end of this article.
Normally, a semantic communication system
should include all three levels in Fig. 1. As a mat-
ter of fact, semantic communication is not a com-
dIfferences between
pletely new idea, which can be traced back to the semAntIc And trAdItIonAl communIcAtIons
seminal works done by Weaver in the 1940s [2]. Traditional communication systems aim to offer a
Then in 1952, Carnap and Bar-Hillel introduced a high data transmission rate and a low symbol (bit)
semantic information theory (SIT) based on logical error rate. However, the basic idea of semantic
probability functions derived from the contents communications is to extract the “meanings” or
of a sentence [4]. Inspired by those pioneering “features” of sent information from a source, and
works, Bao et al. [5] reviewed the existing works “interpret” the semantic information at a destina-
on quantifying semantic information theory, and tion. In this section, the similarities and differences
then proposed a model-theoretical approach for between traditional and semantic communica-
semantic data compression and reliable semantic tions will be discussed.
communications. Recently, enabled by DL tech-
nologies, various semantic communication systems semAntIc source And destInAtIon
were designed for the transmission of text [3, 6–8], In a traditional communication system, the entities
image [9, 10], and speech signals [11]. The objec- at the source and destination are only electronic
tive of this article is to provide an overview of the equipments, which work in a workflow of different
most recent works on DL and E2E communication communication blocks, such as source encoding
based semantic communications. (decoding), channel encoding (decoding), and so
Semantic communication is an interdisciplinary on. As shown in Fig. 2a, data in a traditional com-
research topic, which involves linguistics, computer munication system are compressed by a source
science, and wireless communications. We would encoder and redundancy is added in the channel
like to give detailed explanations for several key encoder to improve its robustness against inter-
terms used in the linguistics to help readers under- ference/noise in the channels. At a destination, a
stand the contents: reverse process proceeds to recover the original-
• Syntax: A set of rules, principles, and processes ly sent data. In such a block-based structure, no
(e.g., word order) that govern the structure of intelligence is involved in signal transmission and
sentences in a given language. reception, and the implicit meanings behind the
• Polysemy: An individual word or phrase that messages are completely ignored at the transmit-
can be used (in different contexts) to express ter and the receiver.
two or more different meanings. On the other hand, a semantic communica-
• Synonym: A word or phrase that gives exactly tion system is a complicated system. The Semantic
or nearly the same meaning as another word or source and destination are agents that need to per-
phrase in the same language. form not only the functions of traditional communi-
• Dialect: Refers to a variant of a language cation terminals, but also various highly intelligent
shared by a particular group of the speakers algorithms. The agents in a semantic communica-
of the language. tion system can be humans, machines, or other
The rest of this article can be outlined as fol- devices with intelligence. Moreover, the semantic
lows. The next section focuses on a comparison source and destination can perceive the environ-
between semantic communications and traditional ment and operate autonomously [1]. A semantic

IEEE Wireless Communications • February 2022 211

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 Noise

Source Channel Physical Channel Source

Bob’s encoding encoding channel decoding decoding Bab’s
automobile autmkobile
was parked was pbrked
there. there.
1000001… 10000010… 10010010… 1000001…
1001001… 10010011… 10001011… 1000101…
1101010… 11010100… 11010100… 1101010…
Source Destination
1010010… 10100101… 10100101… 1010010…
0101110… 01011100… 00010100… 0001010…

(a)
Semantic noise

Semantic
channel
Noise
Bob’s
automobile Semantic Channel Physical Channel Semantic Bob’s car was
was parked encoding encoding channel decoding decoding parked there.
there.

10000010… 10010010…
Semantic feature 10010011… 10001011… Semantic feature
extraction 11010100… 11010100… restoration
10100101… 10100101…
01011100… 00010100…

Semantic source Semantic features Semantic features Destination

(b)

FIGURE 2. A comparison between traditional and semantic communication systems: a) Workflow of a traditional communication sys-
tem; b) Workflow of a semantic communication system.

source is able to extract the semantic features of Semantic noise can cause semantic errors at
raw messages and encode these features into sym- a receiver and induce misunderstanding of the
bols (bits) for transmission. The destination should received messages. On the semantic level, seman-
be able to “understand” and infer the messages sent tic errors can be caused by the mismatch between
by the semantic source. For example, in a natural the background knowledge bases (KBs) used by
language system, there is a syntax for the semantic the semantic source and destination. For instance,
source and destination to understand the meanings the source is an English language system, but in
of words in sentences. Moreover, the agents should the destination only the Chinese language is used.
distinguish even very subtle differences in words, In addition, polysemy and synonym may also
which behave just like a human reading polysemy induce semantic errors. On the technical level (as
and synonym. An example of polysemy is the word shown in Fig. 1), semantic errors may occur from
“cherry,” which becomes a person’s name when symbol or bit errors during transmission due to
“C” is an uppercase letter; otherwise it means a type the noise or interference in the physical channels,
of fruit. Car and automobile are synonyms despite and it is hard to distinguish these errors caused by
the fact that they are different in syntax, and they semantic noise and channel propagation. There-
are the same in a semantic sense. fore, in order to interpret the meanings successful-
ly at a semantic destination, we need to overcome
Semantic Channels with Errors not only physical channel noise, but also semantic
As shown in Figs. 2a and 2b, in addition to a phys- noise in a semantic communication system.
ical channel, a “virtual” channel, that is, a seman-
tic channel exists, through which the semantic Source/Channel Encoding and Decoding
information is transmitted from a source and inter- In traditional wireless communications, data
preted at a destination. Unlike traditional commu- should be compressed by source encoding first
nications, there are two different types of noises and then by channel encoding to combat channel
in semantic communication systems. The first type impairments, aiming to achieve an optimal trans-
is physical channel noise, which exists ubiquitous- mission performance in each processing block. In
ly in wireless communications and is caused by semantic communications, semantic encoding not
physical channel impairments, such as additive only compresses data at the source as much as
white Gaussian noise (AWGN), channel fading, possible, but also extracts the meanings and their
multiple path propagation, and so on. It is noted semantic features of the data. The goals of seman-
that the errors caused by channel propagation tic encoding are twofold [5], that is, maximizing
usually occur before channel decoding and can expected faithfulness in representing observed
be corrected by channel decoding. In addition, worlds and minimizing the amount of data to
the co-channel interferences from different users be transmitted. The semantic features should be
cannot be ignored. The second type of noise is transmitted over a physical channel, and channel
semantic noise, which appears in message inter- encoding should be added to improve the robust-
pretation processes due to the ambiguity existing ness. At a receiver, the received signal is decoded
in words, sentences or symbols used in the sent via channel decoding to extract the semantic fea-
messages [3, 7]. tures before the original messages are recovered.

212 IEEE Wireless Communications • February 2022

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 Empowered by DL-based E2E communications dissimilarity between the two is measured by the Different from tradi-
and natural language processing (NLP) technol- semantic distance, which can be used to evaluate tional communications,
ogies, semantic encoding (decoding) and chan- the distortion between the words on the semantic another important
nel encoding (decoding) can be implemented by level [12, 13]. The average semantic distortion or characteristic feature in
deep neural networks (DNNs). In such a system, error is defined as the average semantic distance
semantic source and destination are auto-encoder in probability, which is statistically expressed by the semantic communica-
and auto-decoder to perform semantic encod- probability of words and the conditional probability tions is that a semantic
ing (decoding) and channel encoding (decoding) of receiving wrong meanings under the condition communication system
jointly, which can achieve a global optimality if of the sent messages. The word error rate is an edit is a knowledge-based
compared to the block-based structure in a tradi- distance normalized by the length of a sentence system. This means that
tional communication system. [6]. Another common measurement is the bilingual
evaluation understudy (BLEU) score, which mea- the semantic source
Background Knowledge Bases sures the similarity between decoded text and raw and destination can
Different from traditional communications, anoth- text. However, because the BLUE score compares establish their own
er important characteristic feature in semantic only the difference between the two text messag- background knowledge
communications is that a semantic communica- es, it cannot distinguish more subtle difference in bases by self-learning,
tion system is a knowledge-based system [1]. This words, such as polysemy and synonym [3]. Thus,
means that the semantic source and destination the sentence similarity is proposed to calculate the just like human brains,
can establish their own background knowledge semantic similarity between the originally sent sen- which form the core of
bases (KBs) by self-learning, just like human tence and recovered sentence [7]. a semantic communi-
brains, which form the core of a semantic com- Image: Two performance metrics for image cation system.
munication system. The KBs are the world models messages were proposed. First, the performance
that the source and destination observed previ- of an image semantic communication system can
ously. The semantic source extracts the seman- be measured by peak signal-to-noise ratio (PSNR),
tic information of the messages based on its KB. which is the ratio between the maximum signal
After receiving the messages, the receiver is able and noise powers [9]. If the mean squared error
to interpret and infer the meanings of sent mes- (MSE) between the transmitted image and the
sages based on the destination KB. There are dif- reconstructed image is smaller, PSNR is larger
ferent types of KBs based on text, image, speech and the reconstruction quality of the image is bet-
or video, and most works in the literature focused ter. PSNR can also be used in video transmission
only on text or image based semantic communi- because video files consist of many image frames.
cations due to mature DL-based NLP and image Furthermore, for image recognition, recognition
processing technologies. The establishment of accuracy is a measurement for a joint transmis-
KBs is a complex and time-consuming process, sion-recognition scheme in an image semantic
which basically is a learning process, just like the communication system [10].
learning process through which humans learn Speech: In the literature, very few works focus
knowledge of the world from a child to an adult. on speech based semantic communications. In
The KBs can learn from the perceived environ- a very limited number of existing works, signal
ment and can continuously expand and update to distortion ration (SDR) was utilized to mea-
their knowledge through training and sharing sure the errors between raw speech vector and
via communications. In addition, the KBs at the restructured speech sequence, where a higher
semantic source and destination may be differ- SDR indicates the fact that the reconstructed
ent because the worlds and environments they speech signal is easy to understand [11]. Another
observed are different and their abilities to under- good metric for speech based semantic communi-
stand things are also different, which may cause cations is perceptual evaluation of speech distor-
a semantic mismatch. However, as shown in Fig. tion (PESQ), as proposed in [11], which evaluates
1, the semantic source and destination can share various speech signal conditions, such as back-
their KBs with each other in order to minimize the ground noise, analog filtering, and so on.
semantic mismatches. Finally, it should be noted that Shannon infor-
mation theory does provide a design guidance for
Performance Metrics semantic communications. The semantic informa-
Traditional communications need to minimize tion of text, image or speech should eventually be
bit-error rate (BER) or symbol-error rate (SER) encoded into bit streams and then transformed
and transmit more bits utilizing as little com- into physical signals for their transmission via com-
munication resource as possible. In semantic munication channels. Thus, modulation, demod-
communications, a receiver is supposed to ulation and other signal processing schemes are
extract the semantic information with the least also required in semantic communications, and
ambiguity of the sent messages, and several advanced wireless communication technologies
performance metrics are used to ensure that can improve the efficiency of semantic communi-
the semantic information is transmitted and cation systems.
retrieved correctly. Different from the perfor-
mance metrics used in traditional communica- An Overview of Semantic Communications
tion systems, the performance metrics used to In the previous section, we compared semantic
measure semantic communication systems are communications with traditional communications
diverse due to different KB types. We will dis- from several aspects, and illustrated how a seman-
cuss this issue in the sequel. tic communication system works. In this section,
Text: For text messages in semantic communi- we will introduce detailed semantic communi-
cations, the performance can be measured by the cation system models, including E2E semantic
similarity between the sent words or sentences and communication systems and multi-user semantic
the interpreted words or sentences. The semantic communication systems.

IEEE Wireless Communications • February 2022 213

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 Retrain channel encoder and decoder when
the communication environment changes

New physical
channel
Outdated
Outdated
destination
source KB
KB
New channel encoder New channel decoder

Physical channel
Text Text

Outdated semantic encoder Outdated channel encoder Outdated channel decoder Outdated semantic decoder

New source New

KB Retrain semantic destination
encoder and decoder KB
New semantic encoder New semantic decoder
with updated KBs

FIGURE 3. A DL-based semantic communication model.

End-to-End Semantic Communications problem is formulated when assigning a binary

codeword for each word, where semantic similar
A generic DL-based E2E semantic communica- words are coded with a short Hamming distance,
tion model is shown in Fig. 3, where the semantic and semantic independent words (i.e., most differ-
encoder (decoder) and channel encoder (decod- ent codewords) are coded with the longest Ham-
er) are implemented by DNNs. With a given static ming distance [12, 13]. For instance, “car” and
source and destination KBs and a communica- “automobile” are coded by “0010” and “0000”,
tion environment, the semantic encoder (decod- and the semantic index of “magician” is “1011”.
er) and channel encoder (decoder) are trained In this way, the words reconstructed at a receiver
jointly by a stochastic gradient descent (SGD) via inverse index assignment have their semantic
algorithm, as shown in the blocks highlighted by similarity very close to the words transmitted by a
the red lines in Fig. 3. Due to the generalization semantic source, in spite of the presence of channel
ability of DNN, it is possible to send a new mes- noise and interference. Semantic index assignment
sage which may not be included in the source is a good way to distinguish semantic similar and
and destination KBs without degrading interpre- semantic independent words when the number of
tation performance. However, if a sent message words is limited. However, the length of a code-
is from a different type of KB, for example, an word is exponentially proportional to the number
image in a linguistic system, it must take a lot of of words, which makes the assignment process
time to retrain the semantic encoder (decoder). extremely time-consuming and complicated.
Owing to dynamic communication environment Recently, DL was proposed to be used in joint
and KBs’ expansion/update, transfer learning is source-channel coding (JSCC), benefiting from
an effective and efficient approach to train the the powerful representation capability of DNNs.
encoder (decoder) due to the flexible structures Inspired by the success of DNNs in NLP, Farsad
of DNNs [3]. The encoder and decoder training et al. [6] proposed a JSCC scheme for text-based
and KB updating processes in a dynamic com- semantic communications, where the encoder
munication environment are denoted by yellow and decoder were implemented by two recurrent
and blue blocks in Fig. 3, respectively. If the com- neural networks (RNNs), and the channel was rep-
munication environment is changing, the channel resented by a dropout layer. Compared to a sep-
encoder and decoder should be retrained using a arate source-channel coding (SSCC) scheme, the
new channel model, while the parameters of the DL-based JSCC scheme offers better performance.
semantic encoder and decoder remain invariant. Although there are some insignificant errors, such
Otherwise, if the semantic source and destination as punctuation errors and so on, the semantic
update their KBs through learning and sharing, information could be conveyed accurately.
the semantic encoder and decoder should be Enabled by intelligent E2E communications, a
retrained based on the new KBs with the given novel framework of semantic communication sys-
parameters of channel encoder and decoder. tems was proposed in [7], which aimed to design a
Next, we will give an overview of semantic joint semantic source and channel coding scheme
communications according to different types of while maximizing system capacity. This work con-
transmitted semantic information, including text, sidered technical level and semantic level jointly. In
image, and speech. addition, based on a transformer and self-attention
Text: An intuitive approach to preserve the mechanism, a destination can easily understand
semantic similarity between two words is to assign long sentences. Considering a dynamic commu-
them similar indexes. A semantic index assignment nication environment with different background

214 IEEE Wireless Communications • February 2022

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 Source Dynamic
Metrics NN Loss function Research content KB References
type environment

Average semantic Binary codeword design for

— — Limited word set — [12]
distortion words

Semantic communication in
Average semantic
— — the presence of an external Limited word set — [13]
error
entity, friend or foe

Text Joint source-channel coding Proceedings of the

Word error rate RNN — — [6]
for text transmission European Parliament

BLEU score and Cross-entropy and Design of DL-based semantic Proceedings of the Transfer
Transformer [3, 7]
sentence similarity mutual information communication systems European Parliament learning

A lite semantic communication Proceedings of the

BLEU score Transformer Cross-entropy — [8]
system design for IoT networks European Parliament

Joint source-channel coding CIFAR-10 image

PSNR CNN Average MSE — [9]
for image transmission dataset
Image Joint image transmission-
Recognition ResNet, CIFAR-10 image
Cross-entropy recognition scheme for the IoT — [10]
accurary CNN dataset
devices

Attention Semantic communication

Speech Edinburgh Transfer
SDR and PESQ mechanism MSE system design for speech [11]
signal DataShare learning
SE network signals

Notation IoT: Internet of things.

“ — “ indicates that the information is not available in the literatures. PSNR: Peak signal-to-noise ratio.
NN: Neural network. CNN: Convolutional neural network.
KB: Knowledge base. MSE: Mean squared error.
RNN: Recurrent neural network. SDR: Signal to distortion ration.
BLEU: Bilingual evaluation understudy. PESQ: Perceptual evaluation of speech distortion.
DL: Deep learning. SE: Squeeze-and-excitation.
TABLE 1. A summary of semantic communication systems.

KBs, the authors in [3] utilized transfer learning to and more difficult to deal with, because its quality
train the semantic encoder (decoder) and channel involves not only speech signal fidelity and loud-
encoder (decoder) DNNs jointly. ness, but also its frequency and tone. Speech signals
Image: The structure of an image-based seman- can convey emotions, such as happiness, sadness,
tic communication system is almost the same as that doubt, and so on. The same text may express dif-
of a text-based semantic communication system. ferent emotions in speech-based semantic com-
The biggest difference between the two lies on the munications. For instance, on one hand, “What is
structure of DNNs. The transformer, dense layer [3, wrong with you?” can express a kind of concern
7] and long short-term memory (LSTM) [6] have for a person’s health condition in a gentle and low
been widely utilized to extract semantic informa- intonation. On the other hand, it is a complaint if
tion from text messages. However, convolutional a person speaks in an anxious and high intonation.
neural network (CNN) has its advantages in image In this case, text cannot express the exact emotion
processing, and thus it is more efficient for image of the transmitter. In addition, it is difficult to rec-
feature extraction. A CNN-based E2E JSCC scheme ognize a dialect in speech signals. Considering the
was proposed for the first time in [9] to transmit nature of speech, Weng et al. [11] utilized attention
high-resolution images in both AWGN and Rayleigh mechanism squeeze-and-excitation (SE) networks to
channels. Compared to most conventional compres- capture imperfections and non-linearities of speech
sion algorithms, such as JPEG and JPEG2000, the signals. It was shown that this system performs bet-
proposed scheme showed a graceful performance ter than a traditional communication system, and is
and did not suffer a “cliff effect”. Moreover, a joint more robust even in a low SNR region. In Table 1,
transmission and recognition scheme was proposed a summary of recently reported works on semantic
in [10] to improve recognition accuracy in the Inter- communications is given.
net of Things (IoT) networks. Dense layer, convolu- So far we have introduced single-modal seman-
tional layer and ResNet were used in image feature tic communication systems. However, multi-modal
extraction and recognition, because video files are semantic communications should not be ignored,
comprised of different image frames, such that the where semantic destination may receive different
image semantic communication systems are basical- types of messages from a source. To the best of
ly able to transmit videos by extracting the feature of our knowledge, there is no specific research focus-
each frame. ing on this aspect in the literature. It is possible for
Speech: Based on advanced NLP technologies, a semantic source to send text (or speech) and the
speech can be translated into text effectively, and destination to receive speech (or text). A meth-
then the text can be transmitted into semantic com- od for this multi-modal semantic communication
munications. However, unlike text that consists of system can be realized based on advanced NLP
characters only, speech signal is more complex technologies, where the sent text (or speech) can

IEEE Wireless Communications • February 2022 215

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 Semantic user 1 Hidden layers
User 1’s

Us
message

e
KB 1 Semantic Channel

r1
’s
encoding encoding

c
ha
Be

nn
Transmitter

am

el
Channel
decoding
Semantic
+
el destination
Semantic
Be
am
ann
ch decoding
Semantic user K ’s User K’s
rK
se message
KB K Semantic Channel U
encoding encoding
Transmitter Intelligent multi-user signals detection

FIGURE 4. A structural architecture of multi-user semantic communication systems.

be translated into speech (or text). coding based multi-user semantic communication
systems for multi-user signal detection. With IR
Multi-User Semantic Communications and multiple antenna technologies, although the
In the previous subsection, several semantic com- KBs of two users are not the same, it is possible to
munication systems were introduced, where DL separate their messages at the receiver.
plays an important role in semantic information
extraction and communication processes. How- Use Cases of Semantic Communications
ever, all aforementioned systems do not involve The use cases of semantic communications are
multi-user transmissions. In general, connectivity extremely important for the implementation of
density in 5G is 106 devices per km2, whereas the semantic communication systems in the future.
connectivity density in 6G networks will grow up Three possible use cases of semantic communica-
to 10 times of 5G, and area traffic density should tions are illustrated in Fig. 5.
be one hundred times of 5G, which requires a
significant improvement in spectrum efficiency. In IoT Networks
this context, NOMA is an efficient way to improve In various data monitoring applications from des-
spectrum efficiency in semantic communication ert, ocean, cities, and home, IoT devices play
systems. As we discussed in the earlier sections, significant roles in 5G and B5G networks. The
the KBs in a semantic communication system may wide proliferation of various intelligent devices,
be very much different from each other, which such as VR/AR glasses, unmanned aerial vehi-
makes it necessary to design multi-user semantic cle (UAV), sensors, and so on, has pushed IoT
communication systems, where users can learn networks to provide more advanced functions,
from each other to communicate as human which in turn consumes more radio resources.
beings. For instance, Alice and Eve are native Furthermore, in status update systems or age of
English speakers who cannot speak Chinese, and information (AoI) aware systems, the IoT devic-
Bob can speak Chinese only. If Alice and Bob talk es should perceive their working environments
to Eve at the same time, Eve can only understand and upload the real-time status of environment
Alice and the Chinese from Bob is ignored. In this information to cloud centers or mobile edge
way, even though the Chinese seems to behave computing (MEC) servers for signal analysis and
like an interference from Bob, Eve can still get processing. Thus, the IoT devices have to sup-
the English messages from Alice without seman- port many sophisticated functions such as intelli-
tic errors. Similarly, in a semantic communication gent monitoring, data process, communications,
system, due to the diversity in KBs, multi-user sig- and so on. Obviously, the data transmitted from
nals can be transmitted using the same channel IoT devices are mostly time-sensitive, which may
resources, such as frequency or time-slot. In this not require a very high data rate but need a low
way, the bandwidth can be saved for an improved latency and high accuracy.
spectrum efficiency. However, multi-user signal Semantic communication is a promising tech-
detection and the complexity of interpretation nology for accurate and real-time data transmis-
process at a receiver are critical issues. sions in IoT networks [8] as it consumes little radio
Here, we propose a structural architecture resource and is relatively insensitive to channel
of multi-user semantic communication systems noise. However, due to limited computation and
based on intelligent radio (IR), as shown in Fig. storage capabilities, the IoT devices cannot use
4. A receiver in IR can estimate the channel state complex DNNs on board, and how to train the
information (CSI) of each user and separate semantic encoder (decoder) and channel encoder
multi-user signals by training an intelligent multi-us- (decoder) is a crucial issue in IoT networks. In order
er signal detection DNN [14]. Then, the separat- to simplify the structure of DNNs, model compres-
ed signals can be decoded by channel decoding sion can be achieved with the help of network
and semantic decoding. Moreover, equipped with sparsification and quantization [8]. Furthermore,
multiple antennas at a transmitter and a receiver, federated learning (FL) and distributed learning
beamforming and precoding techniques can be can be an alternative option to train a DL-based
used to enhance multi-user signal transmissions semantic communication system efficiently. DNN
in a semantic communication system due to the models in FL can be trained for a large number of
spatial diversity gain of MIMO. It is an interesting IoT devices jointly, and the training process can be
research topic to study beamforming and pre- coordinated by a cloud/edge server. The parame-

216 IEEE Wireless Communications • February 2022

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 Cloud center
Monitor and control

Mobile edge server

Monitor and control

Base station

Neural network model

Text
Semantic features

IoT
Networks Monitor
Music UAV
TV
player VR/AR How are
you?

ICV networks
D2D
UAV-aided communications
WiFi
Microwave
oven
Smart factories

FIGURE 5. Three use cases of semantic communications, including IoT networks, ICV networks, and smart factories.

ters of DNNs at the IoT devices are uploaded peri- ences. In semantic communications, due to the
odically to cloud/edge servers for updating, and diversity of KBs, this interference can be mini-
then the updated parameters are fed back to the mized as long as a receiver can understand the
IoT devices, such that each IoT device does not meanings of transmitted messages.
need to consume a lot of computing power for
training. Another advantage is that FL may utilize smArt fActorIes
the diversity in locally sensed data at IoT devices to Semantic communications can also be applied to
speed up the training process. smart factories. Smart factories rely on commu-
nications between machines and the interactions
IntellIgentlY connected vehIcle networks between human and machines. Furthermore,
Another application of semantic communications advanced communication technologies, such as
is intelligently connected vehicle (ICV) networks, 5G and DL-based communication technologies,
where a vehicle can perceive information in the make factories more intelligent, efficient, energy
environment and predict its driving trajectory, saving, and environmentally friendly. In futuristic
traffic flow, network congestion, CSI, and so on. smart factories, unmanned management, real-time
In ICV networks, a vehicle with various sensors control and monitoring are important features to
may generate about tens or even thousands of run machines and equipment. The semantic fea-
gigabytes per day, including videos and images tures of the monitoring information, such as the
of traffic information. Although most of these status of machines, temperature, humidity, and
data are processed at vehicles or discarded, the so on, can be extracted and uploaded to a cen-
amount of remaining data uploaded to road- tral controller or a cloud/edge server to analyze
side units (RSUs) or cloud/edge servers can be the status of materials and the quality of prod-
huge, and the uploading latency should be as ucts. Another important issue in smart factories is
short as possible. Comparisons of average run to control the operation of machines to perform
time between semantic communication systems a specific action. In this sense, semantic control
and traditional schemes in [3, 9] suggest that is an efficient way to achieve goal-centric com-
DL-based semantic communications can help to munications, where semantic information of the
compress and extract semantic information to control signals is conveyed to the machines [15].
reduce latency, so that semantic communications The operational efficiency of smart factories is
are applicable to a large amount data transmis- improved and the communication cost can be
sion in ICV networks with a low latency. In addi- reduced as only the intentions (i.e., the semantic
tion, semantic information is more robust against information) of control signals are transmitted and
channel noise and interferences than bit streams fewer errors will occur.
in traditional communications, which enhances Both ICV networks and industrial Internet
the reliability of data transmission and improves attach great importance to reliability, which can
driving and road safety in ICV networks. More- be ensured by channel encoding and decoding
over, in device-to-device (D2D) based vehicu- in semantic communications, where structured
lar communications, vehicles usually share radio redundancy is introduced to improve the robust-
resources with cellular users in an underlay fash- ness against interference/noise in the channels and
ion, which may cause severe co-channel interfer- increase communication reliability.

IEEE Wireless Communications • February 2022 217

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 The KBs are always
being expended and
Open Issues Effectiveness Level in Semantic Communications
updated frequently, just Despite the fact that semantic communication is The design of any communication systems aims to
like the learning pro- not a completely new research topic, which can minimize the consumption of channel resources,
be traced back to the pioneering work of Weaver such as bandwidth and power. The effectiveness
cess of human beings, [2], there are a variety of challenges to be tackled level in semantic communications is responsible
which makes the shar- before it can be used in real applications. In this for the management of radio resources. Despite
ing process much lon- section, we would like to list several major open the fact that many works have been done on
ger and more difficult. issues for future investigations. radio resource management (RRM) in tradition-
al communication systems, their effectiveness in
Thus, how to commu- Insufficient Theoretical Research on semantic communications remains to be verified,
nicate, share and infer Semantic Communications and more comparisons with traditional communi-
semantic information Classic semantic information theory (CSIT) was cation systems are important for practical appli-
with inconsistent KBs is introduced first by Carnap and Bar-Hillel in 1952 [4] cations of semantic communications. Moreover,
a wide open issue based on logical probability. Inspired by this seminal in resource limited IoT networks, resource con-
in semantic work, some theoretical research works have been sumption for download and parameter upload
done in the literature in the past two decades, such in a DNN model, and gradient back propagation
communications. as [5, 13] and the references therein, but they are (BP) of the parameters from a receiver to a trans-
not sufficient, especially on semantic communica- mitter, should also be considered carefully.
tions in a framework based on DL. Some important
challenges in theoretical research include the lack of Implementation of Semantic Communications
theoretical guidance for joint semantic-channel cod- Today, the research on semantic communi-
ing designs. The technical level and semantic level cations is only in its infant stage. It is widely
should be considered jointly (such as the impact of believed that more theoretical research can defi-
data transmission rate on semantic communications, nitely help to promote real implementation of
how much semantic information can be transmit- semantic communication systems, as theoretical
ted in a wireless channel). More investigations on incompleteness in semantic communications may
SIT under interference channels are also needed, restrict its implementation. In addition, it is still a
plus a specific definition of a semantic channel and question in both industry and academia whether
its capacity, and so on. A general framework for a we really need semantic communications as the
DL-based semantic communication system should existing communication technologies are very
be explored, including its proper performance met- mature. The investigations of DL-enabled seman-
ric, suitable DNN architecture, and so on. tic communication systems are limited not only
by semantic theories, but also by AI hardware,
Inconsistent KBs at Semantic Source and Destination which is required to run DNNs efficiently at a rel-
In semantic communications, the KBs of the seman- atively low cost. At present, a mobile system-on-
tic source and destination are normally inconsistent. chip (SoC) with a dedicated AI core, for example,
Although these KBs can be made more homo- a neural network processing unit (NPU), is able
geneous through KB sharing, it is an extremely to run AI models on embedded AI accelerators.
time-consuming and resource-consuming process The most advanced mobile SoCs, such as Snap-
because the sharing between KBs needs effective dragon 888 and HiSilicon Kirin 9000 manufac-
communications between the semantic source and tured by a 5 nm silicon process, can execute
destination. The KBs are always being expended DL models in even a few milliseconds for image
and updated frequently, just like the learning pro- classification, face recognition, and so on.1 How-
cess of human beings, which makes the sharing ever, these SoCs still cannot meet ultra-low laten-
process much longer and more difficult. Thus, how cy requirements in wireless communications, for
to communicate, share and infer semantic informa- example, 1 ms in 5G ICV networks and 0.1 ms
tion with inconsistent KBs is a wide open issue in in B5G networks. Therefore, it is a big challenge
semantic communications. to develop a DL-based E2E semantic communi-
cation system based on these SoCs, and more
Multi-User Interpretation Algorithm Design advanced microelectronic and chip technologies
As we discussed in the previous sections, multi- are needed to address this problem.
ple users can utilize the same frequency or time-
slot to transmit semantic information due to the Conclusions
diversity in their KBs. However, the complexity of This article gives an overview of the most recent-
semantic information interpretation at a receiver ly reported works on feature extraction based
in a multi-user environment is very high because semantic communications, which are relevant
it must consider multi-user detection, channel to future intelligent communications. Semantic
decoding and semantic decoding jointly. In addi- communication works very much differently from
1 A performance ranking of
tion, the KB at a receiver should include different traditional communication in many aspects, such
mobile SoCs can be found types of data in order to separate multiple users’ as communication channels, source and chan-
in AI-Benchmark (https:// messages. More effective and yet efficient inter- nel encoding (decoding) schemes, performance
ai-benchmark.com/rank- pretation algorithms for joint semantic-channel metrics, and so on. Moreover, the design of an
ing_processors.html), which
provides 46 AI and computer
decoding of an intended user should be designed. E2E semantic communication system is related
vision test results obtained by Although an IR based approach was proposed in to the types of messages transmitted, and thus
neural networks running on the literature, it is just the first step in multi-user the DNN structures of source encoder (decod-
smartphones and measured semantic communication systems. Thus, in our er) and channel encoder (decoder) can be very
in terms of more than 100
different AI performance met-
future work, more research efforts should be much different. In particular, the use cases in IoT
rics, such as speed, accuracy, made on the design of low complexity multi-user networks, ICV networks and smart factories were
initialization time, and so on. interpretation algorithms. discussed for possible implementation of seman-

218 IEEE Wireless Communications • February 2022

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.
 tic communications. In addition, the open issues [12] B. Guler and A. Yener, “Semantic Index Assignment,” Proc. It is widely believed
2014 IEEE Int’l. Conf. Pervasive Computing and Communica-
were summarized to highlight the challenges in tion Workshops (PERCOM WORKSHOPS), Budapest, Hunga- that more theoretical
theoretical research and practical implementation ry, Mar. 2014, pp. 431–36. research can definite-
of semantic communications. In summary, seman- [13] B. Gler, A. Yener, and A. Swami, “The Semantic Communi-
ly help to promote
tic communications will definitely play an import- cation Game,” IEEE Trans. Cogn. Commun. Netw., vol. 4, no.
4, Dec. 2018, pp. 787–802. real implementation
ant role in the development of futuristic AI-based [14] Q. Yu et al., “Intelligent Radio for Next Generation Wireless
communication technologies beyond 5G. Communications: An Overview,” IEEE Wireless Commun., of semantic commu-
vol. 26, no. 4, Aug. 2019, pp. 94–101. nication systems, as
Acknowledgment [15] M. Kountouris and N. Pappas, “Semantics-Empowered
Communication for Networked Intelligent Systems,” 2021; theoretical incom-
The work presented in this article was supported in available: https://arxiv.org/abs/2007.11579v3. pleteness in semantic
part by the Natural Science Foundation of China
(No. U1764263) and Taiwan Ministry of Science Biographies communications may
and Technology (Nos. 109-2221-E-006-175-MY3 Xuewen Luo received his B.E. degree in communications engi- restrict its implemen-
and 109-2221-E-006-182-MY3). neering from Jilin University, Changchun, China, in 2017. He
is currently working toward his Ph.D. degree with the School
tation. In addition, it is
of Electronics and Information Engineering, Harbin Institute still a question in both
References of Technology, Harbin, China. His research interests include
industry and academia
[1] E. C. Strinati and S. Barbarossa, “6G Networks: Beyond Shan- physical layer security, intelligently connected vehicle networks,
non Towards Semantic and Goal-Oriented Communica- mobile edge computing, and semantic communications. whether we really need
tions,” Computer Networks, vol. 190, no. 8, May 2021, pp.
1–17. Hsiao-Hwa Chen [S’89, M’91, SM’00, F’10] is currently a Dis-
semantic communi-
[2] W. Weaver, “Recent Contributions to the Mathematical tinguished Professor in the Department of Engineering Science, cations as the exist-
Theory of Communication,” The Mathematical Theory of National Cheng Kung University, Taiwan. He obtained his B.Sc.
Communication, 1949. and M.Sc. degrees from Zhejiang University, China, and a Ph.D.
ing communication
[3] H. Xie et al., “Deep Learning Enabled Semantic Commu- degree from the University of Oulu, Finland, in 1982, 1985, technologies are very
nication Systems,” IEEE Trans. Signal Process., 2021, doi: and 1991, respectively. He has authored or co-authored over
10.1109/TSP.2021.3071210. 400 technical papers in major international journals and con-
mature.
[4] R. Carnap and Y. Bar-Hillel, “An Outline of a Theo- ferences, six books, and more than ten book chapters in the
ry of Semantic Information,” RLE Technical Reports 247, areas of communications. He has served as the general chair,
Research Laboratory of Electronics, Massachusetts Institute TPC chair, and symposium chair for many international confer-
of Technology, Cambridge MA, Oct. 1952. ences. He has served or is serving as an editor or guest editor
[5] J. Bao et al., “Towards a Theory of Semantic Communica- for numerous technical journals. He is the founding Editor-in-
tion,” Proc. 2011 IEEE Network Science Workshop, West Chief of Wiley’s Security and Communication Networks Journal.
Point, NY, USA, 2011, pp. 110–17. He was the recipient of the best paper award at IEEE WCNC
[6] N. Farsad, M. Rao, and A. Goldsmith, “Deep Learning for 2008 and the recipient of the IEEE 2016 Jack Neubauer Memo-
Joint Source-Channel Coding of Text,” Proc. 2018 IEEE Int’l. rial Award. He served as the Editor-in-Chief for IEEE Wireless
Conf. Acoustics, Speech and Signal Processing (ICASSP), Communications from 2012 to 2015. He was an elected Mem-
2018, pp. 2326–30. ber-at-Large of IEEE ComSoc from 2015 to 2016. He is serving
[7] H. Xie et al., “Deep Learning based Semantic Communica- as TPC Chair for IEEE Globecom 2019. He is a Fellow of IEEE,
tions: An Initial Investigation,” Proc. 2020 IEEE Global Com- and a Fellow of IET.
mun. Conf., Taipei, Taiwan, Dec. 2020, pp. 1–6.
[8] H. Xie and Z. Qin, “A Lite Distributed Semantic Communica- Guo Qing received the B.S. degree in radio engineering from
tion System for Internet of Things,” IEEE JSAC, vol. 39, no. 1, Beijing Institute of Posts and Telecommunications in 1985, and
Jan. 2021, pp. 142–53. M.S. and Ph.D. degrees in information and communication engi-
[9] E. Bourtsoulatze, D. Burth Kurka, and D. Gündüz, “Deep neering from Harbin Institute of Technology in 1990 and 1998,
Joint Source-Channel Coding for Wireless Image Transmis- respectively. He is a professor in the School of Electronics and
sion,” IEEE Trans. Cogn. Commun. Netw., vol. 5, no. 3, Sept. Information Engineering at Harbin Institute of Technology, direc-
2019, pp. 567–79. tor of the Key Laboratory of Wideband Wireless Communica-
[10] C. Lee et al., “Deep Learning-Constructed Joint Transmis- tions and Networks, Heilongjiang Province, China. His research
sion-Recognition for Internet of Things,” IEEE Access, vol. 7, interests include satellite communications, space information
2019, pp. 76 547–61. networks and wireless communication networks, etc. He has
[11] Z. Weng, Z. Qin, and G. Y. Li, “Semantic Communica- published one authored book and more than 200 papers in
tions for Speech Signals,” 2020; available: https://arxiv.org/ journals and international conferences.
abs/2012.05369.

IEEE Wireless Communications • February 2022 219

Authorized licensed use limited to: University of Electronic Science and Tech of China. Downloaded on October 25,2022 at 21:06:53 UTC from IEEE Xplore. Restrictions apply.