Digital Twin Based Beam Prediction:

Can we Train in the Digital World and Deploy in Reality?

Shuaifeng Jiang and Ahmed Alkhateeb
School of Electrical, Computer and Energy Engineering - Arizona State University
Emails: {s.jiang, alkhateeb}@asu.edu

Abstract—Realizing the potential gains of large-scale MIMO digital twin-based communications assume that the BSs have
systems requires the accurate estimation of their channels or information on the position, orientation, dynamics, shapes, and
the fine adjustment of their narrow beams. This, however, is materials of the surrounding objects. With this information, the
arXiv:2301.07682v1 [eess.SP] 18 Jan 2023

typically associated with high channel acquisition/beam sweeping

overhead that scales with the number of antennas. Machine and BSs then construct 3D models of the surrounding environments.
deep learning represent promising approaches to overcome these Using the 3D model and ray tracing, the BSs can simulate the
challenges thanks to their powerful ability to learn from prior channels for each communication link in a digital replica of
observations and side information. Training machine and deep the real world. This simulated channel information can then
learning models, however, requires large-scale datasets that are be exploited to aid various communication tasks in the real
expensive to collect in deployed systems. To address this challenge,
we propose a novel direction that utilizes digital replicas of the world. If the digital replica is adequately accurate, the real-
physical world to reduce or even eliminate the MIMO channel world channel acquisition can even be bypassed.
acquisition overhead. In the proposed digital twin aided communi- The digital replica relies on accurate ray tracing to simulate
cation, 3D models that approximate the real-world communication the channels. However, this ray tracing requires a large compu-
environment are constructed and accurate ray-tracing is utilized to tational overhead. Machine learning (ML) models have demon-
simulate the site-specific channels. These channels can then be used
to aid various communication tasks. Further, we propose to use strated promising results in approximating complex functions
machine learning to approximate the digital replicas and reduce with reduced computational overhead [2]. That motivates us to
the ray tracing computational cost. To evaluate the proposed approximate the digital replica in a data-driven approach with
digital twin based approach, we conduct a case study focusing ML, and solve the real-world communication tasks in an end-
on the position-aided beam prediction task. The results show that to-end manner. To further improve the ML performance, we
a learning model trained solely with the data generated by the
digital replica can achieve relatively good performance on the propose to apply transfer learning and fine-tune the ML model
real-world data. Moreover, a small number of real-world data with a small amount of real-world data.
points can quickly achieve near-optimal performance, overcoming Prior work has investigated the relevant sensing-aided wire-
the modeling mismatches between the physical and digital worlds less communication direction [3]–[6]. In a sensing-aided com-
and significantly reducing the data acquisition overhead. munication setup, the communication devices (typically BSs)
Index Terms—Digital twin, MIMO, machine learning, real-
are equipped with various sensors. These sensors are used to
world data, transfer learning, beam selection.
capture sensing information about the surrounding environ-
ments, which are then utilized to aid assorted communication
I. I NTRODUCTION
tasks [3]–[6]. For example, in [3], [4], the authors employ
Multiple-input and Multiple-Output (MIMO) communication camera and LiDARs at the BSs, and train ML models for
gains have been widely investigated in the last three decades. the blockage detection and future beam prediction tasks. ML
With the multiplexing and array gains of these systems, scaling is particularly useful when the direct relationship between the
the number of antennas up is considered as a key characteris- sensing information (e.g. camera and LiDAR) and the channel
tic of current and future wireless communication systems in is hard to characterize. However, these ML-powered solutions
5G and beyond [1]. Realizing these gains, however, requires may require large datasets that are hard to collect in reality.
some knowledge about the channels of which the acquisition The proposed digital twin-based communications is a step
overhead scales with the number of antennas [1]. The channel forward from sensing-aided communications. The digital twins
estimation/feedback (and beam sweeping) overhead makes it make full use of the sensing data by relating them to the
hard for these systems to continue scaling their gains or to communication channels through accurate ray tracing. The
support highly-mobile applications. Machine and deep learning digital twin-based communications can potentially reduce or
provide a promising path for overcoming these challenges by even eliminate the real-world channel acquisition overhead.
leveraging the prior observations to reduce the channel/beam Moreover, the digital twin can be straightforwardly extended
acquisition overhead. Training machine/deep learning models, to solve various communication tasks. The contribution of this
however, requires large-scale datasets that are hard to collect paper can be summarized as follows: (i) We propose to approx-
in reality without penalizing the system performance. imate real-world communications using digital replicas based
In this paper, we propose a novel direction that exploits on accurate 3D model and efficient ray tracing. (ii) The digital
the digital twin to aid wireless communication systems replicas are exploited to simulate communication channels and
with reduced channel acquisition overhead. The proposed solve real-world communication tasks with reduced/eliminated
 Real World: Digital Replica:
Transmitter Transmitter
: Signal propaga!on law : Ray-tracing
: Communica!on environment : EM 3D models

Optimal beam Optimal beam

Receiver Receiver

Fig. 1. This figure shows a real-world communication system and its digital replica. In the real world, the channel between the transmitter and the receiver is
determined by the communication environment and the signal propagation law. The digital replica employs the 3D model to approximate the communication
environment and the ray tracing to model the signal propagation.

channel acquisition overhead. (iii) We approximate the digital The objective of this paper is to solve the communication task
replicas with ML models to further reduce the computational in (1) while eliminating (or significantly reducing) the channel
complexity. To evaluate the proposed digital twin aided wireless acquisition overhead. To that end, we propose a novel research
communication approaches, (v) we build a digital twin dataset direction that aims to solve (1) by approximating the real world
comprising a real-world dataset (from DeepSense 6G [7]) and with a digital replica.
a digital replica dataset based on accurate 3D ray-tracing.
III. F ROM THE R EAL W ORLD TO THE D IGITAL R EPLICA
II. S YSTEM M ODEL AND P ROBLEM F ORMULATION
In the real world, the communication channels H are de-
In this paper, we consider a general MIMO communication termined by the following two key components: The com-
system where A base stations (BSs) serve B mobile user equip- munication environment including the positions, orientations,
ments (UEs). The a-th (a = {1, . . . , A}) BS is equipped with dynamics, shapes, and materials of the BS, the UE, and other
an antenna array of Na elements. The b-th (b = {1, . . . , B}) objects (reflectors/scatterers) in the surroundings, and (ii) the
UE employs an antenna array of Mb elements. Moreover, laws governing the wireless signal propagation phenomena. Let
we assume that the BSs have knowledge of the surrounding E denote the communication environment and g(·) denote the
environments, which includes information about the positions, signal propagation law, the communication channels can then
orientations, dynamics, shapes, materials of the BS, the UEs, be written as
and the other surrounding objects (that can act as reflec-
tors/scatterers). H = g(E). (2)
Without loss of generality, let Hb,a denote the channel
When the communication environment E and the signal prop-
between the a-th BS and the b-th UE. The channels be-
agation law g(·) are known, the solution T ? to the communi-
tween all the BSs and UEs can be then represented by a
cation task can be obtained by substituting (2) into (1) as:
set H = {Hb,a | b = {1, . . . , B}, a = {1, . . . , A}}. Accurate  
information about the channels H is crucial for realizing T ? = ST g(E) . (3)
the potential gains of the MIMO systems; many essential
communications tasks for MIMO systems, such as precod- Nevertheless, the precise ground-truth communication environ-
ing, beamforming/beam tracking, handover, resource allocation, ment E is difficult to obtain, and the exact expression of the
and interference coordination, requires full or partial channel signal propagation law g(·) remains unclear in complex envi-
information. Let T denote the solution space of one such ronments. To that end, we propose to solve the communication
communication task that requires the channel information H. task in (3) by approximating the communication environment
Let T ? denote an optimal solution (where T ? ∈ T ). Further, E and the signal propagation law g(·) in a digital replica.
let ST denote an existing method that optimally solves the Particularly, we approximate the communication environment
communication task given the channel information H. Then, E with Electro-magnetic (EM) 3D models and the signal
T ? can be written as propagation law g(·) with ray tracing.
EM 3D model E: e The EM 3D models contain informa-
T ? = ST (H) , (1)
tion about the positions, orientations, dynamics, shapes and
While the channel information H is vital for MIMO systems, materials of the BS, the UE, and other surrounding objects
obtaining this channel information often requires a large ac- (reflectors/scatterers). Note that this 3D model information
quisition overhead (beam sweeping, channel training/feedback, can be obtained using several approaches. (i) For static and
etc.) that degrades the overall system efficiency. fixed objects such as the neighboring buildings, the BS can
memorize their position, shapes, and materials information band may require more accurate 3D models and ray tracing
since these objects are not likely to change frequently. (ii) In than the beam prediction task in the mmWave band.
the context of sensing-aided information, the BSs can exploit One potential challenge of the digital twin lies in the high
various sensors such as cameras, radars, and LiDARs to obtain computational complexity of accurate ray tracing. This com-
sensing information on both the surrounding stationery and plexity can even increase when the 3D models have more details
dynamic objects. From this sensing information, the BSs can and contain a large number of interacting objects. As a result,
infer the position, orientation, dynamics, shapes, and materials the digital twin can suffer from high latency, which makes it
of these objects. Moreover, once the BS identifies an object, less suitable for real-time applications. Next, we exploit ML to
the BS can use memorized and/or online data to refine the reduce such computational complexity.
information about this object. For instance, if the BS detects
IV. A PPROXIMATING THE D IGITAL R EPLICA WITH
a car of a certain model, the BS can search for the shape and
M ACHINE L EARNING
material information of this car model in its or online database.
(iii) Thanks to the recent development in the internet of things The computational overhead and consequential latency of
[8], objects with communication capability can report/broadcast (5) limits the feasibility of the digital replicas in real-time
their information that are useful for the 3D model. applications. Therefore, it is interesting to design a function
Ray tracing ge(·): Based on the information in the 3D f (·) that processes Ee and approximates the solution in (5) with
models, the channels H can be modeled using stochastic and lower computational complexity and latency. In this paper, we
deterministic channel models. The stochastic channel models take a data-driven approach and learn f (·) with ML.
assume that the propagation parameters such as pathloss, delay Let f (· ; Θ) denote an ML model with Θ representing the
spread, and angle spread, follow certain probability distri- model parameters, the ML model is developed to learn a
butions. However, it is difficult to define these probability mapping function that takes in Ee and produces a solution Tb
distributions for a specific scenario. By contrast, deterministic that approximates the Te? in (5). The objective of the ML
channel modeling methods like ray tracing do not rely on optimization problem can be written as
assumptions about the probability distributions of the propa- n
e? b e
o
max EE∼ e s(T , T ) | E ,
e Υ (6)
gation parameters. Instead, the ray tracing attempts to track f (· ;Θ)
the propagation paths between each transmit-receive antenna
pair based on the geometry and material information in the where Tb is the output of the ML model f (· ; Θ) given the 3D
3D models, which preserves the spatial and temporal consis- model E.e Υe denotes the underlying probability distribution of
tency. In this process, multiple propagation paths are explicitly the 3D models. E{·} is the expectation operator. The optimal
modeled by considering various propagation effects, such as ML model f ? (· , Θ? ) that solves the ML optimization problem
transmission, reflection, scattering, and diffraction. For each in (6) can be written as
propagation path, the ray tracing produces path parameters n   o
including path gain, propagation delay, and propagation angles. f ? (· ; Θ? ) = arg max EE∼ e?
e s T , f p(E); Θ , (7)
e Υ
e
These propagation path parameters generated by the 3D f (· ;Θ)
model and ray tracing can then be exploited to construct the
channels in the digital replica. The channel impulse response where p(E) e is a function that extracts useful features from
h(t) between a transmit-receive antenna pair in the digital
e the 3D model; depending on the communication configurations
replica can be written as the sum of all the L multi-path and tasks, not all information in the 3D model Ee is useful.
components, which is given by Therefore, we only input the useful features p(E)e to the ML
model.
L
X The optimal ML model f ? (p(E);e Θ? ) can be obtained via a
h(t) =
e αl δ(t − τl )Gt (θlAoD )Gr (θlAoA ), (4) supervised learning approach. First, we randomly sample a total
l=1
number of D 3D models from the 3D model distribution Υ. e For
where αl and τl represent the complex gain and propagation the d-th 3D model sample Ed , we calculate the corresponding
e
delay of the l-th path. The angle of arrival and angle of solution Ted? to the communication task T using (5). This way,
departure are denoted by θlAoA and θlAoD . Gt and Gr are the we can construct datasetof D data points, and the d-th data
radiation patterns of the transmit and receive antennas. point can be written as Eed , Ted? . Then, we train the ML to
With accurate 3D model Ee and ray tracing ge(·), the solution minimize a loss function on this dataset. The loss function
T ? in (3) can be approximated using the solution obtained from measures how much the ML model approximation defers from
the digital replica, Te? , as shown by the digital replica solution, which is given by
 
Te? = ST ge(E)
e . (5) D
X   
Jtrain = − s Ted? , f p(Eed ); Θ , (8)
To investigate the accuracy of the digital replica, we define d=1
s(T ? , Te? ) as the similarity function of Te? and T ? . The accuracy  
requirements on the 3D model and ray tracing can vary in Since the training data for the ML model f p(E); e Θ is
different communication configurations and tasks. For instance, generated from the digital replica, a large amount of training
the channel state information prediction task in the sub-6GHz data can be relatively accessible for the training process.
However, when the ML model is trained solely on the data Y (East)
BS (origin)
generated by the digital replica, the ML model can be biased
by the impairments in the 3D model and/or the ray tracing. X (North)

In this case, the performance of the ML approximation is UE grid in the

BS orienta!on Physical world
user loca!on
limited by the performance of the digital replica solution. To digital replica BS FoV
that end, a small amount of real-world data can be utilized to
fine-tune the ML model that is trained on the digital replica
dataset. This transfer-learning process is expected to calibrate
the impairments brought by the digital replica and improve the
performance of the ML model. The loss function for the transfer
learning can be similarly written as (8). Fig. 2. This figure shows the geometry layout of the real-world (Scenario 1
from DeepSense [7]) data collection environment with the 3D model of the
V. C ASE S TUDY: P OSITION - AIDED B EAM P REDICTION digital replica overlaid on the same layout.

To verify the efficacy of the digital twin-based communica-

tions, we conduct an initial case study where the UE position VI. E XPERIMENTAL S ETUP
is exploited for the mmWave beam prediction task. In this section, we explain the real-world dataset, digital
In our case study, we consider a communication system replica dataset, and the neural network (NN) adopted to evaluate
where a BS with N antenna element is communicating with the proposed digital-twin aided communications.
a single-antenna UE at the mmWave band. For simplicity, A. Real-World Data: DeepSense Scenario 1
we assume the channel between the BS and the UE satisfies
the narrowband block-fading channel model. In the downlink We adopt the DeepSense Scenario 1 for the real-world dataset
transmission, the BS sends a complex symbol s using the [7]. The data collection testbed incorporates a UE (vehicle) and
beamforming vector f ∈ CN ×1 . The downlink receive signal a static BS. The UE carries a GPS sensor, and an omnidirec-
at the UE can be written as tional 60 GHz mmWave transmitter. The BS uses a 16-element
uniform linear array to receive the signals from the UE. The BS
y = hH f s + n, (9) adopts a beam-steering codebook with 16 beamforming vector.
In the data collection process, the vehicle passes the two streets
where h ∈ CN ×1 is the channel vector between the BS and the in front of the BS. At each time step, the BS performs a full
UE. The transmitted symbol s is constrained by the constant beam sweeping to measure the receive powers of all 16 beams,
average transmit power P , i.e., E[|s|]2 = P . The n represents and the UE captures its GPS position. The BS beam power
the additive white Gaussian noise and satisfies n ∼ CN (0, σ 2 ). and UE position measurements are synchronized. The i-th data
We assume that the BS adopt a beamforming codebook F = point can be written as (pi , ui ), where pi ∈ R16×1 contains
{f1 , . . . , fQ } incorporating Q pre-defined beamforming vectors. the power measurements of the 16 beams and the ui ∈ R2×1
To account for the constant-modulus constraint due to the denotes the UE position. Fig. 2 presents a visualization of all
analog beamforming architecture, the beamforming vectors in data points with the color indicating the beam of the highest
F satisfy kfq k2 = 1, ∀q ∈ {1, . . . , Q}. power.
Let Tpr denote the beam prediction task solution space whose
B. Digital Replica (Synthetic) Data
objective is to find the optimal beam f ? out of the codebook
F . The optimal solution of the beam prediction task Tpr given We build a digital replica of the real-world data collection
the channel h can be obtained by maximizing the receive SNR, environment. Fig. 2 shows the 3D model of the digital replica
which is given by overlaid onto the real-world environment. The digital replica
consists of a BS and two UE grids. The UE grids cover
2
q ? = STpr (h) = arg max fqH h , (10) the area where the UE can appear, i.e., the two streets in
q∈{1,...,Q} the real-world environment. The UE grids is discretized into
multiple candidate UE positions with the spacing of 0.1 meter.
where q ? denotes optimal beam index. Substituting the 3D
The channel between each candidate UE position and the BS
model Ee and the ray tracing ge(·) into (10), the beam prediction
is simulated using ray tracing. From the propagation path
solution obtained from the digital replica can be written as e d between
parameters produced by the ray tracing, the channel h
2 the d-th candidate UE position and the BS can be obtained by
qe? = arg max fqH ge(E)
e . (11) (4). Then, for every candidate UE position, we can compute the
q∈{1,...,Q}
power of all the beams in a pre-defined codebook. Lastly, for
Given the two beam indices q ? and qe? , their similarity func- the d-th candidate UE position, we obtain a synthetic data point
tion can then be defined using the Kronecker delta function pd , u
(e e d ) consisting of the beam power vector and the position.
s(q ? , qe? ) = δ(q ? − qe? ). Since we focus on the position-aided Note that, due to the 60 GHz frequency and the geometry, the
beam prediction task in our case study, the feature extraction channels in the real-world data are dominated by the line-of-
function for the ML model is defined as p(E) = u, with u sight (LoS) components. Therefore, in the digital replica, we
denoting the position of the UE. also focus on the LoS paths and neglect the reflectors/scatterers.
 15 given a position, the optimal beam obtained in the real-world
Optimal Beam Index nearest neighbor in digital replica
real dataset should be similar to that indicated by the digital twin.
10 As discussed in Section VI, each data point in the real-world
or digital twin dataset contains a measured beam power vector
and a UE position vector (p, u). For each of the real-world
5
data points (pi , ui ), we find its nearest neighbor (epd , u
e d ) in
the digital replica, i.e., such that the distance between ui and
10 20 30 40 50 60 70 80 90 100 110 e d is minimized. Since the interval of the UE grid in the
u
Time Sample Index (veichle moves from left to right) digital replica is small (0.1 m), the nearest neighbors in the
Fig. 3. This figure shows the optimal beam indices of the real-world data
digital replica can be considered an accurate approximation of
points and their nearest neighbors as the UE passes by the BS. the real-world data points. In Fig. 3, we compare the optimal
beam indices of the real-world data points and their nearest
1 neighbors in the digital twin as the UE passes by the BS from
0.9 the left to the right. It can be seen that the real-world optimal
beam index matches well with the optimal beam index of the
Percentage

0.8

0.7
nearest neighbor in the digital replica. The maximum difference
Top-1
Top-2
between the optimal beam indices obtain from the real world
0.6
Top-3 and the digital replica is two, which translates to around 10◦
0.5 Top-4
Top-5 difference in beam angle.
0.4
Accuracy Relative Receive Power
Next, we directly apply the optimal beams of the nearest
neighbors in the digital replica to the corresponding data points
Fig. 4. This figure shows the top-k accuracy and relative receive power in the real world. Fig. 4 presents the top-k accuracy and relative
performance obtained by the nearest neighbor in the digital replica. receive power obtained by this nearest neighbor approach. It
can be seen that, the top-1 accuracy and relative receive power
are 56.1% and 84.8%, respectively. Despite the relatively low
C. ML Model
top-1 accuracy, the high receive power implies that the nearest
We employ a fully connected NN architecture incorporating neighbor often provides sub-optimal beams with near-optimal
two hidden layers. Each hidden layer has 256 nodes and applies receive power. It is worth highlighting that, with the nearest
the ReLU activation. The input to the network is the position neighbor approach, 84.8% receive power can be obtained
the UE in both Cartesian and polar coordinates. Although this using the digital twin without any beam training or channel
input produces repeated information, we found that they lead estimation.
to more stable ML performance with our dataset. To normalize
the input data, we re-scale the distance in the polar coordinates B. Can the Model Trained in the Digital World Work in
by the maximum distance, and re-scale the x and y in the Reality?
Cartesian coordinates by a shared maximum absolute value. The large computational latency of the digital replica may not
The output layer adopts the standard classification setting, i.e., suit the need for real-time applications. Here, we investigate the
it has 16 nodes and employs the softmax activation function. performance of using the NN model to solve the digital replica
Each node produces a confidence score for one beam in the 16- simulation and the beam prediction task to reduce computation.
beam codebook being the optimal beam. We employ the Adam In Fig. 5, we train the NN model on the real-world data or the
optimizer, and the learning rate is set to 1 × 10−2 and 1 × 10−4 digital replica data, and test the NN model on unseen real-
for the training and fine-tuning. world data. We experiment with two beamforming codebook
in the digital replica, namely the measured codebook and the
VII. E VALUATION R ESULTS uniform beam codebook. In the measured codebook, we extract
In this section, we evaluate the beam prediction performance the beam angles of the beam patterns used in the DeepSense
of the proposed digital twin and ML approaches. We adopt two testbed, and construct a beam steering codebook with these an-
performance metrics: (i) The top-k accuracy is defined as the gles. For the uniform beam codebook, we uniformly discretize
percentage of the test data points whose ground-truth optimal the BS field-of-view into 16 angles and construct a DFT beam
beam lies in the predicted k beams with the highest scores. (ii) steering codebook accordingly.
The top-k relative receive power measures the ratio between the It can be seen from Fig. 5 that training on the digital replica
highest receive power achieved by the top-k predicted beams data using the measured codebook can achieve a relatively good
and the receive power of the ground-truth optimal beam. top-2 accuracy of 91.4% when testing on unseen real-world
data. It is worth noting that this ML approach does not need
A. Does the Digital Twin Matches the Reality? any real-world training data. Furthermore, the NN model can
The key idea presented in this paper is to utilize the digital converge with only 100 data points from the digital replica,
replicas to simulate real-world communication systems. Here, which demonstrates its high data efficiency. It can be also
we first investigate the accuracy of the digital twin in the observed that the (inaccurate) uniform beam steering codebook
context of the position-aided beam prediction task. Ideally, decreases the top-2 accuracy to 84.8%. If we want to rely only
 1 1

0.9 0.9
Accuracy / Relative Receive Power

Accuracy / Relative Receive Power

Transfer learning

0.8 0.8
0.95

0.7
0.7
0.9 measured
codebook
0.6 uniform
0.6 codebook
0.85
0 5 10 15 20
Acc. (trained on real) 0.5 Acc. (trained on real)
0.5 Power (trained on real) Power (trained on real)
Acc. (trained on synth.) measured codeboook 0.4 Acc. (transfer learining) measured codebook
Power (trained on synth.) measured codebook Power (transfer learining) measured codebook
0.4
Acc. (trained on synth.) uniform codebook Acc. (transfer learining) unifrom codebook
Power (trained on synth.) uniform codebook 0.3 Power (transfer learining) unifrom codebook
0.3
0 50 100 150 200 0 20 40 60 80 100
Number of Training Data Points Number of Real Data Points Used for Training

Fig. 5. This figure shows the top-2 performance obtained by training the neural Fig. 6. This figure shows the top-2 performance by applying transfer learning
network on the real-world (real) or digital twin (synthetic) data and testing on to the NN model using the measured and the uniform beamforming codebooks.
the real-world data.

we propose to approximate the digital replica using ML with

on the digital replica data then we need very good modeling to reduce the computational overhead. Lastly, we propose to
for important features such as the beamforming codebook. improve the ML model with a small amount of real-world data
C. Does Transfer-Learning Improve the ML Performance? to compensate for the impairments in the digital replica. To
evaluate the digital twin-based communications, we conduct an
As discussed in Section IV, if the ML model is only
initial case study considering the position-aided beam predic-
trained on the data generated by the digital replica, the ML
tion task. Experiments results can be summarized as follows.
performance will be limited by the modeling impairments of
(i) The optimal beams in the digital replica can match the
the digital replica. To overcome this limitation, we propose
optimal beams in the real world. (ii) Training ML models
to fine-tune the ML model on a small amount of real-world
on the digital replica data can achieve relatively good beam
data after trained on the digital replica data. Fig. 6 shows the
prediction performance. (iii) A small number of data points can
top-2 accuracy and relative receive power performance using
quickly fine-tune the ML performance to near-optimal, and the
this transfer learning approach. As a comparison, we show
digital twin can be used to save real-world training data. (iv)
the performance achieved by directly training the NN on real-
Mismatches between the digital replica and the real world can
world data. It can be seen that a small number of real-world
be calibrated by a small amount of real-world data.
data points (less than 20) can quickly improve the NN to
achieve a near-optimal performance with the transfer learning R EFERENCES
gains. Directly training the NN on real-world data, however, [1] F. Boccardi, R. Heath, A. Lozano, T. Marzetta, and P. Popovski, “Five
requires around 100 data points to converge to the optimal disruptive technology directions for 5G,” IEEE Communications Magazine,
performance. This demonstrates that the digital twin can be vol. 52, no. 2, pp. 74–80, Feb. 2014.
[2] P. Dong, H. Zhang, G. Y. Li, I. S. Gaspar, and N. NaderiAlizadeh, “Deep
used to pre-train ML models to efficiently save the cost and cnn-based channel estimation for mmwave massive mimo systems,” IEEE
effort required by real-world data collection. The uniform beam J. Sel. Top. Signal Process., vol. 13, no. 5, pp. 989–1000, 2019.
steering codebook leads to lower performance compared to the [3] G. Charan, M. Alrabeiah, and A. Alkhateeb, “Vision-aided dynamic
blockage prediction for 6G wireless communication networks,” in 2021
measured beamforming codebook when fine-tuned on a very IEEE Int. Conf. on Commun. Workshops. IEEE, 2021, pp. 1–6.
limited amount of real-world data. However, when the NN [4] S. Jiang, G. Charan, and A. Alkhateeb, “LiDAR Aided Future Beam
model is fine-tuned on more than 20 real-world data points, Prediction in Real-World Millimeter Wave V2I Communications,” arXiv
preprint arXiv:2203.05548, 2022.
the performance of the two codebooks become very similar. [5] U. Demirhan and A. Alkhateeb, “Integrated sensing and communication
This indicates that the mismatches between the real world for 6G: Ten key machine learning roles,” 2022. [Online]. Available:
and the digital replica can be efficiently calibrated by a https://arxiv.org/abs/2208.02157
[6] A. Ali, N. González-Prelcic, and R. W. Heath, “Millimeter wave beam-
small amount of real-world data. selection using out-of-band spatial information,” IEEE Trans. on Wireless
Commun., vol. 17, no. 2, pp. 1038–1052, 2017.
VIII. C ONCLUSION [7] A. Alkhateeb, G. Charan, T. Osman, A. Hredzak, and N. Srinivas,
This paper proposes a novel direction that exploits the digital “DeepSense 6G: A large-scale real-world multi-modal sensing and
communication dataset,” available on arXiv, 2022. [Online]. Available:
twin to aid wireless communications. Using the 3D model of https://www.DeepSense6G.net
the communication environment and ray tracing, the BS can [8] L. Chettri and R. Bera, “A comprehensive survey on internet of things
simulate the communication system in a digital replica, which (IoT) toward 5G wireless systems,” IEEE Internet of Things J., vol. 7,
no. 1, pp. 16–32, 2019.
can be utilized to solve various communication tasks with
reduced or eliminated channel acquisition overhead. Further,