IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL.

21, 2024 3501805

A Radar Target Tracking Algorithm Based on

Learning Displacement
Senlin Xia , Yutao Xiang , Kui Xiong , Graduate Student Member, IEEE,
Shisheng Guo , Member, IEEE, and Guolong Cui

Abstract— This letter considers the problem of target detection subsequent tracking algorithms to accurately track the targets’
and tracking with millimeter-wave (mm-wave) frequency- motions over time. Recently, deep learning-based detection
modulated continuous-wave (FMCW) radar and proposes a methods have been demonstrated to outperform the CFAR
tracking method via learning the displacement of the target
in successive frames. First, a convolutional neural network detectors in real-world environments. In [5], a dual-channel
(CNN), namely, detection and displacement network (DDNet), convolutional neural network (MDCCNN)-based detection
is trained to predict the target positions at current frame and method with the false alarm controllable classifier is pro-
the displacement relative to last frame simultaneously with the posed for complex environment. Wang et al. [6] propose the
range-azimuth (RA) spectrum in two adjacent frames. Then, complex-valued U-Net (CV-UNet)-based clutter suppression
the detections are associated through the predicted displacement
to construct the trajectories. Finally, the effectiveness of the method which considers the amplitude and phase information
proposed method is validated on both simulated and real-world of echoes to improve the performance of target detection.
datasets. In [7] and [8], a deep radar detection network (DRDNet) which
Index Terms— Deep learning, maneuvering target, millimeter- is the fusion of a range-Doppler (RD) detection convolutional
wave (mm-wave) radar, target detection, target tracking. neural network (CNN) and an angle detection CNN is pro-
posed to predict range, radial velocity, azimuth, and elevation
I. I NTRODUCTION of targets at the same time. Despite their respective advantages,
however, all the above detection methods only consider the raw

P OSITIONING and tracking using millimeter-wave (mm-

wave) radar has recently gained significant attention in the
field of intelligent homes, autonomous vehicles, and robotics
data at current frame and the information at previous frames is
ignored. And the association process, such as nearest neigh-
bor data association (NNDA) [9], joint probabilistic density
[1]. The available methods for detection and tracking can be
function (JPDA) [10], and multiple hypothesis tracker (MHT)
divided into two main categories, i.e., tracking-after-detection
[11], plays a crucial role in target tracking. NNDA is simple
(TAD) and tracking-before-detection (TBD) [2]. The tradi-
and effective, but the false alarms may lead to association
tional TAD methods consist of two stages. First, a constant
errors. JPDA relies on the assumption of a known and fixed
false alarm rate (CFAR) detector [3] is designed to get point number of targets, which is always an unknown parameter
target measurements from received radar echoes at the detec- in practical applications. MHT is a powerful data association
tion stage. Then, tracking methods, such as Kalman filter (KF) method for tracking, but the computational complexity grows
and Hungarian algorithm, are used to associate the targets in exponentially with the increase of targets and MHT assumes
successive frames, predict, and update the states of the targets. that targets follow a well-defined motion model.
Compared with the TAD methods, the TBD methods provide In this letter, unlike the traditional TAD methods that
a better detection performance when the signal-to-noise ratio consider detection and tracking as two separate parts and
(SNR) is low [4]. Nevertheless, processing multiframe data also unlike the TBD methods that process multiframe data
without threshold detection results in a very large amount of simultaneously without threshold detection, an end-to-end
computation and data storage for the TBD methods, and the tracking framework based on learning displacement (TBLD)
accumulation process relies on the accuracy of the target’s is proposed. First, a detection and displacement network
a priori knowledge. (DDNet) is designed to take the range-azimuth (RA) data in
In the context of target tracking, a good detection result two consecutive frames as the input. And the DDNet outputs
provides reliable information of the targets which enables the detection results at current frame and the displacement of
Manuscript received 24 June 2023; revised 14 October 2023 and detected targets at current frame relative to the previous frame.
3 December 2023; accepted 10 January 2024. Date of publication 17 January Then, the detections are associated through time via the pre-
2024; date of current version 5 February 2024. This work was supported dicted displacement to form the trajectories, rather than relying
in part by the National Natural Science Foundation of China under Grant
62271126 and Grant 62101098, in part by the China Postdoctoral Science on traditional tracking methods which assume the motion
Foundation under Grant 2022M710622, and in part by the Guangdong models such as the constant velocity (CV) model, the constant
Province Key Areas of Research and Development Program under Grant acceleration (CA) model, constant turn (CT) model, and CT
2020B0909050002. (Corresponding author: Guolong Cui.)
The authors are with the School of Information and Communication Engi- rate and CV (CTRV) model for the targets are known [12].
neering, University of Electronic Science and Technology of China, Chengdu The main contributions of this letter are listed as follows.
611731, China (e-mail: [email protected]; [email protected];
[email protected]; [email protected]; [email protected]). 1) An end-to-end framework that integrates detection and
Digital Object Identifier 10.1109/LGRS.2024.3355246 tracking is proposed for target tracking with the
1558-0571 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: St. Petersburg State University. Downloaded on March 01,2024 at 16:04:55 UTC from IEEE Xplore. Restrictions apply.
3501805 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 21, 2024

mm-wave radar. The framework consists of predicting

the position along with the displacement and association
through the displacement. By employing this framework,
accurate tracking of the maneuvering target can be
achieved without any particular model assumptions.
2) A CNN which predicts detections and displacement in Fig. 1. Overview of the proposed TBLD.
TBLD is proposed to process data in two successive
frames, referred to as DDNet. Through taking the RA there is a target at each range unit and azimuth unit. The
spectrum in two consecutive frames and simultaneous displacement of the kth target at time t relative to time t −1 can
predicting the positions, i.e., range and azimuth, of the be expressed as follows:
targets at current frame, and the relative displacement of
the targets compared to their positions in the previous d t,k = pt,k − pt−1,k = [rt,k − rt−1,k , at,k − at−1,k ] ∈ R2 . (3)
frame at the same time. The displacement label for TBLD can be expressed as Dt ∈
3) Experimental insights and validation on both simu- R H ×W ×2 , where
lated and real-world datasets are conducted for the 
proposed method. Compared with the state-of-the-art  d t,k
, if |h − rt,k | ≤ 1 and |w − at,k | ≤ 1
(SOTA) data-driven single-frame detection methods, the Dt (h, w, :) = τ
 [0, 0], otherwise.
proposed method improves the detection performance.
And TBLD outperforms the traditional KF and extended (4)
KF (EKF) in tracking the maneuvering target.
Here, Dt (h, w, :) denotes all the elements in the hth row and
wth column of Dt , and τ is the scale parameter. The first
II. M ETHOD layer denotes the range displacement, while the second layer
In this section, we present the TBLD framework which represents the azimuth displacement.
consists of DDNet, association through displacement, and
trajectory management, as shown in Fig. 1. The three steps
B. DDNet
of the TBLD framework are specifically discussed in the
following sections. The proposed architecture of DDNet is shown in Fig. 2.
The network takes the RA spectrum at time t M t and the RA
spectrum at time t −1 M t−1 as inputs, and produces a detection
A. Preliminary map for time t St ∈ R H ×W ×2 as well as a displacement map
Consider the target-tracking problem with a mm-wave radar O t ∈ R H ×W ×2 .
working in time division multiplexing-multiple-input multiple- Because the number of targets is small compared to the
output (TDM-MIMO) mode. People are targets and they move RA map, an imbalance exists between the space occupied by
around freely in an open area when the radar collects data. targets and noise. To address this issue, the Dice loss [14] is
First, the RD -azimuth (RDA) map can be obtained by apply- employed as the detection loss function in the training process,
ing the discrete Fourier transform (DFT) to the intermediate defined as follows:
frequency (IF) signal along each sampling dimension [13].
Then, the RDA map is projected onto the RA plane by ℓdet
2
w=1 Pt (h, w, c)St (h, w, c)
P H PW
integrating along the Doppler dimension. In this way, at each X
h=1
time-step t, the RA spectrum is denoted as M t ∈ R H ×W , = 1−
w=1 Pt (h, w, c) + w=1 St (h, w, c)
P H PW 2
P H PW 2
c=1 h=1 h=1
where H and W denote the number of points of range
(5)
sampling and azimuth sampling, respectively. Assuming there
are K targets, the true position of the kth (k = 1, 2, . . . , K ) where St (h, w, c) denotes the value of the element in the hth
target at time t can be expressed as pt,k = [rt,k , at,k ] ∈ R2 , row, wth column, and cth depth of the matrix St .
where rt,k and at,k are the indices of the range and angle bins The L1 loss is used as the displacement loss function, which
where the kth target is located at time t, respectively. The can be expressed as follows:
position label for TBLD can be expressed as P t ∈ R H ×W ×2 ,
and Pt (h, w, c) represents the value of the element in the ℓdisp
h ∈ [1, H ]th row, w ∈ [1, W ]th column, and c ∈ [1, 2]th
P H P W P2
h=1 w=1 c=1 |Dt (h, w, c) − Pt (h, w, 2)Ot (h, w, c)|
depth of the matrix P t , with =
w=1 Pt (h, w, 2)
P H PW
( h=1
1, if |h − rt,k | ≤ 1 and |w − at,k | ≤ 1 (6)
Pt (h, w, 2) = (1)
0, otherwise with Dt (h, w, c) and Ot (h, w, c) denoting the values of the
Pt (h, w, 1) = 1 − Pt (h, w, 2) (2) element in the hth row, wth column, and cth depth of the
matrix Dt and O t , respectively. Thus, the total loss function
where |·| denotes the absolute value of a real number. The for training DDNet can be formulated as follows:
first layer denotes whether there is no target at each range unit
and azimuth unit, while the second layer represents whether ℓtot = ℓdet + ℓdisp . (7)

Authorized licensed use limited to: St. Petersburg State University. Downloaded on March 01,2024 at 16:04:55 UTC from IEEE Xplore. Restrictions apply.
XIA et al.: RADAR TARGET TRACKING ALGORITHM BASED ON LEARNING DISPLACEMENT 3501805

Fig. 2. Architecture of DDNet.

C. Association Through Displacement management measures were conceived to deal with these
After training DDNet by minimizing ℓtot , fix the parameters problems.
of DDNet. The locations with the target classification score of 1) Initialize Trajectories: For every unmatched detection,
DDNet, i.e., St (:, :, 2), greater than 0.5 are considered as the a new trajectory is initialized. Specifically, the position of the
targets. It is reasonable to determine there is a target, because detection is recorded and the consecutive matching frames and
it means the possibility of target is greater than background. trajectory ID are initialized to 0, which can be expressed as
And through cohesion extended targets to points, we obtain follows:
the detections Dt = { p̂t,1 , p̂t,2 , . . . , p̂t,M }, where M denotes )
Tt = Tt−1 ∪ ( p̂t,m , 0, 0), / {m x }min(M,N
m∈ x=1 . (9)
the number of detections at time t, m ∈ [1, M], and p̂t,m =
[r̂ t,m , â t,m ] ∈ R2 . Here, r̂ t,m and â t,m denote the indices of 2) Update Trajectories: The number of consecutive match-
the range and angle bin where the mth detection is located at ing frames of unmatched trajectories are set to 0, and the
time t, respectively. And the displacement corresponding to position vector is updated through the recorded displacement
the detections can be written as Ot = { d̂ t,1 , d̂ t,2 , . . . , d̂ t,M }, vector, which can be written as follows:
where d̂ t,m = τ O t (r̂ t,m , â t,m , :). ξn = 0, / {n x }min(M,N
n∈ )
(10)
x=1
With a sufficiently good detection and displacement pre- )
p̃t,n = p̃t−1,n + d̃ t−1,n , /
n∈ {n x }min(M,N
x=1 . (11)
diction, Hungarian matching algorithm is used to associate
detections across time. The trajectories at time step t − 1 can If a trajectory is matched with a detection, the position,
be expressed as Tt−1 = {( p̃t−1,n , d̃ t−1,n , ξn , i n )}n=1
N
, where N the displacement, and the consecutive matching frames are
denotes the number of trajectories at time t − 1, p̃t−1,n ∈ updated as follows: p̃t,n x = p̂t,m x , d̃ t,n x = d̂ t,m x , and ξn x =
R2 signifies the 2-D position vector of the nth object at time ξn x + 1, where (n x , m x ) ∈ A.
t − 1, d̃ t−1,n ∈ R2 stands for the 2-D displacement vector 3) Output Trajectories: To reduce false trajectories caused
of the nth object at time t − 1, ξn represents the number of by noise, the trajectories are output if they are matched for at
consecutive matching frames for the nth trajectory, and i n is least ξmin consecutive frames, i.e., ξn ≥ ξmin .
the trajectory identification (ID) for the nth trajectory. The cost
matrix for Hungarian matching algorithm can be represented III. E XPERIMENT R ESULTS
as C ∈ R N ×M . The element in the nth row and mth column of A. Evaluation Metrics
C, i.e., the cost of the association between the nth trajectory
and the mth detection can be written as follows: 1) Probability of Target Detection (Pd ): The proportion of
instances where the target is correctly detected within a range
C(n, m) = ∥ p̃t−1,n − ( p̂t,m − d̂ t,m )∥1 (8) of three resolution units. It is utilized to measure the ability
of the system to correctly detect the presence of targets, with
where ∥·∥1 denotes the L1 norm of a vector. The assignment higher Pd indicating better detection performance.
)
results can be formulated as A = {(n x , m x )}min(M,N
x=1 , where 2) Probability of Target False Alarm (P f a ): The rate at
n x , m x , and min(·) denote the index of the xth matched which a target is detected when no target is actually present.
trajectory, detection, and minimum function, respectively. It is a metric for assessing the overall performance of a target
detection system, with lower P f a indicating better detection
performance.
D. Trajectory Management 3) Root Mean Square Error: The square root of the mean
The main problems that trajectory management aims to of the squared errors between the estimated positions and
address include: 1) trajectory initialization; 2) trajectory the ground truth positions. It provides a quantitative measure
updating; and 3) trajectory output. The following trajectory of the average deviation between the estimated and actual

Authorized licensed use limited to: St. Petersburg State University. Downloaded on March 01,2024 at 16:04:55 UTC from IEEE Xplore. Restrictions apply.
3501805 IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, VOL. 21, 2024

Fig. 3. Experimental scene and radar parameters.

Fig. 4. Comparison of detection probability and false alarm rate with SNR
for different methods. (a) Detection probability Pd versus SNR (dB). (b) False
alarm rate P f a versus SNR (dB).

positions for tracking, with lower root mean square error

(RMSE) indicating better tracking performance.

B. Implementation Details
1) Details of Radar: The mm-wave radar features one trans-
mit antenna and four receive antennas, forming a virtual array Fig. 5. Tracking results of TBLD, KF, and EKF for a simulated maneuvering
with four antennas each in the horizontal directions, where target. The arrow indicates the direction of the target’s motion. (a) Trajectories.
the inter-element spacing of receiver is the half wavelength. (b) RMSE against the observation time.
The experimental scenario and the detailed parameters of the
mm-wave radar are shown in Fig. 3. In all the experiments, 2) Comparison of Tracking Performance: To verify the
the noise is modeled as zero-mean Gaussian with variance effectiveness of TBLD, the proposed tracker, i.e., association
σ 2 = 1. The region of interest 8 = [−90◦ , 90◦ ]. The RA through displacement, is compared with CA model-based KF
maps are outputted by the radar at every time-step and have and CTRV model-based EKF using the same detector DDNet.
a dimension of 256 × 256, i.e., H = W = 256, due to the As shown in Fig. 5(a), the target exhibits varying movement
resolution used for the DFT along the fast time, angular, and patterns across the 50 observation frames. Specifically, the
slow time dimensions respectively and integration along the target follows a CV model in the first ten frames, and changes
Doppler dimension. And the SNR is defined as the ratio of to a CT model in the next 20 frames, and finishes the
signal power to noise power on the RA plane. movement with a CA model in the last 20 frames. And
2) Details for Training DDNet: The experiments are carried Fig. 5(b) reports the RMSE versus the frame index. It can be
out on a server with 2 NVIDIA GeForce RTX 4090 GPUs seen that TBLD has a more stable tracking performance than
and Pytorch framework. DDNet is trained at the batch size of KF and EKF especially when the maneuvering target changes
32 for 100 epochs and the Adam optimizer with a learning its motion pattern. And the main advantage of the proposed
rate of 0.001 and weight decay of 10−4 is used with the scale method is that it does not need to know the motion model of
parameter τ = 5. And ξmin is set as 2. targets and set parameters of filters.

C. Numerical Simulations D. Actual Measurements

1) Comparison of Detection Performance: To show the To further verify the effectiveness of the proposed method,
superiority of detection and predicting the displacement from some experiments are carried out in real-world dataset.
successive frames, the SOTA data driven single frame detec- 1) Comparison of Detection Performance: The detection
tion methods which only consider the echoes at current are performance in measured data is shown in Table I, where
taken as the counterparts. As shown in Fig. 4, the detection DDNet has the higher detection probability Pd and the lower
performance of DDNet is compared with U-Net and DRDNet. false alarm rate P f a than the other two methods. The result is
The detection results show that the proposed method improves consistent with Fig. 4.
Pd and has a lower P f a compared with the SOTA data-driven 2) Comparison of Tracking Performance: In Fig. 6(a), two
detection methods, especially when SNR ≥ 5 dB. maneuvering targets freely walk, run, and make turns during

Authorized licensed use limited to: St. Petersburg State University. Downloaded on March 01,2024 at 16:04:55 UTC from IEEE Xplore. Restrictions apply.
XIA et al.: RADAR TARGET TRACKING ALGORITHM BASED ON LEARNING DISPLACEMENT 3501805

TABLE I
D ETECTION R ESULTS IN R EAL -W ORLD DATASET

Fig. 7. RA image and corresponding target trajectories at 200th frame.

the proposed method improves the detection performance

compared with SOTA detection methods and tracks the maneu-
vering target more accurately than traditional KF and EKF
when the target changes its motion mode.

R EFERENCES
[1] J. Pegoraro, F. Meneghello, and M. Rossi, “Multiperson continuous
tracking and identification from mm-wave micro-Doppler signatures,”
IEEE Trans. Geosci. Remote Sens., vol. 59, no. 4, pp. 2994–3009,
Apr. 2021.
[2] W. Yi, Z. Fang, W. Li, R. Hoseinnezhad, and L. Kong, “Multi-frame
track-before-detect algorithm for maneuvering target tracking,” IEEE
Trans. Veh. Technol., vol. 69, no. 4, pp. 4104–4118, Apr. 2020.
[3] A. Coluccia, A. Fascista, and G. Ricci, “Design of customized adaptive
radar detectors in the CFAR feature plane,” IEEE Trans. Signal Process.,
vol. 70, pp. 5133–5147, 2022.
[4] E. Grossi, M. Lops, and L. Venturino, “A novel dynamic programming
algorithm for track-before-detect in radar systems,” IEEE Trans. Signal
Process., vol. 61, no. 10, pp. 2608–2619, May 2013.
[5] X. Chen, N. Su, Y. Huang, and J. Guan, “False-alarm-controllable radar
detection for marine target based on multi features fusion via CNNs,”
Fig. 6. Tracking results of TBLD and KF for two maneuvering targets in IEEE Sensors J., vol. 21, no. 7, pp. 9099–9111, Apr. 2021.
real world. The arrow illustrates the movement direction of the two targets. [6] Y. Wang, W. Zhao, X. Wang, J. Chen, H. Li, and G. Cui, “Nonhomo-
(a) Trajectories. (b) RMSE against the observation time. geneous sea clutter suppression using complex-valued U-Net model,”
IEEE Geosci. Remote Sens. Lett., vol. 19, pp. 1–5, 2022.
[7] D. Brodeski, I. Bilik, and R. Giryes, “Deep radar detector,” in Proc.
the 256 observation frames. And as shown in Fig. 6, TBLD IEEE Radar Conf. (RadarConf), Apr. 2019, pp. 1–6.
can provide lower RMSE than KF and EKF when tracking [8] W. Jiang, Y. Ren, Y. Liu, and J. Leng, “A method of radar target detection
maneuvering targets. The result is consistent with the simula- based on convolutional neural network,” Neural Comput. Appl., vol. 33,
tion in Fig. 5. To find why RMSE is high at some moments, no. 16, pp. 9835–9847, Aug. 2021.
[9] L. Rakai, H. Song, S. Sun, W. Zhang, and Y. Yang, “Data association
we take the target trajectories at 200th frame as an example in multiple object tracking: A survey of recent techniques,” Exp. Syst.
for detailed analysis. The RA image and corresponding target Appl., vol. 192, Apr. 2022, Art. no. 116300.
trajectories is shown in Fig. 7, from which we can see the [10] R. B. Angle, R. L. Streit, and M. Efe, “A low computational complexity
JPDA filter with superposition,” IEEE Signal Process. Lett., vol. 28,
SNR of one target is very low, which causes the predictions pp. 1031–1035, 2021.
of the network is inaccurate. As a result, the RMSE at the [11] L. Zheng and X. Wang, “Improved multiple hypothesis tracker for joint
moment is high. multiple target tracking and feature extraction,” IEEE Trans. Aerosp.
Electron. Syst., vol. 55, no. 6, pp. 3080–3089, Dec. 2019.
[12] X. Li and V. P. Jilkov, “Survey of maneuvering target tracking. Part I.
IV. C ONCLUSION Dynamic models,” IEEE Trans. Aerosp. Electron. Syst., vol. 39, no. 4,
In this letter, a framework for target tracking with mm-wave pp. 1333–1364, Oct. 2003.
[13] X. Fang, J. Zhu, D. Huang, Z. Zhang, and G. Xiao, “E2DTF: An end-
radar is proposed, which takes RA spectrum in two suc- to-end detection and tracking framework for multiple micro-UAVs with
cessive frames as the input and produces the detections FMCW-MIMO radar,” IEEE Trans. Geosci. Remote Sens., vol. 61, 2023,
and corresponding displacement for the current frame. Then Art. no. 2000916.
[14] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-Net: Fully convolutional
the detections are associated through the displacement. neural networks for volumetric medical image segmentation,” in Proc.
Experiments on simulated and measured datasets show that 4th Int. Conf. 3D Vis. (3DV), Oct. 2016, pp. 565–571.

Authorized licensed use limited to: St. Petersburg State University. Downloaded on March 01,2024 at 16:04:55 UTC from IEEE Xplore. Restrictions apply.