Neurocomputing 494 (2022) 43–55

Contents lists available at ScienceDirect

Neurocomputing
journal homepage: www.elsevier.com/locate/neucom

Survey paper

Multi-object tracking in traffic environments: A systematic literature

review
Diego M. Jiménez-Bravo a,⇑, Álvaro Lozano Murciego a, André Sales Mendes a, Héctor Sánchez San Blás a,
Javier Bajo b
a
Expert Systems and Applications Lab, Department of Computer Science and Automatics, Faculty of Sciences, University of Salamanca, Salamanca 37008, Spain
b
Ontology Engineering Group, Artificial Intelligence Department, Universidad Politécnica de Madrid, Madrid 28040, Spain

a r t i c l e i n f o a b s t r a c t

Article history: The use of computer vision techniques to detect objects in images has grown in recent years. These tech-
Received 2 November 2021 niques are especially useful to automatically extract and analyze information from an image or a
Revised 6 April 2022 sequence of them. One of the problems addressed by computer vision is multi-object tracking over
Accepted 17 April 2022
frames sequences. To know the path and direction of objects can be crucial for some areas like traffic con-
Available online 20 April 2022
Communicated by Zidong Wang
trol and supervision; by doing that the system can be able to reduce traffic jams or redirect vehicles over
less condensed areas. These algorithms include several aspects to have in mind in order to start a new
development or research in this area, for instance, is important to review the current state-of-the art
Keywords:
CNNs
techniques, the hardware requirements, the main evaluation metrics, the commonly used datasets,
Datasets among others. Therefore, the objective of this research is to present a systematic literature review which
Evaluation metrics analyzes the recent works developed in the area of multi-object tracking in traffic environments. This
MOT paper reviews the techniques, hardware, datasets, metrics, and open lines of research in this area.
Multi-object tracking Ó 2022 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND
SLR license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Systematic literature review
Traffic environments

1. Introduction way the tracking task can include two main sub-tasks, the detec-
tion of objects and the identification or re-identification of objects
During the last years, the field of computer vision research has between the different frames. Thanks to these two tasks it is pos-
suffered a big evolution and nowadays is one of the most active sible to obtain the exact position of the different objects in an
areas of computer science research. Numerous researches have image and the path they have followed in the field of view repre-
emerged using and proposing novel architectures and methods in sented by the images.
computer vision such as, R-CNN (Region-Convolutional Neural These research areas are extremely useful since they help to
Network) [1], Fast R-CNN [2], Fater R-CNN [3], and YOLO (You Only monitor easily and rapidly open or crowded environments, for
Look Once) [4]. These novel methods are in many cases developed instance, shopping centers, public buildings, squares, streets, high-
by big tech companies, for instance, Facebook [3,4], Microsoft [1], ways, etc. The task of monitoring several moving objects in a
Google, etc. Thus, this underlines the fact of the importance of recording becomes more challenging for a human being as the
computer vision in the actual research overview. Many of these number of objects to track grows [5]. On the contrary, computer
researches are applied in object detection and object tracking; both vision can perform detection and tracking activities over hundreds
of them are extremely related since object tracking depends on of elements with excellent results. This is the field of multi-object
object detection. tracking.
In a simple way, object tracking can be defined as the process of Nonetheless, it is not as easy as it sounds. On the one hand,
tracking an object through different frames in such a way that its detections techniques have to identify where an object is and what
position and direction throughout the sequence is known. In this object it is. In order to do that, the techniques have to determine
the background of the objects and deal with the different poses,
colors, etc. On the other hand, tracking algorithms deal with more
⇑ Corresponding author.
problems, for instance, identify new objects, reidentify lost objects,
E-mail addresses: [email protected] (D.M. Jiménez-Bravo), [email protected] (Á.
Lozano Murciego), [email protected] (A. Sales Mendes), hectorsanchezsanbla-
occlusion, background clutter, and pose changes.
[email protected] (H. Sánchez San Blás), [email protected] (J. Bajo).

https://doi.org/10.1016/j.neucom.2022.04.087
0925-2312/Ó 2022 The Author(s). Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

As we briefly mentioned one of the areas where detection and
RQ5: What are the main open lines of research or issues in this
tracking techniques are used is in the traffic environments. For domain?
many companies and especially city councils it is crucial to know
how vehicles users operate on the road. This information is useful 2.2. Search strategy
to detect dangerous activities on the road, to facilitate the traffic
flow in an intersection or crowded roads, among other uses. Never- Another important issue to consider when designing the review
theless, all the problems mentioned in the previous paragraph process is the search strategy. The search strategy involves the
have to be taken into account. Moreover, the tracking system databases that are going to be used as well as the concepts which
may have to consider the information and data obtained with other are going to be used in the search queries. Hence, the concepts are
sensors or cameras located or not in the same place. going to form a query which is going to be consulted in every one
As we can see the systems that implement detection and track- of the databases selected for the study.
ing technologies are extremely useful for humanity and at the On the one hand, the current study makes use of four databases:
same time its complexity is enormous. However, there is no recent IEEE Xplore, Web of Science, SpringerLink, and Scopus. These data-
systematic review that addresses these systems in the domain of bases are commonly used to publish relevant articles in the topic’s
traffic environments to analyse the current state of the art and research area. Also, the advance query options are very similar
facilitate research in this field. Therefore, we consider that is between them; this helped us to form the queries.
important to review recent articles in the field of multi-object On the other hand, the concepts used are the following ones:
tracking applied to traffic environments to properly know the most
used and novel techniques, the different types of datasets, the
The term ”multi-object tracking” or ”multi object tracking” or
hardware used in the different studies, the evaluation metrics ”MOT” or ”multi-target tracking” or ”multi target tracking” to
commonly used and the open lines of research of the shape of search for papers that study the multi-object tracking problem
things to come. Consequently, we performed a SLR (Systematic Lit- in computer vision.
erature Review).
The term ”traffic” to delimitate the study to those articles that
The rest of the article is structured as follows: Section 2 explains develop a MOT technique in a traffic environment.
the methodology and the review planning implemented for this
The term ”urban” or ”environment” or ”vehicle” or ”city” or
SLR; Section 3 answers the research questions of the SLR; in Sec- ”road” or ”lane” or ”motorway” to refer to the context of ”traffic”
tion 4 we discuss the more relevant aspects of Section 3; and environments.
finally, Section 5 points out the conclusions and future lines of
research. Once the whole process has been defined, the systematic review
process can begin. This process is summarised in 1.
2. Methodology and review process Next, it is possible to begin to answer the research questions
mentioned previously.
For the development of this Systematic Literature Review we
have followed the guidelines of Kitchenham and Charters in [6].
3. Results
To guarantee that there are no other similar state-of-the-art
reviews published in the recent past, we perform a search in the
The area of computer vision research has different sub-areas
most important scientific articles databases. The search was per-
that have or have had great relevance over the years. Perhaps the
formed in databases such as IEEE Xplore, Web of Science, Springer-
first major achievement of computer vision was image classifica-
Link, ACM Digital Library or Scopus within the terms ”slr”,
tion [7]. The classification of images consists of identifying the
”review”, ”survey”, ”state of the art”, ”mot”, ”multi-object track-
object or objects that appear in the image, in this way the output
ing”, ”traffic”, ... After performing the searches, we confirmed that
of the process will be the object that appears in the image or if it
there are not review concerning the research area of this article;
is a multi-label classification, the objects that appear in the image.
hence, this outcome justifies the development of this work.
However, image classification is only able to predict which ele-
Therefore, and following the proposed methodology, we must
ments are present in the image and not where they are, hence
start with the review process. In the following subsections we
the existence of object detection techniques. These techniques
show the most relevant points or issues of the review process. It
detect the exact position of objects in an image and are capable
is important to note that not all of them are included, only those
of detecting an infinite number of object types as long as they have
that we consider most important to illustrate the process.
been considered during the training process.
Object detection techniques have been a great revolution in the
2.1. Research questions
area of computer vision and are one of the fundamental pillars
used in image tracking techniques and therefore in MOT. MOT
The research questions define the outcomes of the study; they
can be defined as the task of locating various entities or objects
establish what we want to survey with this Systematic Literature
by associating an identifier with them and tracking their trajectory
Review. The research questions of this study are related with
through the different frames of a given video or sequence of frames
MOT (Multi-Object Tracking) in traffic environments. The ques-
as input. Hence, in this section we are going to go deeper into the
tions are as follows:
studies carried out in MOT by answering the questions mentioned
in Section 2.1.

RQ1: Which are the main techniques for Multi-Object Tracking
in traffic environments?

RQ2: Which are the devices used for Multi-Object Tracking in 3.1. Which are the main techniques for Multi-Object Tracking in traffic
traffic environments? environments?

RQ3: Which are the main datasets for Multi-Object Tracking in
traffic environments? To answer this question, we believe that is better to introduce

RQ4: Which are the evaluation metrics for Multi-Object Track- the problem formulation of MOT tasks. MOT can be seen as a
ing in traffic environments? sequence of images in which algorithms obtain different observa-
tions of objects that have to match with the states of these objects
44
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

Fig. 1. Review process using the PRISMA 2020 flow diagram.

in the different frames of the sequence. In other words, the MOT objects (real-time processing), and offline tracking can use all the
algorithm has to obtain the different states (trace) of the objects sequence of frames, past, present, and future frames (posterior
in the different images but relating the same object over the differ- processing).
ent images. Nonetheless, something that all the different classifications in
In general terms, the MOT problem can be applied in two differ- the scientific literature [8] have in common is the structure of a
ent situations. These situations depend on the number of cameras MOT algorithm. This structure is illustrated in Fig. 2. Every MOT
that we use for the study. Therefore, we can have MOT with a process starts with a video or a sequence of frames. First, a detec-
single-camera view or MOT with a multi-camera view. The second tion algorithm detects the objects in the different frames; usually,
approach is more difficult than the first one because multi-camera it returns a bounding box per each object detection. The objects
view MOT has to deal with problems with overlapping and non- detected are related to the detector of the MOT, that is to say, if
overlapping cameras, the tracking of objects over different views, the MOT aims to track cars, the object detector will identify cars.
and topology calculation among other problems. The selected Second, those detections are analyzed by the tracking algorithm;
papers of the study address a solution with a single-camera view. this phase usually extracts the features to search an affinity with
However, it is not the only classification of MOT problems. For previous frames detected objects (in the case of online processing)
instance, depending on the initialization method there are and/or with previous or later frames detected objects (in the case
Detection-Based Tracking (DBT) algorithms or Detection-Free of offline processing) in an identification sub-task and associated
Tracking (DFT) algorithms [8]. The first type uses detection algo- those identifications over the different frames to comfort the
rithms to discover the objects in each of the frames. On the other objects paths. This affinity search leads to the association of exist-
hand, DFT algorithms require manual selection of the objects in ing and new objects between frames; in other words, it returns the
the first frame; afterward, the algorithms identify those objects track of the objects. It is important to clarify that these two MOT
in the subsequent frames. DBT is much richer in output terms since processes can be performed separately, first the detection of all
it can detect new objects in the frames sequence. Another classifi- frames and then the tracking of all frames (offline processing), or
cation of MOT approach is to take into account how the algorithms these tasks can be performed repeatedly for each frame (online
use the information while processing. Hence, it is possible to processing). However, in recent years, research is being carried
observe online or offline tracking [8]; online tracking uses only out on simultaneous detection and tracking for each of the frames
the actual or past frames to idnetify or re-identify the detectec [9–12]. We will explain these novel techniques in 3.1.3.
45
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

Fig. 2. MOT process’ structure.

Now that the main structure of MOT systems is clear let’s anal- tional Network) is another architecture usually used for object
yse the different methods and/or techniques used in MOT in urban detection [20]. Ooi et al. [21] use this type of CNN to obtain the
environments. We will analyse the techniques used for the detec- detections of urban vehicles in their study. Other researchers
tion task and afterward the methods used in the tracking part. The applied R-FCN along with a non-CNN solution such as background
resume is available in Fig. 3. removal [22]; they combined the two techniques to increase the
efficiency of the detection task. Also, another powerful CNN is Fas-
3.1.1. Detection techniques ter R-CNN; it has been used by Yu et al. [23] and Liu et al. [24]. Fas-
As we previously explained detection aims to detect the objects ter R-CNN is one of the most powerful architectures for object
of interest in the different frames. DBT systems have to locate all detection however it usually requires high-efficiency resources.
the objects and DFT system only those selected in the first frame. The resources will increase if you use three Faster R-CNN along
However, both types use the same set of techniques to detect the with Inception-ResNet-v2 as Gunduz and Acarman [25] did or with
objects. Our research is focused on urban environments, so the dif- a two-stage Faster R-CNN [26].
ferent studies include in this SLR try to identify cars, trucks, motor- Moreover, we observe that other kind of CNNs are used for this
bikes, vans, cyclists, pedestrians, etc. task, for instance, DenseNet [27], ResNeXt101 [28], MobileNetV3
The current state of the art techniques for object detection in [29], RetinaNet [30], Bi-LSTM (Long Short-Term Memory, this is a
images is based on Convolutional Neural Networks (CNNs). They Recurrent Neural Netowrk (RNN) not a CNN) [31], and CMNet
are a subtype of artificial neural networks (NNs) and they are (Connect-and-Merge convolutional neural network) [32].
applied to different research domains. They are based on convolu- On the other hand, another interesting technique used is back-
tional kernels or filters with weights that slide over the input fea- ground removal. We have observed several studies that use this
tures to provide feature maps. These feature maps allow to identify technique to detect objects in a scene. A key example is [33] which
borders and detect where an object can be. Hence, CNNs offer the uses ViBe background removal for the detection task. Ooi et al. [34]
final detections of the studied objects. use the IMOT background removal along with PAWCS (Pixel-based
It is well known that there are several architectures of CNN; Adaptive Word Consensus Segmenter) to polish the object detec-
however, the most famous one is probably the YOLO algorithm tion. Chandrasekar and Geetha [35] use a fast multi-object tracking
[4]. The performance of this technique is outstanding, and it method using the Three-Frame Differencing Combined-
requires fewer resources than other CNN methods, such us, Faster Background Subtraction (TFDCBS)-coupled-automatic and a fast
R-CNN, Fast R-CNN, . . .Especially version number three (YOLOv3) is Histogram-Entropy-Based Thresholding (HEBT) method together
used in these types of studies for the detection task. This can be with GMPFM-GMPHD filters and a VGG16-LSTM classifier. This last
seen in [13–18]. Furthermore, Zhang et al. [19] propose to use case also uses a CNN as previously explained, however, the back-
YOLOv3 along a PointRCNN (Region based Convolutional Neural ground removal technique slightly increases the performance.
Network) to detect objects with higher accuracy. Finally, we observe two more used techniques to detect vehicles
Nonetheless, other kinds of CNNs are commonly used in this in a scene, Hidden Markov Models (HMM) [36] and PONO (POsition
type of problem. For instance, R-FCN (Region-based Fully Convolu- NOrmalization) [31]. It is important to mention that this last tech-

46
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

Fig. 3. MOT techniques.

nique is combined with the previously mentioned Bi-LSTM. There- the only combination of the Kalman filter with the Hungarian algo-
fore, this technique will focus on the recent features of the objects rithm; Wang and Liu [18] combined both of them along with IoU.
to detect them. Other tracker algorithms used in the literature are the SORT
(Simple, Online, and RealTime) [41] and DeepSORT [42]. However,
the SORT algorithm is just the above combination of the Kalman
3.1.2. Tracking techniques filter and the Hungarian algorithm. Hence, the study developed
As we previously explained tracking techniques aim to related by Federov et al. [26] also uses this tracker method. On the other
detections in different frames which belong to the same object. hand, DeepSORT adds a pre-trained neural net to obtain objects
Online tracking will only take into account the previous detections, features that can be used to associate the same objects over differ-
but offline detection will take into account all detections over all ent frames as the following studies had demonstrated
the frames. In this subsubsection, we will describe the techniques [24,43,29,30].
used in the included papers of this literature review. Another technique is the Field Of View (FOV). Two included
The most used method among the included works is Intersec- research used this technique as their tracking algorithm [44,45].
tion over Union (IoU) tracker. This method can detect when two This technique delimits an area inside the image and that area is
detections in different frames are related by only taking into also divided into smaller areas, depending on the enter and exit
account the previous information. It is based on a identification zone of that area and subareas, the algorithm can track the direc-
hypothesis and relay on the associations made in the previous iter- tion of the detected objects. It is a very basic technique, but it is
ations. As we can see in these studies [27,19,28,25] is commonly useful in road intersections and/or when you do not need to know
used as a tracking method. It is interesting to point out the study the exact position of the detected elements, but you need to know
developed by Hua and Anastasiu [13]; they develop the history- the enter and exit zone.
based IoU (HIoU). HIoU increases the effectiveness of IoU since it Moreover, there are solutions based on NNs like Siamese Net-
can make a relation with objects that have not been detected work [14] and Bi-LSTM [31] networks. Another solution is to use
due to occlusion. Liang et al. [16] also increase the efficiency of graph solutions to track objects between the frames, for instance,
the single IoU method by combining a multi-feature tracking algo- Tian et al. [46] use the Join Domain Information (JDI) along with
rithm based on a KCF (Kernel Correlation Filter). This last method a graph-based solution, and Wen et al. [47] use a Space–Time-
can be used to track objects; for instance, the study developed by View (STV) hyper-graph. Other researchers have used the Markov
Yang et al. [37] used a KCF as their tracking algorithm. decision process [48], trajectory clustering [49], and even brute
Moreover, there is another commonly used method in tracking force [33].
tasks according to the included articles, is the Kalman filter [38]. As
we may know the Kalman filter can estimate the future state 3.1.3. Simultaneous techniques
according to the previous features. It is used in [36,39,32,17,40]. Recent studies in the field of MOT focus on performing detec-
Furthermore, the Hungarian algorithm is widely used in this type tion and tracking tasks simultaneously (also known as join tech-
of studies as we can see in [34,22]. It is even used in combination niques) to avoid some of the problems present in this field (see
with the Kalman filter [21] to increase metrics. However, this is not 3.5) and to perform near real-time tracking, thus decreasing the
47
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

processing time and making the response faster for users. Some of can be activated per second. Hence, as higher the value as higher
the most relevant studies and their techniques are explained the frequency. This feature has relevant importance especially
below. when the processor has to make many operations when training
Zhou et al. [9] have developed a simultaneous MOT process the model. As a result, this characteristic can help to reduce the
based on point-based tracking. Their process is online and is based training and testing tasks; an important issue when addressing
on using the centroid of the detections together with a small offset problems with a huge amount of data.
to associate the different detections across the different frames. We have decided to show the different frequencies in several
Furthermore, their detection process is based on the previous intervals. Hence, it will be easier to analyse the different options.
frames to improve the results. However, their technique does not There is only one study [33] included in the interval [0, 2) GHz, 6
take into account when an object reappears in the image as it is studies [36,?,?,?,?,?] in the interval [2, 2.5) GHz, 4 studies
assigned a new identifier. [13,28,47,25] belong to the interval [2.5, 3) GHz, 4 studies
Other researchers, Wang et al. [10], propose that the detection [15,23,16,26] included in the interval [3, 3.5), 5 studies
and creation of embeddings should be done simultaneously and [35,31,49,39,46] in [3.5, 4), and 2 studies [37,48] included in [4,
shared. In this way, this model can perform these two tasks þ1]. This information is also illustrated in Fig. 4.
together to utilize that information with a fast association method Another important component that usually helps to develop
that results in the tracking of objects in near real-time. The results and train computer vision models is the use of a GPU. GPU comput-
obtained by this technique are therefore faster and with similar ing significantly increases the velocity of the computing process; to
metrics to the state-of-the-art models. do that, the code has to be prepared for GPU computing. Of course,
Wu et al. [11] propose a similar solution to the one already there are differences between different GPUs in performance and
explained in [9]. They make use of tracking offset to improve memory. However, for this type of problem is necessary to have
detection based on the characteristics of the previous frames and GPUs with high memory.
detections. They also propose to merge the detection and tracking During our review process, we have observed that not all the
tasks. Similarly, FairMOT, proposed by Zhang et al. [12], makes use scientific articles made use of GPU during MOT. From the total
of these strategies in combination with CenterNet and re-ID tech- 35 papers included only 12 [13,15,28,31,23,24,16,29,30,17,48,26]
niques to obtain the tracks of different objects. mentioned that they have made use of a GPU (see Fig. 5).
Peng et al. [50] have succeeded in developing a solution that This analysis of some of the components used in the studies
integrates object detection, feature extraction, and data association together with the techniques explained in Section 3.1 allows us
into a single solution called Chained-Tracker. This solution stands to establish which of the systems presented in the reviewed arti-
out because it outperforms current solutions in terms of metrics. cles are intended to be deployed in cloud computing architectures
The Technical University of Munich, however, proposes a quite or in edge computing architectures. Thus, the following articles are
different MOT system [51]. They propose a system that has not clearly focused on the edge paradigm
been trained with tracking data and predicts the next state of the [13,36,14,44,33,35,15,45,49,39,37,16,46]; while these other
objects based on the detections of the current frame. The solution researches focus on cloud services [28,31,23,29,30,17,47,48,26].
has proven to be effective in simple tracking environments.
3.3. Which are the main datasets for Multi-Object Tracking in traffic
3.2. Which are the devices used for Multi-Object Tracking in traffic environments?
environments?
One of the key aspects of every computer science research are
As we have seen in the previous subsection, many of the tech- the reference datasets in the area for testing and evaluating their
niques required high resources to compute and process the data. proposed models. Hence, MOT algorithms relay their effectiveness
MOT datasets are commonly composed of video captures which not only in the selected model or algorithm but also in the dataset
imply high volumes of data that have to be stored and process to used for training and testing. The datasets are a crucial part of the
train the model. However, we will discuss the different datasets development of a system; they allow researchers to compare the
and their characteristics in the next subsection. Nonetheless, these effectiveness of their proposed models against the effectiveness
volumes of data are related to the high computing resources of previous models with the same data.
needed to resolve MOT problems. Consequently, there are several well-known datasets used for
The first important aspect to handle this data is to have an MOT studies. These datasets usually allow performing detection
enought amount of RAM that can load the data and process it in
real-time while training and configuring the different models. This
feature is quite related at the same time with the techniques used
for the MOT task. In other words, if the model or technique
requires much RAM the task will require much RAM.
During the review process we have observed that most of the
studies evaluated mentioned that they had at least 16 GB of RAM
[13,36,44,45,49], 32 GB of RAM [31,23,29,48,26], 64 GB of RAM
[17], or 128 GB of RAM [47,28]. Devices with 64 or 128 GB of
RAM are probably servers’ devices; the use of servers is commonly
used by computer science researchers to train and evaluate their
models and architectures. Another interesting point is that servers
usually have more computational resources to deal with computer
vision tasks among others. On the other hand, we have noticed that
other studies used less RAM, such us, [33] that used 4 GB, [39,46]
that used 8 GB, or [35] with 14 GB of RAM.
Furthermore, another important issue to be taken into account
when confronting MOT is the frequency of the computer processor.
The frequency defines the total number of times that the processor Fig. 4. GHz frecuency intervals’ distribution.

48
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

Fig. 5. GPU’s use between the included articles.

Fig. 6. Datasets used by the included articles.

and tracking activities with several videos recording. Some of the

most important datasets are: MOT15 [52], MOT16 [53], MOT20 question. In this subsection, we what to know the evaluation met-
[54], BU-TIV [55], TUD [56] and PETS2009 [57]. However, only rics commonly used for MOT systems.
MOT16 and BU-TIV are partially prepared for MOT in traffic envi- During the review process of this article, we had found many
ronments; they have some sequences of videos in which the data- papers that do not make use of any evaluation metric
sets are prepper for vehicles detection and tracking. Therefore, [44,30,18,26]; this is not ideal, since every AI system must be eval-
these datasets may not be the ideals for performing a MOT in traffic uated and even compared to other similar solutions to completely
environments. check its performance. Another, issue that we had observed is that
Nevertheless, the research community has released two MOT many researchers use variables, such as, running time or memory
datasets oriented to vehicles identification. These datasets are use to evaluate the system; these metrics can be used to measure
the following: Urban Tracker [58], Ko-PER [59], UAVDT [23], AI City the effectiveness of a model, they only measure how efficient the
2020 [60], KITTI-T [61], nuScenes [62], Waymo [63], and UA- model is in terms of time and memory. However, we cannot forget
DETRAC [64]. Usually, these last two datasets are the most used these variables, especially when we are implementing a system in
in studies involving vehicle tracking. We have summarized the a real environment with limited resources.
most important features of the mentioned datasets in Table 1. On the contrary, many research articles make use of evaluation
The studies included in the review use the following datasets, metrics that are not appropriate for MOT problems. These metrics
KITTI-T, UAVDT, Ko-PER, UA-DETRAC, MOT16, Urban Tracker, are accuracy or mAP (mean Average Precision) among others.
nuScenes, and AI City 2020. Fig. 6 shows the use of the different These variables can be useful for the detection sub-task but, they
datasets in percentages. are not for the general MOT task.
Nonetheless, the vast majority of works use other metrics espe-
cially designed for MOT systems. They are divided into three main
3.4. Which are the evaluation metrics for Multi-Object Tracking in groups explained in detail below:
traffic environments?

VACE metrics: they were proposed by Wu and Nevatia [65] and
Currently, the best way to evaluate an Artificial Intelligence (AI) they measure different types of errors in MOT algorithms. The
system is to use a metric that illustrates to us how much efficiency most relevant VACE metrics are the following ones:
and precision the system has. Nevertheless, depending on the type – FP: the total number of False Positives in the whole video.
of system we are developing and the type of problem we are plan- FPs are determined by the ground-truth bounding boxes that
ning to resolve, we must pay attention to different evaluation met- cannot be associated with any hypothesis.
rics. Consequently, the selection of the correct metric is not a trivial

Table 1
Datasets’ summary in traffic environments.

Dataset Year Properties Training set Testing set

Task Objects Illumination Occlusion Frames Tracks Boxes Frames Tracks Boxes
MOT16 2016 Tracking Pedestrians, vehicles U U 5.3k 467 110k 5.9k 742 182k
BU-TIV 2014 Tracking Pedestrians, vehicles - - - - - 6556 - -
KITTI-T 2014 Tracking Vehicles - U 8k - - 11k - -
UA-DETRAC 2015 Detection, tracking Vehicles U U 84k 5.9k 578k 56k 2.3k 632k
Urban Tracker 2014 Detection, tracking Vehicles U U 8.1k - - 8.1k - -
Ko-PER 2014 Tracking Vehicles - - - - - - - -
UAVDT 2016 Detection, tracking Vehicles - U 70k - - 70k - -
nuSccenes 2019 Detection, tracking Pedestrians, vehicles - U 40k - 1.4M 40k - 1.4M
Waymo 2019 Detection, tracking Pedestrian, vehicles, signs - U 390k - 12.6M 390k - 12.6M
AI City 2020 2020 Tracking Vehicles U U 190k 1.3k - 190k 1.3k -

49
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

– FN: the total number of False Negatives in the whole video. – Identification F1 (IDF1): is obtained following Eq. 5.
The FNs correspond to those hypotheses that cannot be
related to any real bounding box. 2IDPT
IDF1 ¼ ð5Þ
– MT trajectories: Mostly Tracked trajectories, number of 2IDPT þ IDFP þ IDFN
ground-truth trajectories that are correctly tracked in at Along with the articles studied for this review, we observe that
least 80% of the video’s frames. there are some of these metrics that are more frequently used than
– Fragments: total number of trajectory hypothesis that others. For instance, CLEAR metrics are the most used ones; they
covers at most 80% of the video’s frames. One same trajec- are used in 29 of the 35 studies [13,36,27,14,19,33,68,35,15,28,31,
tory can be covered by more than one fragment. 23,39,34,37,24,22,43,16,69,29,46,32,17,21,47,48,25,40]. They are
– ML trajectories: Mostly Lost trajectories, number of followed by the VACE metric with a total of 20
ground-truth trajectories that are correctly tracked in less [13,36,27,14,19,68,35,15,45,28,31,23,39,37,43,69,29,46,25,40] and
than 20% of the video’s frames. by the ID Scores metrics with only 2 studies [14,23]. In Fig. 7 we
– False trajectories: predicted trajectories that do not cover can observe the combination of the metrics used in the different
the trajectory of a trackable object. papers.
– ID switches: number of times that an object is correctly
detected but, the ID has been reasigned incorrectly.
3.5. What are the main open lines of research or issues in this domain?

CLEAR metrics: they were developed for the Classification of
Events, Activities, and Relationships (CLEAR) workshops orga-
Recent advances in the area of computer vision have intensified
nized in 2006 and 2007 [66]. These metrics are based on some
work in this area and have therefore led to the emergence of new
VACE metrics such as FP, FN, Fragments, and ID switches
challenges and lines of research. We can see that from the open
(IDSW). Nonetheless, to obtain CLEAR metrics we must take into
lines of research mentioned by the included papers.
account another metric, IoU (Intersection over Union). IoU helps
Hence, one of the most mentioned open lines is to upgrade the
us to determine when an element and a prediction are related
tracking technique to handle errors in tracking tasks [36]. Nonethe-
or not. That is to say, if an object oi and the hypothesis hi are
less, these types of errors not only may be related to the tracking
matched in the frame t  1 and on frame t the
technique but also to the detection techniques and errors that occur
IoUðoi ; hi Þ >¼ 0:5 then oi and hi are matched in frame t, even
while detection is performed [13,33,31]. Moreover, this detection
if there is another hypothesis hj in which
errors may occur by other reasons, for instance, light conditions
IoUðoi ; hi Þ < IoUðoi ; hj Þ. This process continues for every remain- [30], objects occlusion [44], reflections and shadows [14,15], etc.
ing object and hypothesis. Hence, taking that into account and Further, occlusion is one of the mains problems that researchers
the definitions of the mentioned VACE metrics, the definition in MOT must deal with. Occlusion can happen when a tracked
of the CLEAR metrics are the followings: object is hidden by another tracked one/s or when it is behind
– MOTA: the Multiple Object Tracking Accuracy measures something else. Therefore, while one or both situations happened
the accuracy in a MOT algorithm. It can be calculated follow- the tracked object is lost. The tracking algorithm must take that
ing Eq. 1. into account to handle when the object appears again. This task
ðFN þ FP þ IDSWÞ is called reidentification and it is also one of the open lines of
MOTA ¼ 1  2 ð1; 1 ð1Þ research in several studies [27,31,37,26]. As we can see this task
GT
is extremely related to the improvement of the tracking technique
where GT represents the number of ground truth boxes.
mentioned before. To have a good occlusion and reidentification
– MOTP: the Multiple Object Tracking Precision measures
technique is crucial for a good performance of MOT, especially in
the precision in a MOT algorithm. It is calculated by Eq. 2.
open spaces.
One important issue about this metric is that it is more
Some researchers [46] proposed to use deep future, spatiotempo-
focused on the detection’s quality than on the tracking task.
X ral features, and pose information that may help with the reidentifi-
dt;i cation task. In general terms, the use of deep learning to recognize
t;i objects previously seen may help to resolve this problem.
MOTP ¼ X ð2Þ
ct On the other hand, others think that occlusion can be handle
t with more views [47,44] (Multi Camera Multi Target Tracking,
MCMTT). In other words, they aim to use more cameras to decrease
where ct is the number of matches in frame t, and dð t; iÞ is the
occlusion. However, this approach may lead to other problems, and
bounding box that overlaps the hypothesis i.
it is harder to resolve in computational terms. It is important to

ID Scores: they were proposed as a complement to CLEAR met-
avoid tracking or counting the same objects over the different
rics [67]. They aim to reward those trackers that can follow the
views. Also, there is a big difference if the views are from the same
same object without ID switches for the longest time. As a con-
area or if they are over different spots. So, in this last case, it is also
sequence, ID scores are obtained differently; they construct a
important to consider the reidentification task.
bipartite graph with two different types of vertices. Afterward,
Finally, Zhang et al. [19] propose a 3D multitarget tracking as
considering the connections between the vertices [67] we can
future research; this propose can be interesting to increase the effi-
obtain True Positive ID (IDTP), True Negative ID (IDTN), False
ciency and probably to decrease occlusion. It can be performed by
Positive ID (IDFP), and False Negative ID (IDFN). These metrics
forming 3D images from 2D images or by using other kinds of sen-
allow to calculate of the followings ID scores:
sors, such as Lidars and/or others.
– Identification precision (IDP): is obtained in Eq. 3.
IDPT
IDP ¼ ð3Þ 4. Discussion
IDTP þ IDFP
– Identification recall (IDR): is the result of Eq. 4. From the previous section, we can extract some outcomes that
IDPT make us understand better how research on MOT is doing. In this
IDR ¼ ð4Þ section, we will discuss the more relevant aspects concerning the
IDTP þ IDFN
research questions previously answered.
50
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

Fig. 7. Metrics combination and number of articles.

When we observe the techniques used in the included papers, is recorded from an Unmanned Aerial Vehicle (UAV). Hence, this
we see that the most used method for object detection is the use dataset will be useful if, for instance, you are planning to control
of deep learning techniques based on CNN. As we mentioned the traffic from a drone or other UAV perspective.
before, CNNs shape the state of the art for object detection in Besides, another interesting research question was which met-
images, therefore, is easy to understand why most of the study rics are commonly used when resolving a MOT task in traffic envi-
use this technique. In general terms, CNNs architectures will detect ronments. WE have seen that we have three different types of
better relevant objects in the different frames. However, this tech- metrics, VACE metrics, CLEAR metrics, and ID Scores. However,
nique requires more computational cost than other alternatives. In when we analyze the use of these metrics in the included articles,
consequence, some researchers decided to use less powerful tech- we observe that the most used one is the CLEAR metrics. As we
niques such as background removal to detect objects. Nevertheless, mentioned CLEAR metrics are composed of MOTA and MOTP; accu-
this technique also has a negative outcome, it is unable to detect racy and precision are commonly used metrics in classification
non-moving objects. Consequently, the detection and tracking problems, therefore, it is normal to have and use similar metrics
results decrease slightly. for the tracking problem. Nonetheless, we must not forget that to
Also, it is important to mention that almost all the tracking obtain CLEAR metrics we also need to calculate some VACE met-
techniques can find the associations between objects of the differ- rics. In other words, when we use CLEAR metrics, we also use VACE
ent frames; however, their effectiveness depends on the results of metrics even if we do not show those metrics in the results section.
the detection and identification task. That is to say that the accu- On the other hand, ID Scores metrics seem not to be too useful for
racy and precision obtained from a MOT algorithm are mostly researchers in MOT as the previous metrics are. From our point of
based on the decision of the detection algorithm and the identifica- view, we consider that the use of CLEAR metrics is the most useful
tion algorithm. Consequently, when developing one of these sys- one since it allows you to compare your research with others by
tems, the selection of the detection techniques is crucial for the using a common metric. Also, CLEAR metrics are easier to under-
whole process. stand than ID Scores, since accuracy and precision are terms
Another interesting aspect to have in mind, when we are select- well-known by the computer science community.
ing the MOT’s algorithms, is the hardware resources that we have Finally, we must address the open lines and future research in
to develop the project. It is obvious that if we are going to work the MOT field. We have seen that MOT in traffic environments
with videos and images is important to have enought amount of state of the art still has some open lines of research. It is crucial
RAM to infer the models, especially if we are using NNs. Therefore, to find a good performing algorithm that can handle occlusion
the selection of the algorithm also depends on the RAM and other and reidentification under different conditions, such as, raining
factors such as the frequency of the computer processor. A GPU can and non-raining, day and night, extreme light conditions, etc.
highly increase the training time and the real-time processing. This Moreover, some researchers have already proposed some future
feature can be clearly understood when you realize that the articles solutions like multi-view; nevertheless, this approach still has to
that used GPU also made use of CNNs. improve a lot to obtain results that can compete with the single-
In the previous section, we have explained the different data- view solutions. To do that is important to reidentify objects in
sets available and used for MOT in vehicle traffic environments. the different views.
We have seen that KITTI-T is the most used followed by the UA- Before finishing this section, we would like to mention that
DETRAC dataset. These two datasets are probably the most com- MOT is not only applied in traffic environments for vehicle track-
plete ones and offer videos from different perspectives and areas ing, traffic condition monitoring at intersections or roads, etc.; it
which is useful to train a MOT algorithm under several circum- is a technique that can be used in other domains, one of the most
stances. However, the selection of the dataset also depends on studied is MOT on citizens or pedestrians in public spaces; the first
where are you going to deploy the system. For instance, the Ko- MOT studies were applied to this field of study. Although the case
PER dataset is specially designed for intersections while the UAVDT studies are different, the techniques used for the detection and

51
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

tracking of citizens are practically identical; likewise, the hardware MIAC). PID2019-108883RB-C21/ AEI/ 10.13039/501100011033.
components used to generate this type of model and algorithm are This work was supported by the Spanish Agencia Estatal de Inves-
very similar. The metrics used are the same and CLEAR metrics are tigación. Project Monitoring and tracking systems for the improve-
almost certainly the most widely used in this area of study as well. ment of intelligent mobility and behavior analysis (SiMoMIAC).
As for the datasets, there are specific datasets such as [70,71] PID2019-108883RB-C22/ AEI/ 10.13039/501100011033. The
although some of the datasets mentioned in 1 can also be used research of Diego M. Jiménez-Bravo has been co-financed by the
for the pedestrian MOT. The problems encountered are very simi- European Social Fund and Junta de Castilla y León (Operational
lar, but depending on the environment or scenario where the MOT Programme 2014-2020 for Castilla y León, EDU/574/2018 BOCYL).
is developed, it will be easier or more difficult to deal with prob- The research of André Filipe Sales Mendes has been co-financed by
lems such as occlusion. the European Social Fund and Junta de Castilla y León (Operational
Therefore, we can conclude the discussion by saying that MOT Programme 2014-2020 for Castilla y León, EDU/556/2019 BOCYL).
in traffic environments still has some aspects to improve and that
the work developed by the science community conforms to the CRediT authorship contribution statement
state of the art in MOT techniques. Also, we can see that this is a
trendy line of research due to the number of research articles Diego M. Jiménez-Bravo: Conceptualization, Data curation,
developed in recent years and the new datasets that are actualized Formal analysis, Funding acquisition, Investigation, Methodology,
almost every year to cover new circumstances and to improve the Resources, Software, Validation, Visualization, Writing - original
algorithms. draft. Álvaro Lozano Murciego: Conceptualization, Investigation,
Methodology, Software, Supervision, Validation, Writing - review
5. Conclusions & editing. André Sales Mendes: Funding acquisition, Validation,
Writing - review & editing. Héctor Sánchez San Blás: Formal anal-
A SLR has been carried out in the domain of multi-object track- ysis, Investigation, Validation, Writing - review & editing. Javier
ing in traffic environments. The SLR has been done following the Bajo: Conceptualization, Funding acquisition, Project administra-
guidelines of Kitchenham and Charters in [6]. The review is focused tion, Resources, Supervision, Writing - review & editing.
on the understanding of the MOT problem and the different subar-
eas of this task. Declaration of Competing Interest
The research covers the techniques used in MOT in traffic envi-
ronments and explains the main structure of this kind of system. The authors declare that they have no known competing finan-
We focus our explanation on the two main tasks of a MOT, detec- cial interests or personal relationships that could have appeared
tion, and tracking. We explained that the most used technique for to influence the work reported in this paper.
detection is based on CNNs architectures and that the effectiveness
of the several techniques available for tracking is based on the References
detection and identifiaction outcome. Moreover, we make a dis-
[1] R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate
tinction between different types of MOTs. object detection and semantic segmentation, in: Proceedings of the IEEE
Afterward, we evaluate the hardware used in the included Computer Society Conference on Computer Vision and Pattern Recognition,
papers to perform research in the MOT area. These types of 2014, pp. 580–587, https://doi.org/10.1109/CVPR.2014.81, url:https://arxiv.
org/abs/1311.2524v5.
research usually imply processing big datasets and training big [2] R. Girshick, Fast R-CNN, Proceedings of the IEEE International Conference on
and complex models. We study and resume the RAM, the fre- Computer Vision 2015 (2015) 1440–1448, https://doi.org/10.1109/
quency of the computer processor, and the use or not of a GPU. ICCV.2015.169, url:https://arxiv.org/abs/1504.08083.
[3] S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN: Towards Real-Time Object
Another issue addressed is the different datasets available and Detection with Region Proposal Networks, IEEE Transactions on Pattern
used by the scientific community. We have shown the features of Analysis and Machine Intelligence 39 (6) (2017) 1137–1149, https://doi.org/
some of the most relevant datasets. This section is useful for new 10.1109/TPAMI.2016.2577031, url:https://arxiv.org/abs/1506.01497v3.
[4] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified, real-
researchers in the area to know the commonly used dataset.
time object detection, in: Proceedings of the IEEE Computer Society Conference
Furthermore, another interesting aspect for new researchers is on Computer Vision and Pattern Recognition , 2016- (2016), pp. 779–788,
the metrics used to evaluate the algorithms. We have explained https://doi.org/10.1109/CVPR.2016.91, url:https://arxiv.org/abs/
1506.02640v5.
three different metrics and also, we have mentioned the most used
[5] K.P. Madore, A.D. Wagner, Cerebrum: the Dana forum on brain science 2019.
one among the researchers. These metrics are really useful to com- url:http://www.ncbi.nlm.nih.gov/pubmed/32206165..
pare the research results with the previous state-of-the-art studies. [6] S. Keele, Guidelines for performing systematic literature reviews in software
To conclude, we also study the direction of this research area for engineering, Technical report, Ver. 2.3 EBSE Technical Report. EBSE. url:https://
citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.471..
the coming years. The open lines have been mentioned and dis- [7] Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D.
cussed along with the specific area that has to be developed to Jackel, Backpropagation Applied to Handwritten Zip Code Recognition, Neural
put MOT in traffic environments into the next level. This section Computation 1 (4) (1989) 541–551, https://doi.org/10.1162/
NECO.1989.1.4.541.
can be useful to guide new research in this specific area. [8] W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, T.K. Kim, Multiple object tracking: A
To the best of our knowledge, there is no previous SLR in this literature review, Artificial Intelligence 293. doi:10.1016/j.artint.2020.103448.
specific domain and as we had mentioned, we considered that this url:https://arxiv.org/abs/1409.7618v4..
[9] X. Zhou, V. Koltun, P. Krähenbühl, Tracking Objects as Points, Lecture Notes in
work can be useful to developers and researchers in MOT in traffic Computer Science (including subseries Lecture Notes in Artificial Intelligence
environments. It could be a great initial resource to identify the and Lecture Notes in Bioinformatics) 12349 LNCS (2020) 474–490.
techniques, the hardware needed, the datasets, the metrics, and doi:10.1007/978-3-030-58548-8_28. url:https://arxiv.org/abs/2004.01177v2..
[10] Z. Wang, L. Zheng, Y. Liu, Y. Li, S. Wang, Towards Real-Time Multi-Object
the new lines of research.
Tracking, Lecture Notes in Computer Science (including subseries Lecture
Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12356
LNCS (2019) 107–122. doi:10.1007/978-3-030-58621-8_7. url:https://arxiv.
Funding org/abs/1909.12605v2..
[11] J. Wu, J. Cao, L. Song, Y. Wang, M. Yang, J. Yuan, Track to detect and segment:
This work was supported by the Spanish Agencia Estatal de An online multi-object tracker, in: Proceedings of the IEEE Computer Society
Conference on Computer Vision and Pattern Recognition, 2021, pp. 12347–
Investigación. Project Monitoring and tracking systems for the 12356, https://doi.org/10.1109/CVPR46437.2021.01217, url:https://arxiv.org/
improvement of intelligent mobility and behavior analysis (SiMo- abs/2103.08808v1.

52
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

[12] Y. Zhang, C. Wang, X. Wang, W. Zeng, W. Liu, FairMOT: On the Fairness of Theory, IEEE, 345 E 47TH ST, NEW YORK, NY 10017 USA, 2020, pp. 2066–2071.
Detection and Re-Identification in Multiple Object Tracking, International doi:10.1109/CCDC49329.2020.9164690..
Journal of Computer Vision 129 (11) (2020) 3069–3087. doi:10.1007/s11263- [30] Z. Wang, B. Bai, Y. Xie, T. Xing, B. Zhong, Q. Zhou, Y. Meng, B. Xu, Z. Song, P. Xu,
021-01513-4. url:http://arxiv.org/abs/2004.01888.. R. Hu, H. Chai, Robust and fast vehicle turn-counts at intersections via an
[13] S. Hua, D.C. Anastasiu, Effective vehicle tracking algorithm for smart traffic integrated solution from detection, tracking and trajectory modeling, in: IEEE
networks, in: Proceedings - 13th IEEE International Conference on Service- Computer Society Conference on Computer Vision and Pattern Recognition
Oriented System Engineering, SOSE 2019, 10th International Workshop on Workshops, Vol. 2020-June, 2020, pp. 2598–2606. doi:10.1109/
Joint Cloud Computing, JCC 2019 and 2019 IEEE International Workshop on CVPRW50498.2020.00313..
Cloud Computing in Robotic Systems, CCRS 2019, IEEE; IEEE Comp Soc, IEEE, [31] X. Dong, J. Niu, J. Cui, Z. Fu, Z. Ouyang, Fast Segmentation-Based Object
345 E 47TH ST, NEW YORK, NY 10017 USA, 2019, pp. 67–76. doi:10.1109/ Tracking Model for Autonomous Vehicles, in: Lecture Notes in Computer
SOSE.2019.00019.. Science (including subseries Lecture Notes in Artificial Intelligence and Lecture
[14] Y. Zou, W. Zhang, W. Weng, Z. Meng, Multi-vehicle tracking via real-time Notes in Bioinformatics), Vol. 12453 LNCS, Springer International Publishing,
detection probes and a markov decision process policy, Sensors (Switzerland) 2020, pp. 259–273. doi:10.1007/978-3-030-60239-0_18..
19 (6). doi:10.3390/s19061309.. [32] M.B. Khalkhali, A. Vahedian, H.S. Yazdi, IEEE Transactions on Intelligent
[15] J. Wang, S. Simeonova, M. Shahbazi, Orientation- and scale-invariant multi- Transportation Systems doi:10.1109/TITS.2021.3050878. url:https://www.
vehicle detection and tracking from unmanned aerial videos, Remote Sensing scopus.com/inward/record.uri?eid=2-s2.0-85100831591&doi=10.1109..
11 (18). doi:10.3390/rs11182155.. [33] K. Singh, V. Karar, S. Poddar, Pattern Recognition and Image Analysis 30 (3)
[16] H. Liang, H. Song, H. Li, Z. Dai, Vehicle Counting System using Deep Learning (2020) 416–427, https://doi.org/10.1134/S1054661820030268, url:https://
and Multi-Object Tracking Methods, Transportation Research Record 2674 (4) www.scopus.com/inward/record.uri?eid=2-s2.0-85091005329&doi=10.1134.
(2020) 114–128, https://doi.org/10.1177/0361198120912742. [34] H.L. Ooi, G.A. Bilodeau, N. Saunier, Supervised and unsupervised detections for
[17] L. Lou, Q. Zhang, C. Liu, M. Sheng, Y. Zheng, X. Liu, in: Proceedings of 2019 IEEE multiple object tracking in traffic scenes: A comparative study, in: Lecture
8th Data Driven Control and Learning Systems Conference, DDCLS 2019, Notes in Computer Science (including subseries Lecture Notes in Artificial
Institute of Electrical and Electronics Engineers Inc., 2019, pp. 1012–1017. Intelligence and Lecture Notes in Bioinformatics), Vol. 12131 LNCS, Springer
doi:10.1109/DDCLS.2019.8908873. url:https://www.scopus.com/inward/ International Publishing, 2020, pp. 42–55. doi:10.1007/978-3-030-50347-5_4..
record.uri?eid=2-s2.0-85076438506&doi=10.1109.. [35] K.S. Chandrasekar, P. Geetha, Multiple objects tracking by a highly decisive
[18] K. Wang, M. Liu, YOLOv3-MT: A YOLOv3 using multi-target tracking for vehicle three-frame differencing-combined-background subtraction method with
visual detection, Applied Intelligence doi:10.1007/s10489-021-02491-3.. GMPFM-GMPHD filters and VGG16-LSTM classifier, Journal of Visual
[19] L. Zhang, J. Lai, Z. Zhang, Z. Deng, B. He, Y. He, Multimodal Multiobject Tracking Communication and Image Representation 72. doi:10.1016/j.
by Fusing Deep Appearance Features and Motion Information, Complexity jvcir.2020.102905..
(2020), https://doi.org/10.1155/2020/8810340. [36] S. Chattopadhyay, Q. Ge, C. Wei, E. Lobaton, in: 2015 IEEE Global Conference on
[20] J. Dai, Y. Li, K. He, J. Sun, R-FCN: Object detection via region-based fully Signal and Information Processing, GlobalSIP 2015, Institute of Electrical and
convolutional networks, Advances in Neural Information Processing Systems Electronics Engineers Inc., 2016, pp. 805–809. doi:10.1109/
(2016) 379–387. url:https://arxiv.org/abs/1605.06409v2. GlobalSIP.2015.7418308. url:https://www.scopus.com/inward/record.uri?
[21] H.L. Ooi, G.A. Bilodeau, N. Saunier, D.A. Beaupré, Multiple object tracking in eid=2-s2.0-84964794695&doi=10.1109..
urban traffic scenes with a multiclass object detector, in: Bebis, G and Boyle, R [37] H. Yang, S. Gao, X. Wu, Y. Zhang, Online multi-object tracking using KCF-based
and Parvin, B and Koracin, D and Turek, M and Ramalingam, S and Xu, K and single-object tracker with occlusion analysis, Multimedia Systems 26 (6)
Lin, S and Alsallakh, B and Yang, J and Cuervo, E and Ventura, J (Ed.), Lecture (2020) 655–669, https://doi.org/10.1007/s00530-020-00675-4.
Notes in Computer Science (including subseries Lecture Notes in Artificial [38] R.E. Kalman, A new approach to linear filtering and prediction problems,
Intelligence and Lecture Notes in Bioinformatics), Vol. 11241 LNCS of Lecture Journal of Fluids Engineering, Transactions of the ASME 82 (1) (1960) 35–45,
Notes in Computer Science, UNR; DRI; LBNL; NASA Ames; BAE Syst; Intel; https://doi.org/10.1115/1.3662552.
Ford; Hewlett Packard; Mitsubishi Elect Res Labs; Toyota; Gen Elect, SPRINGER [39] L. Jun, X. Wei-xin, L. Liang-qun, Online Visual Multiple Target Tracking by
INTERNATIONAL PUBLISHING AG, GEWERBESTRASSE 11, CHAM, CH-6330, Intuitionistic Fuzzy Data Association, International Journal of Fuzzy Systems
SWITZERLAND, 2018, pp. 727–736. doi:10.1007/978-3-030-03801-4_63.. 19 (2) (2017) 355–366, https://doi.org/10.1007/s40815-016-0172-2.
[22] H.L. Ooi, G.A. Bilodeau, N. Saunier, Tracking in urban traffic scenes from [40] N.M. Al-Shakarji, F. Bunyak, G. Seetharaman, K. Palaniappan, Multi-object
background subtraction and object detection, in: Karray, F and Campilho, A Tracking Cascade with Multi-Step Data Association and Occlusion Handling,
and Yu, A (Ed.), Lecture Notes in Computer Science (including subseries in: Proceedings of AVSS 2018–2018 15th IEEE International Conference on
Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Advanced Video and Signal-Based Surveillance, IEEE; IEEE Signal Proc Soc,
Vol. 11662 LNCS of Lecture Notes in Computer Science, Assoc Image IEEE, 345 E 47TH ST, NEW YORK, NY 10017 USA, 2019, pp. 423–428.
&s066amp; Machine Intelligence; Univ Waterloo, Fac Engn; Univ Porto, Fac doi:10.1109/AVSS.2018.8639321..
Engn, Dept Elect &s066amp; Comp Engn; Inst Syst &s066amp; Comp Engn [41] A. Bewley, Z. Ge, L. Ott, F. Ramos, B. Upcroft, Simple online and realtime
Technol &s066amp; Sciand Science; Univ Waterloo, Waterloo AI Inst; Univ tracking, Proceedings - International Conference on Image Processing, ICIP
Waterloo, Ctr Pattern Anal &s066amp; Machine Intelligence; Inst S, SPRINGER 2016-Augus (2016) 3464–3468. doi:10.1109/ICIP.2016.7533003. url:https://
INTERNATIONAL PUBLISHING AG, GEWERBESTRASSE 11, CHAM, CH-6330, arxiv.org/abs/1602.00763v2..
SWITZERLAND, 2019, pp. 195–206. doi:10.1007/978-3-030-27202-9_17.. [42] N. Wojke, A. Bewley, D. Paulus, Simple online and realtime tracking with a
[23] H. Yu, G. Li, W. Zhang, Q. Huang, D. Du, Q. Tian, N. Sebe, The Unmanned Aerial deep association metric, Proceedings - International Conference on Image
Vehicle Benchmark: Object Detection, Tracking and Baseline, International Processing, ICIP (2017- (2018)) 3645–3649, https://doi.org/10.1109/
Journal of Computer Vision 128 (5) (2020) 1141–1159, https://doi.org/ ICIP.2017.8296962, url:https://arxiv.org/abs/1703.07402v1.
10.1007/s11263-019-01266-1. [43] X. Hou, Y. Wang, L.P. Chau, Vehicle tracking using deep SORT with low
[24] Z. Liu, W. Zhang, X. Gao, H. Meng, X. Tan, X. Zhu, Z. Xue, X. Ye, H. Zhang, S. Wen, confidence track filtering, in: 2019 16th IEEE International Conference on
E. Ding, in: IEEE Computer Society Conference on Computer Vision and Pattern Advanced Video and Signal Based Surveillance, AVSS 2019, IEEE, IEEE, 345 E
Recognition Workshops, Vol. 2020-June, IEEE Computer Society, 2020, pp. 47TH ST, NEW YORK, NY 10017 USA, 2019. doi:10.1109/AVSS.2019.8909903..
2617–2625. doi:10.1109/CVPRW50498.2020.00315. url:https://www. [44] M. Delavarian, O. Maarouzi, in: Proceedings - 3rd Iranian Conference on Signal
scopus.com/inward/record.uri?eid=2-s2.0-85090154333&doi=10.1109.. Processing and Intelligent Systems, ICSPIS 2017, Vol. 2017-Decem, Institute of
[25] G. Gunduz, T. Acarman, Efficient Multi-Object Tracking by Strong Associations Electrical and Electronics Engineers Inc., 2018, pp. 131–135. doi:10.1109/
on Temporal Window, IEEE Transactions on Intelligent Vehicles 4 (3) (2019) ICSPIS.2017.8311603. url:https://www.scopus.com/inward/record.uri?eid=2-
447–455, https://doi.org/10.1109/TIV.2019.2919473. s2.0-85050820485&doi=10.1109..
[26] A. Fedorov, K. Nikolskaia, S. Ivanov, V. Shepelev, A. Minbaleev, Traffic flow [45] M. Delavarian, O. Marouzi, H. Hassanpour, A multilayer motion direction based
estimation with data from a video surveillance camera, Journal of Big Data 6 model for tracking vehicles at intersections, International Journal of
(1). doi:10.1186/s40537-019-0234-z.. Engineering, Transactions A: Basics 33 (10) (2020) 1939–1950, https://doi.
[27] W. Feng, D. Ji, Y. Wang, S. Chang, H. Ren, W. Gan, Challenges on large scale org/10.5829/IJE.2020.33.10A.12.
surveillance video analysis, in: IEEE Computer Society Conference on [46] W. Tian, M. Lauer, L. Chen, Online Multi-Object Tracking Using Joint Domain
Computer Vision and Pattern Recognition Workshops, Vol. 2018-June of IEEE Information in Traffic Scenarios, IEEE Transactions on Intelligent
Computer Society Conference on Computer Vision and Pattern Recognition Transportation Systems 21 (1) (2020) 374–384, https://doi.org/10.1109/
Workshops, IEEE Comp Soc, IEEE, 345 E 47TH ST, NEW YORK, NY 10017 USA, TITS.2019.2892413.
2018, pp. 69–76. doi:10.1109/CVPRW.2018.00017.. [47] L. Wen, Z. Lei, M.C. Chang, H. Qi, S. Lyu, Multi-Camera Multi-Target Tracking
[28] M. Fernández-Sanjurjo, M. Mucientes, V.M. Brea, A Real-Time Processing with Space-Time-View Hyper-graph, International Journal of Computer Vision
Stand-Alone Multiple Object Visual Tracking System, in: Lecture Notes in 122 (2) (2017) 313–333, https://doi.org/10.1007/s11263-016-0943-0.
Computer Science (including subseries Lecture Notes in Artificial Intelligence [48] J.E. Espinosa, S.A. Velastin, J.W. Branch, Detection and Tracking of Motorcycles
and Lecture Notes in Bioinformatics), Vol. 11678 LNCS, Springer International in Congested Urban Environments Using Deep Learning and Markov Decision
Publishing, 2019, pp. 64–74. doi:10.1007/978-3-030-29888-3_6.. Processes, in: Lecture Notes in Computer Science (including subseries Lecture
[29] T. Zhang, M. Zhao, Multi-Scale Vehicle Detection and Tracking Method in Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 11524
Highway Scene, in: Proceedings of the 32nd Chinese Control and Decision LNCS, Springer International Publishing, 2019, pp. 139–148. doi:10.1007/978-
Conference, CCDC 2020, Chinese Control and Decision Conference, NE Univ; 3-030-21077-9_13..
Chinese Assoc Automat Tech Comm Control &s066amp; Decis Cyber Phys Syst; [49] A.A. Sekh, D.P. Dogra, S. Kar, P.P. Roy, Video trajectory analysis using
Chinese Assoc Automat; Anhui Univ; IEEE Control Syst Soc; NE Univ, State Key unsupervised clustering and multi-criteria ranking, Soft Computing 24 (21)
Lab Synthet Automat Proc Ind; Chinese Assoc Automat, Tech Comm Control (2020) 16643–16654, https://doi.org/10.1007/s00500-020-04967-9.

53
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

[50] J. Peng, C. Wang, F. Wan, Y. Wu, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, Y. Fu, [68] I. del Pino, V. Vaquero, B. Masini, J. Solà, F. Moreno-Noguer, A. Sanfeliu, J.
Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Andrade-Cetto, Low resolution lidar-based multi-object tracking for driving
Joint Multiple-Object Detection and Tracking, Lecture Notes in Computer applications, Advances in Intelligent Systems and Computing, Vol. 693,
Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Springer International Publishing, 2018, pp. 287–298, https://doi.org/
Notes in Bioinformatics) 12349 LNCS (2020) 145–161. doi:10.1007/978-3- 10.1007/978-3-319-70833-1_24.
030-58548-8_9. url:https://arxiv.org/abs/2007.14557v1. [69] L. Wen, D. Du, Z. Cai, Z. Lei, M.C. Chang, H. Qi, J. Lim, M.H. Yang, S. Lyu, UA-
[51] P. Bergmann, T. Meinhardt, L. Leal-Taixe, Tracking without bells and whistles, DETRAC: A new benchmark and protocol for multi-object detection and
Proceedings of the IEEE International Conference on Computer Vision 2019- tracking, Computer Vision and Image Understanding 193. doi:10.1016/j.
October (2019) 941–951. doi:10.1109/ICCV.2019.00103. url:http://arxiv.org/ cviu.2020.102907..
abs/1903.05625 https://doi.org/10.1109/ICCV.2019.00103. [70] P. Dollár, C. Wojek, B. Schiele, P. Perona, Pedestrian detection: An evaluation of
[52] L. Leal-Taixé, A. Milan, I. Reid, S. Roth, K. Schindler, MOTChallenge 2015: the state of the art, IEEE Transactions on Pattern Analysis and Machine
Towards a Benchmark for Multi-Target Tracking. url:http://arxiv.org/abs/ Intelligence 34 (4) (2012) 743–761, https://doi.org/10.1109/TPAMI.2011.155.
1504.01942.. [71] J.F.P. Kooij, N. Schneider, F. Flohr, D.M. Gavrila, Context-based pedestrian path
[53] A. Milan, L. Leal-Taixe, I. Reid, S. Roth, K. Schindler, MOT16: A Benchmark for prediction, Lecture Notes in Computer Science (including subseries Lecture
Multi-Object Tracking. url:http://arxiv.org/abs/1603.00831.. Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8694 LNCS
[54] P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. (PART 6) (2014) 618–633. doi:10.1007/978-3-319-10599-4_40..
Schindler, L. Leal-Taixé, MOT20: A benchmark for multi object tracking in
crowded scenes. url:https://arxiv.org/abs/2003.09003v1..
[55] Z. Wu, N. Fuller, D. Theriault, M. Betke, A thermal infrared video benchmark for
visual analysis, IEEE Computer Society Conference on Computer Vision and Diego M. Jiménez-Bravo studied a degree in Computer
Pattern Recognition Workshops (2014) 201–208, https://doi.org/10.1109/ Engineering Management and Information Systems at
CVPRW.2014.39. the University of the Basque Country (UPV/EHU) (2016).
[56] C. Morris, N.M. Kriege, F. Bause, K. Kersting, P. Mutzel, M. Neumann, http:// Subsequently, he obtained a Master’s degree in Intelli-
arxiv.org/abs/2007.08663TUDataset: A collection of benchmark datasets for gent Systems from the University of Salamanca (USAL)
learning with graphs. url:http://arxiv.org/abs/2007.08663.. (2017). He concluded his formative stage by obtaining a
[57] J. Ferryman, A. Shahrokni, PETS2009: Dataset and challenge, Proceedings of the Ph.D. in Computer Engineering from the USAL (2020).
12th IEEE International Workshop on Performance Evaluation of Tracking and He is part of the research group ESALab, of the same
Surveillance, PETS-Winter 2009 doi:10.1109/PETS-WINTER.2009.5399556.. entity, where he carries out research work under a
[58] J.P. Jodoin, G.A. Bilodeau, N. Saunier, Urban Tracker: Multiple object tracking in contract funded by the Junta de Castilla y León and the
urban mixed traffic, 2014 IEEE Winter Conference on Applications of European Social Fund. His main research interests focus
Computer Vision, WACV 2014 (2014) 885–892 doi:10.1109/
on the field of artificial intelligence, IoT (internet of
WACV.2014.6836010..
things), smart-homes, energy optimisation and social networks, among others. He
[59] E. Strigel, D. Meissner, F. Seeliger, B. Wilking, K. Dietmayer, The Ko-PER
intersection laserscanner and video dataset, 2014 17th IEEE International also collaborates in several research projects within the research group.
Conference on Intelligent Transportation Systems, ITSC 2014 (2014) 1900–
1901 doi:10.1109/ITSC.2014.6957976..
[60] M. Naphade, S. Wang, D.C. Anastasiu, Z. Tang, M.C. Chang, X. Yang, L. Zheng, A. Álvaro Lozano Murciego received the master’s degree
Sharma, R. Chellappa, P. Chakraborty, The 4th AI city challenge, IEEE Computer in intelligent systems and the Ph.D. degree in computer
Society Conference on Computer Vision and Pattern Recognition Workshops engineering from the University of Salamanca, in 2015
(2020- (2020)) 2665–2674, https://doi.org/10.1109/ and 2019, respectively. He is currently working on the
CVPRW50498.2020.00321, url:https://arxiv.org/abs/2004.14619v1. Expert Systems and Applications Laboratory Research
[61] A. Geiger, P. Lenz, C. Stiller, R. Urtasun, Vision meets robotics: The KITTI Group, Computer Science Department, University of
dataset, International Journal of Robotics Research 32 (11) (2013) 1231–1237, Salamanca, as an Assistant Professor. Throughout his
https://doi.org/10.1177/0278364913491297, url:https://journals. training, he has followed a well-defined line of research,
sagepub.com/doi/full/10.1177/0278364913491297. focused on machine learning, route optimization, IoT
[62] H. Caesar, V. Bankiti, A.H. Lang, S. Vora, V.E. Liong, Q. Xu, A. Krishnan, Y. Pan, G.
sensors, and edge computing.
Baldan, O. Beijbom, nuScenes: A multimodal dataset for autonomous driving,
Proceedings of the IEEE Computer Society Conference on Computer Vision and
Pattern Recognition (2019) 11618–11628 doi:10.1109/
CVPR42600.2020.01164. url:https://arxiv.org/abs/1903.11027v5.
[63] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y.
Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S.
Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, D. Anguelov, André Sales Mendes is a Ph.D. student in computer
Scalability in Perception for Autonomous Driving: Waymo Open Dataset, engineering. He has completed a computer engineering
Proceedings of the IEEE Computer Society Conference on Computer Vision and degree and a master’s degree in intelligent systems at
Pattern Recognition (2019) 2443–2451, https://doi.org/10.1109/ the University of Salamanca. He is currently focusing his
CVPR42600.2020.00252, url:https://arxiv.org/abs/1912.04838v7. doctoral studies on artificial intelligence and expert
[64] S. Lyu, M.C. Chang, D. Du, L. Wen, H. Qi, Y. Li, Y. Wei, L. Ke, T. Hu, M. Del Coco, P. robots. He is a member of the Expert Systems And
Carcagni, D. Anisimov, E. Bochinski, F. Galasso, F. Bunyak, G. Han, H. Ye, H. Applications Laboratory research group at the Univer-
Wang, K. Palaniappan, K. Ozcan, L. Wang, L. Wang, M. Lauer, N.
sity of Salamanca. In addition, he has several publica-
Watcharapinchai, N. Song, N.M. Al-Shakarji, S. Wang, S. Amin, S.
tions in JCR indexed journals.
Rujikietgumjorn, T. Khanova, T. Sikora, T. Kutschbach, V. Eiselein, W. Tian, X.
Xue, X. Yu, Y. Lu, Y. Zheng, Y. Huang, Y. Zhang, UA-DETRAC 2017: Report of
AVSS2017 & IWT4S Challenge on Advanced Traffic Monitoring, 2017 14th IEEE
International Conference on Advanced Video and Signal Based Surveillance,
AVSS 2017 doi:10.1109/AVSS.2017.8078560..
[65] B. Wu, R. Nevatia, Tracking of multiple, partially occluded humans based on
static body part detection, in: Proceedings of the IEEE Computer Society
Conference on Computer Vision and Pattern Recognition 1, 2006, pp. 951–958, Héctor Sánchez San Blas is a Ph.D. Student in computer
https://doi.org/10.1109/CVPR.2006.312. engineering. He has completed a Computer Engineer
[66] R. Stiefelhagen, J.S. Garofolo, Multimodal Technologies for Perception of degree and a Master’s degree in Intelligent Systems at
Humans: First International Evaluation Workshop on Classification of Events, the University of Salamanca. During the master’s
Activities and Relationships, CLEAR 2006 Revised Selected Papers, Vol. 4122
degree, he was a collaborating researcher at the Expert
LNCS, Springer, Southampton, 2007. url:https://www.researchgate.net/
Systems and Applications Lab (ESALab) at the same
publication/239577438_Multimodal_Technologies_for_Perception_of_
university, collaborating with research projects related
Humans_First_International_Evaluation_Workshop_on_Classification_of_
Events_Activities_and_Relationships_CLEAR_2006_Southampton_UK_April_6- to the Internet of Things, Virtual Reality applications,
7_2006_Revised_Selec.. and machine learning. Currently, he researches IoT and
[67] E. Ristani, F. Solera, R. Zou, R. Cucchiara, C. Tomasi, Performance measures and neural networks development and research projects
a data set for multi-target, multi-camera tracking, Lecture Notes in Computer focused on artificial vision and Smart Cities.
Science (including subseries Lecture Notes in Artificial Intelligence and Lecture
Notes in Bioinformatics) 9914 LNCS (2016) 17–35. doi:10.1007/978-3-319-
48881-3_2. url:https://link.springer.com/chapter/10.1007/978-3-319-48881-
3_2..

54
D.M. Jiménez-Bravo, Á. Lozano Murciego, André Sales Mendes et al. Neurocomputing 494 (2022) 43–55

Javier Bajo is a Professor at the ETS Ingenieros

Informáticos of the Universidad Politécnica de Madrid.
He is currently Director of the Research Center in Arti-
ficial Intelligence of the UPM. Previously he was a pro-
fessor at the Pontifical University of Salamanca (2003 to
2012) and Director of the Data Processing Center of that
University (2010–2012).He holds a Ph.D. in Computer
Science from the University of Salamanca (with honors)
(2007) and a Master in Electronic Commerce from the
same University (2006). His lines of research focus on
multi-agent systems, social computing and knowledge
representation and reasoning. He has participated in
more than 50 research projects (European, National or Regional) and research
contracts, being principal investigator in 11 of them.

55