MULTIPLE OBJECT DETECTION AND TRACKING

Sriram P
Yutha mesak X Joshua Isaac Raj J
Computer science and
Computer science and Engineering Computer science and
Engineering National engineering
college (An Autonomous National engineering college (An Engineering National engineering
Institution - affiliated to anna Autonomous Institution - college (An Autonomous
university, Chennai) Kovilpatti, affiliated Institution - affiliated
Tamil Nadu. to anna university, to anna university,
Chennai) Kovilpatti, Chennai) Kovilpatti,
[email protected]
Tamil Nadu. Tamil Nadu.
Ms.P.Priyadharshi [email protected] [email protected]
ni
AssistantProfessor,
Computer science and Engineering
National engineering college (An
Autonomous Institution affiliated to
anna university, Chennai)
Kovilpatti,
Tamil Nadu.
[email protected]
 Abstract - Detecting moving objects in an Image video processing in computer vision but Video analyzing and
sequence is a fundamental problem in many vision-based the processing was very difficult to the computer
applications. In particular, detecting the objects and vision.
Moving objects when the camera is moving is a complex
problem. In this study, we propose a symmetric method for The solution: A well-known object recognition
detecting moving objects in the presence of a Masked algorithm that has transformed computer vision is
background. First, a background compensation method called YOLO (You Only Look Once). It works well for
detects the proposed motion region. Next, to accurately
augmented reality, self-driving automobiles, and video
locate the moving objects and try to Mask the other
locations, we propose a convolutional neural network- surveillance applications It is a fantastic option for
based method called YOLOv5 for detecting all objects in the real- time object identification Increasing consumer
image, which is classified and designed explicitly for small involvement with your product may be achieved
objects. Finally, the multiple objects are determined by quickly and reliably by establishing connections
fusing the motion and object detection results. Missed between it and other brands your target audience
detections are recalled according to adjacent frames' already uses. Integrating these brands into your
temporal and spatial information. A dataset is not currently applications for end users results in innovative and
available specifically for moving object detection and engaging new features that set your product apart
recognition, and thus, we have released the dataset from the competition and increase user participation.
comprising three classes with some videos. Our
experiments demonstrated that the proposed algorithm 3.YOLOV5:
could accurately detect moving objects in various scenarios Using an end-to-end neural network to forecast
with good overall performance and the best accuracy. bounding boxes and class probabilities
simultaneously is what Only Look Once (YOLO)
suggests doing. It differs from the strategy used by
Key Words: CNN, yolov5 algorithm, object detection, earlier object detection algorithms, which used
motion caption, moving object detection. classifiers as detectors.
Although it is routine for the human brain,
1.INTRODUCTION identifying things in an image would require more
work for a machine. In this project, we will run an
Multiple object detection and tracking is a vital a end- to-end object identification project on a custom
potential topic of study for computer vision, where it dataset using the most recent YOLOv5
plays essential roles in intelligent video surveillance implementation developed by Ultralytics.
[1,2], robot vision navigation [3,4], virtual reality [5], The predicted probability weighs these bounding
and medical diagnosis (cell state tracking) [6]. In boxes. The approach "only looks once" at the image in
recent years, the development of crewless aerial the detected items after non-max suppression. It
vehicles (UAV) has increased the interest in detecting ensures that each item is only ever recognized once
moving objects in video sequences [7,8]. UAVs have by the object detection algorithm. we analyze and
advanced imaging capacities, where the camera can compare deep learning-based MOT methods
operate with various degrees of movement and according to deep learning functionalities in the
autonomy, but problems occur due to the moving tracking framework. We roughly classify the methods
background and motion blur. In addition, in outdoor into
conditions, due to light, occlusion, and shadows, the Three categories:
appearance of multiple objects can change to affect Multi-object tracking enhancement using deep
the precision of moving object detection. network features, in which the semantic features are
extracted from deep neural Networks designed for
related tasks and used to replace conventional
handcrafted features within the previous tracking
framework. These deep network features are typically
good at tracking performance. Generally, it is hard to
obtain multi-object tracking results by only one
Network because there are some intertwined sub-
modules in MOT tracking. Several works attempt to
implement this target by making some assumptions,
such as Markov property and fixed distributions.

2.PROBLEM STATEMENT Fig1.1 YOLOV5 architecture

The problem: 4.Multiple Object detection:
Nowadays, Accidents and crime and murders were The object detection technique in computer vision
done in public places and high security area’s. We involves identifying and localizing items within an image
can’t stop it but we can try to reducing the faults, or a video. Bounding boxes, rectangular forms
surrounding the objects, are used in image localization to
detect and analyzing the scenes from the surveillance
pinpoint the precise location of one or more items. This
camera video footage. Then we easily doing a image
 procedure is occasionally mistaken with image technique. It is interesting to note that a convolutional
classification or image recognition, which seeks to network modelled after Inception is employed in the
determine which category or class an image or an MultiBox research.
object contained inside an image. The graphic that
follows represents the previous explanation visually. 5.METHODOLOGY ADOPTED
"Person" is the object that Trained found in the
photograph. We will learn about the advantages of Our workflow was divided into two main segments.
object detection in this conceptual blog before learning 1. Optimization of the videos.
about YOLO, a cutting- edge method for object 2. Object Detection and tracking on videos
identification.

Fir 5.1 Proposed Architecture diagram

Video processing consists of signal processing
Fig 4.1 Work flow diagram employing statistical analysis and video filters to
This technology is so effective that deep extract information or perform video manipulation. Basic
learning networks can classify images better than video processing techniques include trimming, image
humans. However, when watching and interacting with resizing, brightness and contrast adjustment, and fade in
the world, people do much more than categorize and fade out, amongst others. More complex video
visuals. Within our field of view, we also localize and processing techniques, or Computer Vision Techniques,
classify each piece. These are far more complex tasks are based on image recognition and statistical analysis to
that machines cannot complete as effectively as people. perform tasks such as face recognition, detecting specific
Successful object detection pushes artificial intelligence image patterns, and computer-human interaction.
(AI) closer to a proper scene understanding. To handle
the challenges of object identification, localization, and Video files can be converted, compressed or
classification, researchers created R-CNNs. In general, decompressed using particular software devices. Usually,
an R-CNN is a particular kind of CNN that can find and compression involves a reduction of the bitrate (the
identify images of items. A collection\a set of bounding number of bits processed per time unit), which makes it
boxes, each of which roughly matches the identified possible to store the video digitally and stream it over
objects, are typically produced as well as a class output. the Network. Uncompressed audio or video usually are
With scores of over 74% map (mean average precision) called RAW streams, and although different formats and
at 59 frames per second on benchmark datasets like codecs for raw data exist, they appear to be too heavy (in
Pascal VOC, it was released at the end of November bitrate terms) to be stored or streamed over the Network
2016. COCO, the study by C. Szegedy et al. regarding in these formats.
SSD: Single Shot MultiBox Detector, established new Video summarization faces the same problem as most
benchmarks for object identification task performance deep learning tasks – it requires vast data. Collecting
and accuracy. summarization labels is time-consuming, and a small
One Shot: this describes how the Network dataset will not be sufficient. Available datasets contain
completes the objectives of object localization and only videos of certain types, which offers a poor
classification in a single forward pass. performance of the model on videos of other categories.
We can apply unsupervised, semi-supervised, or multi-
task learning to deal with this. Computational hardware
and complexity in development are also well-known
problems.
MultiBox: Szegedy et al. bounding .'s box regression
method is known by this name. TSzegedy's work on
Moreover, MultiBox's loss function integrated two
MultiBox, a technique for quick class-independent
essential elements that found their way onto ssd:
bounding box coordinate recommendations, served as
the basis for SSD's bounding box regression
 Confidence Loss: This gauges the Network's trust in the Jaccard index) is greater than 0.5. A solid starting point
computed bounding box's objectivity. Categorical for the bounding box regression process is provided by an
cross- entropy is employed. IoU of 0.5, as seen in the image below. It is a preferable
approach to starting the predictions with random
Location Loss: Location Loss is a metric used to express coordinates.
how far the Network's projected bounding boxes are from Nonetheless, this IoU still needs to be improved. As a
the training set's actual ones. Here, L2-Norm is applied. result, MultiBox attempts to regress using the priors as
The equation for the loss, which expresses how far off our predictions. MultiBox begins with the priors as
prediction "landed," is as follows. Will not go too predictions and tries to regress closer to the actual
mathematical here; read the paper if we are inquisitive and bounding boxes.
want a more exact notation:
Fixed Priors: Unlike MultiBox, each feature map cell has
Multi Colors = confidence loss + alpha * location loss a set of standard bounding boxes with varying sizes and
aspect ratios. These priors were manually (but carefully)
We may balance the location loss's contribution with the chosen, in contrast to They were chosen for MultiBox
aid of the alpha term. Finding the parameter values that because their IoU in relation to the ground truth was
lower the loss function most optimally is the common higher than 0.5. As a result, SSD should theoretically
objective of deep learning, as doing so would improve generalize to any form of input without needing a prior
the accuracy of our predictions. generation pre- training step. For instance, SSD would
calculate f * b * (4
Priors For Multi Box And IOU: + c), values for a given feature map of size f = m * n.
Contrary to what was just said, the logic underlying Suppose we have set up two diagonally opposed
the construction of bounding boxes is considerably more locations (x1, y1) and (x2, y2). The map cell's default
intricate. Stated earlier. However, do not worry; it is still bounding box and c categories are used for each feature
attainable. Priors, also known as anchors in Faster-R-CNN to classify.
parlance, are pre-computed, fixed-size bounding boxes
constructed by the researchers in Multi Box and closely
resemble the distribution of the initial ground truth boxes. Location Loss: SSD computes the location loss using
In actuality, such priors are chosen so that their smooth L1-Norm. Despite this, it is pretty efficient and
Intersection over Union ratio (also known as IoU and gives SSD more freedom because it does not try to
occasionally as the forecast bounding boxes "pixel perfect." despite being
less accurate than L2-Norm, For many of us, a change of
a few pixels would hardly be perceptible.

Classification: SSD performs object classification,

whereas MultiBox does not. As a result, a set of c-class
predictions are produced for each predicted bounding
box, accounting for every potential class in the dataset.
Require training and test datasets with given class labels
and ground truth bounding boxes (only one per bounding
box). The datasets from Pascal VOC and COCO are
helpful places to start.

Map Features: Using several feature maps and MultiBox

enhances the likelihood that any object, big or small,
will eventually be detected, localized, and correctly
classified. Features maps or convolutional block
results represent the image's critical elements at
various scales.

The Hard Negative Mining: Because most of the

bounding boxes during training will have low IoU, we
risk having many negative examples in our training
set. Furthermore, it will consequently be interpreted
as a negative training example. Consequently, it is
advisable to maintain a ratio of harmful to positive
examples of about 3:1 rather than employing only
pessimistic predictions. Must maintain negative
samples since the Network needs to understand what
makes a false detection and be explicitly told what
does not.

According to the authors of SSD, data augmentation

is essential for teaching the Network to become more
resilient to different object sizes in the input, just like
in many other deep learning applications. In order to
achieve this, they created additional training
examples that included random patches and
patches from the
 original image that were cut out at various IoU ratios self-attention module. FSCN is used for global
(e.g., 0.1, 0.3, 0.5.). Each image is horizontally rotated representation extraction and a self-attention
with a chance of 0.5, ensuring that prospective items mechanism for capturing dependencies. Self-attention
are equally likely to appear on the left and the right. mechanisms are fast in matrix computations and
retain the dependencies between frames, making
is' in IFTTT (If This Then That) stands for the them prevalent for video summarization tasks. The
condition, and 'That' stands for the action to be final step is outputting normalized importance scores.
performed if this condition is met. Both of these An LSTM- based discriminator provides a signal for
settings are up to the discretion of the user. That is to training by distinguishing the raw frame features
say, the user inputs boththe condition and the action from the important score-weighted ones. An objective
to be executed if the condition is met. function is composed of three losses: Adversarial loss
– to complete adversarial Network, generator against
the discriminator;
5.1Challenges: Sparsity loss – to limit the selected number of
keyframes; Reconstruction loss – to keep the
Video files can be converted, compressed or summary semantically close to the original video.
decompressed using particular software devices.
Usually, compression involves a reduction of the This research is attractive due to the use of
bitrate (the number of bits processed per time unit), reinforcement learning in the video summarization
which makes it possible to store the video digitally task. In contrast to the one described above, the study
and stream it over the Network. Uncompressed audio applies this method to medical videos. It investigates
or video usually are called RAW streams, if 3D spatiotemporal CNN features are better suited to
representation learning than 2D image features,
And although different formats and codecs for raw data
which are more common. Using reinforcement
exist, they appear to be too heavy (in bitrate terms) to
learning (RL) is nothing new in video analysis tasks: it
be stored or streamed over the Network in these has proven successful in visual tracking, facial
formats. recognition, video captioning and object segmentation.
Most popular FCNs work effectively for semantic
Video summarization faces the same problem as most image feature extraction. The final approach in the paper
deep learning tasks – it requires vast data. Collecting is 3D spatiotemporal U-Net (3DST-UNet) using
summarization labels is time-consuming, and a small spatiotemporal features.
dataset will not be sufficient. Available datasets The role of the RL agent here is to learn policies that
contain only videos of certain types, which offers a maximize the received reward for performing actions.
poor performance of the model on videos of other The action in our case is to select or discard the
categories. We can apply unsupervised, semi- current frame as a keyframe or not.
supervised, or multi- task learning to deal with this.
Computational hardware and complexity in 6.OUTCOME AND FUTURE SCOPE:
development are also well- known problems.
A well-known object recognition algorithm that has
As video relies on users' feedback, subjectivity is one
transformed computer vision is called YOLO (You
of the main challenges since people may consider the
Only Look Once). It is a fantastic option for real-time
different parts necessary for the same video.
object identification applications because it is quick
Depending on a specific business case, this can be
and effective. It has received widespread adoption in
solved with personalization and a content-based
recommender system. Suppose we have several numerous real-world applications and has
ground-truth summaries of the video. In that case, a attained cutting-edge performance on
practical solution is to learn from multiple losses, numerous benchmarks.
each capturing different characteristics of a summary
(summary length, closeness to the original video).

The Network consists of a generator and

discriminator (this is what adversarial means). The
generator is used to predict the importance score for
each frame in the video. Discriminator aims to
distinguish the original feature x from the importance
score weighted feature x. The goal is to retain semantic
information from the original video in summary.

One of the challenges is capturing temporal

relationships in the video, and the generator consists
of a fully convolutional sequence network (FCSN)
Fig 6.1 Output image
and a
YOLO's quick inference speed, which enables it to this paper. On one side, it is far from enough labelled
process images in real-time, is one of its key features. datasets to train satisfied models for tracking under all
It works well for augmented reality, self-driving conditions. A possible way can be paved by the
automobiles, and video surveillance applications.
generative networks, which are outstanding in
Furthermore, YOLO has a straightforward design
promoting the generalization of deep learning models.
and only needs a small amount of training data,
Conversely, to cope with declined tracking results in
making it simple to apply and modify for new jobs.
a complex environment, such as on a moving platform,
the integrated network models must learn the features of
YOLO has shown to be a valuable tool for object
these dynamic scenes. In addition, to adapt to the
identification and has opened up many new
changing conditions further, learning high-order or
opportunities for researchers and practitioners,
online transferred features are expected for the tracked
despite its. objects.
drawbacks, such as difficulty with small objects and
the inability to do fine-grained object classification. It REFERENCES
will be fascinating to see how YOLO and other object
identification algorithms change and advance as the [1] Using deep convolutional neural networks for image
field of computer vision develops. categorization, Krizhevsky, Sutskever, and Hinton.'.
Proc. Neural Information Processing System
Developments, Lake Tahoe, NV, USA, 2012, pp. 1097–
1105.

[2] Fan, L., Huang, W., Gan, C., et al.: 'End-to-end learning
of motion representation for video understanding'.
Proc. IEEE Conf. Salt Lake City, Utah: Computer Vision and
Pattern Recognition, 2018.

[3] Chen, D., Zhu, M., Wang, H.: 'Using a sparse

representation of the PCA subspace for visual tracking, up
to electron. Lett., 2017, 13, (5), pp.392–396
Visual tracking using multi-domain convolutional neural
networks. Nam, H., and Han, B.. Proc. IEEE

[4] Choi, J., Chang, H.J., Fischer, T., et al.: 'Context-aware

Fig 6.2 Output in video deep feature compression for high-speed visual tracking'.
Proc. IEEE Conf. USA, Salt Lake City, Computer Vision and
Pattern Recognition, 2018.
7.CONCLUSION:

Significant advances in deep learning methods are

made in image recognition, object detection and
person reidentification, which also benefit the
development of multi-object tracking. This paper
summarizes deep learning-based multi-object
tracking methods, which are top-ranked in public
benchmarks. The contribution of this.

The paper lies in three aspects. First, the usage of deep

learning for multi-object tracking is organized. The
mechanisms of deep feature transferring, neural
network embedding and end-to-end network training
are analyzed based on existing methods inspired by
the rules to design a new tracking framework. Second,
we investigate the roles of deep networks in tracking
frameworks and explore the issues of training these
networks. Third, comparisons between these multi-
object tracking methods are presented and
reorganized according to standard datasets and
evaluations. The advantages and limitations of the
methods are stressed. From the analysis of
experimental evaluation, there is much room to
improve the tracking results by the deep learning
paradigm. Some valuable insights are given in