Vol 13, Issue 03, MARCH/2022

ISSN NO:0377-9254

Mask RCNN: Object Detection Approach using Machine Learning

Techniques
Padma. E1, Rohith D.N.V2, Sai Charan E.V3.
1
Assistant Professor, Dept. of Computer Science Engineering, Sri Chandrasekharendra Saraswathi
Viswa Maha Vidyalaya, India
2
Student, Dept. of Computer Science Engineering, Sri Chandrasekharendra Saraswathi Viswa Maha
Vidyalaya, India
3
Student, Dept. of Computer Science Engineering, Sri Chandrasekharendra Saraswathi Viswa Maha
Vidyalaya, India
[email protected], [email protected],
[email protected]

Abstract--- The omnipresent and wide 1.INTRODUCTION

applications like video surveillance,
robotics, scene understanding, and This chapter describes the concepts
self-driving systems prompted vast used in the project Object Detection
research in the field of Machine using Machine Learning elaborately.
Learning in the past years. To the Object detection is a combination of
absolute of all the applications of the two tasks – image classification and
Machine Learning which surrounds object localization. Object localization
localization, detection, classification involves finding out the location of one
have gained an amazing research or more objects in an image whereas
momentum. [1] The main aim of this Image classification refers to predicting
project is to build a system that detects the class of any object in the given image.
objects from the image, or a stream of [3] To define object detection, it is
images given to the system in the form locating the presence of any kind of
of previously recorded video or the objects in a bounding box or classes of
real time input from the camera. these objects in the image. Machine
Bounding boxes will be drawn around Learning concepts makes the system
the objects that are being detected by learn on its own from the experiences it
the system.[6] The system will also gains, without the interference of the
classify the object to the classes the external factors.
object belongs. Python Programming
and a Machine Learning Technique An ideal image segmentation
named YOLO (You Only Look Once) algorithm will also segment unknown
algorithm using Convolutional Neural objects, that is, objects which are new or
Network is used for the Object unknown. There are numerous
Detection. applications where image segmentations
could be used to improve existing
Keywords-- Bounding Box,
algorithms by preservation to image copy
Convolutional Neural Network, Machine
detection to satellite imagery analysis to
Learning, Object Detection, YOLO.
on-the-fly visual search and human–
computer interaction.[5] In all of these
applications, having access to
segmentations would allow the problem
to be approached at a semantic level. For
example, in content-based image

www.jespublication.com
PageNo:488
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254
retrieval, the image could is added to the can be simplified using ReLU (Rectified
database as soon as segmentation is done. Linear Unit) that maps negative values to
When a query is processed, it could be 0. Then Pooling Layer collects the
segmented and allows the user to raise resulted Feature Map which is reduced to
query for similar segments in the the smaller sized matrix.[6] This is how
database —e.g., find all of the the features are extracted at the end of the
motorcycles in the database. In human– Convolutional Neural Network and is the
computer interaction, each and every part Fully Connected Layer where the actual
of video frame would be segmented so classification occurs.
that the user could interact at a finer level 1.1. PROBLEM STATEMENT
with other humans and objects in the In image processing techniques to
environment. For example, in the context carry on semantic segmentation we need
of an airport, the security team is to have to an image of definite size.
typically interested in any unattended Variable images of different sizes are not
baggage, some of which could hold appreciated and need to be resize which
dangerous things.[3] It would be causes some errors in segmentation. In
beneficial to make queries for all objects semantic segmentation we can only
which were left behind by a human. Now segment objects by using bounding boxes
a days the most important application of and cannot identify individual instances
image segmentation is in medical of even same classes. The Learning rate
analysis used in semantic segmentation is also
low.
Given a new image, an image 1.2. SOLUTION FOR THE
segmentation algorithm should output in PROBLEM STATEMENT
which the pixels of image belong Mask R-CNN is conceptually simple:
together semantically. Instance Faster R-CNN has two outputs for each
segmentation is challenging because it candidate object, a class label and a
requires the correct detection of all bounding-box offset; to this we add a
objects in an image while also precisely third branch that outputs the object
segmenting each instance. mask.[2] Mask R-CNN is thus a natural
and intuitive idea. But the additional
The YOLO (You Only Look Once) mask output is distinct from the class and
algorithm using Convolutional Neural box outputs, requiring extraction of much
Network is used for the detection finer spatial layout of an object. Next, we
purpose. It is a Deep Neural Network introduce the key elements of Mask R-
concept from Artificial Neural Network. CNN, including pixel-to-pixel alignment,
Artificial Neural Network has three which is the main missing piece of
layers that are, Input Layer, Hidden Fast/Faster R-CNN.
Layer, and the output Layer. [5] Deep
Learning is the part of the Artificial 2.LITERATURE REVIEW
Neural Network that has multiple Hidden
Object detection is a task in computer
Layer that can be used for the Feature
vision that involves identifying the
Extraction and Classification purposes.
presence, location, and type of one or
more objects in each photograph. [4] It is
Convolutional Neural Network (CNN) a challenging situation that involves
is the part of Deep Learning that is used building upon methods for object
in analysis of visual imagery. It has three recognition (e.g., where are they), object
different kinds of layers, they are, localization (e.g., what are their extent),
Convolutional Layer, Pooling Layer, and object classification (e.g., what are
Rectified linear unit Layer. [4] they).
Convolution Layer uses filter and strides In recent years, deep learning
to obtain the Feature Maps. These techniques have achieved state-of-the-art
Feature Maps are the matrix that is as a result for object detection, such as on
obtained after the Convolution Layer. It standard benchmark datasets and in

www.jespublication.com
PageNo:489
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254

computer vision competitions.[6] Most

known as the R-CNN, or Region-Based
Convolutional Neural Networks, and the
most recent technique called Mask R-
CNN that can achieve state-of-the-art
results on a range of object detection.
In the previous study most of them
have concentrated towards Object
detection, Object tracking and Object
recognition for tracking the object using
video sequences. These are discussed as Fig. 2: YOLO Architecture
follows.
[4] The images are divided into SXS
Studies related to object detection.
grid cells before sending to the
The detection of an object in video
Convolutional Neural Network (CNN). B
sequence plays a significant role in many
Bounding boxes per grid are generated
applications. Specifically, as video
around all the detected objects in the
surveillance applications. The different
image as the result of the Convolutional
types of object detection are shown in
Neural Network. On the other hand, the
Fig. 1
Classes to which the objects belong is
also classified by the Convolutional
Neural Network, giving C Classes per
grid.

Fig.1: Types of object detection method

A Review of Detection and Tracking

of Object from Image and Video
Sequences obtaining clear moving target
image. This study only concentrated on
static camera. So need to focus on
moving the camera as well as identify
multiple objects in video frames.
Fig. 3: Data Flow Diagram
3.PROPOSED METHOD The Fig. 3 explains about the Flow of
data in the System. Initially User will be
The Fig. 2 shows the Architecture given an option to choose the type of the
Diagram of the Proposed YOLO Model. File to be given to the System as an input.
Images/Video or Camera are given as the Thus, User can either choose option of
input to the system. As the name suggests File Selection or start the Camera. [6]
You Only Look Once, the input goes The User can choose either Image or a
through the network only once as a result Video and, in the later, User can start the
it detects object and form Bounding Camera module. Once the input is
Boxes. selected, Preprocessing is done, where
the SXS grids are formed. Thus the
resultant which formed with the grids is
send to the Bounding Box Prediction
process where the Bounding Boxes are
drawn around the detected objects. Next
the result from the previous process is
sent to the Class Prediction where the

www.jespublication.com
PageNo:490
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254
Class of the object to which it belongs is information that is neither classified nor
predicted. [5] Then it is sent to the labeled and it allows the algorithm to act
detection process it reduces the on that information without guidance.
clumsiness in the output by forming Here the task of machine is to group
Bounding Boxes in the final Output. unsorted information according to
similarities, patterns and differences
4.IMPLEMENTATION without any prior training of data.
4.2.3.SEMI-SUPERVISED
This chapter explains how LEARNING: semi-supervised learning
implementing is done in this project. algorithms fall somewhere in between
supervised and unsupervised learning,
4.1. OBJECT DETECTION since they use both labeled and unlabeled
[7] This is an implementation of Mask data for training – typically a small
R-CNN on Python using Faster RCNN amount of labeled data and a large
and Binary mask generation. This model amount of unlabeled data. The systems
generates bounding boxes and that use this method are able to
segmentation masks for each instance of considerably improve learning accuracy.
an object in the image. It's based on Usually, semi-supervised learning is
Regional Proposal Network along with chosen when the acquired labeled data
RoI pooling layer. RPN along with RoI requires skilled and relevant resources in
generated bounding boxes are fed into order to train it / learn from it.
Binary mask generating layer where
pixels inside bounding box are used to
create masks.
4.3.CONVOLUTIONAL NEURAL
4.2. MACHINE LEARNING: NETWORKS:
Machine Learning is the field of study
that gives computers the capability to In Deep learning a CNN is a class of
learn without being explicitly Deep Neural Networks. Multilayer
programmed. It is one of the most perceptron’s will refer to the fully
exciting technologies that one would connected networks, that is, each neuron
have ever come across. As it is evident in one layer is connected to all neurons in
from the name, it gives the computer that the next layer. The "fully-connected
makes it more similar to humans: The network” makes them prone to
ability to learn. Machine learning overfitting data. Typical ways of
became very important which is being regularization include adding some form
used today, perhaps in many more places of magnitude measurement of weights to
than one would expect. the loss function. [4] However, CNNs
take a different approach towards
4.2.1. SUPERVISED LEARNING : regularization: they take advantage of the
[3] learning as the name indicates the hierarchical pattern in data and assemble
presence of a supervisor as a teacher. more complex patterns using smaller and
Supervised learning is a nothing but simpler patterns. Therefore, on the scale
training the machine using data which is of connectedness and complexity, CNNs
well labeled that means some data is are on the lower extreme.
already tagged with the correct answer.
After that, the machine is provided with 4.3.1. THE CONVOLUTIONAL
a new set of data. So, that supervised LAYER: It is the core building block of
learning algorithm analyses the training a CNN. The layer's parameters consist of
data (set of training examples) and a set of learnable filters (or kernels),
produces a outcome from labeled data. which have a small receptive field, but
4.2.2.UNSUPERVISED extend through the full depth of the input
LEARNING: [3] Unsupervised volume.[3] During the forward pass, each
Learning trains the machine using filter is convoluted across the width and

www.jespublication.com
PageNo:491
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254

height of the input volume, computing an activation map by setting them to

the dot product between the entries of the zero.[2] It increases the nonlinear
filter and the input and producing a 2- properties of the decision function and of
dimensional activation map of that filter. the overall network without affecting the
As a result, the network learns filters that receptive fields of the convolution layer.
activate when it detects some specific
Other functions are also used to
type of feature at some spatial position in
increase nonlinearity, for example the
the input.
saturating hyperbolic segment and
the sigmoid function. ReLU is often
4.3.2.POOL LAYER: [6] Another preferred to other functions because it
important concept of CNNs is pooling, trains the neural network several times
which is a form of non-linear faster without a significant penalty
downsampling. There are several to generalization accuracy.
nonlinear functions to implement pooling
among which max pooling is the most
common. It partitions the input image 4.4. RCNN :
into a set of non-overlapping rectangles To bypass the problem of selecting a
and, for each such sub-region, outputs the huge number of regions, Ross Girshick et
maximum. al. proposed a method where we use
selective search to extract just 2000
Intuitively, the exact location of a regions from the image and he called
feature is less important than its rough them region proposals. Therefore, now,
location relative to other features, this is instead of trying to classify a huge
the idea behind the use of pooling in number of regions, you can just work
convolutional neural networks. The with 2000 regions. These 2000 region
pooling layer serves very progressively proposals are generated using the
to reduce the spatial size of the selective search algorithm which is
representation, to reduce the number of written below.
parameters, memory footprint and Selective Search:
amount of computation in the network, 1. Generate initial sub-segmentation, we
and hence to also control over fitting. It generate many candidate regions.
is common to periodically insert a 2. Use greedy algorithm to recursively
pooling layer between successive combine similar regions into larger ones
convolutional layers in CNN 3. Use the generated regions to produce
architecture. The pooling operation the final candidate region proposals.
provides another form of translation
invariance.

The pooling layer is independent it

checks deeply throughout the input and
resizes it spatially. The most common
form is a pooling layer with filters of size
2×2 applied with a decisive of 2 down
samples at every input by 2 along with
both width and height, discarding 75% of
the activations.[5] In this case, every max
operation is over 4 numbers. The depth Fig.4: RCNN
dimension remains unchanged.
4.3.3.RE LU LAYER: Re LU is the
abbreviation of rectified linear unit,
which applies the non- 4.5. FAST CNN:
saturating activating function. It
effectively removes negative values from

www.jespublication.com
PageNo:492
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254
Drawbacks of R-CNN to build a faster Faster R-CNN has two networks:
object detection algorithm and it was Region proposal network (RPN) for
called Fast R-CNN. The approach is generating region proposals and a
similar to the R-CNN algorithm. But, network using these proposals to detect
instead of feeding the region proposals to objects. The main different here with Fast
the CNN, we feed the input image to the R-CNN is that the later uses selective
CNN to generate a convolutional feature search to generate region proposals. The
map.[3] From the convolutional feature time cost of generating region proposals
map, we identify the region of proposals is much smaller in RPN than selective
and warp them into squares and by using search, when RPN shares the most
a RoI pooling layer we reshape them into computation with the object detection
a fixed size so that it can be fed into a network. Briefly, RPN ranks region
fully connected layer. From the RoI boxes (called anchors) and proposes the
feature, we use a softmax layer to predict ones most likely containing objects.
the class of the proposed region and also
the offset values for the bounding box. 4.7. MASK RCNN:

It is a conceptually simple, flexible,

The reason “Fast R-CNN” is faster and general framework for object
than R-CNN is because we don’t have to instance segmentation. Our approach
feed the region of 2000 proposals to the efficiently detects objects in an image
convolutional neural network every time. and simultaneously generates a high-
Instead, the convolution operation is quality segmentation mask for each
done only once per image and a feature instance. The method, called Mask R-
map is generated from it. CNN, extends Faster R-CNN by adding a
4.6. FASTER RCNN: branch for predicting an object mask in
parallel with the existing branch for
bounding box recognition. Mask R-CNN
Both of the above algorithms (R-CNN is simple to train and adds only a small
& Fast R-CNN) uses selective search to overhead to Faster R-CNN, running at 5
find out the region proposals. Selective fps. Moreover, Mask R-CNN is easy to
search is a slow and time-consuming generalize to other tasks, e.g., allowing us
process affecting the performance of the to determine the human poses in the
network. Therefore, Shaoqing Ren et al. framework. We show top results in all
came up with an object detection three tracks of the COCO suite of
algorithm that eliminates the selective challenges, including instance
search algorithm and lets the network segmentation, bounding-box object
learn the region proposals. detection, and person key point
detection.[8] Without bells and whistles,
Mask R-CNN outperforms all existing,
Similar to Fast R-CNN, the image is single-model entries on every task,
provided as an input to a convolutional including the COCO 2016 challenge
network which provides a convolutional winners. We hope our simple and
feature map. Instead of using selective effective approach will serve as a solid
search algorithm on the feature map to baseline and help ease future research in
identify the region proposals, a separate instance-level recognition.
network is used to predict the region
proposals.[4] The predicted region
proposals are then reshaped using a RoI
pooling layer which is then used to
classify the image within the proposed
region and predict the offset values for
the bounding boxes.

www.jespublication.com
PageNo:493
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254

should be greater or equal to threshold

value.
6. YOLO then draws bounding box
around the detected objects and predicts
the class to which the object belongs.

Input:

An image is given as an input. On this

Mask RCNN is applied to give us an
Fig. 5: Mask RCNN output.

4.8.BOUNDING BOX
REFINEMENT
This is an example of final detection
boxes (dotted lines) and the refinement
applied to them (solid lines) in the
second stage.

Fig. 7: Input Image

Output:

For each Instance of a class is

identified from the input image an Binary
mask is created after all layers are applied
Fig. 6: Bounding Box Refinement to extract RoI and the mask is applied at
appropriate locations over the RoI.[7] In
Following is the algorithm for our project we are giving different
detecting the object in the Object colored masks for different instances
Detection System. along with their class name and the
maximum probability of their class.
4.9. ALGORITHM FOR OBJECT
DETECTION SYSTEM
1. The input image is divided into SxS
grid. Instance 1 is identified and
2. For each cell it predicts B bounding represented by its mask and class name
boxes Each bounding box contains five and also with the probability of its class.
elements: x, y, w, h) and a box
confidence score.

3. YOLO detects one object per grid

cell only regardless of the number
bounding boxes.
4. It predicts C conditional class
probabilities.
5. If no objects exists then confidence
score is zero Else confidence score

www.jespublication.com
PageNo:494
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254
Till now all the above instances are
related to class Persons and the Instance
4 identified belongs to the class Tie.

Fig.8: Instance 1

Instance 2 is identified and

represented by its mask and class name Fig. 11: Instance 4
and also with the probability of its class.

5. RESULTS & TEST CASES

[3] This section describes different
results obtained by giving various Test
Cases used while Testing the System. We
used pre-trained dataset of COCO model.
Following section will describe the
different Test Cases and the results
obtained.

5.1 TEST CASES

Fig. 9: Instance 2 Table I: Test Cases with Results

Instance 3 is identified and

represented by its mask and class name
and also with the probability of its class.

Fig. 10: Instance 3

www.jespublication.com
PageNo:495
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254

Fig. 13 Image with Overlapping

Objects

Figure 13 illustrates the output

obtained when objects are overlapping.
This shows that partially visible objects
will also be detected by drawing
bounding box around it along with the
label indicating the class to which it
belongs.

5.2 RESULTS
This section describes different results
obtained by giving various Test Cases
described above. Fig. 14: Output obtained in Real-Time

Figure 14 illustrates the output when

Camera is used to detect the object.
Figure 15 illustrates the output generated
when a blur image is given as the input.
Bounding boxes are drawn with no
detected object. This is one of the
drawbacks of the project which given
unsuccessful test result.

Fig. 12: Image with Detecting Object

The Figure 12 illustrates the output of
the Object Detection of the System.

www.jespublication.com
PageNo:496
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254
You Only Look Once (YOLO) to detect
objects from a camera or Image and
Video as input. Although deep learning
and genetic algorithm is an important
problem in data analysis, it hasn’t been
dealt with extensively by the Machine
Learning. The proposed algorithm gives
higher accuracy than the existing
algorithms also.
Fig. 15: Output obtained in Real-
The proposed algorithm achieves an
Time accuracy that is comparable to Raytheon
Technologies’ current multi-tiered
6.CONCLUSION triggering systems, and does so with real-
time detection capabilities involving
minimum detection latency
By using the Mask RCNN for Instance
segmentation the learning rate is
Given the current lack of image
considerately low compared to the
classification, our future work will focus
semantic segmentation. Also we can
on moving the camera as well as identify
resize the image to our ease as we have
multiple objects in video frames. We also
our own set of convolutional neural
plan to look at the proposed algorithm’s
network to handle the input images.[10]
applicability in the object detection.
The instance segmentation can be used in
various fields and technologies like Real
time face detection, Counting the persons
in real time or counting the objects from REFERENCES
an image , Identifying features from an [1] Abdullah, A.Y., Mehmet S.G.,Iman,
image like identifying cancer cells from A., Erkan, B., A Vehicle Detection
an image , can be used in traffic footage Approach using Deep Learning
to identify an required vehicle etc. Methodologies.
Available: arXiv:1804.00429,2, April
However the main characteristic 2018.
feature of deep learning is to compute [2] Jean-Philippe Jodoin, Guillaume-
hierarchical features. With the Alexandre Bilodeau, and Nicolas
implementation of deep learning research Saunier. Tracking All Road Users at
and applications in recent methodology, Multi modal Urban Intersections. IEEE
lots of research works is going to Transactions on Intelligent
implement deep learning methods, like Transportation Systems, 17(11):3241–
convolutional Neural Networks 3251, nov 2016.
[3] Joseph Redmon, Santosh Divvala,
The project is developed with Ross Girshick, and Ali Farhadi. You
objective of detecting real time objects in Only Look Once: Unified, Real-Time
image, video and camera. Bounding Object Detection. 2016 IEEE Conference
Boxes are on Computer Vision and Pattern
drawn as soon as it detects objects along Recognition (CVPR), pages 779–788,
with the label indicating the class to jun 2015.
which the object belongs. We have used [4] “Multiple Object Detection Tracking
CPU for the processing in the project. in Urban Mixed Traffic Scenes”, 2019
IEEE International Conference on Signal
and Image Processing Applications
6.2.FUTURE WORK (ICSIPA).
[5] “Object Detection using Machine
In this paper, we show the usability of Learning”,2019 International Research
machine learning algorithms, specifically

www.jespublication.com
PageNo:497
 Vol 13, Issue 03, MARCH/2022
ISSN NO:0377-9254

Journal of Engineering and Technology Towards Real-Time Object Detection

(IRJET). with Region Proposal Networks. IEEE
[6] Redmon, J., Divvala, S.K., Girshick, Transactions on Pattern Analysis and
R.B., Farhadi, A. You only look once: Machine Intelligence, 39(6):1137– 1149,
Unified, real-time object detection. jun 2017.
arXiv:1506.02640, 2015. [10] “Traffic Prediction for Intelligent
[7] “Real-Time Vehicle Detection in All- Transportation System using Machine
Electronic Tolling System”, 2020 Learning”, 2020 3rd International
Systems and Information Engineering Conference on Emerging Technologies
Design Symposium (SIEDS). in Computer Engineering: Machine
[8] R. B. Girshick. “Fast R-CNN”. Learning and Internet of Things
CoRR, abs/1504.08083, 2015. (ICETCE).
[9] Shaoqing Ren, Kaiming He, Ross
Girshick, and Jian Sun. Faster R-CNN:

www.jespublication.com
PageNo:498