Criminal Detection System using Deep learning

technique
 Abstract
A person is uniquely identifiable by their face, which is an essential component of the human
body structure. As a result, it can be utilized to track down a criminal's identify. With the development
of technology, several public locations now have cameras that can record illegal activity. Using the
previously captured faces and criminal’s images that are available in the crime database, the criminal
face recognition system can be implemented. The implementation of computer vision for the automation
of the criminal detection can substantially improve the crime situation of the city. An automatic criminal
identification system that enhances and upgrades the criminal identification as more effective and
efficient approach through the use of CNN. The CNN model is trained on the criminal dataset and then
deployed for achieving criminal detection. The fuzzy classification approach is also deployed which
considerably enhances the precision of the detection. Once the criminal is detected the system generates
an alert to notify the law enforcement that can significantly reduce the crime levels in the locality. The
proposed approach has been quantified through comparison with the conventional approaches to yield
highly satisfactory outcomes.

Keywords: Face Recognition, Face Detection, Microsoft AZURE Cognitive Services, Face
API, HAAR, Machine Learning, Cloud.
 Chapter 1
INTRODUCTION
 1. INTRODUCTION

Over the years, a lot of security approaches have been developed that help in keeping confidential
data secured and limiting the chances of a security breach. Face recognition which is one of the few
biometric methods that possess the merits of both high accuracy and low intrusiveness is a computer
program that uses a person’s face to automatically identify and verify the person from a digital image or
a video frame from a video source . It compares selected facial features from the image and a face
database or it can also be a hardware which used to authenticate a person. This technology is a widely
used biometrics system for authentication, authorization, verification and identification. A lot of
company has been using face recognition in their security cameras, access controls and many more.
Facebook has been using face recognition in their website for the purpose of creating a digital profile for
the people using their website. In developed countries, the law enforcement create face database to be
used with their face recognition system to compare any suspect with the database. In other hand, in
Malaysia, most cases are investigated by using thumbprint identification to identify any suspect for the
case. However, because of unlimited knowledge through internet usage, most criminals are aware of
thumbprint identification. Therefore, they become more cautious of leaving thumbprint by wearing
gloves except for non-premeditated crimes. This Project to propose a facial recognition system for a
criminal database where the identification of the suspect is done by face matched rather than thumbprint
matched.
Face Recognition for Criminal Identification is a face recognition system in which the security
expert will input an image of the person in question inside the system and the system will first
preprocess the image which will cause unwanted elements such as noise to be removed from the image.
After that, the system will then classify the image based on its landmarks for example, the distance
between the eyes, the length of the jaw line, etc. Then, the system will run a search through the database
to find its perfect match and display the output. This work is focusing on implementing the system for
criminal identification. Current practice of thumbprint identification which is simple and easy to be
implemented can be challenge by the use of latent thumbprint and sometimes cannot be acquired from
the crime scene. The criminals have become cleverer and normally be very careful in leaving any
thumbprint on the scene. This system encompassed face database and an image processing algorithm to
match the face feed with faces stored in the database. There are two parts vital to the success of this
system; detection and recognition. A face detection is one of the most important steps in a face
recognition system and can be classified into four principle categories; knowledge based, feature
invariant, template matching and appearance-based methods . In recognition, two stages are required;
training process and evaluation process. In a training process, the algorithm is fed samples of the images
to be learned and a distinct model for each image is determined while in an evaluation process, a model
of a newly acquired test image is compared against all existing models in the database. Then the near
corresponding model is acquired to determine whether the recognition is triggered . In this stage, a
statistical procedure, Principal Component Analysis (PCA) is used to on a collection of face images to
form a set of basis features, which is called a set of eigenfaces. Any human face can be considered to be
a combination of these standard face.

1.2 Problem Definition

To enhance the process of Criminal Identification system through face recognition proposed model
is using Convolution neural network and Image Segmentation process.
 Chapter 2
LITERATURE SURVEY
 2.1 Literature Survey

Microsoft Kinect Sensor is used for detecting faces[1]. Kinect has a RGB sensor and a IR sensor.
CCTVs have a RGB sensor with IR LEDs for night vision. The device detecting the faces was Kinect
with its vast hardware and sensors. These faces were passed on in a cropped manner to the system. Used
a Ann to train the system using 4 faces. The Ann was trained on Azure. The processing was done by a
PC104+ embedded system. The link between the PC104+ system and the Windows Azure cloud
infrastructure was a 3G wireless connection. The system was overall very fast with speeds sometimes
around 200ms.
The method used several sub methods like skin detection although he most prominent was face
detection [2].The Algorithm used was the Haar Cascade Classifier. HAAR Cascade classifier mainly
works by finding patterns with reference to black and white rectangles. This particular paper used a
advanced implementation which also involved diagonals. Every Face is modeled as an ellipse. Skin
detection is used to speed up haar face detection .This is only detection not recognition.
It is a multistep process where every frame was analyzed [3]. First step was detection using Haar
classifier. Second step was detecting face using both Eigen Faces and Gabor algorithm, third step was
decision making and selecting. The accuracy was around 50 percent because of changes in illumination
pose…etc.
The authors used method which specifically used Microsoft Azure to compute data [4]. This
project main aim was to find objects in the environment for robot to navigate. The paper used the SIFT
and SURF Algorithm as its main object detection strategy .SIFT finds objects by identifying
blobs .Another technique is to try a direct match .The user interface was a windows from C# application.
Authors used the method which used a technique to determine the consistent background [5].
This paper made use of 2 cameras to prevent any disturbance from lighting. The one particular method
they used was background subtraction. The specific algorithm was the MOG background subtraction.
Long term and shorter analysis is done in case someone come back to pick up his luggage. Algorithm
used was the MOG background subtraction.Long term and shorter analysis is done in case someone
With increasing terrorist activities there was augmenting demand for video surveillance. Mostly images
are generally classified based on the value of simple features [6]. It is always better to use features rather
than using pixels as feature based systems always operates much quicker than pixel based systems. In
this approach the algorithm consists of 3 intermediate steps :
 A) By using an intermediate representation for the image, Integral rectangle features can be
computed
very quickly.
B) Adaboost technique can be used for the construction of classifiers that helps us to separate
desired features from the collection of vast no of features. It utilizes a set of positive and negative
images to train.
C) Cascading of different classifiers
A full advantage of Pattern Recognition and Image Processing model can be taken with the help of Open
Computer Vision (OpenCV). To detect human face and achieve fast face detection of the video, mosaic
gray rules were adopted. Three types of main algorithm based on mosaic model are: Gray Rules: The
trisection image model according to the organ segmentation of face is established. Integral Image:
Rectangle feature can be computed very quickly using an intermediate representation for the image.
Frequency Histogram: It removes the non-face region from the image and merge the overlapping face
region.
Changes in the facial appearance occurs due to natural ageing of human [8] [22]. There are
147,784 operational mugshots of 18,007 repeat criminal offenders in the longitudinal face database. It
consists of at least 5 face images of a subject that were collected over at least a 5year time spa. As
elapsed time increases between two face images, population mean trends in genuine scores is estimated
by using multilevel statistical models. COTS-A performance is better than COTS-B.
Haar classifier is used to detect faces in the frames that are come as input [9]. A face detection
method based on AdaBoost was adopted in the paper. Improved AdaBoost algorithm achieves more
robust performance. And high speed over conventional AdaBoost based methods. Study of locality
preserving projection is essential and analyzed the Proposed a method using Bilinear CNN [10]. At each
location convolution layer outputs of two CNNs of the image are multiplied. Image labels are used to
train bilinear CNN model & it requires training network.
Authors proposed a method using deep learning. Automatically collecting and labelling the data
from CCTC videos is done in order to construct a dataset [11]. Face were recognized using VGG Face
Recognition Algorithm. Haar Classifier is used for detecting faces. The paper reported accuracy of
99.2%.Training is an essential part and in this case required 2.6M images of 2.6 lakh people .
This paper classify object detection into many categories like model based system, image
invariance method, example based method , static object detection , moving object detection .Recursive
and non recursive algorithm is used for removing background [12].Object can be tracked using point
tracking ,kernel tracking, edge detection and color. This paper classify object detection into many
categories like model based system, image invariance method, example based method , static object
detection ,moving object detection [13].Recursive and non recursive algorithm is used for removing
background .Object can be tracked using point tracking ,kernel tracking, edge detection and color.
This paper proposes three steps. Steps include face detection, face feature extraction and finally
face recognition [14]. Face was detected and background removed, face features like face cuts and
angles were formatted and styled, while recognition goes ahead and identifies it 3 detection methods
were experimented with namely camshaft algorithm, Haar classifier, vend finding via motion. Camshift
and finding via motion were fast but the most accurate and reliable was Haar classifier.
Authors introduces the Benchmark(IARPA Janus) A (IJBA), face images are manually localized
by publicly available media in the wild dataset (500 subjects) [15] [23]. IJB-A protocol focuses on: the
ability to search for a person’s face in a set of images (search), and (ii) the ability to compare facial
imagery from two persons, and verify whether or not they are the same person (compare). All faces have
been manually localized and have not been filtered by a commodity face detector is a key distinction
between this dataset and previous datasets. In the IJB-A dataset amount of variation in pose, occlusion
and illumination is unprecedented.
Based on conditional random field (CRF) which is combined with the saliency measure which
introduces a new salient object segmentation method [16] [24]. Statistical framework formulate saliency
measure and contrast local features in illumination, color and motion information. Method is efficiently
implemented by using statistical frame work [22]. The integral histogram approach and graph cut
solvers effectively implements the method. The feature includes lab color values and optical formation
which are obtainable in real time.
In the process for detecting the face from the frame that is extracted from the video, author
proposed CIE-Luv color space model [17]. The model is capable of separating out skin and non-skin
regions and detect skin like reasons. Skin detection is carried out on the basis of image size
(height/width>=2/3).Face verification: He says once the skin detection is done, by using variance
formula we can distinguish between face and other parts of body detected in image. The extracted face
from face detection step is verified in database faces to find person and return best match location along
with face recognized. In proposed system the best thing is video is changed to CIE-Luv color model that
works on HSV (Human Vision System) thus accuracy is guaranteed. Variance of test face is calculated
and variance of each face in codebook is calculated, as per the variance of test face, near valued face
from codebook is selected. For face detection viola jones algorithm is used to reduce the computational
complexity.[18]. This algorithm is split into three modules:
1] Integral image: deals with skin color, motion, facial appearance.
2] For face extraction- illumination, noise low regulation are major problem.
3] Problems are overcome by various techniques such as geometric based. Haar wavelet is used
for verifying face
This paper shows two contributions that are made:
A) Assembling of a large scale dataset.
B) Use of deep CNN with appropriate training.
Automation and human combined in the loop to represent large dataset. Data purity and time are the
main focus [19].
The face benchmarks (LFW and YTF) are used to achieve comparable state of result [25]. CNN
feature extractor, a learnable function obtained by composing several linear and nonlinear operators is
used. Euclidean distance used for comparing face in data book and nearest valued face from data book is
returned.
The Deep face work was extended by the DeepID. Dataset
collection included:
1) A list of candidate identity names are bootstrapped and filtered.
2) More images for each identity are collected.
3) Automatic filter is used to improve the purity.
4) Near duplicate removal
5) Final manual filtering.
6) Training: Learning a face classifier, learning a embedding using a triplet loss. The goal
is to minimize the average prediction log-loss .
In distinctive video frames, Author’s proposed algorithm will track the real time moving objects
[20] [25]. The objective is to relate target objects in consecutive video frame for tracking purpose.
Moving object in different frames of video is tracked in the real time using color feature and motion. In
image processing median filtering is used to decrease noise which is a nonlinear operation. Target
representation and localization and filtering and information association is vital in visual tracking.
 Chapter 3
PROPOSED METHODOLOGY
 3.1Proposed Methodology

Fi
g 1 System Overview Design

The system overview diagram provides an overview of the system with the important modules in
the form of blocks. At first the user provides the criminal dataset which is preprocessed and the images
are normalized before sending to the Convolutional Neural Networks to achieve the trained data. The
user then provides the live feed the frames from which are grabbed and preprocessed following that the
CNN trained data is deployed. The Fuzzy classification is utilized which results in the Criminal
Detection and Alert generation.

3.2 Module Description

3.2.1 Module A: Preprocessing

 Image Scaling
 Image Sharing
 Image restoration
 Dataset list formation

3.2.2 Module B: Image Normalization

 Pixel Position
 Color Model
 Model Features
 Region Estimation

3.2.3 Module C: Convolutional Neural Network

 ROI Extraction
 First Layer Convolution
 Fully Connected layer
 Convolution Rate

3.2.4 Module D: Fuzzy Classification

 Test Image data
 Model initialization
 If-then rules
 Criminal Identification
 3.3 Proposed Methodology with relevant Diagrams and Figures

3.3.1 Data Flow Diagrams

3.3.1.1 DFD level 0

Fig 2 DFD level 0

The DFD 0 diagram for the data flow diagrams describes the flow of the approach. The DFD
diagram provides the simplest flow where in the Live feed is provided and the preprocessing and CNN
is implemented and the Criminal Detection and Alert generation is achieved.
 3.3.1.2 DFD level 1

Fi
g 3 DFD level 1
The DFD 1 diagram provides even more details wherein the user provides the Live feed which is
provided for Preprocessing which results in the preprocess list. The image normalization is utilized and
the CNN is deployed through the activation function after which the Fuzzy classification technique is
applied and the Criminal Detection and Alert generation is achieved.
 3.3.1.3 DFD level 2

Fi
g 4 DFD level 2
The DFD 2 diagram is the most detailed wherein the user provides the Live feed from which the
preprocess list is generated and image normalization is utilized through byte array. The system then
deploys CNN through Activation function and dense layer formation following which the fuzzy
classification is applied to get Criminal Detection and Alert generation.
 3.3.2Activity Diagram

Fig 5 Activity Diagram

The activity diagram lists the various activities that are performed in the proposed methodology,
the start state is initiated and the user provides the live feed, preprocessing, image normalization, CNN
which results in the Criminal Detection and Alert generation.

3.3.3 Usecase Diagram

Fig 6 Usecase Diagram

 The Use case Diagram depicts the various use cases that are performed by the user in the
proposed model. The use cases include, live feed, preprocessing, image normalization, CNN and finally
the criminal detection.

3.3.4 Sequence Diagram

Fig 7 Sequence Diagram

The sequence role diagram provides a sequence of the approaches as well as the various roles
performed in the intermediate. In this approach the live feed is provided by the user which is
preprocessed and then normalized. The CNN is implemented using the activation function following
which fuzzy classification is performed using fuzzy list that results in the Criminal Detection and Alert
generation.
 3.3.5 Component Diagram

Fi
g 8 Component Diagram
The component diagram illustrates the important components in the proposed system. In our
approach the important components consist of the Live feed which is interlinked with preprocessing
module and then the CNN, these two modules are further linked to the Fuzzy Classification and the
Criminal Detection and Alert generation module.
 3.3.6 Deployment Diagram

Fig 9 Deployment Diagram

The deployment diagram illustrates the important resources that are utilized for the deployment
purposes. In our approach the system resources consists of Camera, Keras & TensorFlow and the
training dataset along with the Criminal Detection System and the access to the system using the Tkinter
GUI.

3.3.7 Package Diagram

 Fig 10 Package Diagram
The package diagram lists all the important modules and the various packages that are being
utilized to achieve the system, the modules include, Live feed, Criminal Detection and Alert generation,
CNN, whereas the packages interconnected to these modules include, Keras, TensorFlow, OpenCV,
Matplotlib and SKLearn.

3.3.8 State Transition Diagram

 Fig 11: State Transition
The state transition diagram provides the various states that the proposed system goes through.
Initially the start state wherein the user provides the Live feed from which the preprocess list is
generated and image normalization is utilized through byte array. The system then deploys CNN
through Activation function and dense layer formation following which the fuzzy classification is
applied to get Criminal Detection and Alert generation and then reaches the stop state.

3.4 Action Plan

 Fig 12: Action Plan

(1 Week):
Group formation and Sample project topic selection

(2 Week):
Abstract and Synopsis

(3 Week):
Project identification and definition

(4 Week):
Proposal Submission
(1 Week – 4 Week):
Layout and design implementation

(1 Week – 4 Week):
Detailed Design Evaluation

(1 Week – 4 Week):
Design document and methodology

(1 Week – 4 Week):
Testing and experimental result

(1 Week – 4 Week)
Analysis of result and conclusion

(3 Week – 4 Week) :
Report document and entire work
 Chapter 4
SOFTWARE REQUIREMENT SPECIFICATIONS
4.1 Minimum Hardware Specification:

 Processor: Dual Core of 2.2 GHZ

 Hard Disc: 100 GB

 RAM : 2GB

4.2 Software Specification:

 Platform: Python

 Technology : Python 3.9

 IDE: Spyder 5.0

 Database : Mysql 5.0

 Libraries : Keras, Tensorflow

REFERENCES
[1] Dobrea, D. M., Maxim, D., & Ceparu, S. (2013, July). A face recognition system based on a
Kinect sensor and Windows Azure cloud technology. In International Symposium on Signals, Circuits
and Systems ISSCS2013 (pp. 1-4). IEEE.
[2] Abin, A. A., Fotouhi, M., & Kasaei, S. (2009, October). Realtime multiple face detection and
tracking. In 2009 14th International CSI Computer Conference (pp. 379-384). IEEE.
[3] Tathe, S. V., Narote, A. S., & Narote, S. P. (2016, December). Face detection and recognition
in videos. In 2016 IEEE Annual India Conference (INDICON) (pp. 1-6). IEEE.
[4] Lorencik, D., Ondo, J., Sincak, P., & Wagatsuma, H. (2015). Cloud-Based Image
Recognition for Robots. In Robot Intelligence Technology and Applications 3 (pp. 785-796). Springer,
Cham.
[5] Lin, K., Chen, S. C., Chen, C. S., Lin, D. T., & Hung, Y. P. (2015). Abandoned object
detection via temporal consistency modeling and back-tracing verification for visual surveillance. IEEE
Transactions on Information Forensics and Security, 10(7), 1359-1370.
[6] Chatrath, J., Gupta, P., Ahuja, P., Goel, A., & Arora, S. M. (2014, February). Real time
human face detection and tracking. In 2014 international conference on signal processing and
integrated networks (SPIN) (pp. 705-710).IEEE.
[7] Wu, L., Wu, P., & Meng, F. (2010, August). A fast face detection for video sequences. In
2010 Second International Conference on Intelligent Human-Machine Systems and Cybernetics (Vol. 1,
pp. 117-120). IEEE.
[8] Best-Rowden, L., & Jain, A. K. (2018). Longitudinal study of automatic face recognition.
IEEE transactions on pattern analysis and machine intelligence, 40(1), 148-162.
[9] Chavan, V. D., & Bharate, A. A. (2016). A review paper on face detection and recognition in
video. International Journal of Innovative Research in Electrical, Electronics, Instrumentation and
Control Engineering, 4, 97-100.
[10] Chowdhury, A. R., Lin, T. Y., Maji, S., & Learned-Miller, E.(2016, March). One-to-many
face recognition with bilinear cnns. In 2016 IEEE Winter Conference on Applications of Computer
Vision (WACV) (pp. 1-9). IEEE.
[11] Wang, Y., Bao, T., Ding, C., & Zhu, M. (2017, June). Face recognition in real-world
surveillance videos with deep learning method. In 2017 2nd International Conference on Image, Vision
and Computing (ICIVC) (pp. 239-243). IEEE.
 [12] Mishra, P. K., & Saroha, G. P. (2016,

March). A study on video surveillance system for object detection and tracking. In 2016 3rd
International Conference on Computing for Sustainable Global Development (INDIACom) (pp. 221-
226).IEEE.
[13] Goyal, K., Agarwal, K., & Kumar, R. (2017, April). Face detection and tracking: Using
OpenCV. In 2017 International conference of Electronics, Communication and Aerospace Technology
(ICECA) (Vol. 1, pp. 474-478). IEEE.
[14] Rahtu, E., Kannala, J., Salo, M., & Heikkilä, J. (2010, September). Segmenting salient
objects from images and videos. In European conference on computer vision (pp. 366-379). Springer,
Berlin, Heidelberg.
[15] Klare, B. F., Klein, B., Taborsky, E., Blanton, A., Cheney, J., Allen, K., ... & Jain, A. K.
(2015). Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a.
In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1931-1939).
[16] Heshmat, M., Abd-Elhafiez, W. M., Girgis, M., & Elaw, S.(2016, December). Face
identification system in video. In 2016 11th International Conference on Computer Engineering &
Systems (ICCES) (pp. 147-154). IEEE.
[17] Yadhul, K., Lakshmi, P. S., & Saju, A. (2014, February). Face detection and recognition
with video database. In 2014 International Conference on Electronics and Communication Systems
(ICECS) (pp. 15). IEEE.
[18] Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015, September). Deep face recognition. In
bmvc (Vol. 1, No. 3, p.6).
[19] Singh, P., Deepak, B. B. V. L., Sethi, T., & Murthy, M. D. P. (2015, April). Real-time
object detection and tracking using color feature and motion. In 2015 International Conference on
Communications and Signal Processing (ICCSP) (pp.1236-1241). IEEE.
[20] Tian, Y., Feris, R. S., Liu, H., Hampapur, A., & Sun, M. T.(2011). Robust detection of
abandoned and removed objects in complex surveillance videos. IEEE Transactions on Systems, Man,
and Cybernetics, Part C (Applications and Reviews), 41(5), 565-576.
[21] https://azure.microsoft.com/en-in/services/cognitiveservices/face/
[22] https://biometrics.cse.msu.edu
[23] https://www.cv-foundation.org
 [24] https://citeseerx.ist.psu.edu
[25] Deng, W., Chen, B., Fang, Y., & Hu, J. (2017). Deep correlation feature learning for face
verification in the wild. IEEE Signal Processing Letters, 24(12), 1877-1881.

*****