video

Title Page
Title: Video-Surveillance-and-Tracking-System Using Facial Recognition
Authors:

Gourav R [4VV20CS042]

Iftheqar Ahmed [4VV20CS054]

Keerthan K [4VV20CS059]

Hrithik K Gowda [4VV20CS053]

Department:
Computer Science and Engineering
Vidyavardhaka College of Engineering
Mysuru, India
Supervisor:
Dr. Ramakrishna Hegde
Associate Professor
Computer Science and Engineering
Vidyavardhaka College of Engineering
Mysuru, India

[email protected]

Abstract
This project introduces a comprehensive system for real-time location tracking
and access management within premises using a combination of CCTV cameras,
facial recognition technology, and Python-based computer vision and machine
learning techniques. The system's primary objectives are to accurately track the
live location of individuals within the premises and maintain a secure record of
their entries without relying on traditional biometric methods, access cards, or
manual data entry.

video 1
 By harnessing the power of computer vision and machine learning, the proposed
system employs facial recognition algorithms to identify and track individuals
captured by CCTV cameras. Through an innovative integration of Python libraries
and custom-developed algorithms, the system processes the video feed from
these cameras in real-time. It extracts facial features, matches them against a pre-
trained database of authorized individuals, and determines their presence and
location within the premises.
The system's secondary goal is to establish an efficient access management
protocol. Instead of conventional biometric measures or physical access cards,
individuals are granted entry by their recognized faces. The system logs each
entry, associating it with the person's identity and timestamp. This approach
minimizes the risk of unauthorized access, streamlines the entry process, and
reduces the dependency on manual recordkeeping.
The project also encompasses machine learning techniques that enable the
system to continually refine its facial recognition capabilities. By exposing the
system to various lighting conditions, angles, and appearances, its accuracy and
robustness are enhanced over time. The machine learning aspect involves training
the system with diverse datasets and employing techniques such as convolutional
neural networks (CNNs) to improve recognition accuracy.
In conclusion, the proposed system offers a powerful solution for live location
tracking and access management through the integration of CCTV cameras, facial
recognition, Python-based computer vision, and machine learning techniques. By
addressing the limitations of traditional methods and leveraging advanced
technologies, this project contributes to enhancing security, efficiency, and
convenience in premises management.

Acknowledgments
The authors would like to express their sincere gratitude to Dr. Ramakrishna
Hegde, Associate Professor in the Department of Computer Science and
Engineering at Vidyavardhaka College of Engineering, Mysuru, for his valuable
guidance, continuous support, and expert supervision throughout the
development of this video surveillance and tracking system using facial
recognition.

video 2
 The authors would also like to acknowledge the support and resources provided
by the Computer Science and Engineering department at Vidyavardhaka College
of Engineering, which enabled the successful implementation of this project.
Furthermore, the authors would like to thank the open-source community and the
developers of the Python libraries and computer vision tools utilized in this
project, without which the realization of this system would not have been possible.

Table of Contents

1. Introduction
1.1 Motivation
1.2 Objective
1.3 Problem Statement

2. Literature Survey
2.1 Related Work on Facial Recognition and Video Surveillance Systems
2.2 Traditional Face Recognition Algorithms
2.3 Deep Learning for Face Recognition

3. Requirements Specification
3.1 Details on the Requirements and Components of the Proposed System

4. System Design/Methodology
4.1 Explanation of the Methodology
4.2 Overview of the System Architecture and Key Components

5. Implementation
5.1 Details on the Implementation of the Facial Recognition and Video Tracking
System
5.2 Utilization of Python Libraries and Computer Vision Techniques

6. Testing
6.1 Evaluation of the System's Performance and Accuracy
6.2 Handling of Variability and Adaptability

7. Results and Analysis

7.1 Assessment of the System's Capabilities
7.1.1 Face Identification Accuracy
7.1.2 False Positives and Negatives
7.1.3 Speed and Efficiency

video 3
 7.1.4 Data Security and Privacy
7.1.5 Robustness and Adaptability
7.1.6 Integration with Surveillance and Tracking
7.1.7 Alerts and Notifications
7.1.8 Usability and User Interface
7.1.9 Scalability and Maintenance

8. Conclusion
8.1 Summary of the Proposed System and Its Key Contributions
8.2 Future Scope and Improvements

References

List of Tables & Figures

Tables:

bro we need to add some of the details

Figures:
Figure 1: System Architecture Overview
This figure would likely depict the high-level architecture of the proposed video
surveillance and tracking system, highlighting the key components such as CCTV
cameras, facial recognition algorithms, computer vision techniques, and data
storage/processing modules.

Figure 2: Facial Recognition and Tracking Process Flow

This figure would illustrate the step-by-step process of the facial recognition and
tracking mechanism, including steps like image upload, face detection, feature
extraction, matching, and location tracking.
Figure 3: Sample Output Frame with Recognized Faces
This figure would show an example output frame from the system, displaying the
video feed with bounding boxes around the recognized faces and the
corresponding person's name.

Figure 4: Accuracy and Performance Metrics

This figure would present the key performance metrics and analysis results, such
as face identification accuracy, false positive/negative rates, and processing
speed, among others.

video 4
 The exact number and content of the tables and figures may vary depending on
the final implementation and evaluation of the proposed system.

Chapter 1: Introduction

Motivation

Objective

Problem Statement

Motivation:
The introduction highlights the increasing security concerns and
the ever-growing need for advanced surveillance technologies. It
states that the convergence of video surveillance and facial
recognition has emerged as a game-changing solution,
combining the power of video cameras and facial recognition
algorithms to create a comprehensive system that significantly
enhances security, monitoring, and access control in a wide
range of environments.
The document mentions that video surveillance, a staple of security infrastructure
for decades, has evolved from analog closed-circuit television (CCTV) systems to
high-definition digital networks. These systems have been instrumental in
deterring and documenting security breaches, but their effectiveness often
depends on human monitoring and manual review of extensive video footage.

The introduction further emphasizes that facial recognition, once a realm of

science fiction, has evolved into a sophisticated tool with the potential to
revolutionize how we monitor and secure our surroundings. This technology
leverages the unique characteristics of an individual's face to accurately identify
and track them within a network of cameras, providing previously impossible
levels of situational awareness.

The motivation for the proposed system is driven by the need to address the
limitations of traditional video surveillance methods and leverage the capabilities
of facial recognition technology to enhance security, monitoring, and access
control in various environments.

video 5
 Objective:
The primary objective of this project is to develop a
comprehensive system for real-time location tracking and access
management within premises using a combination of CCTV
cameras, facial recognition technology, and Python-based
computer vision and machine learning techniques.
The specific objectives of the proposed system are:

1. Accurately track the live location of individuals within the premises.

2. Maintain a secure record of their entries without relying on traditional

biometric methods, access cards, or manual data entry.

By leveraging the power of computer vision and machine learning, the proposed
system aims to employ facial recognition algorithms to identify and track
individuals captured by the CCTV cameras. Through the integration of Python
libraries and custom-developed algorithms, the system processes the video feed
in real-time to extract facial features, match them against a pre-trained database
of authorized individuals, and determine their presence and location within the
premises.

The secondary objective of the system is to establish an efficient access

management protocol. Instead of conventional biometric measures or physical
access cards, individuals are granted entry based on their recognized faces. The
system logs each entry, associating it with the person's identity and timestamp, to
minimize the risk of unauthorized access, streamline the entry process, and
reduce the dependency on manual recordkeeping.

The system also encompasses machine learning techniques to continually refine

its facial recognition capabilities, enhancing accuracy and robustness over time by
exposure to various lighting conditions, angles, and appearances.

Problem Statement:
The document identifies the limitations of traditional video
surveillance methods and the potential of facial recognition
technology to revolutionize premises management. It states that
the proposed system aims to accurately track the live location of
individuals within the premises and maintain a secure record of

video 6
 their entries without relying on traditional biometric methods,
access cards, or manual data entry.
Specifically, the problem statement highlights the following:

1. Limitations of traditional video surveillance methods:

Dependence on human monitoring and manual review of extensive video

footage

Reduced effectiveness in deterring and documenting security breaches

2. Potential of facial recognition technology:

Ability to accurately identify and track individuals within a network of

cameras

Providing previously impossible levels of situational awareness

The problem statement emphasizes the need to address these limitations and
leverage the capabilities of facial recognition technology to enhance security,
monitoring, and access control in various environments. The proposed system
aims to contribute to this by developing a comprehensive solution that integrates
CCTV cameras, facial recognition, Python-based computer vision, and machine
learning techniques.

By addressing the limitations of traditional methods and harnessing advanced

technologies, the project seeks to enhance security, efficiency, and convenience
in premises management through the implementation of the video surveillance
and tracking system using facial recognition.

Chapter 2: Literature Survey would include the following:

Related work on facial recognition and video surveillance systems:
The literature survey section provides an overview of the related work and
existing approaches in the domain of facial recognition and video surveillance
systems.
Real-world applications of Face Recognition:
The document mentions that face recognition is currently being used in various
applications to make the world safer, smarter, and more convenient, such as:

video 7
 Finding missing persons

Solving retail crime

Security identification

Identifying accounts on social media

School attendance systems

Recognizing drivers in cars

Traditional Face Recognition Algorithms:

The chapter discusses several traditional face recognition algorithms that were
popular during the 1990s and early 2000s, including:

Eigenfaces (1991)

Local Binary Patterns Histograms (LBPH) (1996)

Fisherfaces (1997)

Scale Invariant Feature Transform (SIFT) (1999)

Speed Up Robust Features (SURF) (2006)

These algorithms follow different approaches to extracting and matching
image information with the input image. The traditional algorithms are
generally slower compared to modern face recognition algorithms and cannot
be trained with a single picture of a person.

Deep Learning for Face Recognition:

The chapter also covers some widely used deep learning-based face recognition
systems, such as:

DeepFace

DeepID

VGGFace

FaceNet
These deep learning-based systems utilize facial feature points and
coordinate information to perform face recognition tasks.

How Does a Traditional Face Recognition Model Work?

The document explains the typical steps involved in a traditional face recognition

video 8
 model, including:

Face Detection: Locating faces and drawing bounding boxes around them.

Face Alignment: Normalizing the faces to be consistent with the training

database.

Feature Extraction: Extracting features of faces for training and recognition

tasks.

Face Recognition: Matching the face against known faces in a prepared

database.

This literature survey provides a foundation for understanding the evolution of

facial recognition techniques, from traditional algorithms to more advanced deep
learning-based approaches, and sets the stage for the proposed system's
methodology and implementation.

Chapter 3: Requirements Specification

Details on the requirements and components of the proposed system:
The Requirements Specification chapter outlines the key components and
requirements of the video surveillance and tracking system using facial
recognition, as described in the document.

The proposed system consists of the following main components:

1. CCTV Cameras:

The system utilizes CCTV cameras to capture the video feed from the
premises.

2. Facial Recognition Library:

The system relies on the face recognition library, a wrapper for the dlib
facial recognition functionalities, to perform various face recognition tasks.

3. Python-based Computer Vision:

The system leverages Python-based computer vision techniques,

particularly using the OpenCV library, to access and process the video
frames in real-time.

video 9
 4. Machine Learning Techniques:

The system incorporates machine learning techniques, such as

convolutional neural networks (CNNs), to enable the continuous
refinement of facial recognition capabilities and improve accuracy over
time.

The key requirements and functionalities of the proposed system include:

1. Image Uploading and Processing:

The system allows users to upload a clear image of the target individual,
which is then processed and converted to the required format for facial
recognition.

2. Facial Encoding and Storage:

The system computes facial encodings for the uploaded image and stores
them in an array for subsequent recognition.

3. Real-time Video Processing:

The system utilizes OpenCV to access and process video frames from the
CCTV cameras in real-time.

4. Face Detection and Localization:

The system employs face detection algorithms to identify and locate faces
within the video frames.

5. Facial Recognition and Matching:

The system calculates the facial encodings for the detected faces and
matches them against the stored encodings to identify individuals.

6. Location Tracking and Logging:

The system captures and records the location of the camera, date, and
time for each recognized individual, storing this information in a file.

7. Access Management:

The system grants access to individuals based on their recognized faces,

rather than traditional biometric methods or access cards, and logs each
entry with the person's identity and timestamp.

video 10
 8. Continuous Learning and Adaptation:

The system incorporates machine learning techniques to continuously

refine its facial recognition capabilities, improving accuracy and
robustness over time.

This chapter provides a comprehensive overview of the requirements and

components necessary for the development of the video surveillance and tracking
system using facial recognition, setting the stage for the subsequent system
design and implementation.

Chapter 4: System Design/Methodology

Explanation of the methodology used to develop the system:

The "Video Surveillance and Tracking System Using Facial Recognition" utilizes
the face recognition library, a convenient wrapper for dlib's facial recognition
functionalities, as the core component of the system's methodology.

The system's methodology can be summarized as follows:

1. Image Uploading and Processing:

The user uploads a clear image of the target individual.

The image is processed in the default BGR format using the face
recognition library and then converted to RGB for further analysis.

The facial encodings are extracted from the image and stored in an array.

2. Real-time Video Processing:

The system uses the OpenCV library to access and process video frames
from the CCTV cameras.

Each frame is resized to 1/4 of its original size to improve the Frames Per
Second (FPS) and enhance processing efficiency.

3. Face Detection and Localization:

The face recognition.face_locations() function is applied to the resized

frame to detect and locate the faces.

video 11
 The bounding box coordinates of the detected faces are adjusted by a
factor of 4 to align accurately with the output frame.

4. Facial Recognition and Matching:

For each detected face, the system computes the facial encodings using
the face recognition.face_encodings() function.

The face recognition.face_distance() function is used to calculate the

distance between the test image and all images in the training directory.

The index corresponding to the minimum face distance is identified as the

matching face in the training set.

5. Marking and Tracking:

Upon identifying a match, the system draws a bounding box around the
recognized face using OpenCV and displays the matching name on the
output frame.

The location of the camera, date, and time are captured and recorded in a
file.

6. Continuous Refinement:

The system incorporates machine learning techniques, such as

convolutional neural networks (CNNs), to enable the continuous
refinement of its facial recognition capabilities.

By exposing the system to various lighting conditions, angles, and

appearances, its accuracy and robustness are enhanced over time.

Overview of the system architecture and key components:

The system architecture consists of the following key components:

1. CCTV Cameras: The video feed from the CCTV cameras is the primary input
to the system.

2. Face Recognition Library: The face recognition library, a wrapper for dlib's
facial recognition functionalities, is the core component responsible for facial
encoding, matching, and recognition.

3. OpenCV: The OpenCV library is used for accessing and processing the video
frames, as well as drawing bounding boxes and displaying information on the

video 12
 output.

4. Machine Learning Techniques: The system employs machine learning

techniques, such as CNNs, to continuously refine and improve the facial
recognition capabilities.

5. Data Storage: The system captures and stores the location, date, and time
information for each recognized individual in a file.

This chapter provides a detailed explanation of the methodology used to develop

the video surveillance and tracking system, as well as an overview of the system
architecture and the key components that work together to achieve the desired
functionality.

Chapter 5: Implementation
Details on the implementation of the facial recognition and video tracking system:
The implementation of the "Video Surveillance and Tracking System Using Facial
Recognition" leverages the face recognition library, a convenient wrapper for
dlib's facial recognition functionalities, as the core component.
The implementation process can be summarized as follows:

1. Image Uploading and Processing:

The system allows the user to upload a clear image of the target individual.

The uploaded image is processed in the default BGR format using the face
recognition library and then converted to RGB for further analysis.

The facial encodings are extracted from the image and stored in an array
for subsequent recognition.

2. Real-time Video Processing:

The OpenCV library is used to access the video frames from the CCTV
cameras.

Each frame is resized to 1/4 of its original size to improve the Frames Per
Second (FPS) and enhance processing efficiency.

3. Face Detection and Localization:

video 13
 The face recognition.face_locations() function is applied to the resized
video frames to detect and locate the faces.

The bounding box coordinates of the detected faces are adjusted by a

factor of 4 to align accurately with the output frame.

4. Facial Recognition and Matching:

For each detected face, the system computes the facial encodings using
the face recognition.face_encodings() function.

The face recognition.face_distance() function is used to calculate the

distance between the test image and all images in the training directory.

The index corresponding to the minimum face distance is identified as the

matching face in the training set.

5. Marking and Tracking:

Upon identifying a match, the system draws a bounding box around the
recognized face using OpenCV and displays the matching name on the
output frame.

The location of the camera, date, and time are captured and recorded in a
file.

6. Continuous Refinement:

The system incorporates machine learning techniques, such as

convolutional neural networks (CNNs), to enable the continuous
refinement of its facial recognition capabilities.

By exposing the system to various lighting conditions, angles, and

appearances, its accuracy and robustness are enhanced over time.

Use of Python libraries and computer vision techniques:

The implementation of the video surveillance and tracking system heavily relies on
the following Python libraries and computer vision techniques:

1. Face Recognition Library:

The face recognition library, a wrapper for dlib's facial recognition

functionalities, is the core component used for facial encoding, matching,
and recognition.

video 14
 2. OpenCV:

The OpenCV library is utilized for accessing and processing the video
frames from the CCTV cameras.

OpenCV is also used for drawing bounding boxes around the recognized
faces and displaying the corresponding names on the output frame.

3. Computer Vision Techniques:

The system employs various computer vision techniques, such as face

detection, facial feature extraction, and facial recognition algorithms, to
identify and track individuals within the video feed.

4. Machine Learning:

The implementation incorporates machine learning techniques, particularly

convolutional neural networks (CNNs), to enable the continuous
refinement and improvement of the facial recognition capabilities.

By leveraging these Python libraries and computer vision techniques, the system
is able to perform real-time facial recognition, location tracking, and access
management within the premises, contributing to enhanced security and
efficiency.

Chapter 6: Testing
Evaluation of the system's performance and accuracy:
This chapter focuses on the testing and evaluation of the video surveillance and
tracking system using facial recognition. The key aspects of the performance and
accuracy assessment include:

1. Face Identification Accuracy:

Evaluating how well the system can identify faces and compare them to a
database of known individuals.

2. False Positives and Negatives:

Analyzing the instances of false positives (non-matching faces mistakenly

identified) and false negatives (matching faces not recognized).

3. Speed and Efficiency:

video 15
 Assessing the system's speed and responsiveness in identifying and
matching faces in real-time or near-real-time scenarios.

Handling of variability and adaptability:

The chapter also examines the system's ability to handle various challenges and
maintain its performance and accuracy over time. The key aspects related to
variability and adaptability include:

1. Handling Variability:

Evaluating how well the system adapts to changes in ambient factors, such
as illumination, posture, and facial expressions.

2. Adaptability:

Determining the system's capability to adjust to changes in individuals'

appearances over time, as their faces may change due to aging, hairstyle
modifications, or other factors.

The testing and evaluation process aims to ensure that the video surveillance and
tracking system can reliably identify and track individuals in a wide range of
conditions, providing accurate and efficient performance. The assessment of the
system's adaptability to variability is crucial to ensure its robustness and long-
term effectiveness in real-world environments.
By thoroughly testing the system's performance, accuracy, and adaptability, the
authors can identify any areas for improvement and make necessary adjustments
to enhance the overall effectiveness of the video surveillance and tracking system
using facial recognition.

Chapter 7: Results and Analysis

This chapter focuses on the assessment and analysis of the video surveillance
and tracking system's capabilities and performance in various aspects:
Face Identification Accuracy:

Evaluation of the system's ability to accurately identify and match faces with
the pre-trained database.

Analysis of the overall face identification accuracy achieved by the system.

video 16
 False Positives and Negatives:

Assessment of the instances where the system mistakenly identifies non-

matching faces (false positives).

Evaluation of the cases where the system fails to recognize matching faces
(false negatives).

Examination of the system's effectiveness in minimizing such errors.

Speed and Efficiency:

Evaluation of the system's responsiveness and processing speed in real-time

or near-real-time scenarios.

Assessment of the impact of optimization techniques, such as frame resizing,

on the overall system performance.

Data Security and Privacy:

Verification of the system's compliance with relevant standards, data

protection laws, and privacy regulations.

Evaluation of the security measures implemented to protect the facial

recognition data and ensure data privacy.

Robustness and Adaptability:

Assessment of the system's ability to handle variability in factors such as

lighting, posture, and facial expressions.

Evaluation of the system's adaptability to changes in individual appearances

over time.

Integration with Surveillance and Tracking:

Examination of the system's accuracy in tracking and following individuals

across multiple video streams or frames.

Assessment of the integration and coordination with other surveillance and

tracking technologies.

Alerts and Notifications:

Evaluation of the system's ability to provide real-time alerts and notifications

upon the identification of recognized individuals.

video 17
 Analysis of the accuracy and reliability of the alerts and notifications.

Usability and User Interface:

Assessment of the system's user-friendliness and ease of use for

configuration and administration.

Evaluation of the training requirements for users to effectively operate and

interpret the system's results.

Scalability and Maintenance:

Examination of the system's capacity to accommodate a growing database of

faces and adapt to changing requirements.

Assessment of the system's maintenance and update processes to address

security vulnerabilities and enhance performance over time.

The analysis of these key capabilities and performance aspects will provide a
comprehensive understanding of the video surveillance and tracking system's
effectiveness, identify areas for improvement, and guide future development and
optimization efforts.

Chapter 8: Conclusion
Summary of the proposed system and its key contributions:
In the concluding chapter, the authors provide a summary of the proposed "Video
Surveillance and Tracking System Using Facial Recognition" and highlight its key
contributions:

1. Comprehensive System for Location Tracking and Access Management:

The proposed system combines CCTV cameras, facial recognition

technology, and Python-based computer vision and machine learning
techniques to create a comprehensive solution for real-time location
tracking and access management within premises.

2. Accurate Tracking of Individual Locations:

The system accurately tracks the live location of individuals within the
premises, without relying on traditional biometric methods, access cards,
or manual data entry.

video 18
 3. Secure Entry Management:

The system establishes an efficient access management protocol by

granting entry based on recognized faces, minimizing the risk of
unauthorized access and streamlining the entry process.

4. Continuous Refinement of Facial Recognition:

The system incorporates machine learning techniques, such as

convolutional neural networks (CNNs), to continuously refine its facial
recognition capabilities and improve accuracy over time.

5. Enhanced Security, Efficiency, and Convenience:

By addressing the limitations of traditional methods and leveraging

advanced technologies, the proposed system contributes to enhancing
security, efficiency, and convenience in premises management.

Future scope and improvements:

The chapter also discusses the potential future scope and improvements for the
video surveillance and tracking system:

1. Multidisciplinary Approach:

The authors suggest that the development of a comprehensive

surveillance system requires a multidisciplinary approach, involving
expertise in areas such as signal and image processing, computer vision,
networking, communications, pattern recognition, and sensor
development and fusion.

2. Integrated Surveillance System:

The authors mention the possibility of using multiple cameras at various

positions and angles to address the issue of disparate facial positions or
image depths, and then applying intelligent techniques to merge the
camera feeds for improved robustness, efficiency, and error-proneness.

3. Expansion to Different Environments:

The authors express the potential to deploy the technology in various

environments, such as workplaces, classrooms, and other sensitive
locations, to improve security for staff members and other individuals.

video 19
 4. Continued Research and Development:

The authors highlight the need for ongoing research and development to
further enhance the system's capabilities, address emerging challenges,
and stay ahead of evolving security and surveillance requirements.

By summarizing the key contributions of the proposed system and outlining the
future scope and potential improvements, the authors conclude the report on a
forward-looking note, emphasizing the valuable role that video surveillance and
facial recognition technologies can play in enhancing security, efficiency, and
convenience in various settings.

video 20