Smart Security Surveillance System with Face and

Action Recognition
Sairaj P. Rajput Vilas B. Rabad
[email protected] [email protected]

Tushar Kalaskar Dr Pravin Futane

[email protected] [email protected]

Department of Information Technology,VIIT, Pune

I. INTRODUCTION
Abstract—
In the current scenario, security is an
The growth of smart surveillance systems has essential aspect of our daily lives. However,
become necessary in recent years to guarantee the
security and safety of different surroundings. This many households find traditional security
study provides a novel way to machine learning- camera systems to be cost-prohibitive. To
integrated smart security surveillance using Python address this issue, this work focuses on
and related modules. Action and facial recognition presenting a solution that is both cost-effective
are among the system's primary functions. and incorporates facial and action recognition
Convolutional Neural Networks (CNN) are utilized
for reliable and effective identification procedures.
systems.
The work suggested, the solution improves The work proposes the introduction of a
performance and accuracy in real-time security surveillance system capable of
surveillance scenarios by utilizing the robust recognizing both the actions of intruders and
machine learning algorithms combined with the faces of individuals. Upon detection, the
Python's ecosystem, which includes libraries like system will promptly notify the owner,
TensorFlow, OpenCV, and Keras. The system
improves its recognition skills through ongoing providing real-time updates on the actions
learning and adaptation, which helps it recognize performed by the intruder.
intricate movements and recognize people in To achieve this, the dataset is developed and
observed situations. employs technologies such as Mediapipe,
Additionally, by allowing the system to promptly CNNs (Convolutional Neural Networks), and
send alerts and notifications to specific recipients other Python modules. Mediapipe is utilized
in the event of detected security breaches or
suspicious activity, the integration of Twilio API for extracting landmarks from body postures,
promotes seamless communication. This feature which are then used to predict the position of
improves the system's ability to respond quickly individuals.
and effectively to possible security threats. The main objective of the work is to offer a
cost-effective, user-friendly system capable of
Keywords: Python, Machine learning, facial and action recognition, adaptable to
CNN, Twilio API, Open-CV, LBPH,
Mediapipe,ANN. various scenarios and environments.
 II. LITERATURE SURVEY

The system described in this work uses the contingent upon the size and quality of the
GSM system for messaging services. For camera’s frame.[3]
intruder detection this work implements the
This study aims in developing the system
hardware components like sensors,Sim900
which aims in providing the home security
GRPS/GSM Module. The system is very
surveillance system. The project possesses a
beneficial for people who want to safeguard
distinct advantage in terms of time and
their properties and restrict access. This system
efficiency compared to similar systems
is very affordable and easily operated, so that
discussed. Unlike conventional surveillance
anybody whether rich or comfortable, young or
systems where all recorded videos must be
old can make use of this system. But this
manually reviewed to identify intrusions, the
system has the drawback when there is no
proposed system offers immediate detection.
network in such situations this system will fail.
This is facilitated by the generation of an email
[1]
for the user, containing vital information such
The work introduces a security monitoring as the time of intrusion and accompanying
system leveraging IoT (Internet of Things) images. Furthermore, the proposed system is
technology. The system comprises Raspberry designed to minimize false alarms by adjusting
Pi 3, Arduino, PIR sensor, webcam, and a the threshold pixel value of intruder images.
buzzer. Notably, the novelty of this system lies Users also benefit from the ability to observe
in its incorporation of human detection intrusions in real-time through live video,
capability through the use of the Histogram of enhancing the overall outcome of the
Oriented Gradients (HOG) and Support Vector project.[4]
Machine (SVM) methods, with the buzzer
The system outlined in this work integrates
serving as a warning mechanism for the
PIR sensors, RFID technology, Arduino
homeowner.
microcontrollers, and hazard lights, in addition
Simulation results demonstrate that the to OpenCV integration. Specifically, the
system can swiftly detect intruders within system utilizes the Haar Cascade Frontal Face
seconds, achieving an accuracy of 90% with a Detection algorithm for facial recognition.
processing time of approximately 2 seconds.
This comprehensive system aids in the
The work suggests that future research will
detection of potential culprits and promptly
delve into exploring alternative feature
sends images to the system owners, facilitating
extraction and classification methods aimed at
immediate action. Its versatility allows for
further enhancing the accuracy of intruder
implementation in various settings, including
detection.[2]
home security and secure locations such as
The study presents a system featuring a banks and jewelry shops.[5]
Graphical User Interface (GUI) and achieving a
The work describes how the system operates
remarkable accuracy of 94.5% in face
by detecting the motion of intruders through
recognition. Initially, users are required to
PIR sensors, capturing their images
input individual data, allowing the model to
accordingly. Leveraging the Haar Cascade
undergo training. Subsequently, the system
algorithm enhances computational efficiency
utilizes this data to recognize individuals. It’s
and accuracy in human face detection and
noted that the accuracy of the system is
recognition. Remarkably, the system achieves
an image processing rate of approximately 28 This work uses ESP32-CAM module which
images per second for the entire process. offers a versatile solution for animal intrusion
Utilizing IoT technology, the system detection, applicable across various sectors. It
promptly sends alert emails to the owner, promises to enhance wildlife preservation,
containing the latest captured image of the protect agricultural assets, fortify property
intruder. This low-cost solution proves to be security, prevent infrastructure damage, and
fast, highly accurate, and efficient in providing support environmental research. Despite its
alerts, effectively serving as a monitoring advantages, such as cost-effectiveness and
system. Its convenience makes it adept at scalability, addressing limitations like detection
addressing security concerns, ultimately range and weather sensitivity is crucial. With
contributing to the reduction or prevention of careful planning and optimization, this
break-ins.[6] technology can significantly contribute to
safeguarding assets and preserving wildlife and
ecosystems.[7]

Reference No. Limitations Future Scope

[1] i. Constant internet connection i. SIM900-GPRS module and
microcontroller can be use to
ii. The system is limited to the area with communicate.
the GSM network .
ii. By connecting to a GSM network,
the mobile device serves as a
worldwide controller.
[2] i. Look into feature extraction and
classification techniques.
[3] i. The accuracy of the system is contingent
upon the size and quality of the camera’s
frame
[7] i. The ESP32-CAM module have i. Farmers can use the system to
limitations in detecting animals beyond a safeguard their crops
certain distance
ii. Homeowners in rural areas can
ii. Variations in weather conditions and deploy the system
lighting can affect the accuracy
iii. Researchers can utilize the data
collected by the system for studying
animal behavior
[5] i. Accuracy of PIR sensor may vary based i. Implement optimizations to reduce
on distance latency

ii. Relies on cost-effective components ii. Investigate advanced motion

like Arduino Uno detection algorithms and sensor
technologies.
iii. Designed for smaller-scale
applications.
[9] i.Having redundant and irrelevant features i. Utilizing various feature selection
 algorithms to enhance intrusion
detection

ii. Transitioning from single classifiers

to hybrid classifiers can improve
performance evaluation
Gaps identified: intrusion detection, they could not have
sophisticated capabilities like remote access
1] Cost: In the work, it was discovered
and real-time monitoring.
that most of the systems were expensive due to
use of sensors and arduino uno.
2] Video Poor Quality: Poor video quality
B. Systems for Video Surveillance:
results in inaccurate predictions.
A video surveillance system / CCTV is
composed of a system of cameras,
III. EXISTING SYSTEM monitors/display units, and recorders. Cameras
The development of intruder detection systems may be either analog or digital with a host of
has been essential in the field of security to possible design features which will be
protect different settings from possible discussed momentarily. These systems can be
dangers. To meet the many security needs of applied to both interior and exterior areas of a
contemporary society, a range of technologies building or property.
and approaches have evolved, from Privacy Concerns: Using surveillance cameras
conventional alarm systems to sophisticated can raise privacy issues for employees and the
smart intrusion detection systems. These public, especially if cameras are placed in
solutions are designed to improve overall sensitive areas. Dependence on Technology:
security posture, provide quick response Surveillance systems rely on technology which
mechanisms, and provide timely intrusion can fail, be hacked, or suffer from technical
detection. In order to guide the development of issues like poor image quality.
more reliable and effective security measures,
it is critical to evaluate and comprehend the C. Systems of Security Based on IoT:
capabilities and limitations of current intruder IoT-based security systems leverage
detection systems as technological interconnected sensors, cameras, and devices
breakthroughs continue to change the to monitor and secure physical spaces. These
landscape of security solutions. systems utilize IoT connectivity to enable
remote monitoring and control, allowing users
Let's now move on to describe how the current to access surveillance feeds and receive alerts
systems are used. from any internet-connected device. Machine
learning algorithms may be employed for
anomaly detection and predictive analytics,
A. Conventional alarm systems: enhancing the system's ability to detect and
Conventional alarm systems, which use sensors respond to security threats in real-time.
like motion detectors, door/window sensors,
and glass break detectors, have long been used D. Biometric Access Control Systems:
as a means of detecting intruders. When these Biometric access control systems use distinct
systems detect movement or illicit access biometric identifiers, like fingerprints, face
within a monitored area, they sound an alert or recognition, or iris scans, to confirm people's
alarm. Although they work well for simple identities. These systems provide a high degree
of accountability and security because C. Video Processing
biometric data is hard to falsify or duplicate. Observe live video footage processing. The
Organizations can improve security at entrance motion detector algorithm detects any invasion
points and access control mechanisms and by utilizing the MOG2 algorithm for
guarantee that only authorized personnel are background subtraction, enabling the detection
allowed access to secured areas by integrating of moving objects against a static background.
biometric access control with intruder
detection systems. D. Face recognition
Upon detecting a moving person, the system
crops the area around them and tracks their
IV. METHODOLOGY coordinates. Utilizing the Haar cascade
algorithm for face detection and a CNN model
The proposed system in this research work for recognition, the system identifies the
comprises three key modules: face recognition, person. If authorized, it sends a message
action recognition, and message sending. regarding the activity to the authorized
Operating via the smartphone application IP individual.
WEBCAM, the system detects moving objects. E. Action recognition
Upon detecting an intruder, it utilizes the
Twilio API to send a message alert to the The system obtains landmarks of the person,
owner. representing the coordinates of each body part.
Based on these coordinates, it predicts the
person’s position.
F. Sending message
Finally, the system saves frames where
intruders are detected and sends a message to
the system owner.

System used 2 deep learning algorithms:

Action Recognition Model:

An artificial neural network (ANN) is an
Fig 1: System Architecture Diagram algorithm that mimics the human brain.
Artificial Neural Networks contain artificial
neurons, which are called units. These units are
System workflow: arranged in layers, constituting the entire
A. Starting server in IP Webcam artificial neural network in the system.
Begin by logging into IP WEBCAM, then Commonly, the artificial neural network will
initiate the server from the application. have one input layer, one middle layer, and a
third output layer, with a hidden layer between
B. Get the IP Url from IP Webcam them. The input layer accepts data from the
Retrieve the IP WEBCAM server IP and outside world, the hidden layer preprocesses
paste it into the UI Url field, adjusting the the data, and the output layer provides the
camera’s position as needed. response in the form of response data. Each of
these connections has weights that determine
the influence of one unit on another unit. As
the data transfers from one unit to another, the
Face Recognition Model:
neural network learns more about the data,
eventually resulting in an output from the A Convolutional Neural Network (CNN) is a
output layer. specialized type of artificial neural network
commonly used for tasks like image
The proposed system utilizes the ANN recognition, including face recognition. CNNs
Algorithm for predicting the position of a consist of multiple layers, including
person using landmarks. convolutional layers, pooling layers, and fully
connected layers. These networks
The following steps are taken for automatically learn spatial hierarchies of
constructing the action recognition model features from input images. In face recognition,
based on the ANN Algorithm: CNNs analyze facial features at different levels
1. Creation of the dataset: A total of 200 of abstraction, enabling accurate and reliable
images are collected for running, standing, face recognition. Despite facing challenges in
walking, and waving activities. These images predicting unauthorized individuals, a
are split into 75% for training and 25% for secondary model based on the LBWH
testing purposes. algorithm of OpenCV is trained to address this
issue.
2. Generation of CSV File: The
coordinates of each part of the training data are 1. Creation of the dataset: A total of 150
generated using the mediapipe library, and images of 4 persons are collected and split into
each class registers the coordinates. The model 70% for training and 30% for testing. These
is then trained based on the CSV file. images are converted into grayscale images of
size 64 by 64.

2. Creation and training of the CNN

Model: The CNN model is created and trained
on the dataset achieving an accuracy of 94.03%
with 12 epochs.
Fig. 2.Action Model Summary
3. LBPH Model: A model based on the
LBWH algorithm is trained on the same
dataset.

4. Saving: Both models are saved in

separate directories.
Fig. 3.Face-model Summary
5. Loading the models: First, the
confidence value predicted by the LBPH-based
3. Creation of the ANN Model and model is obtained. If the value is greater than
training: The ANN model is created and 85, the person is unrecognized, and the second
trained based on the CSV file data. model is not used. Otherwise, if the value is
less than 85, the person is recognized, and their
face can be predicted.
Saving detected Frames:
This system makes use of a max heap data
structure, which holds timestamps, frames, and
the percentage change relative to the base
frame. Elements are arranged in the heap
according to the largest percentage difference
between the current frame and the base frame.
The motion detection MOG2 algorithm is used
by the camera to recognize the presence of a Fig. 5. Testing on sample
person when they enter its field of view. After
B. Face Recognition Model:
drawing a box around the moving object, the
system saves the frame along with the relative The CNN model achieves an accuracy of
94.04%. This model trained on 105 images and
percentage change from the base frame. The
45 images in 70% and 30%. As shown on Fig.
top 30 frames from the maximum heap will be
6.
saved when a player exits the field.

V. RESULT
A. Action Recognition Model: Fig. 6. Testing of face recognition model
The action recognition model achieved the
accuracy of 93.00% accuracy and testing
accuracy of 93.07% as shown in Fig.4. The
model trained with 200 epochs, 16 batch size. C. Message Sending API:
You can see the result as shown in Fig. 5.
If all models show positive results then the
system sends SMS to the owner mobile
number as shown on Fig. 7. This system uses
Twilio API for sending messages regarding
changes in camera configuration and changes
in base frame.

Fig. 4. Metrics
 [2] Nico Surantha and Wingky R.
Wicaksono, “An IoT based House Intruder
Detection and Alert System using Histogram of
Oriented Gradients”, Jakarta 11480, Indonesia,
DOI:10.3844/jcssp.2019.1108.1122

[3] Akula Surya Teja, Ginni Chandra

Mohini, Dannana Dhanunjay, Dr P M
Manohar, “Realtime Intrusion Detection
System Using OpenCV”, Vol. 10 No. 2S
(2023): Special Issue 2,
DOI:10.17762/sfs.v10i2S.1324.

[4] G.H.A.Chaminda, “CCTV Intruder

Fig. 7 Message sending API Testing Detection System", 2 August 2021,
2016/MIT/009.

[5] M. V. Dd Prasad, N. Sai Kiran, Ch

Sumanth, T. V. V. R. N. Sri Harsha, Sk Hasane
Ahammad, “Video surveillance-based security
VI. CONCLUSIONS system using OpenCV and Arduino uno”,
2021, Vol. 8, No. 5, 1522-1528 ref. 22 ref.,
The proposed system integrates face
ISSN:2148-9637
recognition, action recognition, and message
sending modules, operating through the IP [6] G. Mallikharjuna Raoa, Haseena
WEBCAM smartphone application.
Palleb, Pragna Dasaric, Shivani Jannaikode,
Leveraging artificial neural networks (ANN)
“Implementation of Low Cost IoT Based
for action recognition and convolutional neural
Intruder Detection System by Face
networks (CNN) for face recognition, the
Recognition using Machine Learning”, Vol.12
system achieves impressive accuracies of
No.13(2021), 353-362,
93.00% and 94.04%, respectively. By
DOI:10.17762/turcomat.v12i13.8295.
efficiently detecting intruders and sending
timely alerts via the Twilio API, the system
enhances security measures in diverse settings. [7] Pradeep S, Nikhil Raghav V, Bharath
With its robust performance and streamlined Kumar S P, “Animal Intrusion Detection Using
workflow, the system holds promise for ESP32 Cam and OpenCV”, Volume 8, Issue
effective deployment and widespread adoption. 10, October – 2023, ISSN:2456-2165

[8] Quamar Niyaz, Weiqing Sun, Ahmad Y

Javaid, and Mansoor Alam, “A Deep Learning
Approach for Network Intrusion Detection
VII. REFERENCES System”, Toledo, OH-43606, USA,
DOI:10.4108/eai.3-12-2015.2262516
[1] Nwalozie G. C et al, “Enhancing
Home Security Using SMS-based Intruder
[9] Nutan Farah Haq,Musharrat Rafni,
Detection System”, Vol.4 Issue.6, June- 2015,
Abdur Rahman Onik, Faisal Muhammad Shah,
pg.1177 – 1184, ISSN:2320–088X
Md. Avishek Khan Hridoy, “Application of
Machine Learning Approaches in Intrusion
Detection System: A Survey”, Vol. 4, No.3,
2015, DOI:10.14569/IJARAI.2015.040302.