B.

Tech IoT Programming

Project Report
On

SMART LOCK MODEL BASED ON FACE

RECOGNITION AND LIVENESS DETECTION
by
Group Leader:
Ayush Bhagat (123102045) [email protected] 8287568041
Other Team Members:
Priyansh Raj (123102038) [email protected] 9508116425
Vivek kumar(123102034) [email protected] 8750660406
NitishKumar(123102041)[email protected] 9044867488
Himanshu singh(123102012)[email protected] 8969665649

Branch: CS A(03)
Subject: IoT Programming (CSPC 209)

1
 Abstract
This project implements a smart lock system based on facial detection, recognition,

and liveness detection models to provide a modern security solution. The system

integrates two models: one for face detection and recognition, and another for

liveness detection to prevent spoofing. The system captures live input from a

camera, detects the face, and compares it to pre-registered images of authorized

users. If a match is found, the system proceeds to verify the liveness of the detected

face, ensuring it is not a photo or video used for spoofing attempts. If an

unauthorized or fake face is detected, a snapshot is taken, and the user is alerted via

a Telegram bot, with the option to remotely unlock the door through Telegram.

For facial recognition, the face_recognition library is used, which is based on dlib

library and employs a CNN model for accurate recognition. The system also improves

recognition accuracy by using multiple images of authorized users. For anti-spoofing,

a custom-trained model based on YOLOv8X-cls is used to detect whether the face is

real or fake. OpenCV is employed for handling live video processing, NumPy for

numerical operations, and TeleBot for Telegram bot communication. The deep

learning models are supported by TensorFlow and Keras, providing the necessary

framework for performing both face recognition and liveness detection.

Keywords: Live face detection and recognition, liveness detection, face_recognition

library, dlib, YOLOv8X-cls, TeleBot, TensorFlow, Keras.

2
1. Introduction
In the modern world, where facial recognition systems are used in critical applications
for user access control and authentication, there is an increasing demand for an
enhanced security system based on facial recognition that is reliable. Traditional
security methods like PINs or passwords are vulnerable to hacking or theft, and even
conventional facial recognition systems can be fooled by spoofing attempts, such as
using photographs or videos of authorized users. Studies suggest that top-tier facial
recognition algorithms can have error rates of around 5% when subjected to spoofing
attacks, underscoring the vulnerabilities these systems face, especially when
unauthorized individuals attempt to use fake images or videos for access.
This project addresses these challenges by implementing a smart lock system that
integrates facial detection, recognition, and liveness detection to provide a modern
security solution that is both reliable and convenient. The system utilizes two models:
a face_recognition model based on dlib for face detection and recognition, and a
YOLOv8X-cls-based liveness detection model. It captures input from a live camera
feed, detects the face, and matches it against a pre-registered database of authorized
users. If the face matches an authorized user, the system performs liveness detection
to ensure the face is real, preventing spoofing attempts. If a fake or unauthorized user
is detected, the system captures an image and sends it to the user via Telegram,
allowing the user to decide whether to lock or unlock the door remotely via the
Telegram bot.
The system incorporates several libraries and frameworks: OpenCV for live camera
feed and image processing, NumPy for handling numeric operations, and
face_recognition for implementing face detection and recognition using the dlib
CNN model. Multiple images of each user are used to improve recognition accuracy.
TeleBot facilitates communication with users for sending alerts, images, and enabling
remote door unlock commands. The deep learning models are supported by
TensorFlow and Keras, powering both facial recognition and liveness detection
models.
Unique Functionality of the Project:
1. Combined face detection and liveness detection using two models.
2. Real-time alerts based on live camera feeds.
3. Remote access control through Telegram.
4. Multiple image training for improved recognition accuracy.
5. Use of YOLOv8X-cls for advanced liveness detection.

3
 6. Robust anti-spoofing mechanism based on deep learning.

2. Project Related Work

4
 A. Author Link of Year Data Features Model/ Results Drawback
published Sensors/ in their
work Used Project
(Research
Paper/
Patent)
Muhammad Face 2020 Hierarchica Raspberry Resilience Limited
Waseem Recognitio l Network Pi to Dataset
n for Smart (HN) Random
Sundar Ali Door Lock Framework Pre- Faces No
Khowaja System trained Liveness
using Architectu Detection
Farhan Bashir Hierarchica res &
l Network | FaceNet
IEEE
Conference
Publication
| IEEE
Xplore
Zhiguo Zhu Application 2020 Hybrid OpenCV Cost- Lack of
of attitude Approach - effectiven Specificity
Yao Cheng tracking Retro and Efficient ess
algorithm Modern Attitude Absence of
for face Lock Tracking Hybrid Experiment
recognition System: Algorith Design al Data
based on m
OpenCV in (EATA)
the
intelligent
door lock -
ScienceDir
ect
Phawinee, Face 2019 Control Raspberry Real- Vulnerabilit
Suphawimon recognition Sensing Pi Time y to
in an Module: Image Spoofing
Cai, JinFangb intelligent Camera Streamin
door lock Mobile Sensor g: Complexity
Guo, ZheYu with Applicatio of Setup
5
ResNet n CNN and
model Maintenanc
based on e
Drawbacks in these existing projects
1. Vulnerability to Spoofing: Without robust liveness detection, the system can be easily
fooled by images or videos of authorized users. This oversight leaves it open to spoofing
attacks, potentially allowing unauthorized access.
2. Complex Setup and Maintenance: Implementing the system requires technical expertise,
including software installation and hardware configuration. This complexity may pose
challenges for users without a technical background.
3. Dataset Limitations: The system's accuracy depends on the quality and diversity of the
training dataset. A limited or biased dataset can lead to lower recognition rates for users
whose features differ from those in the dataset.

3. Project
This project is based on creating a smart lock system to establish a reliable
security system that can be used in different fields by integrating face detection,
face recognition, and liveness detection. The system uses deep learning models
and combines them with real-time monitoring using the camera. The project
overcomes the problems faced with the current security system, such as
spoofing attacks.

Key Features:

1. Face Detection and Recognition: The system captures live video input
using a camera and processes the frames to detect faces. Once a face is
detected, the system matches it against authorized user images, which
have been pre-stored as embeddings. This step utilizes the
face_recognition library, which is based on dlib and employs a
convolutional neural network (CNN) model for accurate face detection
and recognition.

2. Liveness Detection (Anti-Spoofing): To prevent spoofing attacks, the

system incorporates a YOLOv8 model trained on a custom dataset
specifically designed for detecting fake representations. This ensures the
system can distinguish real faces from those that are static, like photos or
videos, by analyzing various visual features.

6
 3. Alert and Remote Control via Telegram: When an unauthorized or
spoofed face is detected, the system takes a snapshot and sends it to the
user via Telegram using the TeleBot library. The user can then remotely
unlock or keep the door locked using the bot interface.

Technical Components:

1. OpenCV: Used for real-time video capture and image processing. It

detects faces in each frame from the live feed, enabling face recognition
and liveness detection.

2. YOLOv8: A state-of-the-art model employed for object detection and

classification, particularly useful in identifying spoofing attempts by
analyzing visual characteristics. This model is trained on a custom dataset
to enhance its accuracy in recognizing unauthorized access.

3. face_recognition Library: This library simplifies the implementation of

deep learning models for facial recognition. The system uses its CNN-
based architecture to accurately identify and verify faces against the
stored images.

4. TeleBot (Telegram API): The TeleBot library facilitates interaction with

the user. When the system detects a suspicious face, an alert, along with
the captured image, is sent to the user. The user can then control the lock
remotely via Telegram commands.

3.1 Conceptual Design Diagram

Workflow / Architecture:
1. Camera Activation: The system begins by capturing live video input
through a connected camera, providing a continuous stream of footage for
analysis.
2. Face Detection: Utilizing OpenCV, the system detects and locates faces
within the video feed, identifying regions of interest where faces are
present.

7
3. Face Recognition: The detected face is extracted, and the
face_recognition library compares it with stored images of authorized
users, using multiple images per user to enhance recognition accuracy.
4. Authorization Check:
o If the face matches an authorized user:
 Liveness Detection: The system performs liveness detection
using a YOLOv8 model trained on a custom dataset to verify
the face is live, analyzing dynamic traits such as blinking or
head movement.
 Unlock Door: If the face is confirmed as live, the door is
unlocked for access.
o If the face is not recognized as authorized or is deemed a spoof:
 Capture Snapshot: The system takes a snapshot of the
detected face for documentation.
 Notification: A notification, along with the snapshot, is sent
to the user via a Telegram bot.
 User Decision: The user can then decide to lock or unlock
the door remotely through the Telegram interface.

8
 89=

3.2 Working Example (Add Real time Photographs of the working project)

9
4. Experimental Setup

10
4.1 Libraries and Frameworks:
 OpenCV: Used for real-time computer vision tasks, including face
detection and image processing.
 face_recognition: A high-level library for face recognition based on dlib,
utilizing a CNN model for accurate face detection and recognition.
 YOLOv8: Employed for the liveness detection model, trained on a
custom dataset to distinguish live faces from spoofed images.
 NumPy: Used for numerical operations and efficiently handling arrays,
especially in image processing tasks.
 TensorFlow and Keras: Utilized for building and training deep learning
models, including the liveness detection model based on YOLO
architecture.
 TeleBot: A library for integrating Telegram messaging functionality,
enabling notifications and remote control commands.
Models:
1. Face Recognition Model: The face_recognition library employs a CNN-
based architecture for efficient face detection and recognition, ensuring
high accuracy through the use of multiple images per user.
2. Liveness Detection Model: A YOLOv8 model trained on a custom
dataset to analyze dynamic features and confirm the authenticity of the
detected face, preventing spoofing attempts.
Integration:
 The system integrates the above components, allowing for real-time video
capture, processing, and interaction through a Telegram bot. The
workflow facilitates face detection, recognition, liveness verification, and
user notifications, creating a comprehensive smart lock solution.

4.2 Experimental Results:

11
Fig a. Confusion Matrix for anti spoofing model

Fig b . Test Result for trained model for spoof detection

12
5. Comparison with the Existing Work

6. Conclusion and Future Work

The smart lock system effectively utilizes advanced technologies such as face
detection, recognition, and liveness verification to enhance security for user
authentication. By integrating the face_recognition library, which is based on
dlib and utilizes a CNN model, the system achieves high accuracy in facial
recognition while maintaining efficiency. The incorporation of real-time
monitoring and remote access through a Telegram bot significantly enhances
user convenience and security.

Future Work:

Several improvements can be considered for further enhancing the system:

1. Advanced Anti-Spoofing: Implementing more sophisticated measures,

such as 3D recognition and training on more advanced architectures like
ResNet50, to improve the detection of spoofing attempts.

13
 2. Database Integration: Adding a database for managing user data and
access logs would facilitate better security management and adaptability
over time.

3. User Interface Enhancements: Developing the Telegram interface to

include features like live video streaming and customizable notifications
for a better user experience.

4. Model Optimization: Focusing on optimizing machine learning models

for enhanced speed and accuracy in face recognition and liveness
detection, ensuring compatibility with various hardware systems.

5. Smart Home Integration: Expanding the system’s capabilities to work

seamlessly with other smart home devices, providing a more
comprehensive security solution.

References:

1.Romit Ganjoo, Anjali Purohit(2021)

https://ieeexplore.ieee.org/abstract/document/9358795/authors#aut
hors
2. Deepak B. Desai, S.N. Kavitha (2019)
https://ieeexplore.ieee.org/abstract/document/9065873

3. Satyam Mishra, Nguyen Thi Bich Thuy & Cong-Doan Truong(2023)

https://link.springer.com/chapter/10.1007/978-3-031-45140-9_7

4. Keyurkumar Patel; Hu Han; Anil K. Jain(2016)

https://ieeexplore.ieee.org/abstract/document/7487030

5.Polash Kumar Das; Bin Hu; Chang Liu; Kaixin Cui; Prabhat
Ranjan; Gang Xiong(2019)
https://ieeexplore.ieee.org/abstract/document/8955089

6. Sachin Sharma; Mayank Sharma; Gopal Sharma; Akash Bhasney(2024)

https://ieeexplore.ieee.org/abstract/document/10489006

14
1.

15