Real-time Attendance Tracking System based on

Face Recognition
Duc M. Truong1,5, Quoc D. Truong2,5,, Duc Trinh3,5, Thai V. Pham4,5
1,2,3,4
University of Information Technology, Ho Chi Minh City, Vietnam
5
Vietnam National University, Ho Chi Minh City, Vietnam
Email: [email protected], 2 [email protected], 3 [email protected], 5 [email protected],
1

Abstract— E-learning is an online educational system work, we store and transfer data from the device to the
using technological and electronic devices through the offline data processing section using Kafka. We utilize
internet. Using e-learning educational systems, learners YOLOv8 as the face detection component, which Kafka
can refer to documents and communicate directly with sends to webface_r50 for face data processing. This data is
teachers to effectuate the acquisition of knowledge. In supplied to the attendance section via Spark Streaming and
this paper, the method for creating an intelligent is kept in MongoDB. This section of the study will highlight
querying system for e-learning is proposed. The the relevant research for this project and is organized into
knowledge base of this system is organized based on the three sections: "Other Attendance Systems," "Transmission,
Processing of Face Data," and "Face Recognition." These
integrating of ontology and the structure of database.
are a few studies that touch on our investigation.
Some searching issues for knowledge content, such as
studying and resolving knowledge searches based on A. Facial Recognition
classification of knowledge, are studied and solved. This The following procedures are used to tackle face
method is applied to build an intelligent querying system recognition problems: face detection, facial feature
on the course of Database Foundation in Information extraction, and facial feature categorization. Face detection
Technology (IT) curriculum at university. This system aims seeks to locate and draw a bounding box around the target's
to support students to review lessons and understand face, from which facial characteristics are extracted to
more the knowledge of courses by their self-learning. produce face vectors, which are then categorized in the
The experimental results show that it would be expected classification stage. Numerous research have been conducted
to contribute in supporting online-based learning on face detection since it is one of the most crucial aspects of
facilities for students. face processing.
In the facial feature extraction section, we use the
Keywords— Intelligent searching, Ontology, webface_r50 model from the InsightFace: 2D and 3D Face
Knowledge engineering, E-learning, Database Systems. Analysis project [1]. The model promises to solve the
I. INTRODUCTION problem of recognizing small faces with similar features.
Current attendance is an important and repetitive task in YOLOv8 is a real-time object detection system that uses
human resource management. From a repetitive task, we a single neural network to detect objects in images and
need a system that can automate that task. However, videos. It is a significant improvement over previous
traditional attendance methods such as fingerprint, card, etc. versions of YOLO, with faster detection speed and improved
accuracy.
have inherent problems such as: Delay due to time to find
devices or large number of employees making devices not Webface_r50 is a deep learning model for face analysis.
enough to serve, inaccurate due to the influence of external It can be used to extract facial features, such as the eyes,
factors such as environment, device quality, fingerprint nose, and mouth. These features can then be used to identify
quality or may be cheated by clocking in with cards. In a person or to compare two faces.
addition, these methods are not flexible when they cannot be This time, for our project, we utilize YOLO too, but
applied to mobile work. we're using face detection version eight. The new feature that
Face recognition applications are a modern attendance is seen to be exceptional is that the v8 version is much faster
method that solves the remaining problems of traditional than the v7 version, allowing it to process picture data in real
attendance, including the following advantages: quick and time and provide accurate and timely object detection. In
accurate, flexible to apply to different working environments order to analyze picture data for face analysis, we use
without being limited by devices as well as no need for other webface_r50. This allows us to extract facial characteristics
attendance devices such as cards. for comparison and user identity authentication.
From here, we propose a proposal for an application that YOLOv8 and webface_r50 together provide a potent face
applies face analysis technology to attendance work with the recognition tool. Numerous applications, including security,
aim of maximizing the working time of human resources but monitoring, and customer service, are possible for it.
still maintaining the accuracy of the work. In addition, the
application also serves the purpose of applying the latest B. Transferring and Data Processing
technologies in the world to improve the quality and
efficiency of daily tasks, which in this case is attendance. In this project, we use Kafka to push data which Spark
Streaming processes and stores into MongoDB and Redis.
II. RELATED WORK Kafka and Spark Streaming are not new technologies, so
Numerous research on face recognition has been there have been many studies on their use for face
conducted in relation to attendance or similar issues. In this recognition.
 [2] proposes a video analysis framework that collects records. It also helps to minimize the possibility of
video from multiple cameras and analyzes it using Apache impersonation and false attendance. This system has high
Kafka and Apache Spark Streaming. It uses Kafka to accuracy and does not require special hardware to deploy.
transmit video data from clients to cloud servers, uses Spark In our project, we combine the technologies mentioned
Streaming to process the streaming data, and calls Chainer above to improve the speed and accuracy of face recognition.
to perform image recognition. In addition, the deployment of the face recognition part on
[3] proposes a method for face recognition on live video the web makes it easier to access and install the system on
streams of pedestrians using Apache Spark Streaming, the systems of companies than before.
Kafka, and OpenCV on a distributed platform. Processing
and analyzing video from multiple sources are a time- III. METHOLOGY
consuming process when using the Cloud or any high- A. System Architecture
performance single computer. Multi-stream applications that
require low-latency data exchange between multiple In this realm of real-time attendance tracking using facial
concurrent processes are also inefficient with MapReduce. recognition, the pivotal factors of scalability and speed take
A real-time event analysis solution is needed in a Hadoop precedence. The integration of Apache Kafka and Apache
distributed environment. The results show that this method Spark Streaming stands as instrumental solution to address
has high accuracy, low latency, and good scalability. these imperatives effectively []. This integrated system not
only meets the challenges of scalability and speed but also
C. Others attendance system
establishes a robust foundation for enhancing attendance
Face recognition-based attendance or timekeeping management in academic and organizational contexts []. The
systems are no longer new, face recognition devices have
also been commercially available and there are also studies system can be divided into the following modules:
to improve the speed and accuracy of this technology. 1) In this realm of real-time attendance tracking using
[4] describes a method for an Automatic Attendance facial recognition, the pivotal factors of scalability and speed
System using Face Recognition Techniques This system take precedence. The integration of Apache Kafka and
records the times of student arrival and exit in a classroom Apache Spark Streaming stands as instrumental solution to
setting then extracts and compares facial characteristics address these imperatives effectively []. This integrated
using the Principal Component Analysis (PCA) technique. system not only meets the challenges of scalability and speed
The eigenface technique, first presented by Kirby and
but also establishes a robust foundation for enhancing
Sirovich in 1988, is employed by the author for facial
identification. Using this technique, faces composed of attendance management in academic and organizational
eigenvectors, or eigenfaces, are computed by analyzing contexts []. The system can be divided into the following
facial photographs. The presence and identity of a face are modules:Front-end (mobile and web application): All users
determined by comparing eigenfaces. The author suggests interact with the system through the front-end. The following
using facial recognition technology in educational and information will be displayed for an employee: a table that
organizational settings to regulate attendance by tracking records every attendance received so far with filtering
people's admission and departure timings. When compared functions, each record shows their employee id, checked
to conventional attendance systems, the early experiment's
findings demonstrate enhanced performance in attendance activity, timestamp, recognition confidence score and
estimation. Face identification techniques from pictures or tracked processed image with bounding box.
video frames are the focus of current development. The 3) Back-end: A microservices-based back-end
author plans to employ user, system, and administrator
architecture is proposed as an effective paradigm for
interaction in future work to increase facial recognition
efficiency. handling complex streaming flows for serving the task of
real-time facial recognition and attendance tracking. This
[5] discusses a model that uses Eigenface values, architecture integrates Kafka as a dependable and
Principal Component Analysis (PCA), and Convolutional
expandable event streaming system, enabling the
Neural Networks (CNN) to construct an automated
attendance system for kids in a classroom. The procedures uninterrupted passage of video frames over its dispersed
required to construct an intelligent attendance system are channels. After being ingested, these frames pass through
covered in this article, including registration, face detection, Spark Streaming to a Facial Recognition module, which
face recognition, classroom camera validation, and makes use of parallel processing power to effectively train
attendance marking. Facial recognition is an efficient and infer detection and embedding models for facial
methodology for handling student data and attendance in this recognition. The outcomes of this recognition procedure are
system. It also aids in reducing the likelihood of forgeries then sent to a specific module in charge of tracking
and fraudulent attendance. This system may be deployed
attendance. Furthermore, the processed image frames are
without specific hardware and has a high accuracy rate.
conveyed to a separate Kafka consumer that is connected to
[5] talks about a model to implement an automatic a WebSocket communication layer. This consumer
attendance system for students in a classroom by using orchestrates the interaction with the frontend, thereby
Eigenface values, Principal Component Analysis (PCA), and
enabling real-time display and visualization of facial
Convolutional Neural Networks (CNN). This paper
describes the necessary steps to build an intelligent recognition outcomes.
attendance system, including registration, face detection, 4) Facial Recognition Module: This is where the
face recognition, confirmation by the classroom camera, and
computationally demanding face recognition operations will
attendance marking. This system uses face recognition as an
effective model for managing attendance and student be carried out. Every active camera will have a separate
computing thread that connects to the back-end server to chosen embedding model, having been trained on a dataset
calculate attendance and receives real-time video feed encompassing 600,000 records from the WebFace dataset.
directly from the relevant camera. The following information
is received by each thread of the facial recognition server 2) CelebA
from the back-end: employ identifier, sending timestamp,
The CelebA dataset [] is an extensive set of face images
and the corresponding video frame as binary base64 encoded
used as a standard for a number of computer vision tasks,
string. especially those related to facial attribute recognition. Over
Database: MongoDB serves as a persistent data store, 200,000 celebrity photos, each labeled with 40 attributes like
allowing the system to seamlessly check and insert gender, hair color, and facial expression, are included in this
dataset, which was released by Zhanpeng Zhang and Ping
attendance records. It stores information regarding
Luo []. The photographs exhibit diversity, encompassing an
attendances (such as employee id/email, images for extensive array of poses, backgrounds, and lighting
recognition, checked activity with the corresponding conditions. CelebA is now a widely used tool for machine
checked timestamp, device ID), and devices (device ID with learning model evaluation and training, particularly for tasks
its corresponding Kafka streaming topic). Figure. 2 shows a involving attribute prediction and facial analysis [].
simplified schema of the database. Researchers and industry professionals frequently use this
dataset to create and evaluate algorithms meant to
comprehend facial features and attributes in practical, large-
scale settings.
B. Facial Recognition

Fig. 2. The structure of Face Recognition Module

Fig. 1. The structure of Database

B. Integrating ontology Rela-Ops model and the structure 1) Detection:

of database The face detection phase is an integral component, which
makes use of the sophisticated capabilities of the YOLOv8
IV. DATASET AND MODELS face model []. The model has been smoothly integrated into
A. Dataset the ONNX format [] to improve computational performance.
This adaptation is designed to significantly improve
1) Webface processing speed, ensuring swift and accurate detection of
The WebFace dataset, a cornerstone in the field of facial facial features within the given images or video frames.
recognition, has significantly influenced the training of
2) Recognition:
advanced embedding models, including iResNet.
Comprising an extensive repository of facial images In the recognition phase of our facial recognition system,
collected from diverse sources on the internet, WebFace we employ the iResNet embedding model. This model is
captures a broad spectrum of facial expressions, poses, and pretrained on an extensive dataset containing 600,000
records from the WebFace dataset. To enhance
lighting scenarios. The dataset is meticulously curated to
computational efficiency, we leverage TensorRT for the
encompass a wide array of identities, fostering the porting process, contributing to accelerated inference speed.
development of robust and generalized facial embedding During the initial face registration process for new
models. In our research, we leverage the iResNet embedding employees, their facial embedding vectors are stored in a
model, renowned for its exceptional performance, which has Redis stack. Subsequently, during the deployment of the
undergone rigorous training on the extensive and diverse attendance tracking system, we conduct facial recognition by
WebFace dataset. This exhaustive training process endows comparing the provided facial embedding vector with all
the iResNet model with the capability to discern intricate stored vectors in Redis, utilizing cosine similarity [] as the
metric. The resulting group of vectors, organized by
facial features and nuances, allowing it to generate highly
employee ID, undergoes further refinement through the
discriminative and representative facial embeddings for RANSAC [] reranking layer. This reranking process can be
subsequent recognition tasks. Our study highlights the employed to robustly estimate the similarity between facial
synergy achieved by integrating iResNet with the WebFace features, culminating in the identification of the top-ranking
dataset, showcasing a potent combination of state-of-the-art label. The label with the highest similarity score is selected
model architecture and a comprehensive, well-curated as the predicted label for the given facial embedding vector.
dataset. Specifically, in our research, iResNet serves as the
 A∗B embeddings, prioritizing vectors with higher inlier counts for
similarity=cos (θ )= subsequent analysis.
| A|∗¿ B∨¿ ¿
C. Evaluation
Given a registered candidate facial embedding vectors
domain K = (candidate_id, embedding), an input video CelebA dataset are used to evaluate the proposed facial
frame f. The following algorithm retrieves the disired labels recognition model due to its accurate reflection on real-world
scenarios.
that is matched with faces appearing in the video frame f.
Input: The raw video frame that is streamed from a V. EXPERIMENTAL AND TESING
camera device as f .
Output: The predicted labels with the processed frame A. Architecture of the proposed system
from K which is suitable to frame f. The proposed system uses face recognition to
Algorithm 4.2 automatically handle attendance of employees. Taking the
Faces := {} // Set of faces detected from f. actual face recognition algorithm as a black box, the
Faces-Emb := Embedding(Faces) // Embedding vector temporal monitoring infrastructure is grounded upon two
of faces extracted from f principal procedural facets. Firstly, newly appointed
Retrival-Emb := {} // Set of vectors with scores, that is employees are required to enroll their facial biometrics
similar to the embedding vector of a corresponding face concomitantly with the assignment of their employee
Faces[i]. code, serving as a distinctive identifier. This process
Group-Emb := {} // Set of Emb group by candidate ID constitutes the registration phase within the attendance
with average similarity score. tracking system. Secondly, the camera devices are always
Rerank-Emb := {} // Set of Group-Emb after applied to on, recording and streaming frames throughout the
the RANSAC reranking layer. entirety of each day, making it easier to conduct temporal
Result : = {} // The predicted label of each face data collection and thorough surveillance. These two
appearing in frame f. processes collectively establish a robust attendance
Step 1: tracking framework, incorporating continuous video
 Using the YOLOv8 face model mentioned in 4.2 streaming and facial recognition functionalities.
with threshold t to extract faces from f, then,
append the extracted faces into Faces set.
Faces[f] := Detect(f, t)
 Update Faces
Step 2:
 Using the pretrained IResNet model to embed the
faces in Faces set into the semantic representation as
vectors.
for face in Faces do
Faces-Embface := Embedding(face)
End for Fig. 3. The structure of Attendance tracking system
 Retrieve the embedding vectors with corresponding
labels that is similar to the input vector and assign B. Application Interface
them to Retrieval-Emb set. We used the JavaScript ReactJs framework to create a
 Update Retrieval-Emb web application. The system's web application interface is
displayed in Figure 5. We designed the administrator and the
Step 3: Group the content from Retrieval-Emb to employee as the two actors in this system. Employee’s
Group-Emb by candidate ID with mean similarity attendance tracking data is recognized and send a
score. notification on web to that employee. Administrators get
for face in Faces do real-time access to the attendance tracking sheet, where they
Faces-Embface := Embedding(face) may view data such as the employee's photo taken while
End for Step 4: Return Results. tracking, the time and date, and the fundamental

3) Reranking:
Our proposed reranking layer leverages RANSAC
algorithm [] to robustly assess the similarity of an input
image with a curated list of satisfying embedding vectors
corresponding images. This involves determining
correspondences based on a weighted sum of squared
differences in a small neighborhood around interest points. Fig. 4. The interface of Attendance tracking page
Subsequently, the algoritm is applied to estimate the
geometric transformation, adeptly discerning and Administrators enroll each employee through the
categorizing outliers and inliers. Following this, the facial Enrollment Tab. To make sure this is the correct employee,
vectors that satisfying a given threshold are sorted in the administrator will input the employee data. Then they
descending order based on their inlier count. This intricate will scan the face of employee to change the image to data.
process ensures a refined and robust evaluation of the facial Subsequently, they will scan the employee's face to convert
the picture to data. Kafka receives such data, and Spark
Structured Streaming's integrated attendance tracking fine-tune the processing pipeline and reduce latency, making
models will carry out the recognition procedure. Mongoose the system even more responsive in real-time scenarios.
DB will store the outcome. The Enrollment Interface is Integrating additional security features, such as anti-spoofing
displayed in Figure 6. measures, could also be considered to fortify the system
against potential vulnerabilities. Moreover, scalability
considerations could be addressed to accommodate larger
datasets and handle a higher volume of concurrent users.
Overall, the presented attendance tracking system serves
as a foundation for future developments in facial recognition
technology, providing a platform for ongoing innovation and
improvement in the realm of real-time, intelligent attendance
monitoring.
ACKNOWLEDGMENT
This research is supported by the VNUHCM-University
of Information Technology's Scientific Research Support
Fig. 5. The interface of Enrollment page
Fund.
REFERENCES
To assist administrators with viewing live cameras in
real time, we have created a Camera Viewer page. The
system will select the data from a camera once [1] "InsightFace: 2D and 3D Face Analysis Project," 2024.
[Online]. Available:
administrators have chosen the name of the camera list. https://github.com/deepinsight/insightface. [Accessed
Kafka will transfer the camera data to Spark Structured 22 1 2024].
Streaming. Following that, the data was split into two flows: [2][[1]] Ichinose, Ayae and Takefusa, Atsuko and Nakada,
the Normal Camera View and the Tracking Camera View. If Hidemoto and Oguchi, Masato, "A study of a video
analysis framework using Kafka and spark streaming,"
a camera identifies someone as an employee in that system, 2017 IEEE International Conference on Big Data (Big
the attendance tracking models will receive the data from Data), pp. 2396-2401, 2017.
the second source, recognize the person, and save the [3][[2]] Vikas Tripathi, Durgaprasad Gangodkar, Devesh
information into a Mongoose database. Next, the label, Pratap Singh, Dibyahash Bordoloi, "Using Apache
which is the employee ID, and the recognition process's Spark Streaming and Kafka to Perform Face,"
Webology, vol. 18, no. 5, pp. 3416-3422, 2021.
confidence are the data that the viewer has output. [4][[3]] Kar, N., Debbarma, M. K., Saha, A., & Pal, D. R.,
Additionally, two sources' data is upstream from Kafka. "Study of implementing automated attendance system
using face recognition technique," International
Journal of computer and communication engineering.,
vol. 1, no. 2, pp. 100-103, 2012.
[5][[4]] Sawhney, Shreyak and Kacker, Karan and Jain,
Samyak and Singh, Shailendra Narayan and Garg,
Rakesh, "Real-Time Smart Attendance System using
Face Recognition Techniques," 2019 9th International
Conference on Cloud Computing, Data Science &
Engineering (Confluence), pp. 522-525, 2019.
[6][[5]] Sari, Diana and Mirza, A. Haidar, "THE
DETECTION OF FACE RECOGNITION AS
EMPLOYEE ATTENDANCE PRESENCE USING
THE YOLO ALGORITHM (YOU ONLY LOOK
ONCE)," Jurnal Darma Agung, vol. 30, p. 41, 2022.
[7][[6]] A. R. Lahitani, A. E. Permanasari and N. A.
Fig. 6. The interface of Camera Viewer page Setiawan, "Cosine similarity to determine similarity
measure: Study case in online essay assessment," 2016
VI. CONCLUSION AND FUTURE WORK 4th International Conference on Cyber and IT Service
Management, Bandung, Indonesia, 2016, pp. 1-6
In conclusion, the presented attendance tracking system
[8][[7]] Martin A. Fischler and Robert C. Bolles. 1981.
harnessing facial recognition technology, built upon the Random sample consensus: a paradigm for model
integration of Kafka and Spark Streaming, demonstrates a fitting with applications to image analysis and
powerful and efficient solution for real-time processing of automated cartography. Commun. ACM 24, 6 (June
video frames. Leveraging the YOLOv8 face detection and 1981), 381–395.
embedding similarity comparison for recognition, the system
successfully employs cosine similarity scores to group labels
and enhance prediction accuracy through RANSAC-based
reranking. The robustness of the system is reflected in its
ability to handle continuous streams of video frames,
enabling swift and accurate attendance tracking.
In the future, we would explore enhancements in both the
accuracy and efficiency of the facial recognition module.
Further research may focus on refining the detection and
recognition models, potentially incorporating advanced
techniques or newer models to improve overall performance.
Additionally, optimization strategies could be explored to