Received 13 October 2022, accepted 15 November 2022, date of publication 18 November 2022,

date of current version 1 December 2022.

Digital Object Identifier 10.1109/ACCESS.2022.3223374

Design for Visitor Authentication Based on Face

Recognition Technology Using CCTV
HYUNG-JIN MUN 1 AND MIN-HYE LEE 2
1 Department of Information and Communication Engineering, Sungkyul University, Anyang, Gyeonggi 14097, South Korea
2 Center for General Education, Wonkwang University, Iksan, Jeollabuk 54538, South Korea

Corresponding author: Min-Hye Lee ([email protected])

This work was supported by Wonkwang University in 2020.

ABSTRACT Recently, image recognition technology using deep learning has improved significantly, and
security systems and home services that use biometric information such as fingerprints, iris scans, and face
recognition are attracting attention. In particular, user authentication methods that utilize face recognition
have been studied at length. This study presents a visitor authentication technology that uses CCTV with a
Jetson Nano and webcam. In the preprocessing phase for face recognition, face data with 7 features that can
be identified as a person are collected using CCTV. The collected dataset goes through the annotation process
to classify the data, and facial features are detected using deep learning. If there are four or more detected
features, the image data is determined to be a person, and the visitor’s face is matched with stored user data
in detail using 81 feature vectors. Additionally, the security of the access control system was enhanced by
implementing logging functions such as recording the face of the visitor, the number of visitors, and the
time of the visit. This paper implements a visitor authentication system using a Jetson Nano and evaluates
performance by analyzing the accuracy and detection speed of the system. The tiny-YOLOv3 in the Jetson
Nano was effective in real-time verification for the real-time face authentication system with an average
detection speed of 6.5 FPS and 86.3% accuracy. Through this study, we designed a system based on deep
learning technology that recognizes and authenticates the face of a user during the visitor access process and
controls user access.

INDEX TERMS Face recognition, deep learning, visitor authentication, security, YOLO, Jetson nano.

I. INTRODUCTION user. So, it is often used for the purpose of crime prevention,
As the need for security spreads around the world, many facility safety, and fire detection. Currently, there has been
smart home services using CCTV (Closed-circuit Televi- researches about Fire Detection or weapon detection using
sion) are being installed and operated [1]. According to a CCTV applying deep learning technique such as CNN [2],
report by the global statistics site Statista (Fig. 1), the global [3], [4].
smart home security market is expected to increase from In particular, CCTV in smart home services has the
$9,903 million in 2017 to $35,619 million in 2024, with its additional functions of automatically applying user access
compound annual growth rate (CAGR) increasing by 20.1%. controls, automatically searching the surroundings at a deter-
CCTV can record the situation outside a building or in mined time, tracking a target, and increasing image resolu-
the absence of people and transmit this information to the tion. As artificial intelligence (AI) has developed, systems
that have introduced intelligent object recognition technology
The associate editor coordinating the review of this manuscript and have been continuously studied. However, most commercial
approving it for publication was Michele Nappi . CCTV systems are focused on simple functions, so it is

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
124604 VOLUME 10, 2022
H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

family’s return home on the system manager’s smart-

phone, check access information in real time, and ana-
lyze access patterns or provide additional services in
the case of long-term absence.
In this study, we utilize a Jetson Nano and YOLO, which
is an object detection deep learning model that can recognize
visitors in real time, in order to develop a system that can
check for intruders, authorized visitors, and peeking over a
person’s shoulder.
There are various types of microprocessors (microcon-
FIGURE 1. Security of smart home market prospect.
trollers) that connect various sensors and are portable such
as Arduino, Raspberry Pi, Panda, Jetson Nano, and so on.
In order to determine the suitability of hardware for building
difficult to confirm whether a targeted object is a person or and utilizing the proposed model, two types of micropro-
not, and there is a limit to responding to the situation even cessors that are suitable for deep learning were analyzed in
when a person is recognized. the experiment. First, the portable, inexpensive, and com-
Therefore, we propose a system that can identify visitors monly used Raspberry Pi was used, and secondly, the recently
by applying AI technology to CCTV and provide services released Jetson Nano with a built-in GPU was used.
according to the situation. The proposed system uses You We implemented a stand-alone CCTV mechanism by using
Only Look Once (YOLO) [5] and OpenCV with the image Webcam and Jetson Nano. This proposed system is used
input from the CCTV camera in front of the doorto determine to detect visitors’ faces and to identify whether faces are
whether or not a human face is recognized, whether or not a registered member or not.
person is an intruder, and whether or not that person is trying The following are the contributions of this paper.
to steal the door code by peeking over the registered user’s
shoulder. 1. The system was developed at a low cost since it is stand
This system can provide an alarm notification to users by alone and not server -based.
recognizing situations when there is an outsider beside or 2. Experimenting Raspberry Pi and Jetson Nano in same
behind the registered person (such as a family member) who environment, showed the different accuracy.
may be attempting to steal the door code. Recently, there have 3. The purpose of visitor authentication was achieved by
been a number of incidents of outsiders entering an apartment face detection and face recognition.
building along with a woman who is alone at night. In such 4. The system was proven safe from various attacks, pro-
a case, photographs and logs about the access situation are vided conveniency, and allowed visitor authentication.
transmitted to the building manager to prevent intruder access The structure of the paper is as follows. Chapter 2 intro-
and to be used as evidence in the case of an incident response. duces related research including image processing technol-
The following are the requirements that the proposal sys- ogy and user authentication technology. Chapter 3 designs a
tem must satisfy: visitor face authentication system using CCTV and presents
1. It needs to identify an intruder or companion : If the the components necessary for deep learning, the type of learn-
companion is a family member, the alarm notification ing required for the system. Chapter 4 introduces and presents
is displayed as a family member. If the companion is the protocol of the proposed system. And in Chapter 5,
not a family member, a long password is required to the user face authentication technique and the proposed
prevent peeking. system are evaluated and analyzed, and conclusions are
2. It needs to check if someone is peeking at the pass- drawn.
word : The system detects the companion’s eyes and
determines if he or she is peeking over the registered II. RELATED WORKS
user’s shoulder. If there is a passer by, the alarm goes A. USER AUTHENTICATION
off and asks for a long password. When entering a short Identification and authentication are required to ensure that
password, a notification message is printed informing users have access and permissions to certain information.
the user that, ‘‘the password is too short.’’ The user authentication approach is accomplished in four
3. It needs to recognize the Registered person’s face : ways [6]. Authentication is divided into knowledge-based
Family members register photos in advance. When a authentication (what you know, i.e. a password), ownership-
visitor is authenticated based on OpenCV, the door is based authentication (what you have, i.e. a credit card
opened through multi-factor authentication when enter- or mobile phone), and object characteristic-based authen-
ing a password at the front door. tication (what you are, i.e. biometric data). The fourth
4. Various services such as remote door opening and approach to user authentication includes a combination
intruder detection needs to be available : If necessary, of the other three methods and is known as multi-factor
the system include the ability to be notified of the authentication.

VOLUME 10, 2022 124605

 H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

user characteristics. Biometrics is a representative object

characteristic-based authentication method. Biometrics dis-
tinguish physical and behavioral features. Physical features
include face recognition using face shapes and thermal
images of each person, iris recognition, vein recognition,
fingerprint recognition, and other methods including retina
and hand shape. Behavioral features include speech recog-
nition, gait recognition, and signature recognition. Recently,
companies have begun using behavioral features such as gait,
FIGURE 2. Password based authentication method [8].
signature, and voice recognition to enhance biometric accu-
racy [12], [13], [14]. Face recognition is more convenient than
fingerprint, iris, voice, and vein recognition. Iris recognition
is more accurate than other biometrics, and the installation
1) KNOWLEDGE-BASED AUTHENTICATION
cost is less than a speech recognition system.
A simple and convenient user authentication method that is Recently, a permanent and low-cost fingerprint recognition
often used is ID/password-based authentication. Password method or Face ID method, based on the user’s face, have
authentication techniques, such as Fig. 2, pass the hash value been implemented and used almost universally to identify the
of the password [7] to the server when the user enters their owner of the phone and allow smartphone use and access.
ID and password. It then verifies the hash value of the input
password against the hash value of the original password
provided at the time of registration 4) MULTI-FACTOR AUTHENTICATION
The safety of password-based authentication depends on Knowledge-based, or password-based, authentication is vul-
the management of the database (DB) in which passwords are nerable because it allows short, insecure passwords which
stored. There have been numerous cases in which attackers often are never changed from the password that was used
linked to insiders have leaked users’ passwords by using a to register with. Therefore, additional authentication mea-
member information table containing member information. sures are needed to overcome the vulnerability of passwords.
In general, the password registered by the user is not stored Multi-factor authentication is an effective authentication
in the database. This is to protect the password from leakage technique that allows users to further enhance security by
in the member information table. But the resulting value of a authenticating with ID/Password and then using another addi-
hash function, which is a one-way function, is stored in the tional authentication method. Additional methods of multi-
member information table instead of the password. Vulner- factor authentication include ownership-based authentication
abilities, such as crashes in the MD5 hash function, which or object characteristic-based authentication. In particular,
generates 128-bit hash values, have led to the MD5 hash func- smartphones are being used as a means of authentication with
tion being widely replaced by stronger hash functions such such features as fingerprint authentication, face authentica-
as SHA-256 and SHA-512 [9]. However, whenever a user tion, OTP, or random number input through SMS.
enters a password, it can still be recorded by someone peeking
over the shoulder of the user or by recording the input when B. IMAGE PROCESSING AND ARTIFICIAL INTELLIGENCE
entering a password through Google Glass. In addition, social TECHNOLOGY
engineering attacks based on user information or password 1) OBJECT DETECTION TECHNOLOGY
brute force attacks are also possible. If the member informa- The traditional object detection methods such as Haar-like
tion table where the password is stored is leaked, the user’s features, scale invariant features transform (SIFT), and his-
password can be leaked through a Rainbow Table attack by tograms of oriented gradients (HOG) require the trouble
using the hash value generated in advance [10], [11]. of manually extracting and adjusting features of an image
according to the detection target [15]. Object detection tech-
2) OWNERSHIP-BASED AUTHENTICATION nology is growing rapidly due to the proposal of a deep
Ownership-based user authentication is an authentication learning method that can collectively process data at once
technique that identifies a user’s possessions. Posses- from feature extraction to classification.
sions used for authentication can include identification Object detection research using deep learning started
cards, encryption keys, smartphones, credit cards, and OTP with the proposal of a Convolutional Neural Network
(one-time passwords). After password-based authentication, (CNN) model, followed by Regions with CNN features
additional authentication techniques such as SMS authentica- (R-CNN) [16], fast R-CNN [17], and faster R-CNN [18].
tion or OTP authentication are required. An R-CNN-based deep learning model is a model that detects
objects by applying four techniques: region proposal cre-
3) OBJECT CHARACTERISTIC-BASED AUTHENTICATION ation, feature extraction, classification, and linear regression.
Object characteristic-based authentication is an authen- However, the R-CNN-based model has a problem in that
tication method which authenticates users based on each process is executed separately, which requires more

124606 VOLUME 10, 2022

H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

computing power. But, the faster R-CNN model introduced eyebrows, nose, mouth, and chin can be analyzed to extract
another region proposal network (RPN) to solve the problem characteristic information. The extracted feature information
of inefficiency in learning and running time. is compared with the facial feature information stored in
the database in order to recognize the face and confirm the
2) OBJECT DETECTION TECHNOLOGY IN A REAL-TIME identity of the user.
ENVIRONMENT The video security market was able to detect objects with
The R-CNN-based deep learning model extracts and classi- motion analysis and rule-based pattern analysis by early
fies features from the proposed region. Since the detection 2010, but false positives were frequent with an error percent-
and recognition processes have to be individually trained, age of 28.2%. But with the development of deep learning
the problem exists that the computation speed is slow and technology, a breakthrough improvement was made, reducing
optimization is difficult. YOLO is a model that integrates the error rate from 18.4% in 2012 to 3.57% in 2015. Addi-
object detection and recognition into one system to improve tionally, the accuracy of face recognition increased by about
the shortcomings of the R-CNN model. The input image is 95%.
divided into a grid, and each grid cell calculates whether In general, CNN-based deep learning models are used to
the object exists within it and the confidence score for the detect faces and apply them to face recognition systems.
object’s existence. Then, the reliability of the object detection Included among these are face mask detection systems for
is calculated and the object in the area is recognized. Since preventing the spread of infectious diseases and restricting
this model is faster than the R-CNN model, it can be applied access [26], age estimation studies on face images, with
to the security and authentication system used in real-time various neural network algorithms applied, for use in intelli-
environments to detect objects quickly and easily. Beginning gent security, financial transactions, and so on [27], criminal
with YOLOv1, the YOLO model was extended to YOLOv4 to identification using MTCNN algorithm research [28], and
compensate for the low object detection accuracy compared face recognition in smart door lock systems [29]. All of these
to the fast detection speed and to detect more classes. As the are being researched to apply to various intelligent security
version number increased and performance improved, it was systems.
necessary to reduce the computational complexity by simpli-
fying the network layer for real-time detection in a low-spec C. ACCESS CONTROL SYSTEM
system environment. Accordingly, Tiny-YOLO was devised An access control system, a type of physical security, is used
as a model for low-spec real-time object detection. in many places to block unauthorized users from entering,
YOLOv3 uses Darknet-53 as the backbone network. such as buildings or offices.
By applying the Feature Pyramid Network (FPN) concept, A representative example of this is shown in the method
each object is detected by feature maps at three scales created of controlling the entrance and exit of a car to a parking lot.
in the process of passing the image through each neural net- A camera is used to recognize a car’s license plate number
work. In order to detect small objects, an object is detected by and then check whether it is registered in the system, or an
increasing the size of the feature map by upscaling it. Three RFID-based smart card system is used. This RFID-based
Anchor boxes will be generated for each scale, and a total of contactless smart card is attached to the windshield of the
nine Anchor boxes will be selected. Fig. 3 shows the structure car and used to determine whether car access is permitted.
of the YOLOv3 model [19], [20], [21]. Tiny-YOLOv3 is a Convenience is guaranteed and for easy authentication, the
model with the YOLOv3 structure that reduces computing same contactless smart card access is possible even inside the
resources and increases speed through the simplification of building when entering the office. In the case of contactless
the Convolutional and Pooling processes. Tiny-YOLOv3 is smart cards, a security algorithm is applied, so it is safe from
less accurate than the YOLOv3 model, but it is faster and can illegal copying or hacking without the encryption algorithm,
be used in a limited detection environment [22], [23], [24]. special key, and mutual authentication. Smart door locks
In order to be applied to an actual industrial field, even are widely used to control access by entering passwords in
if the accuracy of its performance is somewhat low, it is apartments or houses. But with the development of biometric
necessary to reduce the cost of processing the data with authentication technology, authentication with a user’s fin-
higher speeds and multiprocessing. Therefore, for real-time gerprint, or iris recognition with a camera, the service for
application in the proposed system, it is appropriate to use controlling access by confirming that the user is registered
Tiny-YOLO, which has a high processing speed, even if the has become commonplace. Recently, various methods have
accuracy is somewhat low [25]. also been developed to control access to prevent the spread
of infectious diseases (Fig. 4).
3) SECURITY SERVICE WITH FACE RECOGNITION Additional research is now being conducted to confirm
TECHNOLOGY APPLIED the approach of a user with the PIR proximity sensor, or to
Face recognition technology is a technology that scans recognize the user’s voice with a microphone, and identify the
and recognizes face shapes and thermograms with thermal user with additional authentication methods such as questions
infrared rays. After taking a picture of the face with a camera, and answers [30]. Face recognition technology using deep
either 68 or 81 key points on areas such as the eyes and learning is being used for user authentication by recognizing

VOLUME 10, 2022 124607

 H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

FIGURE 3. Structure of YOLOv3.

registration and processing. In detail, it consists of (1) deep

learning model training, (2) visitor face detection, (3) face
recognition and access verification, and (4) monitoring.
In the deep learning model training portion, a learning
model is obtained by using Tiny-YOLOv3 for face features
data sets that are labeled with 7 classes. In the visitor face
detection portion, we create the face recognition security
system. This includes a webcam to act as the CCTV, a servo
motor to monitor the surroundings, and an infrared sensor
for human body detection. Each of these is connected to the
FIGURE 4. Access control using a smart mirror. Jetson Nano board. First, a human face is detected from the
real-time image received from the webcam using the learning
model. When it is confirmed that a human face is present, the
people in access management systems. In general, access detailed features of the face are detected using the landmark
control has previously been performed through security staff algorithm in the recognition and security portion.
or visitor tags, but problems may arise due to the reduction The detected detailed features are compared with the fam-
of security staff or loss of visitor tags. Accordingly, face ily of face data stored in the database of the monitoring
recognition technology is emerging as an alternative technol- system, and security is canceled in case of a member, while
ogy for effective access control. Recently, due to COVID-19, an alarm notification is displayed to the user in case of an
it has been partially applied for purposes such as creating outsider. Lastly, log information such as the face and number
access lists, wearing masks, and measuring body temperature. of people in question is stored in the database through the
Object detection technology using such deep learning can wireless LAN. The results can be checked later.
also be applied to security management systems. In addition,
when entering a number in front of the front door, face recog- 1) DESCRIPTION OF EACH COMPONENT
nition is emerging as an alternative to the act of password (1) Deep learning model training component (derivation of
peeking by outsiders. When entering a password, it can be a deep learning training model) : Desktop - Save images of
used to develop a system that sounds an alarm by determining family and acquaintances in a dataset in advance and learn
whether there is a person behind the user and whether that them using the Tiny-YOLOv3 deep learning model
person is a companion. (2) Video data collection component: Collects the visitor’s
video using the webcam CCTV.
III. DESIGN FOR PROPOSED SYSTEM & DEEP LEARNING (3) Control and processing components: A database is built
A. SYSTEM STRUCTURE on the Jetson Nano board where the collected video images
The overall structure of the system proposed in this paper is are stored and processed.
shown in a block diagram as shown in Fig. 5. The composi- (4) Monitoring and post processing: The processing result
tion of the system is largely divided into parts in charge of is transmitted to the manager’s smartphone and processed.

124608 VOLUME 10, 2022

H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

FIGURE 5. Proposed system structure diagram.

camera to scan for visitors, and an infrared sensor is added for

visitor detection. The system begins operation when a visitor
is detected through the infrared sensor. The image received
from the CCTV installed in front of the door is detected using
a deep learning model that has been pre-trained. By using the
face detection system, key points that can identify a visitor’s
face are searched and compared with the face data stored
in the system database to determine whether the visitor is a
member or not. MariaDB is installed in the system to store
video images and access information collected from CCTV.
An Internet connection is required for notification of visitors
FIGURE 6. The structure of hardware. and remote monitoring of users. A wireless LAN card has
been added to the Jetson Nano board to connect to the smart
home. Additional authentication is required to reduce the
2) HARDWARE CONFIGURATION DIAGRAM false acceptance rate (FAR) of the visitor’s face recognition.
The operation of the entire system for face detection, recog- Finally, the touch screen outputs a message to the user on the
nition, and log storage uses NVIDIA’s Jetson Nano board as screen, and if necessary, a keypad that can be input on the
the main system. The Jetson Nano board is a development screen is displayed, where the password can be input with a
kit optimized for artificial intelligence learning with Ubuntu touch.
18.04 on a Quad-core ARM A57 CPU @1.43 GHz, with a
128-core Maxwell GPU, and 4GB of 64-bit LPDDR4. B. DEEP LEARNING-BASED TRAINING MODEL
• NVIDIA’s Jetson Nano Board 1) DATASET COLLECTION AND PREPROCESSING
• Logitech C270 HD Webcam To create a dataset, we had Google crawl over 500 face
• HDMI LCD Touch Screen Monitor images with various features on the web. Considering that the
• Servo motor place where the system is installed is a specific environment,
• Human body detection infrared sensor 500 additional images of people entering and exiting the
• Logitech Speaker building were collected. It obtained face images which have
Fig. 6 is an example of the hardware implementation form various looks (e.g wearing eye glasses, mask, cap) in addition
of the proposed system. Logitech’s C270 HD Webcam is to gender, age and lateral angle of the face in order to decrease
connected to the Jetson Nano board to be used as the CCTV false positive ratio in face detection and to increase accuracy.

VOLUME 10, 2022 124609

 H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

TABLE 1. Classification table for face features.

FIGURE 7. Annotation of the visitor sample result screen.

For smooth learning in the Jetson Nano, the image was

resized to a standard size of 400×320. In addition, the images
collected in consideration of the influence of the environment
were augmented through various angles of rotation, inversion,
and brightness adjustment. Out of a total of 2,000 images,
1,400 were used as a training dataset, 400 were classified as
a validation dataset, and 200 were classified as a test dataset.

2) ANNOTATION PROCESS (CLASSIFICATION)

We classify facial features using YOLO Marker, an open
source tool for YOLO labeling distributed for free on
AlexeyAB GitHub. In order to more accurately determine that
it is a human face, 7 features of the face region are selected
and classified as shown in Table 1. The data set is labeled
with the index, class number, and bounding box coordinate
values and stored in a txt file for each image. Then, a list file
for the entire image is also created. Fig. 7 shows the matching
structure for the result of labeling using YOLO Marker. There
is a class number that was initially set for labeling, and
then the square coordinate values for each labeling class are
shown. For example, in the case of the first value [4 0.5 0.1 FIGURE 8. Tiny-YOLO v3 architecture.
0.2 0.3], the class name is 4, the second is the x value, the
third is the y value at the center point of the rectangle (0.5,
0.1). The fourth value is the width at 0.2, and the last value is architecture of deep learning model for face detection
the height at 0.3. (The width of the entire screen is set to 1.0, obtained from learning of dataset. We outputted results
and the height is set to 1.0.) Class number 2 indicates a mask, through input, Convolution step 13, MaxPooling step 6,
and class number 4 indicates a face or head. Up-sampling step 1, step 2 of detection and output.
It can recognize human faces through training model
3) FACE DETECTION APPLYING TRAINING MODEL using 7 features. It detects faces which is the most important
Training of the Tiny-YOLOv3 model for face features was part that is critical to recognize human among extracted fea-
carried out on the Jetson Nano. We shown in Fig. 8 the tures and make attributes like eyes, nose, mouth as the main

124610 VOLUME 10, 2022

H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

TABLE 2. Definition of tables in the database.

FIGURE 9. Inference result using face feature learning model.

priorities. If there are four features or more, including face

feature, it transfers extracted face area data to the authen-
tication system. Fig. 9 shows the inference result using the
training model.

IV. STRUCTURE AND PROTOCOL OF THE PROPOSED

SYSTEM
A. USER AUTHENTICATION SYSTEM FIGURE 10. Flow chart of the proposed system.
A MariaDB is built on the Jetson Nano and a database for
visitor access control is created. A table for storing access-
related information is created like Table 2. The ‘‘pictureTBL’’ Algorithm 1 is a pseudo-code that recognizes visitors with
records the image of the visitor, their identity, the number of the PIR sensor and the CCTV camera installed on the main
simultaneous visitors, and the time of the visit. In addition, door. If it does not recognize a visitor after a long distance
for future post-processing, visit records are registered in the or while wearing a mask, it will send a message to the user
‘‘visitorLogTBL’’ and managed separately. If there are two so that the visitor’s face can be checked. In this process,
or more visitors, the name and information of the front-most it photographs the visitor, creating a visit history. It unlocks
visitor is recorded. the door when it recognizes a visitor’s face and recognizes
the visitor as a family member. When the initial identification
B. PROTOCOL fails, it requests additional identification.
Fig. 10 is a flow chart that shows the process of the proposed [Step of visitor recognition]
system in action. Algorithm 1 is the pseudo-code suggested 1. As shown in Fig. 10, the visitor may be a family member
for the flow chart of the proposal system in Fig. 10. or a guest for the purpose of visiting, a passerby at the front

VOLUME 10, 2022 124611

 H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

Algorithm 1 Visitor Authentication Based on Face Recogni- and if the visitor is a family member, the door lock password
tion is pressed.
Let be PIR in operation: Recognizing visitor by PIR sensor. 2. If the visitor is a guest or family member, the system
Let be CCTV in operation: take visitor’s picture by CCTV checks the visitor. In order to check the visitor’s face, the
model finds a region of the face to recognize the face. How-
01: while System running do ever, if it cannot detect that face region, the face will not
02: if front door waiting time > Threshold then be detected, and the system will output the message ‘‘Please
03: Guest is VIsitor come closer’’ or deliver that message through the speaker.
04: else 3. If the region of the face of the visitor is found, then the
05: Guest is passer-by face image of the visitor is stored.
06: goto line 01 [Step of face recognition]
07: end if 1. Recognize a face in the CCTV image.
2. If a face is not recognized, check whether a mask is worn,
08: while Visitor’s face non-detection do and if the face is not recognized due to a mask, output the
09: Alarm(‘‘Please come closer’’) message ‘‘Please pull down your mask’’ or deliver it through
10: end while the speaker.
11: Save(photo of Visitor) 3. If a mask is not worn, move to the step of finding the
face region again and take a picture of it again.
12: if Visitor’s face recognition then 4. If the face is now recognized, check whether it is a family
13: if Visitor is Family then member stored in the database.
14: Open the Door [Steps for post-authentication process]
15: else 1. If the visitor is a registered family member, authenticate
16: Alarm(‘‘Who are you?’’) them through simple authentication and automatically open
17: end if the front door.
18: else 2. In the case of a guest, the face image of the visitor is
19: if mask wearerthen delivered to the manager. The manager may then open the
20: Alarm(‘‘Please take down your mask’’) front door after confirmation. If the system fails to recognize
21: goto line 11 a face registered in advance, the door lock password may be
22: else entered to authenticate the visitor and then open the front
23: gotoline 08 door. In the proposed system, the door lock is a device
24: end if that inputs the password through a touchpad and opens the
25: end while front door when the correct password, which was stored in
advance, is input. The front door lock system authenticates
visitors when the password entered as a number string on the
touchpad matches the password stored in advance.
door, or a courier delivering goods. Only after the visitor Fig. 11 is a chart that shows the process of checking how
is standing at the front door for a certain period of time many visitors are at the front door. In this way, it is possible
within range of the proximity sensor is the visitor recog- to confirm how many visitors there are. This function is
nized. The system then wakes up from the standby state and necessary to ensure that when a visitor enters a password into
operates. the door lock, no one is peeking from behind. The proposed
2. The system takes a picture of the visitor with the system can identify the number of visitors through CCTV
CCTV camera to recognize the visitor, and detects the face and protect against peeking, which is a social engineering
of the visitor based on the learning model trained by deep attack [31].
learning. Algorithm 2 is the pseudo-code that figures out the number
3. After the system detects the face, it takes a picture with of visitors in front of the main door in Fig. 11
the center of the face and stores it in the system. This is stored When face detection fails to authenticate a family member,
for the purpose of collecting information about suspicious the proposed method requires an additional authentication.
people. This is not a problem if there is only one visitor, but if
[Step of visitor decision] there are multiple visitors or passing people, the password
1. The visitor may be a courier who leaves a delivery at may be exposed. In other words, determining the number
the front door, or a guest coming into the house. If the visitor of visitors helps to avoid shoulder surfing. If the num-
is a family member or a guest who wants to enter the house, ber of visitors detected by video through CCTV is two
they can ring the doorbell or enter the door lock password. or more, it senses shoulder surfing from behind. It deter-
The system checks whether the visitor is a member of the mines whether or not a visitor is shoulder surfing based
business or a visitor based on how long the person waits at on his or her position, face size, and the focus of an eye
the front door. If the visitor is a guest, the doorbell is pressed, from behind. If the shoulder surfing attack is detected,

124612 VOLUME 10, 2022

H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

Algorithm 2 Visitor Shoulder Surfing Attack Protection

01: procedure input Password()
02: Input CCTV captured photo??
03: Input Visitor’s detected face image

04: if number of Visitor >2 then

05: if shoulderSurfingAttack(image) then
06: Display(‘‘Please input long & complex password’’)
07: Input(password)
08: end if
09: else
10: Input(password)
11: end if

01: procedure shoulderSurfingAttack(image)

02: size=other_face
//the face of the person in the back
03: focus=eye_position
//the eye of the person in the back
04: peek=false
05: delay(300)
06: if other face >size then
07: peek=true
08: else if other eye - focus < threshold then
09: peek=true
10: end if
FIGURE 11. The function of the number of visitors. 11: return peek

the system request the visitor enter a lengthy or complex

password. are possible. For example, users can be authenticated by using
[Steps for CCTV image capturing] photographs of family members. Therefore, the proposed
1. Using CCTV, faces are found in the image using a model system identifies the changing of feature points on the face
that has been trained in advance with deep learning. and confirms whether or not the face is a photo, in which case
2. The number of faces in the recorded image is user authentication is blocked.
determined. Fig. 12 shows the process of checking whether a visitor
<In cases of at least two visitors> is a family member using CCTV at the front door. The
If the number of visitors is two or more, the system checks proposed model detects the face of a visitor in the image
whether one visitor is peeking. taken through CCTV with a model trained by deep learning.
Shoulder surfing attack? If the person at the back comes Authentication is completed when the face image taken with
near the front door or looks at the door lock over the front CCTV is compared to the face information of family mem-
visitor’s shoulder, it is suspected that they are peeking. Than, bers registered in advance, and the comparison value is higher
the door lock display will display the alert message ‘‘Please than the set threshold. Then, the information of visitors is
enter the password carefully so that no one else can see.’’ The registered in the database access log table and the next step is
user then protects against this social engineering attack by performed. If additional authentication is required, the door
entering the password carefully. lock password may be entered to open the front door lock and
<In the case of a single visitor, or not peeking> authentication is completed.
3. If there are more than two visitors but no peeking is
detected, or if there is only one visitor, the message ‘‘Please, V. ANALYSIS AND EXPERIMENTAL RESULTS OF THE
input the password’’ is displayed. PROPOSED SYSTEM
[Steps for password input] A. ANALYSIS OF USER AUTHENTICATION SYSTEM
1. The visitor inputs the password on the touchpad. When the visitor’s face is recognized with a CCTV image,
2. If the password entered matches the password stored in 81 feature points are detected using the model that was trained
the database, the front door opens. on the file shape_predictor_81_face_landmarks.dat. Using
Because this is a system that determines whether or not to the HoG feature as a face detector, a linear classifier is used to
open the door lock through face recognition, various attacks find the landmarks of the face such as the eyes, nose, mouth,

VOLUME 10, 2022 124613

 H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

FIGURE 12. Flow charts to detect faces and authenticate family members at the front door.

TABLE 3. FAR and FRR of HoG according to decision threshold. Algorithm 3 Part of the Source Code for Visitor Authentica-
tion Using Deep Learning
Let be detector = dlib.get_frontal_face_detector()
Let be sp = dlib.shape_predictor(’face_landmarks.dat’)

01: face_rec = dlib.face_recognition_model_v1(’dlib_face_

recognition_v1.dat’)
02: def find_faces(img): ∼ #function definition
chin line, and eyebrows. For user authentication, the faces 03: def encode_faces(img, shapes): ∼ #function definition
of registered members or family members are stored in the 04: webcam = cv2.VideoCapture(0)
database in advance. The vector values of the feature points in ∼
the family member’s face images are stored in the familyTbl 05: while webcam.isOpened():
table in the database. This method extracts the vector values 06: rects, shapes, _ = find_faces(img)
of feature points in the visitor’s face image and compares 07: descriptors = encode_faces(img, shapes)
those with the vector values of the feature points of family 08: fig, ax = plt.subplots(1, figsize=(10, 10))
members stored in familyTbl. When the threshold value is 09: for i, desc in enumerate(descriptors):
within the authentication range, authentication is performed. 10: if min_dist < threshold: # authentication
So, if a visitor’s face is very similar to a registered family ∼
member’s face, it may be recognized as a family member. 11: ax.add_patch(rect)
That is, there is a False Acceptance Rate (FAR). In addition, ∼
despite being a family member, a False Rejection Rate (FRR) 12: plt.show()
may occur when a face is obscured by a mask, glasses,
or other facial covering and some of the vector values of
feature points are distorted. Table 3 shows the evaluation indi-
cators, FAR, FRR, and Precision according to the threshold from the family photo. It is the source code that extracts the
of similarity in the HoG algorithm using Dlib [31]. feature in the video image collected by the webcam when a
Thus, because there is FRR and FAR, additional authenti- visitor is waiting at the main door and confirms if the image
cation methods are required. is the same person by comparing the vector value of the
Algorithm 3 is part of Python code that extracts 81 facial previously saved feature of the family to the extracted vector
features. It uses the Dlib library to extract and save features value.

124614 VOLUME 10, 2022

H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

FIGURE 15. YOLO and Tiny-YOLO’s training loss function graph.

(a) YOLOv3, (b) Tiny-YOLOv3.

The x-axis displays the number of repetitions, and the y-axis

displays the average loss rate. The number of iterations of
FIGURE 13. Results of the detection of 81 feature points on the face.
the model trained with the YOLOv3 algorithm was 3,400 and
the average loss was 0.9384. The number of iterations of the
model trained with the Tiny-YOLOv3 algorithm was 182,100
and the average loss was 1.6490.
When the average loss rate has a relatively linear shape,
the result of a good training model is obtained. As shown
in Fig. 17-(a), YOLOv3 derives a training model that is
meaningful even when the number of repetitions is 3,400
times. However, in the case of Fig. 17-(b), the number of
repetitions was increased because Tiny-YOLO did not come
up with a meaningful training model. Fig. 17-(b) shows that
180,000 times were performed, but there was no difference in
the loss rate.
C. EXPERIMENT RESULTS
The proposed system enables real-time processing of visitor
FIGURE 14. Visitor’s face recognition result screen.
face recognition with the Jetson Nano as shown in Fig. 16.
It detects faces, eyebrows, masks, and eyes. It collects face
In the proposed model, a pass number can be additionally images which have various looks such as wearing eye glasses,
requested or the administrator can perform an additional mask, cap and make up. Moreover, it can identify gender, age,
verification procedure. and lateral angle of the face in order to decrease false positive
The 81 features from the family are saved in the system as ratio of face detection and to increase accuracy.
vector previously. The two members of family is registered The Raspberry Pi, which is cheaper than the Jetson Nano,
in Fig 13. A face is detected with a webcam (CCTV), and was also tested in the same environment. The model applied
81 features of Person 0 and Person 1 are found by facial to the Raspberry Pi was trained with Tiny-YOLOv3. Three
recognition technology and similarity with the vector value faces are detected, 4 face features are detected for the left
of the family stored in the system is calculated and recognizes person, 3 face features are detected for the middle person,
as family member. and 3 face features are detected for the right person. Figure 17
In case of Fig. 14, The method is the way that finds 81 fea- shows the authentication by finding the 7 features and deter-
tures and authorizes they are family like Fig.13. In Fig. 14, the mining whether they are the same as the registered person.
name of a family member is displayed on the face in screen. The left face of Fig. 17 is detected with Raspberry Pi and
According to Fig.14, When two visitors are in front of the shows the probability of whether a feature exists.
CCTV, their faces are detected and they are asked to lower Fig. 18 shows the actual inference result and FPS output of
their masks in order to extract accurate face features from the the two algorithms. Fig 18-(a) shows the YOLOv3 execution
proposed model. In addition, this is a screen that captures the result screen and Fig 18-(b) shows the Tiny-YOLOv3 execu-
feature points of the face when the mask is lowered. tion result screen.
B. DATASET-BASED LERNING RESULTS AND VI. COMPARATIVE ANALYSIS AND DISCUSSION OF THE
EXPERIMENTS PROPOSED SYSTEM
1) LEARNING RESULT AND EXPERIMENTS USING DATASET A. COMPARATIVE EVALUATION OF SYSTEM
Fig. 15 shows a graph of the loss function of the trained Table 4 shows the average speed (FPS) and average accu-
images by applying 7 face feature classification data sets racy (%) when running YOLOv3 and Tiny-YOLOv3 on the
to both YOLOv3 and Tiny-YOLOv3 models, respectively. Jetson Nano board and detecting face features. Although the

VOLUME 10, 2022 124615

 H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

TABLE 4. The comparison and analysis of processing speed and accuracy

about each algorithm on each system board.

TABLE 5. Recognition Speed of a single user.

FIGURE 16. Screen that detected the faces of three visitors at the front
door.
TABLE 6. Performance Comparison.

accuracy of face feature detection was calculated by taking

the average of the face classes that were recognized in all
faces. As shown in Fig 18, when YOLOv3 was run with
the Jetson Nano, the average detection speed was 2.4 FPS
and the accuracy was 90.3%. When Tiny-YOLOv3 was
run with the Jetson Nano, the average detection speed was
FIGURE 17. Feature detection and face authentication in Raspberry pi.
6.5 FPS and an accuracy of 86.3%. Although this was slightly
less accurate, it was found to be about three times faster in
terms of speed. As a result of executing Tiny-YOLOv3 on
the Raspberry Pi, it was confirmed that the calculation result
of frames per second was less than 1 in real-time video, but
the detection result was derived and the accuracy was 76.3%
(Table 4).
FPS means the number of frames performed per
second. As a test result, the average processing speed of
Tiny-YOLOv3 is 6.5FPS, which means the speed performing
average 6.5 frames per second. Therefore, it was found that
the time taken for face recognition in the access system could
be shortened compared to other algorithms and hardware by
using the Jetson Nano board and Tiny-YOLOv3.
We measured the face recognition speed and stated the
situation about detecting Single user and two or more users
in Table 5.
As the result of comparison, there is a little difference
between 1 and 2 or more, but no difference in average FPS.
Table 6 is the result of comparing processing speed of
YOLO which is the core algorithm and Faster R-CNN which
is the representative object extraction algorithm.
FIGURE 18. YOLO and Tiny-YOLO execution results and FPS comparison. We compared YOLO and Faster R-CNN algorithm in GTX
1080Ti desktop environment and in Jetson system board
environment. In inference time in GTX1080Ti desktop envi-
performance is lower than that of the Jetson Nano, a cheap ronment, YOLO is 3 times faster than Faster R-CNN. The
Raspberry Pi was also used and compared together. The processing speed of Tiny-YOLOv3 is 2.7 times faster than

124616 VOLUME 10, 2022

H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

that of YOLOv3 in the Jetson Nano system board, which can various algorithms that have improved and developed CNNs
stand alone due to the characteristics of the suggested system. capable of image processing have been proposed.
In this paper, we introduce various object detection algo-
B. EVALUATION OF ANALYSIS OF SECURITY FIELD
rithms including CNN. CCTV detects the face of a visitor
The requirements mentioned in the introduction are satisfied and then recognizes 81 feature points in the face to create a
as follows. set of vector values based on the features. Members of the
First, the proposed model can be used to identify the num- family are registered in advance with face images. In this
ber of visitors based on faces found with CCTV. If the number study, we designed and implemented a system that opens the
of people decreases after a certain period of time, it is judged front door after recognizing a new visitor as a member of the
to be a passerby. Otherwise, it is judged to be a visitor or an family when the difference in recognized facial feature vector
intruder. values between the CCTV image and the image stored in the
Second, if the visitor at the front touches the password for database is smaller than the threshold.
user authentication, the person behind the visitor can peek at The proposed model was applied to a microprocessor that
it. So, it is necessary to check whether the visitor standing is portable and operates various sensors and the system was
behind the front person is a family member or a companion. tested. Although the system was applied to two representative
If the companion’s face is perceived to be approaching the microprocessors, the Raspberry Pi and the Jetson Nano, it was
door, or if the CCTV recognizes that the eyes of the visitor difficult to apply in real life because the operation was very
are trying to steal the password, it outputs a message to the slow and the desired result was not obtained in the cheap and
front visitor on the display warning them of this. In other widely used Raspberry Pi. However, the desired result was
words, if the visitor standing behind is judged to be peeking, derived from the Jackson Nano with a built-in GPU.
the visitor standing in front receives a ‘‘Your password is too Specifically, in the proposed model, YOLOv3’s inference
short’’ message when entering a registered password. time was 2.4 FPS with an accuracy of 90.3%, and Tiny-
Third, family members are registered in advance. It learns YOLOv3’s inference time was 6.5 FPS with an accuracy of
from the registered photos, determines whether a face has 86.3%.
been registered in advance from the video image entered into Future studies include the development and research of a
the CCTV, and authenticates the visitor. system that recognizes faces when controlling access to shops
When authentication is performed, a probabilistically cal- or restaurants, automatically registers frequent visitors, iden-
culated result is derived by comparing the CCTV image with tifies the number of visitors, stores visiting records, and con-
a previously registered face. If the calculated result is higher trols access according to visitors. In addition, it is necessary
than the threshold, primary authentication is performed, but if to develop an access authentication system that can enable
it is not 1, secondary authentication is required because incor- easy access with face recognition rather than fingerprints or
rect authentication may occur. There are various secondary passwords, and has a low false detection rate.
authentication methods, but in this study, secondary authenti-
cation can be performed by entering a short password. When
REFERENCES
this authentication is completed, the door is opened.
[1] T.-H. Tsai, C.-C. Huang, C.-H. Chang, and M. A. Hussain, ‘‘Design of
Fourth, the image taken by CCTV in the proposed model
wireless vision sensor network for smart home,’’ IEEE Access, vol. 8,
can be transmitted to the manager’s phone through the smart pp. 60455–60467, 2020, doi: 10.1109/ACCESS.2020.2982438.
home service. The manager can detect intruders by checking [2] K. Muhammad, J. Ahmad, I. Mehmood, S. Rho, and S. W. Baik,
the image. If password authentication fails or a family mem- ‘‘Convolutional neural networks based fire detection in surveillance
videos,’’ IEEE Access, vol. 6, pp. 18174–18183, 2018, doi:
ber’s visit is not registered, the visitor cannot enter the house 10.1109/ACCESS.2018.2812835.
with authentication. [3] M. T. Bhatti, M. G. Khan, M. Aslam, and M. J. Fiaz, ‘‘Weapon detection
In this case, it checks the visitor with a smartphone and in real-time CCTV videos using deep learning,’’ IEEE Access, vol. 9,
pp. 34366–34382, 2021, doi: 10.1109/ACCESS.2021.3059170.
opens the door remotely. The result of the proposed method
[4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ‘‘ImageNet classification
is that the important image is stored on the server and the with deep convolutional neural networks,’’ in Proc. Adv. Neural Inf. Pro-
authenticated visitor’s name is stored in a log file. This log cess. Syst., 2012, pp. 1097–1105.
file can also provide additional services when family mem- [5] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, ‘‘You only look once:
Unified, real-time object detection,’’ in Proc. IEEE Conf. Comput. Vis.
bers return home, access information, access pattern analysis, Pattern Recognit. (CVPR), May 2016, pp. 779–788. [Online]. Available:
or have a long absence. https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/
Redmon_You_Only_Look_CVPR_2016_paper.pdf
VII. CONCLUSION [6] S. W. Shah and S. S. Kanhere, ‘‘Recent trends in user authentication—
Deep learning, a major technology of artificial intelligence, A survey,’’ IEEE Access, vol. 7, pp. 112505–112519, 2019, doi:
10.1109/ACCESS.2019.2932400.
is growing rapidly as it is applied to voice recognition and [7] K. M. Renuka, S. Kumari, D. Zhao, and L. Li, ‘‘Design of a secure
image recognition. In particular, deep learning technology password-based authentication scheme for M2M networks in IoT enabled
in the field of image recognition is being applied as a core cyber-physical systems,’’ IEEE Access, vol. 7, pp. 51014–51027, 2019,
doi: 10.1109/ACCESS.2019.2908499.
technology to autonomous driving and crime prevention mon-
[8] H.-J. Mun, ‘‘Biometric Information and OTP based on authentication
itoring systems, which are emerging as future industries. mechanism using blockchain,’’ J. Converg. Inf. Technol., vol. 8, no. 3,
As for deep learning models in the field of image recognition, pp. 85–90, 2018, doi: 10.22156/CS4SMB.2018.8.3.085.

VOLUME 10, 2022 124617

 H.-J. Mun, M.-H. Lee: Design for Visitor Authentication Based on Face Recognition Technology Using CCTV

[9] A. Mohammed Ali and A. Kadhim Farhan, ‘‘A novel improvement with an [26] D. R. Shenvi and K. Shet, ‘‘CNN based COVID-19 prevention system,’’ in
effective expansion to enhance the MD5 hash function for verification of Proc. Int. Conf. Artif. Intell. Smart Syst. (ICAIS), Mar. 2021, pp. 873–878,
a secure E-Document,’’ IEEE Access, vol. 8, pp. 80290–80304, 2020, doi: doi: 10.1109/ICAIS50930.2021.9396004.
10.1109/ACCESS.2020.2989050. [27] S. Han, ‘‘Age estimation from face images based on deep learning,’’ in
[10] H.-J. Mun, S. Hong, and J. Shin, ‘‘A novel secure and efficient hash func- Proc. Int. Conf. Comput. Data Sci. (CDS), Stanford, CA, USA, Aug. 2020,
tion with extra padding against rainbow table attacks,’’ Cluster Comput., pp. 288–292, doi: 10.1109/CDS49703.2020.00063.
vol. 21, no. 1, pp. 1161–1173, Mar. 2018, doi: 10.1007/s10586-017-0886- [28] Q. Guo, Z. Wang, C. Wang, and D. Cui, ‘‘Multi-face detection algo-
4. rithm suitable for video surveillance,’’ in Proc. Int. Conf. Comput. Vis.,
[11] J. Shin, Z. Liu, C. M. Kim, and H.-J. Mun, ‘‘Writer identification using Image Deep Learn. (CVIDL), Chongqing, China, Jul. 2020, pp. 27–33, doi:
intra-stroke and inter-stroke information for security enhancements in 10.1109/CVIDL51233.2020.00013.
P2P systems,’’ Peer-to-Peer Netw. Appl., vol. 11, no. 6, pp. 1166–1175, [29] M. Waseem, S. A. Khowaja, R. K. Ayyasamy, and F. Bashir, ‘‘Face
Nov. 2018, doi: 10.1007/s12083-017-0606-0. recognition for smart door lock system using hierarchical network,’’ in
[12] Science, ICT & Technology Trends. (Aug. 2019). Spreading Proc. Int. Conf. Comput. Intell. (ICCI), Seri Iskandar, Malaysia, Oct. 2020,
the Introduction of Facial Recognition and Seeking Ways to pp. 51–56, doi: 10.1109/ICCI51257.2020.9247836.
Revitalize it in Korea. [Online]. Available: http://www.itfind.or.kr/ [30] H.-J. Mun and K.-H. Han, ‘‘Design for access control system
admin/getFile.htm?identifier=02-001-190725-000007 based on voice recognition for infectious disease prevention,’’
[13] H. Li, J. Hu, J. Yu, N. Yu, and Q. Wu, ‘‘UFaceNet: Research on multi- J. Korea Converg. Soc., vol. 11, no. 7, pp. 19–24, Jul. 2020, doi:
task face recognition algorithm based on CNN,’’ Algorithms, vol. 14, no. 9, 10.15207/JKCS.2020.11.7.019.
p. 268, Sep. 2021, doi: 10.3390/a14090268. [31] M. Alsawwaf, Z. Chaczko, M. Kulbacki, and N. Sarathy, ‘‘In your face:
[14] L. L. Chambino, J. S. Silva, and A. Bernardino, ‘‘Multispectral facial Person identification through ratios and distances between facial features,’’
recognition: A review,’’ IEEE Access, vol. 8, pp. 207871–207883, 2020, Vietnam J. Comput. Sci., vol. 9, no. 2, pp. 187–202, May 2022, doi:
doi: 10.1109/ACCESS.2020.3037451. 10.1142/S2196888822500105.
[15] K. Kim, B. Lee, and J. W. Kim, ‘‘Feasibility of deep learning algorithms
for binary classification problems,’’ J. Intell. Inf. Syst., vol. 23, no. 1,
pp. 95–108, Mar. 2017, doi: 10.13088/jiis.2017.23.1.095.
[16] R. Girshick, J. Donahue, T. Darrell, and J. Malik, ‘‘Rich feature hierarchies HYUNG-JIN MUN received the B.S. and mas-
for accurate object detection and semantic segmentation,’’ in Proc. IEEE ter’s degrees in mathematics from Chungnam
Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 580–587. National University, South Korea, in 1996 and
[17] R. Girshick, ‘‘Fast R-CNN,’’ in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2002, respectively, and the Ph.D. degree in com-
Dec. 2015, pp. 1440–1448.
puter science from Chungbuk National University,
[18] S. Ren, K. He, R. Girshick, and J. Sun, ‘‘Faster R-CNN: Towards real-
time object detection with region proposal networks,’’ IEEE Trans. Pattern
in 2008. He was an Assistant Professor and an
Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, Jun. 2016. Associate Professor at Yanbian University Sci-
[19] M. Lee, and H. Mun, ‘‘Comparison analysis and case study for deep ence and Technology, China, in 2009 and 2010,
learning-based object detection algorithm,’’ Int. J. Adv. Sci. Converg., respectively. Currently, he is an Assistant Profes-
vol. 2, no. 4, pp. 7–16, Dec. 2020, doi: 10.22662/IJASC.2020.2.4.007. sor with Sungkyul University. His research inter-
[20] H. Chen, Z. He, B. Shi, and T. Zhong, ‘‘Research on recognition method ests include personal privacy, user authentication, network security, AI, and
of electrical components based on YOLO v3,’’ IEEE Access, vol. 7, blockchain technology.
pp. 157818–157829, 2019, doi: 10.1109/ACCESS.2019.2950053.
[21] X. Jiang, T. Gao, Z. Zhu, and Y. Zhao, ‘‘Real-time face mask detection
method based on YOLOv3,’’ Electronics, vol. 10, no. 7, p. 837, Jan. 2021,
doi: 10.3390/electronics10070837. MIN-HYE LEE received the B.S. degree in
[22] M.-H. Lee, J.-Y. Kang, and S.-J. Lim, ‘‘Design of YOLO-based removable computer information engineering from Kunsan
system for pet monitoring,’’ J. Korea Inst. Inf. Commun. Eng., vol. 24, no. 1, National University, South Korea, in 2006 and
pp. 22–27, Jan. 2020, doi: 10.6109/jkiice.2020.24.1.22. 2010, and the master’s and Ph.D. degrees in elec-
[23] J. Redmon and A. Farhadi, ‘‘YOLOv3: An incremental improvement,’’
tronic engineering from Wonkwang University,
2018, arXiv:1804.02767.
[24] W. Fang, L. Wang, and P. Ren, ‘‘Tinier-YOLO: A real-time object
in 2012 and 2018, respectively. She was a Post-
detection method for constrained environments,’’ IEEE Access, vol. 8, doctoral Associate at Kunsan National University,
pp. 1935–1944, 2020, doi: 10.1109/ACCESS.2019.2961959. in 2019. Currently, she is an Assistant Professor
[25] K. M. Lee, H. Song, J. W. Kim, and C. H. Lin, ‘‘Balanced performance with Wonkwang University. Her research inter-
for efficient small object detection YOLOv3-tiny,’’ in Proc. Korean Soc. ests include image processing and AI, biomedical
Broadcast Eng. Conf. Anseong, South Korea: The Korean Institute of engineering, and healthcare.
Broadcast and Media Engineers, Nov. 2018, pp. 117–118.

124618 VOLUME 10, 2022