FACE DETECTION AND COUNTING

Shyam Sundar Singh1 2

, Vineet Singh
1
Springer-Verlag, Computer Science Editorial, Heidelberg, Germany
{alfred.hofmann,ralf.gerstner,anna.kramer}@springer.com
2
Springer-Verlag, Technical Support, Heidelberg, Germany
[email protected]

Abstract. This study investigates the use of face detection and counting
techniques in the crowd, with an importance on the Viola-Jones algorithm,
methodologies, covering techniques such as deep learning-based
approaches, feature extraction, and one-to-one or many-to-one matching
algorithms. The Viola-Jones method is well-known for its speed and
accuracy in face identification, making it a good choice for real-time
applications. The review highlight the relevance of adaptation to changing
environmental conditions, occlusion resistance, and real-time performance
in face identification systems. The analysis highlights the need of including
explainability and moral issues into these systems. Future research
recommendations include improving adversarial adaptability,
experimenting with multi-modal fusion approaches, and establishing
continuous learning mechanisms. The challenges, trends, and future
directions in this field are also discussed and the documented in the paper.
Keywords: face detection, face Counting, Viola-Jones algorithm, deep
learining

1 Introduction

The goal of people counting systems is to automatically estimate the population in

public or enclosed spaces [1]. The objective is to develop methods and systems for
accurately and efficiently recognizing and counting the number of human faces in
images or video frames. These systems find application in various fields such as
malls, traffic management, and auditoriums, where large crowds are present. The
aim is to provide an automated system that detects and counts people, ensuring
accuracy while maintaining speed and efficiency.
Face Detection is a branch of computer science that identifies the size and
placement of human faces in random photographs. It recognizes and detects
characteristics that resemble the human face while ignoring everything else, such
as the body, trees, automobiles, and buildings. Understanding Human face
recognition is an important and growing research subject in computer vision [2].
Machines can distinguish things in photos and movies thanks to computer vision.
Machines with CV skills identify and classify many items faster than humans.
Human face identification and verification are major study fields in the computer
vision field, and it has various applications, such as security cameras in malls and
streets [3]. Face Detection determines if an image contains faces or not and if
faces are discovered in the picture, the position with a bound box for each face in
the image is returned. Face detection is a fundamental component of face
verification in a variety of applications.
Deep neural network (DNN) advancements pose the challenge of massive demand
for data annotations. However, collecting item-level bounding box annotations,
which are often required for training DNN-based object detection algorithms, is
both expensive and time-consuming, especially for photos containing hundreds of
objects [4].

2 Literature survey

2.1 History Perspective

Face identification has evolved dramatically over time, from early rule-based
algorithms and handmade feature approaches to the transformational age of deep
learning. Eigenfaces and feature-based techniques established the groundwork in
the 1980s and 1990s, with the pioneering Viola-Jones algorithm following in
2001. This approach, which uses Haar(The algorithm employs Haar-like features
to represent local image characteristics)- features and cascaded classifiers, enabled
real-time face identification. Machine learning pioneered techniques such as
Support Vector Machines (SVMs). However, the scene evolved dramatically with
the introduction of deep learning in 2012. Convolutional Neural Networks (CNNs)
and later models such as YOLO and SSD accelerated face detection into real-time
applications. Ongoing developments include developing deep learning
architectures, tackling issues such as posture variations, and researching new
datasets, indicating a promising future for face identification [5]. In Fig.1, it's
evident that the landscape of image processing underwent significant
transformations throughout the 2010s.
 3

16.4
17
13 11.2
9 7.4 6.7
5
5 3.57
Error %

1
2012 2013 2014 GoogLe ResNet- Human
AlexNet[ Clarifia VGG-16 Net-19 152 [11]
7] [8] [9] [10]
Se- 16.4 11.2 7.4 6.7 3.57 5
ries
1

Model &Years

Fig 1. Error based years [5]

The acceptable categorization error rate in this period was approximately 25%.
However, with the introduction of AlexNet in 2012, a deep convolutional neural
network (CNN), this rate dramatically improved to 15.3%, marking a substantial
milestone as it surpassed existing algorithms by more than 10.8%. Notably,
AlexNet's performance secured it the winning title in the ILSVRC that year.
Subsequent advancements in the field further refined accuracy: ZFNet achieved an
error rate of 14.8% in 2013, GoogLeNet/Inception reduced it to 6.67% in 2014,
and ResNet achieved an impressive 3.6% error rate in 2015. This progression
underscores the rapid evolution and impact of deep learning in image processing
[5].

2.2 Traditional Approaches

Traditional approaches to face identification evolved throughout time, before the

dominance of deep learning techniques. These approaches used handmade
features, rule-based systems, and traditional machine learning algorithms.
Template matching, an early approach, entailed comparing a specified face
template to various sections of an image. Eigenfaces and other feature-based
approaches rely on techniques like Principal Component Analysis to detect
specific facial traits. Local Binary Patterns (LBP) and Histogram of Oriented
Gradients (HOG) used texture and gradient information, respectively, to recognize
faces. The revolutionary Viola-Jones technique, unveiled in 2001, used Haar-like
features with a cascade of Ad boost-trained classifiers to enable real-time face
identification. Colour-based techniques, rule-based systems, and other classic
methods have all contributed to the subject. However, these techniques struggled
to deal with differences in scale, posture, lighting, and occlusions. The limitations
4

of old approaches led the door for the birth of deep learning, namely convolutional
neural networks, which have subsequently revolutionized face identification with
increased accuracy and robustness [6].

2.1.1 Viola Jones Algorithm

The Viola-Jones algorithm, created in 2001 by Paul Viola and Michael Jones, is a
landmark face identification approach recognized for its efficiency. It uses Haar-
like characteristics and integral pictures to perform fast computations. The
technique distinguishes between face and non-facial areas using a cascade of
Adaboost-trained classifiers. Its cascade structure enables rapid rejection of non-
face regions, making it ideal for real-time applications such as video surveillance
and facial recognition. Despite being an early approach, Viola-Jones laid the
groundwork for contemporary face detection techniques [7]. InFig.2 the Viola-
Jones method uses four Haar features: edge, linear, centre, and diagonal [7].

Fig.2. Four Haar Features [7]

2.2.2 Feature Based techniques and limitations

Feature-based face detection approaches, which are a fundamental strategy in the

growth of facial recognition systems, entail identifying particular facial
characteristics such as the eyes, nose, and mouth. However, these approaches have
significant drawbacks. Their sensitivity to changes in position, lighting conditions,
and facial emotions might compromise their performance in real-world
applications. Furthermore, manual feature engineering, a critical component of
these strategies, necessitates domain knowledge and may lack adaptation to varied
datasets. The inherent constraints of dealing with variances, as well as the
 5

requirement for user intervention, have driven a shift toward more complex and
automated technologies, such as machine learning and deep learning, with the goal
of improved face identification accuracy and resilience [6].
6

2.3 Machine Learning based approches

Machine learning methods have been useful in improving facial detection

algorithms. Face recognition technologies progressed from rule-based and
handcrafted feature techniques to more data-driven and automated approaches as
machine learning gained popularity. Support Vector Machines (SVMs) emerged
as a key tool throughout this evolution. Faces were identified using SVMs, which
learnt discriminatory patterns from attributes extracted directly from photographs
[15].
Support Vector Machines excel at determining appropriate decision boundaries
across classes, making them useful for facial categorization problems. These
models are trained on labeled datasets, learning to discriminate between face and
non-facial areas using characteristics extracted from the training pictures. SVMs'
flexibility and generalization skills helped them gain popularity in the early phases
of incorporating machine learning into face identification systems [16].
While SVMs were a big step forward, the later advent of deep learning, namely
Convolutional Neural Networks (CNNs), altered the face identification
environment even more. CNNs outperformed by automatically learning
hierarchical features from raw pixel data, eliminating the need for laborious
feature engineering. This change to deep learning architectures represented a
paradigm shift, resulting in the creation of extremely accurate and efficient face
identification models.
 7

3 Methodology

3.1 Deep Learning Technique

Deep learning, namely Convolutional Neural Networks (CNNs), has transformed

face identification by automating feature extraction from raw visual data. CNNs
extract hierarchical features, allowing for precise and efficient detection of face
patterns. Object identification frameworks such as Faster R-CNN and SSD use
region-based CNNs to improve bounding box prediction for faces. Real-time
systems, such as You Only Look Once (YOLO), split pictures into grids to
forecast bounding boxes and class probabilities. Innovations such as Multi-task
Cascaded Convolutional Networks (lutional Networks (MTCNN) and Focal Loss
in Retina Net demonstrate continued progress. These techniques improve
accuracy, speed, and flexibility, making deep learning crucial in face identification
for applications ranging from surveillance to human-computer interaction [8].

Fig.3 Face recognition block diagram [8]

The face recognition block diagram in Fig. 3 illustrates the core components of a
typical face recognition system. It typically includes stages such as face detection,
feature extraction, and classification. Face detection identifies facial regions,
feature extraction captures distinctive facial attributes, and classification matches
them against known identities for recognition.
8

14%

5% CNN
Auto Encoder
47% DBM
14% GAN
Hybrid
Reinforcement Learning
11% Other

5% 4%

Fig.4 Different deep learning architectures for face recognition [8]

In Fig.4. different deep learning architectures for face recognition encompass

CNN (Convolutional Neural Networks) for feature extraction, Autoencoders for
unsupervised learning, DBM (Deep Boltzmann Machines) for probabilistic
modeling, GANs (Generative Adversarial Networks) for generating realistic faces,
Hybrid models combining various techniques, and Reinforcement Learning for
adaptive recognition strategies, offering diverse approaches to address face
recognition challenges.

3.1.1 Convolutional Neural Network

The Convolutional Neural Network is the most widely used deep learning
technique for image identification, classification, pattern detection, and feature
extraction from images. There are several types of CNN algorithms. However, two
kinds are shown here to describe the CNN algorithm. Two types of algorithms
exist: feature extractor and classifier. CNN's name is derived from Convolution is
a mathematical linear process between two matrices. In CNN, one matrix
represents the picture, while the other is the kernel (operator). An image is a
matrix with either a single channel (gray-scale) or three channels (color), with
each entry representing a pixel. The dimensions of the picture matrix are
(HxWxD). Most popular kernels, including edge detectors and other operators,
employ a 3 × 3 size kernel. However, M and N are arbitrary. The term D refers to
the kernel's depth or dimension. In this example, H represents height, W
represents width, and D represents the RGB color channel for the image. The
grayscale picture channel has one, whereas the color image channel has three
RGB color channels. The kernel (operator) is a matrix with dimensions of MxNxD
[9].
 9

Fig.5. CNN architecture consists of four layers: convolutional, pooling, non-linear,

and fully linked layers. The parameterized first two layers. The first is a basic
CNN architecture, whereas the other two are not parameterized. Parameterized
layers include convolutional and fully linked layers. Non-parameterized layers
include both nonlinear and pooling layers. However, the design may alter based on
the issue needs. Researchers modified the CNN design and created several FR
architectures, including VGGNet and Google Net. Deep facial recognition
algorithms typically require supervision to function well [9].

Fig.5 Basic CNN architecture [8]

3.2 Face Counting Techniques

Face counting algorithms have emerged to solve the challenging challenge of de-
termining the number of faces in pictures or video frames. This branch of com-
puter vision has applications in a variety of domains, including crowd monitoring,
surveillance, and social behavior analysis. The strategies used in face counting are
diverse, with each adapted to address distinct issues.

Density-based approaches use statistical techniques to determine the spatial distri-

bution of faces in a picture. Kernel Density Estimation (KDE) and Gaussian Mix-
ture Models (GMM) are examples of density-based approaches that use feature
distribution to estimate face counts. These techniques are especially beneficial in
congested environments when standard counting becomes difficult [5].

Regression-based techniques take a different approach, using machine learning

models to directly estimate face count based on picture attributes. These models
are trained using labeled datasets to understand the intricate link between visual
attributes and face counts. Regression-based approaches are flexible and scalable,
allowing for adaption to various datasets and settings [10].
10

Deep learning, particularly Convolutional Neural Networks (CNNs), has emerged

as a dominant paradigm in face counting. CNNs use hierarchical characteristics to
detect detailed patterns in congested surroundings. Deep learning models' capacity
to automatically understand and represent complex relationships in data has con-
siderably increased the accuracy of face counting systems. Researchers investi-
gated alternative architectures and configurations to improve the performance of
CNNs for face counting applications [8].

Detection and aggregation methods combine face detection models and counting
procedures. Individual faces in an image are identified by models such as Faster
R-CNN or Single Shot MultiBox Detector (SSD), and their counts are combined
to calculate the overall face count. This method is helpful in cases requiring exact
face localization, such as security and surveillance applications [1].

Crowd counting datasets are essential for developing and evaluating face counting
systems. Datasets such as ShanghaiTech and UCF_CC_50 include annotated in-
stances of crowded settings, making model training and testing easier under a vari-
ety of scenarios. These datasets improve the resilience and generalizability of face
counting techniques .

Real-time video analysis adds a dynamic component to face counting, allowing

systems to constantly monitor face presence in busy areas. This capacity is espe-
cially useful in scenarios requiring real-time insights into crowd dynamics, such as
public events, transit hubs, and smart city applications .

Face counting presents a wide variety of challenges. Variations in crowd density,

occlusions, and the necessity for precise counting in various settings provide chal-
lenges for researchers. The dynamic nature of real-world contexts necessitates
adaptation and robustness in face counting systems. Occlusions, which hide the
face partially or completely, make precise counting even more difficult. Further-
more, differences in lighting conditions, viewpoints, and the presence of various
demographic factors add levels of complexity.

Fig.6. depicts how to overcome these issues, ongoing research is investigating

unique ways that combine traditional methodology with the potential of deep
learning. Hybrid models, which combine the advantages of old approaches and
new deep learning architectures, seek to improve the accuracy and efficiency of
face counting systems across a wide range of applications.
 11

Fig.6. to count face counting systems

12

4. CURRENT CHALLENGES AND FUTURE DIRECTIONS

The field of face identification and counting has various modern issues that drive
academics to continue exploring and innovating. One pressing worry is the detect-
ing systems' capacity to adapt to a wide range of ambient variables, from illumina-
tion and weather fluctuations to complex background clutter. Furthermore, the
complex challenge of controlling occlusions and overlapping faces in crowded set-
tings necessitates advanced detection techniques. Achieving real-time speed with-
out sacrificing precision is a constant challenge, especially in dynamic contexts.
Furthermore, establishing the generality of face detection models across numerous
populations, including various age groups, races, and gender presentations, is criti-
cal for equitable and impartial performance.

Researchers are ready to examine cutting-edge opportunities for development as

they design future routes. Adversarial robustness is a significant emphasis, trying
to strengthen face detection systems against manipulative attacks while improving
security and dependability. The integration of information from several modali-
ties, including as optical and thermal data, is a potential field for improving accu-
racy, particularly under adverse climatic settings. Prioritizing explainability and
ethical concerns in face detection algorithms is critical, since it requires openness
and responsibility in algorithmic decision-making.

Continuous learning and adaptation procedures are intended to allow face detec-
tion systems to adjust dynamically to changing situations without requiring regular
retraining. Furthermore, the development of privacy-preserving approaches, such
as federated learning and on-device processing, is critical for addressing privacy
problems in face detection applications. The development of benchmark datasets
that include a wide range of demographics is critical for ensuring that face detec-
tion algorithms generalize well across varied groups, hence reducing biases and
promoting inclusion [12].

Human-centric evaluation measures are expected to play an important role in im-

proving the assessment of face detection system performance. These KPIs will be
more closely aligned with user experiences, taking into account the impact of false
positives and false negatives on end users. Simultaneously, an investigation of the
societal impact of broad face detection use would help academics get a thorough
knowledge of topics such as surveillance, civil liberties, and unforeseen conse-
quences [13]
 13

5. PERFORMANCE EVALUATION METRICS

Performance evaluations should be quantitative. The report should show how

many items were accurately recognized and how many false positives were
generated. The assessment should allow one-to-one, one-to-many, and many-to-
one matches. It should also be scalable to bigger test regions or numerous 3D
scenarios without compromising tracking capacity. The system shown in figure
7 and figure 8 includes three types of performance indicators: detection-based,
tracking-based, and perimeter intrusion detection metrics. Detection-based metrics
assess the performance of a System Under Test (SUT) on individual frames of
video sensor data. They do not track the IDs of items during the exam. Each item
is evaluated separately to ensure a match between the SUT and Ground-truth (GT)
system for each video frame. The experiment's performance score is calculated by
averaging individual frame performance over all frames. Tracking-based metrics
compare GT and SUT tracks based on best correspondence, considering both the
identification and entire trajectory of each item during the test run. The best
matches are used to calculate error rates and performance indicators, as explained
below. The perimeter intrusion detection measure detects objects that enter a
certain region [14].

(a) (b) (c)

Fig.7. (a) One-to-one matching, (b) many-to-one matching and (c) one-to-many
matching [17].

In Fig.7(a) One-to-one matching entails comparing attributes of identified faces in

current and prior frames to generate correspondences that help in tracking and
counting. This procedure guarantees that each identified face is appropriately
connected with its counterpart across frames, allowing for more precise
monitoring and counting of persons.

In Fig.7(b) Many-to-one matching, which includes linking several detected faces

in a current frame with a single face or identity in a prior frame. This approach is
beneficial in situations when faces may briefly obscure each other or when several
14

detections refer to the same person. By combining several detections into a single
identification, tracking, and counting algorithms become more accurate.

Fig.8.One-to-one matching, many-to-one matching, and one-to-many matching

[14].
 15

6. CONCLUSION

To summarize, the landscape of face identification and counting presents consid-

erable obstacles as well as interesting areas for future study. Key findings under-
line the need of flexibility to multiple environmental situations, resolving chal-
lenges like as occlusions and overlapping faces, attaining real-time performance,
and assuring generalizability across populations. The combination of explainabil-
ity and ethical issues appears as a vital component in the development of transpar-
ent and accountable face detection systems.

Future research should focus on improving adversarial resilience, investigating

multi-modal fusion for greater accuracy, and building continuous learning mecha-
nisms that can dynamically adjust to changing situations. Integrating privacy-pre-
serving approaches and creating benchmark datasets with diverse demographics
are critical steps toward reducing biases and assuring inclusion. Human-centric as-
sessment criteria, as well as an in-depth analysis of the societal effect of face iden-
tification technologies, can help us gain a better grasp of their ramifications.

Future development recommendations include developing cooperation among re-

searchers, industry stakeholders, and legislators to set ethical rules and standards.
The emphasis on the human-centric component in the evaluation of face detection
systems is critical for connecting technical progress with social ideals. As im-
provements continue, a comprehensive strategy encompassing technological, ethi-
cal, and sociological elements will be required to move the profession ahead re-
sponsibly and ethically.
16

References

[1] E. D. a. L. C. Xi Zhao, "A People Counting System based on Face Detection

and Tracking in a Video," IEEE, vol. Advanced Video and Signal Based Sur-
veillance, p. 69, 2009.

[2] M. P. Tofiq Quadri, " Face Detection and Counting Algorithms Evaluation
using OpenCV and JJIL," in GITS-MTMIAt, Udaipur, Rajasthan, December
2015," vol. 2, p. 42.

[3] M. S. Yehea Al Atrash, "Detecting and Counting People’s Faces in Images

Using Convolutional Neural Networks," IEEE, no. Palestinian International
Conference on Information and Communication Technology (PICICT),, p.
116, 2021.

[4] u. H. ,. X. H. a. L.-P. C. Yi Wang, " A Self-Training Approach for Point-Su-

pervised Object Detection and Counting in Crowds," IEEE TRANSACTIONS
ON IMAGE PROCESSING, vol. 30, no. 2876, p. 2876, 2021.

[5] "https://anyconnect.com/blog/the-history-of-facial-recognition-
technologies," [Online].

[6] Y. S. a. H. C. Jing Huang, "Improved Viola-Jones face detection algorithm

based on HoloLens," EURASIP Journal on Image and Video Processing,
2019.

[7] L. W.-y. Ming, "Face Detection Based on Viola-Jones Algorithm Applying

Composite Features," International Conference on Robots & Intelligent Sys-
tem (ICRIS), p. 45, 2019.

[8] A. A. F. D. S. MD. TAHMID HASAN FUAD, "Recent Advances in Deep

Learning Techniques for Face Recognition," IEEE Access, p. 23, 2021.

[9] P. W. S. C. H. Xudong Suna, "Face detection using deep learning: An im-

proved faster RCNN approach," Elsevier, pp. 42-50, 2018.

[10] J. J. L. C.-J. K. K. Sung Eun Choi, "Age face simulation using aging func-
tions on global and local features with residual images," Expert Systems with
Applications, pp. 80:107-125., 2017.

[11] J. R.-d.-S. R. V. M. C. Gabriel Hermosilla, "A comparative study of thermal

face recognition methods in unconstrained environments," Pattern Recogni-
 17

tion, pp. 45( 7): 2445-2459, 2012.

[12] M. H. Mamta, "A new entropy function and a classifier for thermal face
recognition," Engineering Applications of Artificial Intelligence, pp. 36: 269-
286., 2014.

[13] A. Y. Y. ,. G. John Wright, IEEE TRANSACTIONS ON PATTERN ANALYSIS

AND MACHINE INTELLIGENCE, vol. 31, p. 457, 2, FEBRUARY 2009.

[14] R. B. S. Afzal Godil, "Performance Metrics for Evaluating Object and Hu-
man Detection and Tracking Systems," Tracking Systems, pp. 7972,,3-4 .