Smart Crowd Analyzer

Submitted by:
2017-EE-02 Mehrunisa Ashraf
2017-EE-07 Hamda Anees
2017-EE-24 Haris Rafique
2017-EE-36 Shahkar Ul Hassan

Supervised by:
Dr. Ubaid Ullah Faiz
Sir Ali Shafique

Department of Electrical Engineering

University of Engineering and Technology Lahore
 Abstract
This paper proposes a novel system for people counting based on Artificial Intelligence (AI),
which develops algorithms to count and monitor numerous people entering a certain area at
a certain timeframe. This project is a foundation of prior research, coupled with the addition
of embedded features comprising group detection, emotion detection, bi-directional counter,
regular customer detection, staff exclusion along with customer’s physical features such as age,
height, gender determination. The analysis of the physical features of the customers, provides
with an analytical report that helps retailers to optimize their business performance, such as
stock management, public space designing and staff planning. Furthermore, the algorithm will
also comprehend efficient marketing schemes, and provide the retails with sales conversion on
daily, weekly, monthly and yearly bases. The proposed work is aimed to gain deep analysis and
tracking in retail operations. Also, aid investors in tracking their business (especially when they
have chain stores), automate task distribution, and ensure proper working environment.
Contents

Abstract i

1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Brief Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Project Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Problem Statement 2

3 Literature Review 3
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.2 Global use of People Counting Systems . . . . . . . . . . . . . . . . . . . . . 3
3.3 Project Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.4 Classification of people Counting Systems . . . . . . . . . . . . . . . . . . . . 4
3.5 Project Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.6 Project Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4 Project Overview and Objectives 7

4.1 Project Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4.2 Project Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

5 Methodology/Architecture 8
5.1 Door Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.1.1 Bi-Directional Counter . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.1.2 Age Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5.1.3 Gender Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.4 Height detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.5 Emotion detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.6 Group of People Detection . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.7 Regular Customer Detection . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.8 Staff Exclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5.1.9 Heat map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2 Analysis Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2.1 Statistics Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5.2.2 Sales Conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

6 Project Milestones and Deliverables 11

ii
Table of Content iii

6.1 Project Milestone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

6.2 Project Deliverable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

7 Block Diagram 13

8 Flow Chart 14

9 Work Division 15
9.1 Research-Base Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
9.2 Hardware/Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

10 Costing 16
10.1 Project Equipment Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

References 17
Chapter 1

Introduction

1.1 Background
Over the last decade, retailers are facing significant obstacles, such a staff planning, unavailabil-
ity of stocks, accumulation of unwanted items, and the inability to accurately forecast demand,
etc. It has been known, that without adequate preparation and strategy, the attempt to capture
more of the market is useless.

1.2 Brief Overview

Nowadays customer has an abundance of choices and their preferences changes, making it dif-
ficult for retailers, to keep up with the trends as well as customer’s shopping behavior. Complex
retail operations and management is also not an easy task for retailers. All these problems result
in poor performance/sales, and ultimately, result in profit loss. Thus, to prevent these issues, we
propose an effective method, based on crowd features analysis and AI, for better retail opera-
tions, customer satisfaction, and profit.

1.3 Project Proposal

This proposed project is modeled upon a people counter, namely, ”Smart Crowd Analyzer”,
which is a bidirectional wireless crowd analysis device based on a smart networked video-
camera that analyses crowd features using artificial intelligence and deep learning algorithm.
The project is based on prior research with the addition of embedded features such as group
detection, age detection, height detection, gender determination,emotion detection, regular cus-
tomer detection, and staff exclusion. After analysis of the physical traits, an analytical report is
generated, that provides retailers with deep insights analysis and tracking in retail operations,
and comprehends efficient marketing schemes that optimize sales, and business performance.

1
Chapter 2

Problem Statement

Over the last decade, retailers are facing significant obstacles, such a staff planning, unavailabil-
ity of stocks, accumulation of unwanted items, and the inability to accurately forecast demands,
etc. It has been known, that without adequate preparation and strategy, the attempt to capture
more of the market is useless.

This problem is targeting many major manufacturing companies, store managers (supermarket
and shopping malls), retailer, shipping companies, transportation office, hospitals and medical
care facilities etc. When a customer shops from a store, many questions come in mind of re-
tailers such as what items were purchased? At what time? How long did they shop? and so,
that retailers struggle to understand what data to focus on. To make it more difficult, nowadays
trend changes within a couple of seconds. Customer have an abundance of choices to make and
their preferences changes according to the trend at that moment, making it harder for retailers,
to keep up with the comping changes in the trend as well as customer’s shopping behavior.

Complex retail operations and management is also not an easy task for retailers. All these
problems result in poor performance/sales, and ultimately, result in profit loss. Thus, to prevent
these issues, we propose an effective method, based on crowd features analysis and AI, for better
retail operations, customer satisfaction, and profit.

2
Chapter 3

Literature Review

3.1 Overview
The notion of a simple people counter can go a long extends in solving pesky situations we,
as humans, encounter. For instance, a supermarket company, being unaware of how many cus-
tomers visit during the peak hours of the market itself, can transpire into problems such as
attaining lack of analysis regarding which stocks are more coveted, followed by the more un-
popular choices, which ultimately leads to the wastage of stock items, sales stagnation, and may
lead to a loss in overall profit for the company running the market.

3.2 Global use of People Counting Systems

Given how global business has blossomed over the years, it is imperative for various companies
and firms to get to grips with the latest technological advancements that may help in maintain-
ing the development of their respective businesses, no matter how small or trivial the instalment
may seem. In other words, stock handling, as well as stock distribution to different franchises
can be monitored efficiently by knowing how many people are interested in products. Having
the knowledge of how many people come and purchase such items can be of much benefit. An-
other main prerogative of this project is to be of importance in other various locations, such as
the aforementioned football stadiums, in restaurants/ café’s, concerts, libraries, museums etc.
[4] Acquiring information of the number of people being present at such places can prove to
be statistically significant, enabling the owners to survey and compare the influx of the public
at such places.[5] This will allow them to keep track of how these places are doing in terms of
attracting the public and what amendments do they need to make in order for a larger influx of
people.

3
Chapter 3. Literature Review 4

3.3 Project Research

People monitoring and counting is an active area of research and has seen several developments
since the inception of deep learning. The typical people counting processes involved a person
standing with a clicker physically counting people as they passed. As an individual can imag-
ine, this is a job that is tedious and prone to errors. However, as technology has blossomed over
the past few decades, various algorithms to employ people counting is now available, utilizing
different types of cameras, sensors, and trackers. Single beam infrared sensors project a beam
of infrared light across a doorway. When a person breaks the beam of light by passing through
the doorway, their presence is detected. Single beam sensors only offer limited accuracy as they
cannot distinguish multiple people walking together. They also do not record whether a person
is entering or leaving the room.

3.4 Classification of people Counting Systems

Thermal cameras use people’s body heat to detect their presence. They use motion to dis-
tinguish people and can record which direction a person is moving in.[4] However, they can
struggle to count people accurately if they stand still, walk close to each other, and are also
prone to error when, for example, the door of an air-conditioned building is opened on a hot day.

CCTV people counters rely on installing software into existing CCTV systems. As CCTV
systems typically use one camera to count people, they suffer from reduced accuracy due to a
lack of depth perception and interference from environmental factors such as shadows. instal-
lations are typically aimed at security concerns and are often not positioned in the optimal way
for people counting duties. There is also a growing concern over privacy and GDPR compliance
due to the fact CCTV cameras capture images of people’s faces, which enables identification.

Stereo vision people counting systems use two cameras to provide depth perception and im-
proved counting accuracy compared with single-camera solutions. Stereovision is more expen-
sive than other people counting technologies and is still limited by environmental conditions
such as ambient lighting conditions. [4] It must be understood that each method has its lim-
itations, which, due to the nature of the equipment, typically concerns accuracy and privacy
protection of people moving around a space.

3.5 Project Proposal

Thus, in-order to prevent such, we have pitched the idea to design a state-of-the-art people
counter, namely the Smart Crowd Analyser. In simple context it is a real-time system for in-
door people counting, based on a slightly-aligned wireless sensing-video camera. There is a
booming interest in video-based solutions akin to people monitoring, as well as counting in
Chapter 3. Literature Review 5

business and security applications. Unlike the classic sensor-based implementations, these ones
allow for more versatile functionalities, (such as emotional detection, individual tracking as
well as group detection, bi-directional counter, age, height, gender determination, along with
that, regular customer detection and staff exclusion solution has also been employed, for better
functioning of people counter system.

3.6 Project Features

In group counting, the project will have the capability to count large groups of people as well
as individuals. This feature will display and will depict the data on how many families/large
gatherings have been visiting a specific location at a specific time. On the other hand, physical
characteristics (age, height and gender) will result in knowing what age group of the common
people is interested in certain products, coupled with the dominant gender of these people. A
more obvious feature of this analyser is that it will have the capability to distinguish the persons
entering or leaving a particular place, so that a person, is not counted twice in the total amount.
This will also help in time spent by each individual at a shop in, suppose, a shopping mall, and
other tricky places, where finding the number of people may prove to be difficult at hindsight.

3.7 Conclusion
Through this project we gain deep insights into our retail operations and helps in producing
marketing initiatives and develop a comprehensive understanding of customer behavior and in-
teraction within our stores or in shopping malls and public environments.[4] Which will help in
increases sales and create mass productivity, hence, business will flourish. This project will also
aid our investors in tracking their business especially when they have a chain stores.

Our service includes:

• Stock Management

• Sales Conversion

• Visual Surveillance

• Behavioral Analysis of Customers

• Staff Schedule Planner

• Public Space Designing

• Retail Analysis Report

Chapter 3. Literature Review 6

Further improvements can be made in data collection section, such as setting goals, and main-
tain customer record with the products and creating a backup on Cloud. In case of Security, we
can add Tracking detector and an Object detector, in case of theft.

In Public sector, our project can be utilized in such a way, it may help during voting period.
Managing data which will ease the workers and will save time in generating result. Similarly, it
can be used in Administration Sector such as NADRA. Or in daily used public service system
such as, in bus terminals.

Adopting deep learning algorithms, the product features an accuracy of 97% or greater, a smart
crowd analyzer that tops the industry of crowd analysis devices.
Chapter 4

Project Overview and Objectives

4.1 Project Overview

The objective of the project is to develop an algorithm to count and monitor multiple people
and through their traits, we predict the need for availability of that precise (size) of the item and
through analyses of the sales, we comprehend the restocking of items.

4.2 Project Objectives

The goal of this project is to use open-Frameworks C++ application on the Raspberry Pi to
handle the camera input, image abstraction, tracking, and people counting. Some of the features
of the project are as mentioned below:

• Detect the gender of people visiting.

• Detect the age of people with respect to classified age groups.

• Detect the height of customers, for future stock management.

• Detect the regular customers, for bridging the gap between retailers and customers. Also
generating some discounts or priority for them.

• Omit staff from detection, in order to make counting of customers more precise.

• Detect groups of people visiting a shop. Hence, will be able to comprehend the days/ the
month, when families or groups mostly visit a shop/retail.

• Heat-map will help us in public space designing.

• Generate real time stats and analysis report.

• Generate daily, weekly, monthly and yearly chart reports.

7
Chapter 5

Methodology/Architecture

We will use openFrameworks C++ application on the Raspberry Pi handling the camera input,
image abstraction, tracking and people counting. We will:

1. Capture images from the PiCamera (Raspberry Pi connected)

2. Pass them through a series of openCV Algorithms

3. Detect the Gender, Emotions, Age, height and Groups of people by algorithms.

4. Generate real-time stats.

5. Generate daily, weekly,monthly and yearly Chart Report.

The methodology adopted for the features are categorized as the following below:

5.1 Door Counting

5.1.1 Bi-Directional Counter
Through HoG (History of Orientated Gaussians) and SVN (Support Vector Machines) detector
we will compute the tracking. The contour finder passes the bounding boxes of the contours
to the Tracker object which converts them into blobs. The next task is to count them and not
just counting the number of blobs that appear, we require more details about the blobs and
their transition through our camera scene. When a new blob is detected, our system will log its
initial position and will continue to track its current positions. Once it will reach a customizable
evaluation box, the system will then evaluate which direction the blob has been travelling in and
final width of the blob as it will enter the box. [2]

5.1.2 Age Detection

Once facial features (e.g. eyes, nose, mouth, etc.) are localized by PiCamera, their sizes and
distances measured, ratios between them are calculated. Then, face classification is done into
different age categories according to self-made rules. PiCamera will detect facial features live

8
Chapter 5. Methodology/Architecture 9

time and will compare it with models on which the system would be trained. LBP descriptor
variations and a dropout-SVM classifier will be used to increase the accuracy of detection. [1]

5.1.3 Gender Detection

Webers Local texture Descriptor will be used for gender recognition, demonstrating near-perfect
performance on the FERET benchmark. Popular Labeled Faces in the Wild (LFW) benchmark,
primarily used for face recognition, will be used for this project. The method is a combination
of LBP features with an AdaBoost classifier. As with age estimation, the main focus will be on
the Adience set which contains images more challenging than those provided by LFW.[1]

5.1.4 Height detection

Through HoG (History of Orientated Gaussians) and SVN (Support Vector Machines) detector
[3], tracking will be computed. The height of the person will be obtained by measuring the
height of the contour of that person, after getting the coordinates from the bounding box pixel
height is estimated. Afterward, some correlation could be estimated between pixel height and
real-world height. [2]

5.1.5 Emotion detection

We will examine Visual features of image. We will use Kaggle’s Facial Expression Recognition
Challenge and Karolinska Directed Emotional Faces (KDEF) datasets for detecting emotions.
Corpus Data consists of fifteen actors in sessions with markers on the face, head, and hands,
which provide detailed information about their facial expression and hand movements. The
actors show selected emotions and also improvised scenarios designed to express specific types
of emotions (happiness, anger, sadness, frustration and neutral state). So, training our system
with these models will help us to detect and classify the emotions in real time as well. [6]

5.1.6 Group of People Detection

HoG (History of Orientated Gaussians) and SVN (Support Vector Machines) detector will help
to compute the tracking (similar to counter and height detection). In this case, contour depen-
dence on each other is checked with time. From the initial position to the final position. If their
contours have some relatable movement and are close to each other, it means that they are of the
same group. Hence, this way, a group of peoples in the area will be detected.

5.1.7 Regular Customer Detection

Datasets will be generated containing the different faces of customers coming inside. Then, if
that person comes again and his facial features will be matched with previously-stored datasets.
It will be deduced that the person is a regular customer in a retail store.And we can also Find
regular customer by tracking the WiFi Ip of the customer and if the person with same WiFi ip
enters again , regularity would be found.

5.1.8 Staff Exclusion

This solution will work same as to Regular Customer Detection. Dataset consisting of staff’s
faces will be provided to the system. Then, if the staff member enters or leaves the store, his face
Chapter 5. Methodology/Architecture 10

will be detected and will be matched with previously-stored datasets. If the following match is
true, his entry/exit will be excluded from the total visitor traffic count, delivering real visitor
count. With this technique, we will be receiving only pure conversion results

5.1.9 Heat map

Heatmap helps us in occupancy management. It shows the area more filled and the path the
customers take. Which side of shop/platform attracts them more. We can generate heat map
with WiFi IP of customers or by camera . For camera we will have to use multicam. This will
help in occupancy management.

5.2 Analysis Report

5.2.1 Statistics Generation
Data will be displayed in Chartjs.org. ChartJS is a lightweight javascript library that uses
HTML5 and Canvas to generate and render charts. It will display data in a number of different
chart types such as pie, bar, line and polar. It will obtain data from the database (MySQL, PHP,
and Javascript) and then generate, display and update charts. This will include the outcomes
of door counting and the business trends with the efficiencies of stocks etc. This will be our
final outcome which can be delivered in MOBILE APP or as a monthly report. The following
services will also be provided:

• Stock Management

• Staff Management

• Discount/Deals Management

• Business Trend Monitoring

5.2.2 Sales Conversion

Sale conversion is a ratio of number of people doing shopping to number of people entering
the shop. It tells about the effectiveness of stock . Number of people entering the shop will
be counted by door counter and number of people doing shopping can be counted by tracking
shopping bags or number of transactions being made at counter. We can use any of this method
to count number of people doing shopping.
Chapter 6

Project Milestones and Deliverables

6.1 Project Milestone

F IGURE 6.1: Working Progress of Methodology

6.2 Project Deliverable

The project provides its users with vivid of benefits such as:

• Combination of embedded features such as group counting, age, height, gender determi-
nation, regular customer detection, staff exclusion.

• In Group Detection, our study has the capability to count groups of people rather than
picking them off as individuals. It will comprehend the number of families/groups that

11
Chapter 6. Project Milestones and Deliverables 12

visit a specific location and a specific time and using Machine Learning algorithms may
introduce family/friends discounts on specific days/timings.

• The People Counting algorithm will distinguish the person from entering and leaving a
particular place, so that they are not counted twice in the total.

• Physical traits of crowd such as age, height, and gender, will result in generating data,
such as comprehending what sort of age group is interested in certain products, coupled
with the dominant gender of the people.

• Furthermore, it will generate an accurate retail statistics report, monitoring the transac-
tions, number of visitors, and management of staff schedule, and stock, also providing
public space design suggestions and visual surveillance service, on daily/weekly/month-
ly/yearly basis.

This study hopes to benefit its retailers, with deep insights into retail operations and motivate
effective marketing initiatives that would increment their sales. This study is also helpful for re-
searchers and socialists, helping them develop a comprehensive understanding of customer be-
havior and interaction within stores or in shopping malls and public environments. This project
also targets enthusiastic investors, interested in tracking their business especially when they have
chain stores.
 Chapter 7

Block Diagram

Database
Age and
Gender
Estimation
Image Face Normalized Feature
Aquisition Detection Images Extraction

Camera Distortion Height

Calibration Compen- Mea-
Parameters sation surement
PiCamera Generate
Statistic
Algorithm
Supervised Report
Proxemics Cluster for each Group
Trajectories Correlation
Extraction Solution Application Detection
Clustering
separately

Face
Image Face Face
Feature
Aquisition Detection Recognition
Extraction Regular
Customer
Detection
Training
Set

13
Chapter 8

Flow Chart

Acquiring relevant datasets (start)

Age detection (1) Gender detection (2) Height detection (3)

Group detection (4)

Statistics generation (5)

Sales conversion (6) Heatmap generation (7)

Completion

14
Chapter 9

Work Division

A work-division strategy was introduced for implementing on the project. The purpose of this
strategy is to distribute tasks fairly among each member according to their capability. After
proper planning and consultation, we categorized our project work into two main fractions:-

1. Research-based

2. Hardware/Implementation

9.1 Research-Base Work

This deals with all the effort and study before the implementation of the project. All members
were assigned different part of research work that they will also be implementing later on. A
comprise and thorough research and analysis was done before further execution of the project.
Meeting and detailed discussion hours were also arranged to achieve the goal.

Member Name Assigned Work

Mehrunisa Ashraf Case Study of Sales Analysis
Hamda Anees Case Study of Retail and Business Analysis
Haris Rafique Case Study of Bi-directional Counter, Regular Customer, Staff Exclusion
Shahkar Ul Hassan Case Study of Age,Gender,Height,Group Detection

9.2 Hardware/Implementation
After careful planning and research work, each member started implementation of their follow-
ing tasks. Meetings and detailed online discussion hours were also arranged to achieve the goals.

Member Name Assigned Work

Mehrunisa Ashraf Sales Analysis Compilation
Hamda Anees Retail and Business Analysis Compilation
Haris Rafique Bi-directional,Regular Customer,Staff Exclusion Compilation
Shahkar Ul Hassan Age,Gender,Height,Group Detection Compilation

15
Chapter 10

Costing

The table below shows the price of the hardware equipment and materials purchased for the
completion of the project. This list is made after carefully analyzing the project requirements
and the market values of the materials.

10.1 Project Equipment Details

Item Name Type No.of Units Per Unit Cost Total

NVIDIA Jetson Nano Development Kit Equipment 1 30,000 30,000
USB Camera Equipment 1 22,000 22,000
HDMI Display Equipment 1 15,000 15,000
NVIDIA Jetson Nano Accessories Kit Equipment 1 3000 3000
Stationary Miscellaneous 1 5000 5000
Printouts and Documentations Miscellaneous 1 5000 5000

16
References

[1] H. Bischof B. Wachmann, W. Kropatsch. ”evaluation of people counting systems”, january

2001. [Online]. Available:https://www.researchgate.net/publication/
246112302_Evaluation_of_people_counting_systems, Accessed June 20,
2020.

[2] M. Akcay H.H. Cetinkaya. ”people counting at campuses”, procedia - so-

cial and behavioral sciences, vol. 183, pg. 732-736,may 13, 2015. [On-
line]. Available:https://www.sciencedirect.com/science/article/pii/
S1877042815030967?via%3Dihub, Accessed June 20, 2020.

[3] Nithya Roopa. S. ”emotion recognition from facial expression using deep learn-
ing”,international journal of engineering and advanced technology (ijeat), pg.2249 – 8958,
vol. 8, issue 6. August 2019.

[4] Dr Anuradha S G Shivashree G. ”crowd analysis using computer vision techniques”, in-
ternational journal of engineering research in computer science and engineering (ijercse)
,vol 5, issue 4, april 2018. [Online]. Available:https://ieeexplore.ieee.org/
document/5562657/, Accessed June 20, 2020.

[5] G. Levi T. Hassner. “age and gender classification using convolutional neural net-
works”, ieee workshop on analysis and modeling of faces and gestures (amfg),
ieee conference on computer vision and pattern recognition (cvpr),boston, 2015.
[Online]. Available:https://talhassner.github.io/home/projects/cnn_
agegender/CVPR2015_CNN_AgeGenderEstimation.pdf, Accessed June 20,
2020.

[6] R. Zhang Y. Zhao, J. Qi. ”cbhe: Corner-based building height estimation for com-
plex street scene images”, www ’19:the world wide web conference, pg. 2436-2447,
may 13–17, 2019. [Online]. Available:https://dl.acm.org/doi/10.1145/
3308558.3313394, Accessed June 20, 2020.

17