Group Project

Vital Sign Monitoring System using IoT Cameras

COMP7310 2023 Fall Course Project

Introduction
In this project, you will work to build an end-to-end system for monitoring vital signs (heart rates) using an
IoT camera. The project is related to Remote photoplethysmography (rPPG), a technology that uses
videos of faces to extract health parameters, such as heart rate. Although we don’t cover video-based
sensing in our lecture, rPPG is relevant in the sense that it is another contactless technology for vital sign
monitoring.

Video-based rPPG is a widely studied area. In this project, to simplify the tasks, we limit the scope of the
problem to the following:

1. We assume the subject is not moving (too much), e.g., sitting and working in front of a computer.

2. We assume the user is close enough to the camera so that the image resolution is sufficiently high.
Specifically, we can assume the target is roughly around 1 meter to the camera.

3. We assume the user’s face appears in the center of the image most of the time.

4. Last, we assume good lighting condition.

You are required to team up with (up to) 4 classmates to complete the project. By the end of this project,
you will be able to learn:

How to extract heart rate from videos of a face;

How to use an IoT data and fetch video data from it;

How to connect an IoT device to a server via a wireless link and how to use MQTT for IoT data
communication;

How to visualize data properly;

How to evaluate a system and analyze its performance.

💡 Note: If the above topic doesn’t excite you too much, you are allowed to propose your own idea.
However, please write to the instructor first to get approval if you want to propose your own
topic.

Tasks
We will provide each group one IoT camera (ESP32 Wrover CAM) for your usage throughout the project.
While it is also possible to use the camera on your laptop/smartphone to finish the project, having a real
IoT widget makes it feel more real and fun.

Group Project 1
 We can manage the development into four parts:

(1) Data Acquisition: As the first step, you will need to fetch data from the IoT camera and, if necessary,
perform certain local processing on the IoT device. You need to familiarize yourself with the IoT camera
and its specifications, and then develop your code to fetch data from the camera. Your code may need to
support flexible configurations of parameters such as sampling rates. We will give you a simple example
of camera web server for your understanding of ESP32 utilisation.

(2) Data Transportation: You are then required to send the video data from the IoT camera back to a
server. In this project, you are suggested to use your own laptop to serve as a server. And you are
required to use the MQTT protocol for data transportation. The data should be transmitted via a wireless
connection.

(3) Data Analytics: Once you have the data, now we need to think about the core question How to
design an efficient and effective algorithm to extract heart rates from videos of a face? To do so, you will
need to first understand the concept and key principle of heart rate estimation from video data and then
come up with your algorithm design. You can either employ a learning or non-learning algorithm for this
project. However, non-learning design is strongly recommended due to the limited dataset and computing
resources available.

We have also provided a small dataset for your usage, mainly for the purpose of evaluating your design,
rather than for training. The dataset comes with ground truth labels.

(4) Data Visualization: Finally, you are going to build a frontend dashboard to visualize your results,
including the real-time videos and the extracted heart rate information. The dashboard can also serve as
a control panel to start/stop the system and change its configuration. It can be built as a Web App, or a
Desktop/Mobile App.

Finally, you are required to put everything together and deliver an end-to-end system that runs in real
time.

Evaluation
The evaluation of your system must include at least two parts:

(1) Offline evaluation using the provided dataset: You need to run your algorithm over the provided data
with ground truths to report the accuracy and detection rate for heart rate estimation.

The test dataset contains 8 pieces of sample videos, which last one minute each. You can download the
dataset along with a jupyter notebook which can help you get to know the dataset more quickly. The
download link is given below.
Group Project
The two evaluation metrics are defined as below.

Error: The Absolute Error between your estimated heart rates and the ground truth heart rates.

Group Project 2
 Detection Rate: Given a certain period of time (e.g., a certain video), how often can your algorithm
detect and report the heart rates.

(2) Online evaluation: Evaluate your real-time end-to-end system by

plotting a graph showing the detection results over 1.5 min for each of your team members;

reporting the respective detection rate for each testing case as well as an overall detection rate for all
the team members.

You can report more metrics, for example, the run-time latency, and system complexity, etc.

Additional Support
About Camera
To help you come in handy with ESP32 Wrover CAM more quickly, please follow the tutorial in the given
link.

Getting Started with the Freenove ESP32-Wrover CAM Board | Random Nerd Tutorials
Get started with the Freenove ESP32-Wrover CAM board. It features an OV2640 camera, several
GPIOs to connect peripherals, and a built-in programmer with a USB connector.

https://randomnerdtutorials.com/getting-started-freenove-esp32-wrover-cam/

Any related setting up questions can be posted on Moodle.

When you flash your device in succeed, you can open your browser with the camera server front-end. You
can change the parameter settings on the left side and get more familiar with the IoT device.

About Dataset
To facilitate the development and validation of your algorithm, we will provide a small dataset, extracted
from the MMPD: Multi-Domain Mobile Video Physiology Dataset, with special support to this course from
Jiankai Tang and Yuntao Wang from Tsinghua University.

Group Project 3
 Note: To use this dataset in your project, you are required to make the following statement in your
project report:

We hereby acknowledge that the provided data from the MMPD dataset is used
for and within this project only. All the data that we have downloaded and used will
be removed after the completion of this course. [Put down all your team members’
names.]

You may also find the https://github.com/ubicomplab/rPPG-Toolbox useful for this project, which is also
developed jointly by the authors of MMPD dataset.

About MQTT
Message Queuing Telemetry Transport (MQTT) "lightweight" communication protocol based on the ()
mode, which is built on the TCP/IP protocol and was released by IBM in 1999.
The biggest advantage of MQTT is that it provides real-time and reliable messaging services for
connecting remote devices with very little code and limited bandwidth.

As a low-overhead, low-bandwidth instant messaging protocol, it has a wide range of applications in the
Internet of Things, small devices, mobile applications, etc.

1. ESP32 + MQTT

Tutorial:
Control ESP32 from ANYWHERE in the World - Step-By-Step
Tutorial: https://www.youtube.com/watch?v=z53MkVFOnIo
Introduction of MQQT: https://www.youtube.com/watch?v=p3vJxGKWDIg

Using ESP32 as a MQTT client to publish data

https://iotdesignpro.com/projects/how-to-connect-esp32-mqtt-broker
https://github.com/CurlyWurly-1/ESP8266-WIFIMANAGER-
MQTT/blob/master/MQTT_with_WiFiManager.ino

API document: https://pubsubclient.knolleary.net/api#loop

2. Install MQTT On Your Device

AWS IoT | ESP32CAM working as a publisher on

MQTT: https://www.youtube.com/watch?v=7_3qbou_keg

Group Project 4
 MQTT + AWS: https://www.youtube.com/watch?
v=SDrkv2hUzAc&list=PLnq7JUnBumAzGQNb04Fufs9jlJQos-
Txm

AWS IoT | Python MQTT Client for Multiple

ESP32CAM: https://www.youtube.com/watch?v=-0L0PrfuSEo

ESP32 + UWB | Indoor Positioning + Unity

Visualization: https://www.youtube.com/watch?v=c8Pn7lS5Ppg

Github: https://github.com/0015/ThatProject
3.1 Install MQTT Broker

Video: https://www.youtube.com/watch?v=AD1YvjmRiR4
Doc: https://subscription.packtpub.com/book/application-
development/9781787287815/1/ch01lvl1sec12/installing-a-mosquitto-broker-on-
macos

Mosquito man: https://mosquitto.org/man/mosquitto_pub-1.html

For windows: https://www.youtube.com/watch?

v=hyJhKWhxAxA&t=0s

Group Project 5
 For ubuntu: https://www.youtube.com/watch?v=KOohvO4ngTw

For AWS: https://www.youtube.com/watch?

v=SDrkv2hUzAc&list=PLnq7JUnBumAzGQNb04Fufs9jlJQos-
Txm
3.2 Build Your Own Client
This is a tool to test the MQTT as a client (http://mqtt-explorer.com/) on Mac ( // The tool is not good for
high-speed message transmission testing. For such a case, using python to implement client:
Using python to implement a MQTT cklient for data receiving.

Tutorial:
https://www.youtube.com/watch?v=kuyCd53AOtg
https://github.com/code-and-dogs/mqtt-python

pip install paho-mqtt

https://github.com/eclipse/paho.mqtt.python#installation

Submission
You are required to submit the following documents for grading:
1) A 4-page report summarizing their design, implementation, and evaluation. The report should include a
contribution statement clarifying the contribution made by each team member, and data usage statement
as mentioned above.
2) 3-minute video introducing your system.

3) Your source code of the entire system.

Grading Criteria
The course project will be graded out of a total of 30 points, distributed among the following components:

1. System Components (8 points): The project should include all four specified components as outlined
in the project instructions.

2. Real-Time Functionality (3 points): The system should be able to run in real-time, ensuring that data
processing and visualization occur without significant delays.

3. Algorithm Performance (8 points): The performance of the heart rate estimation algorithm will be
evaluated based on its accuracy, robustness, and efficiency.

4. System Implementation (3 points): The implementation of the system will be assessed based on
factors such as system architecture, interface design, and code structure. The code should be well-
organized, modular, and adhere to best practices.

5. System Evaluation (4 points): The project should include a comprehensive evaluation of the system's
performance, discussing the accuracy of the heart rate estimation, any limitations or challenges

Group Project 6
 encountered, and potential areas for improvement.

6. Project Report (2 points): The project report should be clear, concise, and well-structured. It should
provide a summary of the design, implementation, and evaluation processes.

The project presentation (the video submission) will be graded separately, weighing 5 points in total.

Conclusion

By following the instructions outlined in this document, your group will be able to successfully complete
the project. Each member should take responsibility for their assigned tasks and collaborate effectively to
ensure the smooth execution of the project. As required by the University, each member will be graded
individually.
Please post on Moodle for any questions. Enjoy, and good luck with your project!

Group Project 7