University of Omar Al-Mukhtar

Faculty of Engineering
Mechanical Engineering Department

A Computer Vision Model for Classifying and Sorting

Waste in OMU
A Project Submitted in Partial Fulfillment of the Requirements for the
Bachelor Degree (B.Sc) in Mechanical Engineering

By
Muntasarbellah I.M.

Suliman A. M.

By Supervised
Dr. Gebreel Abdalrahman

Mr. Yusef B.

Al Bayda, Libya
March 2023
Abstract

This project leverages Edge Impulse to develop, train, and test a machine vision
model aimed at automating the sorting of two types of waste: bottles and cans. Conducted
at OMU, the project involves creating a comprehensive dataset and employing a TinyML
algorithm to facilitate the sorting process. The trained machine learning (ML) model is
deployed onto an ESP32-CAM module, forming the core of an intelligent sorting device.
The device integrates two servo motors and an ultrasonic sensor to execute the
classification and sorting tasks. Following rigorous testing, the ML model achieved an
impressive accuracy rate of 98%. The sorting device operates efficiently, accurately
executing actions based on the ML model's predictions. This implementation
demonstrates the potential of TinyML and Edge Impulse in creating effective, low-power,
embedded AI solutions for real-world applications.

2
Acknowledgment

We are deeply grateful for the opportunity to complete this project. Our deepest thanks go
to Allah for providing us with the strength and guidance necessary to persevere.

We extend our sincere appreciation to our esteemed supervisor Dr. Gebreel Abdalrahman and
Mr. Yusef B. their patience and insightful guidance were instrumental in shaping this project and
navigating its challenges. His constant encouragement and belief in our abilities kept us
motivated throughout the research process.

We are also incredibly thankful for the unwavering support of our families and friends.
Their belief in us even during challenging times fueled our determination to succeed.

Finally, we acknowledge each other. This project wouldn't have been the same without the
mutual support, collaboration, and friendship we shared. Working together as a team was a
rewarding experience that fostered personal growth and a sense of accomplishment.

In conclusion, we recognize the collective effort that went into this thesis and express our
deepest gratitude to all those who contributed to its completion

3
 Table of contents
Abstract ................................................................................................................................. 2

Acknowledgment .................................................................................................................. 3

List of figures ........................................................................................................................ 6

Chapter one ........................................................................................................................... 7

Introduction ........................................................................................................................... 7

1.1 Machine learning (ML) ................................................................................................... 8

1.2 Problem statement ......................................................................................................... 11

Chapter Two ........................................................................................................................ 12

Literature review ................................................................................................................. 12

Chapter Three ...................................................................................................................... 13

Methodology ....................................................................................................................... 13

3.1 Sample preparation ....................................................................................................... 14

3.2 Designing Impulse ........................................................................................................ 15

3.3 Generating Features ...................................................................................................... 16

3.4 Training model .............................................................................................................. 17

3.5 Testing model................................................................................................................ 18

3.6 Deploying model ........................................................................................................... 18

3.7 Arduino IDE.................................................................................................................. 19

Chapter Four........................................................................................................................ 20

Results and discussion......................................................................................................... 20

4.1 confusion matrix ........................................................................................................... 20

4
4.2 generate feature ............................................................................................................. 20

4.3 System electronic circuit & components ...................................................................... 21

4.4 Conclusion and Recommendations ............................................................................... 24

References ........................................................................................................................... 25

Appendices .......................................................................................................................... 26

5
List of figures

Figure 3.1 Edge impulse processing

Figure 3.2 bottle can in different conditions

Figurer 3.3 A structure fornthe distribution of data set used in this project

Figure 3.4 impulse block (a) input blocks, (b) processing block , (c) learning block

Figure 3.5 image data visualization using edge impulse feature explorer

Figure 3.6 over and under fitting cases by learning curve

Figure 3.7 Servo motors control flowchart

Figure 4.1 Machine learning vision confusion matrix

Figure 4.2 corrected and incorrected label features

Figure 4.3 Schematic diagram of project circuit and components

Figure 4.4 showcase the system after it`s assembly

Figure A.1 preparing of training set

Figure A.2 Preparing of testing set

Figure A.3 create an impulse/building AI model

Figure A.4 feature extraction based on bottle & can category

Figure A.5 Neural Network settings of training

Figure A.6 downloading the Edge Impulse AI model in the form of Arduino IDE library

Figure C.1 Dimension Diagram

6
 Chapter one

Introduction

Effective waste management is a pressing challenge faced in the Middle East and North
Africa (MENA) region. Currently, more than 480 million inhabitants are living in 23 MENA
region countries, and the average waste generation rate is around 1 kg/capita/day. This means
that there will be more than 480,000 tons of Municipal Solid Waste (MSW) generated every day.

Table 1-1 shows the annual waste generation in MENA region in 2019 from [1]. In the
specific case of Libya, the waste statistics for the years 2019 and 2020 indicate a population of
approximately 6,754,507. The average Municipal Solid Waste (MSW) generation per capita per
day during that time was 0.77 kg, resulting in a total waste generation of 1.9 million tons per
year.

Table 0-1 Annual waste generation in MENA region in 2019 from (Thabit, etall.,2022)

while specific statistics on waste generation in Libya and the Arab world are not provided
and available, additional research can be conducted to gather relevant data. This information will
provide valuable context for the project, allowing for a better understanding of the waste
management scenario and the potential impact of the computer vision model in waste
management efforts.

7
 The MENA region, including countries like Libya, grapples with multiple waste
management drawbacks, including limited resources, inadequate infrastructure, and a lack of
data. To address these challenges, innovative solutions and technology are required. In line with
this objective, a project is being undertaken at Omar Al-Mukhtar University (OMU) that
leverages machine learning and computer vision technology.

The primary goal of this project is to develop a computer vision model for waste
classification and sorting. By utilizing machine learning algorithms and computer vision
techniques, the model aims to accurately classify different types of waste materials. This
technology will enable efficient waste sorting and contribute to the overall improvement of waste
management practices at OMU.

By implementing this innovative approach, OMU seeks to enhance waste management

processes, reduce environmental impact, and promote sustainable practices. The utilization of
machine learning and computer vision technology offers the potential to transform waste
management systems, enabling better resource allocation and fostering the transition towards a
circular economy.

The developed computer vision model can serve as a blueprint for waste management
systems in other educational institutions and broader communities. By sharing the knowledge
and insights gained from this project, one can contribute to the advancement of waste
management practices, promoting sustainability, and driving positive environmental change.

The project aims to utilize machine learning and computer vision technology to develop a
computer vision model for waste classification and sorting in OMU. By implementing this
innovative solution, it will be easy to optimize waste management processes, enhance recycling
efforts, and create a cleaner and healthier environment within the university campus. The project
has the potential to serve as a foundation for future waste management initiatives and contribute
to sustainable practices in the MENA region and beyond.

1.1 Machine learning (ML)

Machine learning, situated within the domain of artificial intelligence, is witnessing rapid
growth as it facilitates computers to acquire knowledge from data or prior information, thereby

8
enhancing their performance progressively without explicit programming. This process primarily
revolves around the utilization of algorithms capable of refining the parameters of a model by
discerning patterns inherent within training data or accumulated experiences. These models can
be categorized as descriptive, aimed at extracting insights from data, or predictive, focused on
forecasting outcomes. Significant examples of machine learning algorithms encompass decision
trees, support vector machines (SVMs), and convolutional neural networks (CNNs) [2]. ML
algorithms are categorized based on the nature of the problem and the data given and the desired
outcome of the algorithm, there are four distinct categories of ML algorithms namely,
Supervised learning, unsupervised learning, semi-supervised learning, and reinforcement
learning [3].

Supervised learning: constitutes a fundamental task within the domain of machine

learning, wherein the objective is to learn a function that maps input data to corresponding output
values based on a set of labeled input-output pairs. This approach relies on labeled training data
to deduce the underlying function and is inherently oriented towards achieving specific task
objectives. Common applications of supervised learning encompass classification and regression
tasks. An illustrative example is text classification in sentiment analysis, wherein the algorithm is
trained on labeled data to discern sentiment categories within textual content.

Unsupervised learning: constitutes a pivotal aspect of machine learning, characterized by

the analysis of datasets devoid of explicit labeling, thus operating without direct human
intervention. This approach is fundamentally data-centric, aiming to uncover inherent patterns
and structures within the data. It serves various purposes such as feature extraction, trend
identification, clustering, and exploration of the dataset.

Semi-supervised learning: represents a hybrid approach amalgamating elements of both

supervised and unsupervised methodologies, leveraging both labeled and unlabeled data for
model training. This method finds utility in scenarios characterized by scarcity of labeled data
juxtaposed with abundance of unlabeled data. Notably, it offers potential for superior predictive
performance compared to utilizing solely labeled data. Applications of semi-supervised learning
span various domains including machine translation, fraud detection, and text classification.

9
 Reinforcement learning: serves as a pivotal paradigm within machine learning,
facilitating the optimization of software agents and machines towards achieving optimal
behavior within designated contexts or environments, thereby enhancing operational efficiency.
Rooted in the principles of reward and penalty, its overarching objective revolves around
maximizing rewards or minimizing risks encountered during decision-making processes. This
approach finds applicability in training artificial intelligence (AI) models aimed at augmenting
automation or streamlining the efficiency of complex systems, including but not limited to
robotics, autonomous driving, and supply chain logistics. However, it is not typically favored for
addressing rudimentary problems within the domain.

Applications of machine learning exhibit the capability to expedite data acquisition,

discern patterns, and execute actions autonomously, thereby reducing reliance on conventional
cloud networks. This autonomy stems from the strategic deployment of machine learning
algorithms in proximity to data sources, a paradigm known as edge computing. Edge devices
encompass a diverse array of hardware, including sensors, cameras, mobile devices, and
embedded systems. These devices are characterized by their compact size and low power
consumption, rendering them advantageous for edge computing applications [4].

Tiny ML represents an emerging domain at the intersection of Machine Learning and

Embedded Systems, focusing on the deployment of lightweight models on edge devices. This
field seeks to investigate the feasibility and efficacy of various model architectures for execution
on resource-constrained hardware. The adoption of Tiny ML algorithms offers several distinct
advantages when implemented on edge devices [5][6].

1. The security and privacy of data are addressed through an alternative approach in
TinyML. Instead of transmitting data streams to cloud-based servers for processing,
inference tasks are executed locally on embedded microcontrollers. This localization of
computation ensures that sensitive data remains confined to the device and its
immediate physical environment, bolstering both privacy and security measures.
2. The optimization of power consumption stands as a notable benefit conferred by Tiny
ML algorithms. This efficiency is primarily attributed to the minimal or absence of

10
 data transmission requirements, resulting in markedly reduced power usage compared
to traditional approaches.
3. High availability with low latency: Inference has a local latency of a few milliseconds
and is not affected by network latency or availability.

In this study, Machine vision models, which includes Computer vision (CV) and Machine
Learning (ML) model, is implemented to classify waste at OMU.

1.2 Problem statement

This project aims to develop a computer vision-based system to classify different types of
waste (bottle, can) at Omar AL-Muktar University by using machine learning and image
recognition technology. The machine vision model is created and trained by the Edge Impulse
platform to identify and categorize various waste materials. Then introduce the machine learning
model and deploy it into the hardware such as ESP-32 AI thinker cam, ultrasonic sensor, and two
servo motors.

11
Chapter Two
Literature review

Although the ML is widely used for many applications, a few studies are conducted for using
ML in the waste management.

Olscher et al. [7] investigated the use of spectroscopic methods to detect tracer materials in
plastic waste. These tracers, designed with unique spectroscopic properties, are incorporated into
plastics during production. The study highlights the potential of spectroscopic methods to
enhance plastic waste sorting processes, addressing challenges faced by traditional recycling
methods. the paper provides an overview of spectroscopic methods and materials for marker-
based sorting, emphasizing the need for further research to address economic feasibility and
scalability.

Balagugan et al [8] used the machine learning to select and separate waste. The results showed
mixing waste in bins has a fewer ratio of error using ML.

Singh, S. et al. [9] used a deep learning algorithm for sorting waste. The findings demonstrated
that when deep learning is used, the strategy outperforms other approaches; for example, the
accuracy and specificity of image processing increased by 7% and 11%, respectively.

Balakrishnan et al. [10] developed an Artificial Intelligence (AI) and induction algorithms. using
machine learning techniques can be effectively applied to container sorting that applying the
sorting system. The study concluded that the AI systems can contribute to improving waste
management, reducing the amount of waste sent to sanitary landfills, increasing resources,
participation and money, and protecting the general environment.

Myra.G.has [11] evaluated using AI algorithms in terms of strength and weaknesses. Wastes
classified by the algorithm are aluminum, cans, plastic, cutlery, and plastic bottles. Using
intelligent waste separator (Iws) algorithm do not depend on people. Consequently, avoiding
mixing waste in bins has a fewer ratio of error. However, the capacities of waste separator do not
allow in obtaining information and its response was slow.

12
Chapter Three
Methodology

Edge Impulse [12] is a cloud-based platform that enables the development, training, and
deployment of machine learning models on edge devices. Over 1,000 enterprises worldwide use
Edge Impulse, as it provides an advanced machine learning development experience that can be
applied broadly to embedded devices for sensors, audio, and computer vision applications.
Additionally, the use of TinyML on the Edge Impulse platform can expedite the development
process. Edge Impulse has a variety of professional and industrial use cases, including human
and animal sensing, asset tracking and monitoring, and predictive maintenance.

Typically, the classification ML models used in Edge Impulse are trained on image
datasets to generate predictions. The ML model can be developed using a dataset collected by an
edge device such as a camera, labeled, trained, and then deployed on edge devices, as illustrated
in Fig. 2.1. Edge Impulse's drag-and-drop interface allows for easy addition of additional
sensors.

Figure 3.1 Edeg impulse processing

To use Edge Impulse, an account is required, and an edge device must be connected to
collect data for the Edge Impulse data acquisition process. In this study, the Edge Impulse
platform is used to design the proposed machine learning model in the following steps:

13
3.1 Sample preparation

In this project, a dataset is constructed to segregate waste by collecting pictures and images
to train the model to classify waste in OMU campus. To prepare the data set, several samples of
plastic bottles and Aluminum can from varies brands and conditions were selected some of the
samples are shown in the Fig. 2.2
`

Figure 3.2 bottle and cans in different conditions

A total of 2301 image were captured in different lighting conditions and angle to
ensure variety in dataset. Images are classified approximately to 1000 for each of the two labels
bottle and can and about 250 images for the unknown label.

A training set and testing set are created from the image that are uploaded to the edge
impulse platform. In this study, 80 percent of the dataset were used in training the model while
the remaining 20 percent are used to test the model. Figure 2.3 shows the distribution of dataset
in training and testing set.

14
 Dataset
100%
(2301)

Testing set Training set

20% 80%
(463) (1838)

Figure 3.3 A structure for the distribution of dataset used in this project

3.2 Designing Impulse

The term "impulse" in the context of Edge Impulse refers to a set of actions performed on
input data to prepare it for machine learning, along with the Learning Block that defines the
algorithm for the classification. The impulse design process consists of three main blocks: the
input block, the processing block, and the learning block.

The input block specifies the type of data used to train the model, such as images or time
series of sound, vibration, or movement. In this project, "image data" is selected as the input.

The resize mode for the input images is set to 160x160 with a "fit shortest axis" option,
which standardizes the input image size for the ML model, as shown in Fig. 2.4 (a). The "image"
processing block, depicted in Fig. 2.4

Figure 3.4 Impulse blocks: (a) input block, (b) processing block, (c) learning block

15
 Edge Impulse provides the pre-trained MobileNetV2 model as a classification model. Due
to its reduced size and complexity, the MobileNetV2 model is well-suited for deployment on
embedded systems and mobile devices, such as smartphones. Therefore, this project utilizes the
MobileNetV2 model for training the proposed ML model. As shown in Fig. 2.4 (c), the learning
block in this project employs the transfer learning approach. Transfer learning is a computer
vision technique that allows for the development of a new vision ML model by leveraging the
features of a pre-trained model. This approach enables faster training and the use of a smaller
dataset to develop the new model.

3.3 Generating Features

For generating the features of images, the Edge Impulse uses two different techniques
namely, Gray scale and RGB based. Gray scale is a technique used to generate features from
images. Each pixel in an image is represented by a single value, which represents the intensity of
the pixel. This single value represents the shade of grey, ranging from black (0) to white (255).

When processing images, Edge Impulse uses its built-in feature generation functionality to
extract relevant features from the images. For each image, the feature size is calculated by
multiplying the image size (e.g. 160 x 160) by the number of color channels (one for gray scale).
This results in a feature vector with a specific number of values, such as 25,600 values for a
160x160 image. This feature vector, with 25,600 values, represents the image in a format that
can be used by machine learning models to learn patterns and make predictions.

Enabling the algorithm to learn patterns and accurately classify the type of defect. The
feature data can be visualized and explored using the feature explorer Edge Impulse before the
training process as shown in Fig. 2.5. All the images in the project, along with their
corresponding labels, could be seen in detail using the feature explorer. It can therefore confirm
that the images were correctly labeled, and the features were generated correctly

16
 Figure 3.5 Image data visualization using Edge Impulse feature explorer

3.4 Training model

The proposed ML model is trained using the MobileNetV2 model and various hyper
parameters, such as the learning rate, dataset size, and number of epochs. To assess the model's
performance and prevent over fitting or under fitting, a validation set comprising 20% of the total
dataset is used.

As shown in Fig. 2.6, an over fitted model exhibits good performance on the training data
but poor generalization to other data. Conversely, an under fitted model demonstrates poor
performance on both the training data and other data.

To improve the quality of the model, two key techniques are employed: data augmentation
and auto-balance dataset. Data augmentation involves applying small, random alterations to the
training images, which increases the diversity of the dataset. The auto-balance dataset technique
ensures that the number of images in each label category is balanced, reducing the risk of the
model being biased towards any one label.

17
 These measures help to ensure that the trained model achieves good performance on the
training data as well as effective generalization to other data.

Figure 3.6 Over and Under fitting cases by learning curve [13]

3.5 Testing model

The performance of the ML model is evaluated by testing it on a set of unseen data, known
as the testing set. In this project, the testing set represents 20% of the entire dataset and is not
used during the training process.

To test the proposed ML model, the predicted labels for the images in the testing set are
compared to their actual labels. This allows for an assessment of whether the model has over
fitted or under fitted the training data, and whether it performs effectively with new, previously
unseen data. By evaluating the model's performance on the testing set, the researchers can
determine the model's generalization capabilities and ensure that it is not simply memorizing the
training data, but can accurately classify new, unseen examples.

3.6 Deploying model

In this stage, the trained ML model was deployed onto the ESP32 AI Thinker camera
model, leveraging the Edge Impulse platform. The ESP32 AI Thinker camera model was chosen
for its compact size, low power consumption, and Wi-Fi capabilities, making it an ideal choice
for IoT applications. The deployment process involved uploading the trained model to the Edge
Impulse platform, which then generated optimized code for the ESP32 microcontroller. The code

18
was then flashed onto the ESP32 board, enabling the camera to run the ML model in real-time.
This deployment allowed for the classification of plastic bottles and cans to be performed
directly on the device, reducing latency and enabling real-time decision-making, and the Edge
Impulse AI model is deployed into esp32 AI Thinker model in the form of Arduino IDE library.

3.7 Arduino IDE

The Arduino Integrated Development Environment - or Arduino Software (IDE) - contains

a text editor for writing code, a message area, a text console, a toolbar with buttons for common
functions and a series of menus [14]. In this research, it connects to the ESP32 cam hardware to
upload programs and communicate with the hardware such as servo motors and ultrasonic
sensor. Figure 3.7 shows the code flowchart to control the servo motors based on the
classification results (i.e., bottle or can). The control code is provided in B.1 Appendix B.

Figure 3.7 Servo motors control flowchart

19
Chapter Four
Results and discussion

The results of the machine learning of edge impulse model and the sorting system setup
will be discussed in this chapter. More details is available in A.1 Appendix A.

4.1 confusion matrix

Figure 4.1 shows the confusion matrix of the machine vision model that uses 30 training
cycles, learning rate of 0.0005 and color depth is grayscale. After testing the model, he accuracy
of the model of 98.1 % and loss only 0.04.

Figure 4.1 Machine vision confusion matrix

The term loss in edge impulse refers to the amount of error between the predicted and the
actual label of the test set. F1 score indicates how accurate the machine model classifies
individual label relative to number of samples.

4.2 generate feature

In this project, the validation set samples for the AI model has generated features and
plotted it on a graph as shown in Fig. 4.2.

20
 Figure 4.2 Corrected and incorrected label features

Figure 4.2 demonstrates the result of the AI classification algorithms and measures the
convergence of test sample features in different labels.

4.3 System electronic circuit & components

The project control system contains many electronic devices and sensors that are wired
together to create a functional electrical circuit as shown in Fig. 4.3.

Power
supply

Ultrasonic sensor
+ve -ve

Main servo
motor

Tilt servo
motor
Esp32 AI
thinker
camera

Figure (4.3) Schematic diagram of project circuit and components

21
 Table (4.1) describes the component in the electronic circuit of the machine vision project.
The control device of sorting system id depicted in Fig.4.5. More details about the circuit
component are provided in C.1 Appendix C.

Table (4.1) the component of the project electronic circuit

Component
Task and description
notation
The camera is micro controller and has the capability to run the AI Edge
1.Esp32 AI
Impulse model and it has 9 IO pins and sends a PWM signal to servo
thinker cam
motors to carry it task.
2.Ultra sonic It functions as proximity/distance sensor that its objective in the system is
sensor to activate the camera classifier code to starting prediction.
3.Main Its Torque is 2.5 (kg-cm). After model prediction, the servo rotates in two
servo motor directions (180o for bottle, and 90o for Can).
4.Tilt servo It is used to rotate the object holder to put the object in the basket. The tilt
motor angle of the holder is 10o.
Most of the component need a current and voltage between 3-6v to power
5.Power them. The ESP 32 cam module can power the component, but the
supply brownout of images can be occurred. So, external power supply (5v) is
added to the system to power servo motors and the sensor.

22
Figure (4.5) Showcase the system after it’s assembly

23
4.4 Conclusion and Recommendations

This project successfully developed a waste classification model based on the machine
learning technique (i.e., a machine vision) using and deployed the model on an ESP32-CAM
platform. A waste classification hardware system is designed and created. The system
demonstrates a good ability to separate bottles and cans. However, improvement in areas like
response time and prediction speed is needed. Additionally, the model's current capabilities
might be limited by factors like:

• Data Representativeness: The current data collection strategy might have

inadvertently introduced biases based on time of collection, location, or image
capture practices. Future studies can employ a more controlled and systematic
approach to ensure a representative sample of OMU's waste stream. Techniques
like stratified sampling, capturing location-specific waste, and standardizing image
capture can be implemented.
• Data Volume and Augmentation: The current dataset size and diversity might limit
the model's ability to generalize to unseen waste scenarios. Future studies can focus
on expanding the dataset with a wider range of waste items, variations within each
category, and consider data augmentation techniques to artificially increase data
diversity.
• Hardware Limitations: While the ESP32-CAM served as a valuable starting point,
its processing power and camera resolution might limit future advancements. For
studies aiming at increased model complexity, faster response times, or handling
higher resolution images for detailed waste analysis, exploring alternative hardware
platforms with more powerful processors and high-resolution cameras might be
necessary.

By focusing on enhancing data representativeness, exploring data augmentation

techniques, and potentially upgrading hardware in future studies, researchers can significantly
improve the system's performance, accuracy, and generalizability. This will lead to a more robust
and efficient waste classification system, contributing to OMU's sustainability goals by
facilitating accurate waste sorting and diverting recyclable materials from landfills.

24
References

1 Thabit, Q.;Nassour, A. and Nelles, M. (2022) Facts and Figures on Aspects of Waste
Management in Middle East and North Africa Region.
2 S. E. Jozdani, B. A. Johnson, and D. Chen, “Comparing deep neural networks, ensemble
classifiers, and support vector machine algorithms for object-based urban land use/land cover
classification,” Remote Sens., vol. 11, no. 14, pp. 1–24, 2019, doi: 10.3390/rs11141713.
3 Smola, Alex, Introduction to machine learning. 2008.
4 N. J. Nilsson, Introduction to machine learning. An early draft of a proposed textbook. 1998.
5 L. Martin Wisniewski, J. M. Bec, G. Boguszewski, and A. Gamatié, “Hardware Solutions for
Low-Power Smart Edge Computing,” J. Low Power Electron. Appl., vol. 12, no. 4, pp. 1–24,
2022, doi: 10.3390/jlpea12040061.
6 D. Sharma and S. Sarkar, “Enabling Inference and Training of Deep Learning Models for AI
Applications on IoT Edge Devices,” Internet of Things, pp. 267–283, 2022, doi:
10.1007/978-3-030-87059-1_10.
7 Olscher, C., Jandric, A., Zafiu, C., Pavlicek, A., & Part, F. (2023). Marker materials and
spectroscopic methods for sorting plastic waste. NanoTrust-Dossiers, 60en. DOI:
10.1553/ita-nt-060en
8 Balagugan, Raja S, Maheswaran T, Savitha S., Implementation of Automated Waste
Segregator at Household Level. International Journal of Innovative Research in Science,
Engineering, and Technology. ISSN (Online): 2319-8753. ISSN (Print): 2347-6710
9 Singh, A., Aggarwal P., & Arora, R., (2016). 2016 5th International Conference on
Reliability, Infocom Technologies and Optimization (ICRITO) (Trends and Future
Directions), Sep. 7-9, 2016, AIIT, Amity University Uttar Pradesh, Noida, India
10 Balakrishnan, Kavya & T B Swathy, Rosmi & T D, Subha. (2016). Automatic Waste
Segregator and Monitoring System. Journal of Microcontroller Engineering and Applications
11 Myra g. Flores et al: literature review of automated waste segregation system.
12 Edge Impulse, https://studio.edgeimpulse.com/
13 S. Z. Ms. Ir. Ewine Smits, Lucia Falcioni, “Deep Learning: Findin0gThat Perfect Fit,” 2020.
https://www.compact.nl/articles/deep-learning-finding-that-perfect-fit/.
14 https://www.arduino.cc/en/software

25
Appendices

Appendix A

Figures A1 to A6 illustrate the workflow for developing the multi-classification machine

learning model implemented in this study. These figures provide a visual representation of

the key steps involved in the model development process.

Figure A.1 Preparing of training set

26
 Figure A.2 Preparing of testing set

Figure A.3 create an impulse/building AI model

27
Figure A.4 feature extraction based on bottle & can category

28
Figure A.5 Neural Network settings of training

29
 Figure A.6 downloading the Edge Impulse AI model in the form of Arduino IDE library

Appendix B

B.1. Sorting and control system code:

#include <recyclenoob_inferencing.h>
#include "edge-impulse-sdk/dsp/image/image.hpp"

#include "esp_camera.h"
#include <ESP32Servo.h>

30
//#define CAMERA_MODEL_ESP_EYE // Has PSRAM
#define CAMERA_MODEL_AI_THINKER // Has PSRAM

#if defined(CAMERA_MODEL_ESP_EYE)
#define PWDN_GPIO_NUM -1
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 4
#define SIOD_GPIO_NUM 18
#define SIOC_GPIO_NUM 23

#define Y9_GPIO_NUM 36
#define Y8_GPIO_NUM 37
#define Y7_GPIO_NUM 38
#define Y6_GPIO_NUM 39
#define Y5_GPIO_NUM 35
#define Y4_GPIO_NUM 14
#define Y3_GPIO_NUM 13
#define Y2_GPIO_NUM 34
#define VSYNC_GPIO_NUM 5
#define HREF_GPIO_NUM 27
#define PCLK_GPIO_NUM 25

#elif defined(CAMERA_MODEL_AI_THINKER)
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
31
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22

#else
#error "Camera model not selected"
#endif

/* Constant defines -------------------------------------------------------- */

#define EI_CAMERA_RAW_FRAME_BUFFER_COLS 320
#define EI_CAMERA_RAW_FRAME_BUFFER_ROWS 240
#define EI_CAMERA_FRAME_BYTE_SIZE 3
// Define the pin connected to the Trig pin of the ultrasonic sensor
#define TRIG_PIN 13

// Define the pin connected to the Echo pin of the ultrasonic sensor
#define ECHO_PIN 12

// Minimum distance in centimeters to trigger the camera

#define MIN_DISTANCE 5

// Maximum distance in centimeters to trigger the camera

32
#define MAX_DISTANCE 10
Servo servo;
Servo tiltServo;
//LiquidCrystal lcd(2, 16, 12, 13, 14, 15);

// bool objectDetected = false; // Initialize flag variable

// bool shouldRunClassifier = true;
/* Private variables ------------------------------------------------------- */
static bool debug_nn = false; // Set this to true to see e.g. features generated from the raw
signal
static bool is_initialised = false;
uint8_t *snapshot_buf; //points to the output of the capture

static camera_config_t camera_config = {

.pin_pwdn = PWDN_GPIO_NUM,
.pin_reset = RESET_GPIO_NUM,
.pin_xclk = XCLK_GPIO_NUM,
.pin_sscb_sda = SIOD_GPIO_NUM,
.pin_sscb_scl = SIOC_GPIO_NUM,

.pin_d7 = Y9_GPIO_NUM,
.pin_d6 = Y8_GPIO_NUM,
.pin_d5 = Y7_GPIO_NUM,
.pin_d4 = Y6_GPIO_NUM,
.pin_d3 = Y5_GPIO_NUM,
.pin_d2 = Y4_GPIO_NUM,
.pin_d1 = Y3_GPIO_NUM,
.pin_d0 = Y2_GPIO_NUM,
.pin_vsync = VSYNC_GPIO_NUM,
.pin_href = HREF_GPIO_NUM,
.pin_pclk = PCLK_GPIO_NUM,

33
 //XCLK 20MHz or 10MHz for OV2640 double FPS (Experimental)
.xclk_freq_hz = 20000000,
.ledc_timer = LEDC_TIMER_0,
.ledc_channel = LEDC_CHANNEL_0,

.pixel_format = PIXFORMAT_JPEG, //YUV422,GRAYSCALE,RGB565,JPEG

.frame_size = FRAMESIZE_QVGA, //QQVGA-UXGA Do not use sizes above
QVGA when not JPEG

.jpeg_quality = 12, //0-63 lower number means higher quality

.fb_count = 1, //if more than one, i2s runs in continuous mode. Use only with JPEG
.fb_location = CAMERA_FB_IN_PSRAM,
.grab_mode = CAMERA_GRAB_WHEN_EMPTY,
};

/* Function definitions ------------------------------------------------------- */

bool ei_camera_init(void);
void ei_camera_deinit(void);
bool ei_camera_capture(uint32_t img_width, uint32_t img_height, uint8_t *out_buf) ;

/**
* @brief Arduino setup function
*/
void setup()
{
// put your setup code here, to run once:
Serial.begin(115200);
servo.attach(15); // Attach servo to pin 15
tiltServo.attach(14);
// Initialize ultrasonic sensor pins

34
 pinMode(TRIG_PIN, OUTPUT);
pinMode(ECHO_PIN, INPUT);
// Set up the LCD's number of columns and rows:
// lcd.begin(16, 2);
// // Print a message to the LCD.
// lcd.print("Classifier Results");
//comment out the below line to start inference immediately after upload
while (!Serial);
Serial.println("Edge Impulse Inferencing Demo");
if (ei_camera_init() == false) {
ei_printf("Failed to initialize Camera!\r\n");
}
else {
ei_printf("Camera initialized\r\n");
}

ei_printf("\nStarting continious inference in 2 seconds...\n");

ei_sleep(1000);
}

/**
* @brief Get data and run inferencing
*
* @param[in] debug Get debug info if true
*/
void loop()
{

// instead of wait_ms, we'll wait on the signal, this allows threads to cancel us...
if (ei_sleep(5) != EI_IMPULSE_OK) {
return;

35
 }

snapshot_buf = (uint8_t*)malloc(EI_CAMERA_RAW_FRAME_BUFFER_COLS *
EI_CAMERA_RAW_FRAME_BUFFER_ROWS *
EI_CAMERA_FRAME_BYTE_SIZE);

// check if allocation was successful

if(snapshot_buf == nullptr) {
ei_printf("ERR: Failed to allocate snapshot buffer!\n");
return;
}

ei::signal_t signal;
signal.total_length = EI_CLASSIFIER_INPUT_WIDTH *
EI_CLASSIFIER_INPUT_HEIGHT;
signal.get_data = &ei_camera_get_data;

if (ei_camera_capture((size_t)EI_CLASSIFIER_INPUT_WIDTH,
(size_t)EI_CLASSIFIER_INPUT_HEIGHT, snapshot_buf) == false) {
ei_printf("Failed to capture image\r\n");
free(snapshot_buf);
return;
}
// Read distance from ultrasonic sensor
long distance = readUltrasonicDistance();
// Print the distance
Serial.print("Distance: ");
Serial.print(distance);
Serial.println(" cm");

// Check if an object is within the acceptable range

36
 if (distance >= MIN_DISTANCE && distance <= MAX_DISTANCE) {
// Object detected within acceptable range, run classifier

// Run the classifier

ei_impulse_result_t result = { 0 };

EI_IMPULSE_ERROR err = run_classifier(&signal, &result, debug_nn);

if (err != EI_IMPULSE_OK) {
ei_printf("ERR: Failed to run classifier (%d)\n", err);
return;
}

// print the predictions

ei_printf("Predictions (DSP: %d ms., Classification: %d ms., Anomaly: %d ms.): \n",
result.timing.dsp, result.timing.classification, result.timing.anomaly);

#if EI_CLASSIFIER_OBJECT_DETECTION == 1
bool bb_found = result.bounding_boxes[0].value > 0;
for (size_t ix = 0; ix < result.bounding_boxes_count; ix++) {
auto bb = result.bounding_boxes[ix];
if (bb.value == 0) {
continue;
}
ei_printf(" %s (%f) [ x: %u, y: %u, width: %u, height: %u ]\n", bb.label, bb.value,
bb.x, bb.y, bb.width, bb.height);
}
if (!bb_found) {
ei_printf(" No objects found\n");
}
#else
for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {

37
 ei_printf(" %s: %.5f\n", result.classification[ix].label,
result.classification[ix].value);
}
#endif

#if EI_CLASSIFIER_HAS_ANOMALY == 1
ei_printf(" anomaly score: %.3f\n", result.anomaly);
#endif

// Check if the first object is detected with high confidence

if (result.classification[0].value > 0.5) {
if (strcmp(result.classification[0].label, "unknown") == 0) {
// Stop the motor
servo.write(0); // Or whatever action you want to take
// Optionally, you can also perform other actions or print messages here
} else {
// Perform actions for other recognized objects
servo.write(180); // Or other actions
delay(5000);
}
// shouldRunClassifier = false;
// objectDetected = true; // Set flag to true when an object is detected
//servo.write(180); // Rotate servo to 180 degrees
// lcd.setCursor(0, 1); // Set cursor to the second row
// lcd.print("Object detected: 1"); // Display result on LCD
// Wait for servo to reach 180 degrees
tiltServo.write(40); // Rotate tilt servo to 10 degrees
delay(2000); // Wait for tilt servo to reach 10 degrees
tiltServo.write(0); // Return tilt servo to 0 degrees
delay(2000); // Wait for tilt servo to return to 0 degrees
servo.write(0); // Return servo to 0 degrees

38
 }

// Check if the second object is detected with high confidence

if (result.classification[1].value > 0.5) {
// shouldRunClassifier = false;
// objectDetected = true; // Set flag to true when an object is detected
servo.write(90); // Rotate servo to 90 degrees
// lcd.setCursor(0, 1); // Set cursor to the second row
// lcd.print("Object detected: 2"); // Display result on LCD
delay(5000); // Wait for servo to reach 90 degrees
tiltServo.write(40); // Rotate tilt servo to 10 degrees
delay(2000); // Wait for tilt servo to reach 10 degrees
tiltServo.write(0); // Return tilt servo to 0 degrees
delay(2000); // Wait for tilt servo to return to 0 degrees
servo.write(0); // Return servo to 0 degrees
}

// else {
// // Check if both servos have returned to 0 degrees
// if (servo.read() == 0 && tiltServo.read() == 0) {
// shouldRunClassifier = true; // Resume running the classifier
// objectDetected = false; // Reset flag when both servos have returned to 0 degrees
// }
// }
free(snapshot_buf);
}
// Add a delay between consecutive detections to avoid rapid processing
delay(5000); // Adjust delay time as needed
}
long readUltrasonicDistance() {
// Send a short pulse to trigger the ultrasonic sensor

39
 digitalWrite(TRIG_PIN, LOW);
delayMicroseconds(2);
digitalWrite(TRIG_PIN, HIGH);
delayMicroseconds(10);
digitalWrite(TRIG_PIN, LOW);

// Measure the duration of the echo signal

long duration = pulseIn(ECHO_PIN, HIGH);

// Calculate distance in centimeters

long distance = duration * 0.034 / 2;

return distance;
}
/**
* @brief Setup image sensor & start streaming
*
* @retval false if initialisation failed
*/
bool ei_camera_init(void) {

if (is_initialised) return true;

#if defined(CAMERA_MODEL_ESP_EYE)
pinMode(13, INPUT_PULLUP);
pinMode(14, INPUT_PULLUP);
#endif

//initialize the camera

esp_err_t err = esp_camera_init(&camera_config);
if (err != ESP_OK) {

40
 Serial.printf("Camera init failed with error 0x%x\n", err);
return false;
}

sensor_t * s = esp_camera_sensor_get();
// initial sensors are flipped vertically and colors are a bit saturated
if (s->id.PID == OV3660_PID) {
s->set_vflip(s, 1); // flip it back
s->set_brightness(s, 1); // up the brightness just a bit
s->set_saturation(s, 0); // lower the saturation
}

#if defined(CAMERA_MODEL_M5STACK_WIDE)
s->set_vflip(s, 1);
s->set_hmirror(s, 1);
#elif defined(CAMERA_MODEL_ESP_EYE)
s->set_vflip(s, 1);
s->set_hmirror(s, 1);
s->set_awb_gain(s, 1);
#endif

is_initialised = true;
return true;
}

/**
* @brief Stop streaming of sensor data
*/
void ei_camera_deinit(void) {

//deinitialize the camera

41
 esp_err_t err = esp_camera_deinit();

if (err != ESP_OK)
{
ei_printf("Camera deinit failed\n");
return;
}

is_initialised = false;
return;
}

/**
* @brief Capture, rescale and crop image
*
* @param[in] img_width width of output image
* @param[in] img_height height of output image
* @param[in] out_buf pointer to store output image, NULL may be used
* if ei_camera_frame_buffer is to be used for capture and resize/cropping.
*
* @retval false if not initialised, image captured, rescaled or cropped failed
*
*/
bool ei_camera_capture(uint32_t img_width, uint32_t img_height, uint8_t *out_buf) {
bool do_resize = false;

if (!is_initialised) {
ei_printf("ERR: Camera is not initialized\r\n");
return false;
}

42
camera_fb_t *fb = esp_camera_fb_get();

if (!fb) {
ei_printf("Camera capture failed\n");
return false;
}

bool converted = fmt2rgb888(fb->buf, fb->len, PIXFORMAT_JPEG, snapshot_buf);

esp_camera_fb_return(fb);

if(!converted){
ei_printf("Conversion failed\n");
return false;
}

if ((img_width != EI_CAMERA_RAW_FRAME_BUFFER_COLS)
|| (img_height != EI_CAMERA_RAW_FRAME_BUFFER_ROWS)) {
do_resize = true;
}

if (do_resize) {
ei::image::processing::crop_and_interpolate_rgb888(
out_buf,
EI_CAMERA_RAW_FRAME_BUFFER_COLS,
EI_CAMERA_RAW_FRAME_BUFFER_ROWS,
out_buf,
img_width,
img_height);
}

43
 return true;
}

static int ei_camera_get_data(size_t offset, size_t length, float *out_ptr)

{
// we already have a RGB888 buffer, so recalculate offset into pixel index
size_t pixel_ix = offset * 3;
size_t pixels_left = length;
size_t out_ptr_ix = 0;

while (pixels_left != 0) {
out_ptr[out_ptr_ix] = (snapshot_buf[pixel_ix] << 16) + (snapshot_buf[pixel_ix + 1]
<< 8) + snapshot_buf[pixel_ix + 2];

// go to the next pixel

out_ptr_ix++;
pixel_ix+=3;
pixels_left--;
}
// and done!
return 0;
}

#if !defined(EI_CLASSIFIER_SENSOR) || EI_CLASSIFIER_SENSOR !=

EI_CLASSIFIER_SENSOR_CAMERA
#error "Invalid model for current sensor"
#endif

44
Appendix C

C.1. Component datasheet:

• ESP32-CAM Development Board

INTRODUCTION
ESP32-CAM is a low-cost ESP32-based development board with onboard camera, small. It
is an ideal solution for IoT application, prototypes constructions and DIY projects.
The board integrates WiFi, traditional Bluetooth and low power BLE, with 2 high-
performance 32-bit LX6 CPUs. It adopts 7-stage pipeline architecture, on-chip sensor, Hall
sensor, temperature sensor and so on, and its main frequency adjustment ranges from
80MHz to 240MHz.
Fully compliant with WiFi 802.11b/g/n/e/i and Bluetooth 4.2 standards, it can be used as a
master mode to build an independent network controller, or as a slave to other host
MCUs to add networking capabilities to existing devices
ESP32-CAM can be widely used in various IoT applications. It is suitable for home smart
devices, industrial wireless control, wireless monitoring, QR wireless identification,

45
wireless positioning system signals and other IoT applications. It is an ideal solution for
IoT applications.

C.1 Dimension Diagram

Notes:
1.Please be sure that the power supply for the module should be at least 5V 2A, otherwise
maybe there would be water ripple appearing on the image.
2.ESP32 GPIO32 pin is used to control the power of the camera, so when the camera is in
working, pull GPIO32 pin low.

46
3.Since IO pin is connected to camera XCLK, it should be left floating in using, and do not
connect it to high/low level.
4.The product has been equipped with default firmware before leaving the factory, and we
do not provide additional one's for you to download. So, please be cautious when you
choose to burn other firmware.

FEATURES
• Up to 160MHz clock speed Summary computing power up to 600 DMIPS
• Built-in 520 KB SRAM, external 4MPSRAM
• Supports UART/SPI/I2C/PWM/ADC/DAC
• Support OV2640 and OV7670 cameras, Built-in Flash lamp.
• Support image WiFI upload
• Support TF card
• Supports multiple sleep modes.
• Embedded Lwip and FreeRTOS
• Supports STA/AP/STA+AP operation mode
• Support Smart Config/AirKiss technology
• Support for serial port local and remote firmware upgrades (FOTA)

SPECIFICATION
• SPI Flash: default 32Mbit
• RAM: built-in 520 KB+external 4MPSRAM
• Dimension: 27*40.5*4.5（±0.2）mm/1.06*1.59*0.18”
• Bluetooth: Bluetooth 4.2 BR/EDR and BLE standards
• Wi-Fi: 802.11b/g/n/e/i
• Support Interface: UART, SPI, I2C, PWM
• Support TF card: maximum support 4G
• IO port: 9
• Serial Port Baud-rate: Default 115200 bps

• Image Output Format: JPEG( OV2640 support only ), BMP, GRAYSCALE

47
• Spectrum Range: 2412 ~2484MHz
• Antenna: onboard PCB antenna, gain 2dBi
• Transmit Power: 802.11b: 17±2 dBm (@11Mbps); 802.11g: 14±2 dBm (@54Mbps);
802.11n: 13±2 dBm (@MCS7)
• Receiving Sensitivity: CCK, 1 Mbps : -90dBm;
CCK, 11 Mbps: -85dBm;
6 Mbps (1/2 BPSK): -88dBm;
54 Mbps (3/4 64-QAM): -70dBm;
MCS7 (65 Mbps, 72.2 Mbps): -67dBm
• Power consumption: Turn off the flash: 180mA@5V. Turn on the flash and adjust the
brightness to the maximum:
310mA@5V
Deep-sleep: the lowest power consumption can reach 6mA@5V
Moderm-sleep: up to 20mA@5V
Light-sleep: up to 6.7mA@5V
• Security: WPA/WPA2/WPA2-Enterprise/WPS
• Power supply range: 5V
• Operating temperature: -20 °C ~ 85 °C
• Sorage environment: -40 °C ~ 90 °C, < 90%RH
• Weight: 10g

SHIPPING LIST
• ESP32-CAM Development Board x1

48
\

• HC-SR04 Ultrasonic Sensor Module User Guide

49
 Ultrasonic is an excellent way of figuring out what’s in the immediate vicinity of your
Arduino. The basics of using ultrasound are like this: you shoot out a sound, wait to hear it echo
back, and if you have your timing right, you’ll know if anything is out there and how far away it
is. This is called echolocation and it’s how bats and dolphins find objects in the dark and
underwater, though they use lower frequencies than you can use with your Arduino.

HC-SR04 Ultrasonic Sensor is a very affordable proximity/distance sensor that has been
used mainly for object avoidance in various robotics projects. It has also been used in turret
applications, water level sensing, and even as a parking sensor. Table (c.1) shows the module
Specifications.

Table (c.1) the ultrasonic module Specifications

Electrical parameters Value
Operating Voltage 3.3Vdc -5Vdc
Quiescent current <2mA
Operating Current 15mA
Operating Frequency 40KHz
Operating Range & Accuracy 2cm ~ 400cm ( 1in ~ 13ft) ± 3mm
Sensitivity -65dB min
Sound Pressure 112dB
Effective Angle 15
Connectors 4-pins header with 2.54mm pitch
Dimension 45mm x 20mm x 15mm
Weight 9g

50
 • Servo motors

The unit comes complete with 30cm wire and 3 pin 'S' type female header connector that
fits most receivers, including Futaba, JR, GWS, Cirrus, Blue Bird, Blue Arrow, Corona, Berg,
Spektrum and Hitec.

This high-speed standard servo can rotate approximately 120 degrees (60 in each
direction). You can use any servo code, hardware or library to control these servos, so it's great
for beginners who want to make stuff move without building a motor controller with feedback &
gear box, especially since it will fit in small places. The MG995 Metal Gear Servo also comes
with a selection of arms and hardware to get you set up nice and fast!

Specifications

* Weight: 55 g

* Dimension: 40.7 x 19.7 x 42.9 mm approx.

* Stall torque: 8.5 kgf·cm (4.8 V ), 10 kgf·cm (6 V)
* Operating speed: 0.2 s/60º (4.8 V), 0.16 s/60º (6 V)
* Operating voltage: 4.8 V a 7.2 V
* Dead band width: 5 µs
* Stable and shock proof double ball bearing design

51
* Temperature range: 0 ºC – 55 ºC

52