AI VIET NAM – RESEARCH TEAM

Data Augmentation Using Background

Replacement for Automated Sorting of Littered
Waste
May 4, 2024

Date of publication: 12/08/2021

Authors: Arianna Patrizi, Giorgio Gambosi and Fabio Massimo Zanzotto
Sources: Journal of imaging (ic-ETITE)
Data sources (if Litted Waste
any):
Keywords: automated waste sorting; convolutional neural networks; background
replacement; data augmentation; computer vision; deep learning; multi-
class classification
Summary by: Truc Truc Cute

1. Introduction:

• Although the efficiency of sorting and recycling has been improved through enhancing modern
waste treatment plants, another issue is emerging. More and more waste is being littered in
the environment instead of being properly collected. Oceans are filled with plastic debris,
spoiling the beauty of coastlines. However, while this is a significant problem for oceans,
it becomes gigantic for land. The presence of large plastic debris in soil appears to be 40
times greater than in the oceans. Pieces of solid waste are commonly found on streets, urban
parks, beaches, forests, and many other places, indicating the footprint of modern human
society. Precious glass bottles, plastic bottles, cans, and cardboard boxes are all discarded,
and their potential value is lost. Moreover, the COVID-19 pandemic has exacerbated waste
management, due to the uncontrolled increase of household waste during social distancing
measures.
• The solution to retrieving large-scale waste cannot solely rely on the current trend of enhanc-
ing traditional plants for managing solid waste with automated waste sorting systems based
on image recognition techniques. In fact, waste has surpassed traditional recycling manage-
ment and can be transformed into valuable new materials. Current waste recognition systems
aim to reduce the number of people working in these unhealthy waste treatment plants. For
example, waste pickers in the largest waste processing areas in Brazil have faced or are still
facing various illnesses due to working conditions, primarily musculoskeletal disorders, but
also viral infections, and emotional vulnerabilities. Therefore, current waste recognition sys-
tems as well as current datasets for training focus on sorting activities at waste treatment
plants, where solid waste runs on conveyor belts and is then sorted into different types. For
this reason, current datasets contain waste images taken on a uniform background. In fact,

1
AI VIETNAM aivietnam.edu.vn

conveyor belts typically have a uniform color. However, systems derived from these datasets
can only be used when solid waste has reached its final destination. Searching and sorting
solid waste in natural and urban environments may pose several challenges for these systems
trained on existing datasets.
• This paper has introduced a method for building automated waste recognition systems that
can be used to identify and sort litter directly at the location where it is found. Our method
is based on BackRep, a novel data augmentation procedure that expands existing datasets
by cropping solid waste in images taken on a uniform background (white) and placing it on
more complex backgrounds. For our purposes, these complex backgrounds represent locations
where solid waste is commonly discarded. To test our data augmentation procedure, we have
created a new dataset in realistic settings.

2. Datasets:

• To test waste recognition systems in normal settings, the authors relied on an existing dataset
- CompostNet - primarily used for training, and they collected a testing dataset where waste
images are taken where waste is most commonly found.
• The paper referenced the CompostNet dataset to assess the effectiveness of our data augmen-
tation method in leveraging existing datasets to train waste classifiers that can be applied to
identify and classify waste in a general setting.
• In the CompostNet dataset, waste images are captured on a white background. For this
reason, these images are oriented to facilitate the construction of waste classification models
that can be used for sorting waste on conveyor belts. In fact, the CompostNet dataset com-
plements the TrashNet dataset by adding a compost class. The TrashNet dataset consists
of 2527 images belonging to six different material classes: paper, glass, plastic, metal, card-
board, and trash. These images represent damaged objects, which can be associated with
the concept of waste. They were produced as part of activities reported in the literature, and
they were captured from various angles to expand (and improve) the dataset with additional
images representing the complex shapes of damaged waste. Their initial idea of using the
Flickr Material Database and images from Google ImagesTM was not applicable. In fact,
these images did not accurately represent the state of wasted goods as most of them represent
undamaged or completely unused objects. The TrashNet dataset has been widely used. The
set of additional images in CompostNet has similar characteristics to the rest of the TrashNet
dataset. There are 177 compost images with a white background.
• In the experiments of the paper, the authors utilized a split ratio of 70 percentage - 13
percentage - 17 percentage for training, validation, and testing, respectively, as in [reference].
All categories were included. The validation split would be useful for tuning hyperparameters
during the learning phase. Thus, the model would be tested on these samples, ensuring that
the gold test set remains unused.
• To evaluate whether the waste recognition system can provide good results for waste in
normal settings, the authors prepared a testing dataset: the Littered Waste Testset. In our
dataset, images of waste items are captured in more normal settings, i.e., the background is
not a neutral white. This aims to represent waste where it is most usually littered and not
on the conveyor belt of a waste management plant.
• The Littered Waste Testset comprises 114 images categorized according to the classes in
the CompostNet dataset. A group of 4 volunteers took the pictures in indoor and outdoor
settings with various backgrounds and lighting conditions. The group used their mobile
phones to capture the images, which were then resized to 500 × 400 pixels. The distribution
AI VIETNAM aivietnam.edu.vn

of waste classes was determined by the volunteers. These are types of waste commonly found
in urban environments and in their homes. Volunteers were asked to produce at least ten
samples for each class.
• This new test set may confuse waste classification systems trained on conveyor-belt-oriented
datasets. In fact, parts of the background may inaccurately influence the behavior of the
convolutional neural network, as the texture and color could resemble other types of waste.

3. Methods and implementations:

• The main goal of the research is to build automated waste sorting systems to identify and clas-
sify waste retrieved at its disposal site, as represented in the littered waste testset. Therefore,
the authors first introduce BackRep, their data augmentation method, to leverage existing
categorized waste datasets. Secondly, the authors briefly summarize AlexNet and Incep-
tionV4, two advanced systems applied to waste classification. These latter systems have
been used in conjunction with BackRep to demonstrate its feasibility for the primary objec-
tive.

4. Result:

• The experiments aim to explore whether the Background Replacement method for data
augmentation helps adapt the conveyor system to recognize litter in real-life scenarios.
• For these reasons, the authors employed the best-performing models - AlexNet and Incep-
tionV4 enhanced with BackRep. The paper divides four configurations of the two systems:
NotAug, BackRep, BackRep and NotAug, and LittleAug. NotAug is the baseline system.
BackRep is the system when the input is augmented with the BackRep method. BackRep
and NotAug are a combination of NotAug and BackRep, where the input data is pure and
augmented. Finally, LittleAug is a model introduced with augmentation methods: images
are represented in two ways: simple and flipped horizontally and vertically. The model is
used for comparison with the augmentation method.
• The paper experiments on different datasets: conveyor data and waste data. The data
is trained in the conveyor direction using 4 configurations 5 times and tested on both the
conveyor dataset and the waste dataset. The paper reports the Accuracy, Macro Average F1,
and Micro Average F1 scores. The F1 score is an evaluation method to assess classification
performance because it relies on both Recall and Precision, and the final score is inspired
by lower values with arithmetic mean. The difference between micro-average F1 and macro-
average F1 is that the former is the average of F1 scores without weighting each class, while
the latter is the average of weighted F1 scores.

5. Discussion:

• The experiments in the paper have a solid foundation with the best-performing waste clas-
sification systems being used. In fact, the performance of both AlexNet and InceptionV4 is
comparable, with the metrics provided in the original paper. The results are updated with 5
runs on different seeds.
• InceptionV4 performs better in tests on conveyor-oriented datasets compared to AlexNet.
Specifically, AlexNet achieved an accuracy of 0.683, while InceptionV4 achieved 0.844. The
complexity of InceptionV4 is assessed with a performance improvement of 16
• However, the assumptions are affirmed: conveyor-oriented systems do not apply to littering
situations. In fact, both AlexNet and InceptionV4 are noted to have decreased performance
AI VIETNAM aivietnam.edu.vn

when tested on the waste classification configuration. AlexNet decreases by over 0.55 points
in accuracy from 0.683 in the conveyor system test to 0.138 in the waste classification test.
The InceptionV4 model experiences a similar decline - around 0.44 in accuracy - from 0.844
to 0.409.
• The BackRep model is a solution to leverage conveyor-oriented datasets, achieving 0.138 in
the waste classification test. The InceptionV4 version experiences a performance decline of
0.44 in accuracy from 0.844 to 0.409.
• The BackRep model is a solution to leverage conveyor-oriented datasets to enhance waste
classification systems for littering scenarios. The application of BackRep enhances the results
across waste classification configurations in the systems. The growth in accuracy is higher in
the best-performing models, notably InceptionV4: 0.442 and 0.409 in the waste classification
configuration, and this difference in accuracy is quite evident.
• However, BackRep is not perfect; in fact, images may be misinterpreted, leading to erroneous
cropping. The BackRep system, like littered waste, has its limitations. A significant portion
of litter is represented by microplastics. In reality, these pieces are too small and require
alternative methods that do not rely on image recognition techniques.

6. Conclusion:

• The current amount of waste is overwhelming. The paper introduces BackRep, a model
designed to adapt to Automated Waste Sorting (AWS) systems for waste classification. The
authors also introduce a dataset to test AWS systems for waste classification. The experi-
ments indicate that AWS systems trained on augmented datasets outperform significantly the
waste classification systems trained on current datasets. Therefore, this could be a solution
to build identification systems for suburban and wild environments.
• The BackRep model can enable the development of automated waste sorting systems in waste
classification settings. These systems can be used for small robots, which can be employed
to clean up forests, beaches, urban environments, and other areas to remove waste.
• Certainly, the BackRep model is a general data augmentation model that can be used for
various image recognition tasks. Therefore, this model opens up a new avenue for research.