Segmentation Approaches for Crop Detection from

Satellite Images
PROJECT REPORT

In partial fulfilment for the award of the degree

Of

BACHELOR OF ENGINEERING
IN
COMPUTER SCIENCE AND ENGINEERING

Under the guidance of Submitted by

Ms. Preetpal Kaur Buttar Jashanpreet Singh (GCS-2030458)
Assistant Professor Sunny (GCS-1940032)
CSE Department Harjeet Singh (GCS-2030498)

SANT LONGOWAL INSTITUTE OF ENGINEERING AND TECHNOLOGY,

LONGOWAL – 148106, DISTRICT- SANGRUR, PUNJAB, INDIA
May, 2023

1
 CERTIFICATE

This is to certify that the project report entitled “Segmentation Approaches for Crop Detection
from Satellite Images” submitted to department of Computer Science and Engineering, SLIET, in
partial fulfillment for the award of the degree of Bachelor of Engineering in Computer Science and
Engineering, is a record of work carried out by Jashanpreet Singh (GCS-2030458), Sunny (GCS-
1940032) and Harjeet Singh (GCS-2030498) under the supervision and guidance of Ms. Preetpal
Kaur Buttar. The results presented to this have not been submitted previously to this or any other
University for B.E. and any other degree.

Jashanpreet Singh Sunny Harjeet Singh

GCS/2030458 GCS/1940032 GCS/2030498

This is to certify that the above statement made by the candidates is correct to the best of my
knowledge.

Ms. Preetpal Kaur Buttar

(Supervisor)
Assistant Professor, Dept. of C.S.E.
Sant Longowal Institute of Engg. &
Tech.
Sangrur (Punjab)

Date:

2
 Acknowledgement
We express our sincere thanks to Ms. Preetpal Kaur Buttar, Assistant Professor Computer
Science Department, SLIET. She is our assigned project mentor, and guided us throughout our
project. She inspired us to work with satellite imagery, deep learning, and introduced us to main
objectives of project work.
We, students of degree final year, Computer Science and Engineering have successfully
completed the project on “Segmentation Approaches for Crop Detection from Satellite Images”
under our project guide Ms. Preetpal Kaur Buttar guidance.

3
 Abstract

The information about where crops are distributed is useful for agri-environmental assessments, but
is chiefly important for food security and agricultural policy managers. The quickness with which
this information becomes available, especially over large areas, is important for decision makers.
Methodologies have been proposed for the study of crops. Most of them require field survey for
ground truth data and a single crop map is generated for the whole season at the end of the crop
cycle and for the next crop cycle a new field survey is necessary. Here, we present models for
recognizing different types of crops. The models were trained on satellite images captured by
Sentinel2. The algorithms used for classification were LightGBM and Temporal CNN (TempCNN),
and the spectral information captured in the B02, B03, B04, B05, B06, B07, B08, B8A, B11, and
B12 bands of Sentinel 2 images was used.

4
 Table of contents

Sr. No. Contents Page No.

1 Introduction 7
1.1 Sentinel-2 7
1.2 Eo-learn 8
2 Problem Definition 9
3 Objectives 10
4 Methodology 11
4.1 Data acquisition from Sentinel-Hub 12
4.2 Adding GT and EO data 13
4.3 Visualizations and feature/label engineering 14
4.4 Sampling 17
4.5 Model training 17
4.6 Validation and Evaluation 19
5 Result and screenshots 24
6 Conclusion and Future Work 27

5
 List of Figures

Sr. No. Contents Page No.

1 Subarea 11
2 Boundary outline of AOI 12
3 Split AOI with reference to Austrian national borders 12
4 GT data workflow 13
5 EOpatch 13
6 EOpatch visualization 15
7 EOpatch NDVI visualization 15
8 Data masks 16
9 Leaf-wise tree growth 18
10 TempCNN architecture 19
11 LightGBM evaluation 20
12 TempCNN evaluation 20
13 LightGBM Confusion matrix 21
14 TempCNN confusion matrix 22
15 Predicted EOpatch 24
16 Predicted EOpatch vs GT 25
17 Zoomed comparison 25

6
1. Introduction

The availability of open Earth observation (EO) data through the Copernicus and Landsat
programs represents an unprecedented resource for many EO applications, ranging from ocean
and land use and land cover monitoring, etc. Remote sensing has several potential applications
in the agricultural field since it allows us to observe the distribution, area, and phenological
states of crops.
However, achieving an accurate classification of crops can be difficult due to the phenological
stages of crops among other factors. The temporal changes in the crop phenology identified by
satellite images monitoring during the agricultural cycle can help to achieve better
classifications. Data from unmanned aerial vehicles and satellite images provide information
for crop monitoring, but frequent data acquisition along the entire growth period of the crop at
least during the key stages of growth and development is necessary. A methodology was
proposed that considers samples or observations of the plots in different phenological stages
within the agricultural cycle and uses two different types of models to classify crop types.
LighGBM machine learning model developed by Microsoft is one of the models that is
implemented for the task of classification, other is a deep learning model architecture known
as TempCNN which is supposed to work on temporal dimension.
1.1 Sentinel-2: -
European space agency's Sentinel-2 comprises of constellation of two identical satellites in the
same orbit, phased at 180 to each other. It aims at monitoring variability in land surface
conditions. Each satellite carries an innovative wide swath high-resolution multispectral imager
with 13 spectral bands for a new perspective of our land and vegetation. Wide swath of Sentinel-
2 is 290 km and has a 5-day revisit time.
Each satellite has a high level of autonomy, so that they can operate without any intervention
from the ground for periods of up to 15 days. The ‘carpet mapping’ imaging plan requires
acquisition, storage and transmission of or 1 TB per day. This massive data blast results from
the combination of the 290 km swath with 13 spectral channels incorporating four visible and
near-infrared bands at 10 m resolution, six red-edge/shortwave-infrared bands at 20 m and three
atmospheric correction bands at 60 m.

7
1.2 eo-learn: -
eo-learn is a collection of open-source Python packages that have been developed to seamlessly
access and process spatio-temporal image sequences acquired by any satellite fleet in a timely
and automatic manner. eo-learn is easy to use, it’s design modular, and encourages collaboration
– sharing and reusing of specific tasks in a typical EO-value-extraction workflows, such as
cloud masking, image co-registration, feature extraction, classification, etc. The library is
written in Python and uses NumPy arrays to store and handle remote sensing data.
eo-learn is divided into several subpackage according to different functionalities and external
package dependencies. Therefore, it is not necessary for user to install entire package but only
the parts that he needs.
The following are available subpackage:
o eo-learn-core - The main subpackage which implements basic building blocks
(EOPatch, EOTask and EOWorkflow) and commonly used functionalities.
o eo-learn-coregistration - The subpackage that deals with image co-registraion.
o eo-learn-features - A collection of utilities for extracting data properties and feature
manipulation.
o eo-learn-geometry - Geometry subpackage used for geometric transformation and
conversion between vector and raster data.
o eo-learn-io - Input/output subpackage that deals with obtaining data from Sentinel Hub
services or saving and loading data locally.
o eo-learn-mask - The subpackage used for masking of data and calculation of cloud
masks.
o eo-learn-ml-tools - Various tools that can be used before or after the machine learning
process.
o eo-learn-visualization - Visualization tools for core elements of eo-learn.

8
2. Problem Definition
It is necessary to anticipate crop yield and production across a specified area and to ascertain
how much of the crop will be harvested under particular circumstances. The identification of
water shortages or surpluses, weed infestations, insect damage, hail damage, and plant
populations can all be done using remotely sensed images.
In this project, we have developed two different systems involving different models, both uses
images taken by Sentinel-2 satellite and maps the agricultural land area. Also, the system is
able to categorize different types of crops on the field. The proposed systems are trained using
a LightGBM architecture & TempCNN architecture and attains a high accuracy for the task of
agricultural land cover.

9
3. Objective
1) To study the relevant literature for agriculture land cover mapping using satellite imagery.
2) To create a dataset by fetching Sentinel-2 satellite images over an area of interest.
3) To apply different models on the dataset.
4) To compare the performance of the trained models on test images.

10
 4. Methodology
Dataset Sample: -

Fig1: - Subarea

Sample image from the dataset of Sentinel-2 imagery, showing a subarea (part of an EOpatch).
Sentinel-2 has 13 spectral bands having 10, 20, or 60 meters in pixel size. For this project, both 10
and 20 meters in pixel size bands are used.

11
Project includes these main parts - Data acquisition from Sentinel-Hub, Adding GT and EO data,
Visualizations and feature/label engineering, Scaling and model training, Validation and evaluation

4.1 Data acquisition from Sentinel-Hub: -

We start with selecting an area of interest that can be used for training or inference. A polygon
of area of Austria will be used for this purpose, having an extent of 20664m x 20561m.

Fig2: Boundary outline of AOI

Fig3: Split AOI with reference to Austrian national borders

12
4.2 Adding GT and EO data: -
We define two different EO-Workflows for uploading ground truth as well as earth observation
data (satellite data) to be used our project. Data has been taken from Geopedia. Firstly, LPIS
has been added and then we calculated and added area ratio, finally saving task/EOpatch.
For EO data, firstly we load Sentinel2 data with 10 resolution bands, 0.8 cloud coverage,
additional feature mask and 5 threads. Secondly, convert cloud mask and combine few other
features into one, creating L1C data masks, for easy use. Sen2Cor classification map has been
created and combined with ‘IS_DATA’ feature to define valid data mask. We also add L2A Val
mask and combine all data masks. Then, we check if EOpatch’s time-series is valid or not and
filter out scenes including too many invalid pixels. Indices such as NDVI, NDWI and Euclidean
norm are calculated. EOpatches has been also filtered based on lpis threshold, for this project
we can included only EOpatches containing more than 13% agricultural area.

Fig4: GT data workflow

Fig5: EOpatch

13
4.3 Visualizations and feature/label engineering: -
In this part, we visualize a sample EOpatch and different indices as well as valid data masks.
Various info such as MY-S2-L2A-BANDS, NDVI, NDWI, NORM is concatenate into
single feature for easier handling of data and perform temporal interpolation ((filling gaps
and resampling to the same dates). As the LPIS data is divided into 200 different crop type
classes, which is too detailed for achieving accurate prediction results. Therefore, we group
these classes into 14 groups based on similar spectral characteristics. After the grouping, the
data set stored in vector format is converted into a raster format. Each EO pixel can be
assigned to a crop type value and a buffer of 1 pixel (10m) size is applied to each individual
field in the border area.
4.3.1 Normalized Difference Vegetation Index (NDVI): - The Normalized Difference
Vegetation Index (NDVI) is the most commonly used vegetation index for observe greenery
globally. In general, Healthy vegetation is good absorber of electromagnetic spectrum in
visible reason. Chlorophyll contains in a greenery highly absorbs Blue (0.4 - 0.5 µm) and
Red (0.6 - 0.7 µm) spectrum and reflects Green (0.5 – 0.6 µm) spectrum. Therefore, our eye
perceives healthy vegetation as green. High reflectance in NIR and high absorption in red
spectrum, these two bands are used to calculate NDVI. So, following formula gives
Normalized Difference Vegetation Index (NDVI) –
NDVI = (NIR – Red) / (NIR + Red)

4.3.2 Normalized Difference Water Index (NDWI): - Normalize Difference Water Index
(NDWI) is use for the water bodies analysis. The index uses Green and Near infra-red bands
of remote sensing images. The NDWI can enhance water information efficiently in most
cases. It is sensitive to build-up land and result in over-estimated water bodies. The NDWI
products can be used in conjunction with NDVI change products to assess context of
apparent change areas. Water bodies having low reflectance. It only reflects within visible
portion of the electromagnetic spectrum. There is no reflection in Near Infrared (NIR) and
beyond. The formula of NDWI then modified by Xu. It uses Green and SWIR band -
MNDWI = (Green – SWIR) / (Green + SWIR)

14
Fig6: EOpatch visualization

Fig7: EOpatch NDVI visualization

15
Fig8: Data masks

16
4.4 Sampling: -
A spatial sampling of the EOPatches is performed by randomly taking a subset of pixels from a
patch to use in the machine learning training and testing while only considering classes that are
represented to a certain quantity of pixels. We consider a threshold of 1000 pixel is necessary
for a class to be considered in sampling. Labels that are not in both train and test data and also
mask out samples where features include NaN values are masked out. A total of 444766 samples
were taken out of which dataset includes 352679 training and 92087 test samples. Sample
dataset is further visualized to check if generated dataset is well balanced.

4.5 Model training: -

In this part, features are standardized by removing the mean and scaling to unit variance as well
as perform one hot encoding. Two different models are used for detection namely, LightGBM
and TempCNN. State-of-the-art LightGBM is a fast, distributed, high-performance gradient
boosting framework based on decision tree algorithms, used for many ML tasks. As novel
competitor, TempCNN DL architecture is used, that is supposed to exploit the temporal
information of satellite image time series. Both models are set-up , trained and saved at end of
this portion.
4.5.1 LightGBM: - LightGBM is a gradient boosting framework that uses tree-based learning
algorithms. It is designed to be distributed and efficient with the following advantages:
• Faster training speed and higher efficiency.
• Lower memory usage.
• Better accuracy.
• Support of parallel, distributed, and GPU learning.
• Capable of handling large-scale data.

LightGBM does not grow a tree level-wise — row by row — as most other
implementations do. Instead, it grows tree leaf-wise. It chooses the leaf it believes will
yield the largest decrease in loss. It utilizes two novel techniques called Gradient-Based
One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) which allow the
algorithm to run faster while maintaining a high level of accuracy.

17
 Fig9: Leaf-wise tree growth

4.5.2 TempCNN: - Temporal 1D-CNNs (TempCNNs) where convolutions are applied in the
temporal domain have proven to be effective for handling the temporal dimension for
(general) time series classification, and 3D-CNN for both the spatial and temporal
dimension in video classification. Consequently, these TempCNN architectures, that could
make the most of the temporal structure of SITS (satellite image time series) have started to
be explored in remote sensing where convolutions are applied across the temporal
dimension alone.
o Architecture: - The architecture is composed of three convolutional layers (64
units), one dense layer (256 units) and one Softmax layer. Three convolutional filters
are consecutively applied, then one dense layer, and finally the Softmax layer, that
provides the predicting class distribution. To control for overfitting, we use a
regularization mechanism: dropout with a dropout rate of 0.5.

18
 Fig10: TempCNN architecture

4.6 Validation and Evaluation: -

LightGBM and TempCNN architecture models are tested on test dataset and evaluated based
on few methods – accuracy, confusion matrix, precision, F-1 score, recall.

1. Confusion matrix - A confusion matrix is an N X N matrix that is used to evaluate the

performance of a classification model, where N is the number of target classes. It
compares the actual target values against the ones predicted by the ML model. As a
result, it provides a holistic view of how a classification model will work and the errors
it will face. It results in four potential outcomes:
o True positive (TP) - the number of true results when the actual observation is
positive.
o True negative (TN) - is the number of true predictions when the observation is
negative.
o False positive (FP) - the number of incorrect predictions when the actual observation
is positive.
o False negative (FN) - is the number of incorrect predictions when the observation is
negative.

2. Precision - Precision is the analysis of the true positives over the number of total
positives that are predicted by the machine learning model. The formula for precision is
written as TP/(TP+FP). This indicator allows you to calculate the rate at which your
positive predictions are actually positive.

19
3. Recall - Recall is the measure of the TP over the count of the actual positive outcomes.
The formula to calculate Recall is TP/(TP+FN). This parameter assesses how well the
ML model can analyze the input and identify the actual result.

4. F1 score - The harmonic mean of precision and recall is the F1 score. It is used as an
overall indicator that incorporates both precision and recall. This harmonic mean
analyzes both false positives and false negatives and performs well on an imbalanced
dataset. The formula to calculate it is 2(p*r)/(p+r), where r is the recall and p is precision.

Fig11: LightGBM evaluation

Fig12: TempCNN evaluation

20
Overall Classification accuracy and weighted F1-score of LightGBM comes out to be 94.9% and
94.6% respectively. Whereas, overall classification accuracy and weighted F1-score of TempCNN
comes out to be 94.6% and 94.4% respectively. It can be clearly seen that LightGBM has slightly
higher accuracy and F-12 score than TempCNN model, but when models are evaluated for each
class, it can be seen that for some specific classes TempCNN has outperformed LightGBM model.

Fig13: LightGBM Confusion matrix

21
 Fig14: TempCNN confusion matrix

The validation of the models shows that for most of the groups both perform very well. However,
there seem to be differences in their confusion for certain classes:
1) In this specific case orchards might catch your attention mostly. LightGBM performs
worse than TempCNN. But more interesting than the overall accuracy is, that
LightGBM classifies actual orchards as grass a lot (low recall) while, no other class
is mistaken as orchards (high precision). In contrast TempCNN recognizes actual

22
 orchards well (high recall) but identifies actual grass as orchards frequently (lower
precision). Generally, confusion with grass class is not surprising, as there is a lot of
it between the individual trees.
2) There is also poor performance received for potatoes in both models as their
cultivation practices are quite similar to peas.
3) Poor performance for the group Other is expectable in consequence of its diverse
class composition.

23
5. Result and screenshots
For predicting the whole AOI, LightGBM model is further use for predicted due to higher
overall accuracy. A workflow is defined for making prediction on existing EOpatches and
using LightGBM model, EOpatch is predicted and visualized as well as compared with
ground truth data.

Fig15: Predicted EOpatch

24
Fig16: Predicted EOpatch vs GT

Fig17: Zoomed comparison

25
It can probably be seen in the randomly chosen section of the AOI there are certain patterns of
misclassified pixels:

• There are complete fields mistaken as another crop group. In this case the algorithm got
confused because of similar spectral characteristics. Evaluation part already showed an
overview of the frequency and combination of these incidents.
• Misclassified single pixels are usually located at the border of the respective fields. Here the
"mixed-pixel-problem" impacts the prediction results. For the modeling these pixels were
excluded, as they may include spectral reflectance values of different vegetation types and
thereby confuse the algorithm.

26
6. Conclusion and Future Work
Both the models detect the land cover as well as differentiate crop types into defined classes,
indicated by using different color codes. By comparing predicted images of both the models
with each other, it clearly shows that the LightGBM model has better accuracy in prediction
for all classes and fall behind TempCNN for some specific categories. This project further
implemented on larger area can help detect land covers in different regions along with types of
crops and help in management of agricultural land as well as planning for future yield based
on prediction. Planning can be done based on predicted results remotely. As of today, various
satellites have been launched in atmosphere and there is no lack of resources for satellite
imagery such as Sentinel-2 satellite. System can be implemented for live surveillance of land
and can be made accessible through web, reducing much extensive work of management.

27
 References

1. Peressutti, D. (2019, May 15). Introducing eo-learn - Sentinel Hub Blog - Medium.

Medium. https://medium.com/sentinel-hub/introducing-eo-learn-ab37f2869f5c

2. Sentinel-2 L2A. (n.d.). https://docs.sentinel-hub.com/api/latest/data/sentinel-2-l2a/

3. GeoPandas 0.13.0 — GeoPandas 0.13.0+0.gaa5abc3.dirty documentation. (n.d.-a).

https://geopandas.org/en/stable/index.html

4. GISGeography. (2022). Sentinel 2 Bands and Combinations. GIS Geography.

https://gisgeography.com/sentinel-2-bands-combinations/

5. Pelletier, C., Webb, G. I., & Petitjean, F. (2019b). Temporal Convolutional Neural Network

for the Classification of Satellite Image Time Series. Remote Sensing, 11(5), 523.

https://doi.org/10.3390/rs11050523

6. Coordinate Reference Systems — QGIS Documentation documentation. (n.d.).

https://docs.qgis.org/3.28/en/docs/gentle_gis_introduction/coordinate_reference_systems.

html#:~:text=A%20Coordinate%20reference%20system%20(CRS)%20defines%2C%20

with%20the%20help,Systems%20and%20Projected%20Coordinate%20Systems.

7. Kshetri, T. (n.d.). NDVI, NDBI & NDWI Calculation Using Landsat 7, 8.

www.linkedin.com. https://www.linkedin.com/pulse/ndvi-ndbi-ndwi-calculation-using-

landsat-7-8-tek-bahadur-kshetri

28