ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Contents lists available at ScienceDirect

ISPRS Journal of Photogrammetry and Remote Sensing

journal homepage: www.elsevier.com/locate/isprsjprs

An unsupervised domain adaptation deep learning method for spatial and

temporal transferable crop type mapping using Sentinel-2 imagery
Yumiao Wang a, b, Luwei Feng c, *, Zhou Zhang d, Feng Tian c, e
a
Department of Geography and Spatial Information Techniques, Ningbo University, Ningbo 315211, China
b
Key Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Natural Resources, Shenzhen 518034, China
c
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China
d
Department of Biological Systems Engineering, University of Wisconsin-Madison, Madison, WI 53706, USA
e
Hubei Luojia Laboratory, Wuhan 430079, China

A R T I C L E I N F O A B S T R A C T

Keywords: Accurate crop type mapping is essential for crop growth monitoring and yield estimation. Recently, various
Crop type mapping machine learning methods have been increasingly used for crop type mapping, but they often lose their validity
Unsupervised domain adaptation when directly applied to other regions and years due to differences in the distribution of source and target data,
Time-series imagery
that is, domain shift. To address the problem, we developed a deep adaptation crop classification network
Transfer learning
(DACCN) based on the idea of unsupervised domain adaptation (UDA). The proposed DACCN mainly consists of
two parts, a feature extractor that converts the original input into high-level representations, and a domain
aligner where the discrepancy between source and target distributions is measured using multiple kernel variant
of maximum mean discrepancy (MK-MMD). Four states in the United States (U.S.) Corn Belt and two provinces in
northeastern China were used as study areas, where samples used for model building and accuracy evaluation
were collected based on time-series Sentinel-2 imagery and reference maps in 2018 and 2019. Then, three ex­
periments were designed to verify the transferability of DACCN across space, time, and space–time, respectively.
In each experiment, the proposed DACCN was compared to deep crop classification network (DCCN), a model
with a similar structure to DACCN but without the domain adaptation mechanism, and two machine learning
methods, random forest (RF) and support vector machines (SVM). The experimental results showed that DACCN
outperformed other models in most transfer cases with overall classification accuracies ranging from 0.835 to
0.922. The DACCN also performed better in spatially continuous mapping with its predicted crop type maps more
consistent with the reference ones. As an innovative application of transfer learning in crop type mapping, the
methodology proposed in this study effectively addressed the problem of missing labels in target domains and
alleviated the negative impact of domain shift.

1. Introduction mapping a more challenging task.

Satellite remote sensing, as an effective earth observation technique,
Population growth and climate change pose considerable threats to provides a promising data source for crop type mapping. Coarse reso­
food security, and there is an increasing demand for timely and reliable lution imagery, such as from the Advanced Very High Resolution
knowledge about crop geographical distribution to develop yield pre­ Radiometer (AVHRR) and the Moderate-resolution Imaging Spectror­
diction models, generate agricultural statistics, and support food- adiometer (MODIS), has been utilized to generate cropland extent at
security decision making (Wardlow and Egbert, 2008; Zhu et al., broad scales. Particularly, each crop has its unique phenology, and time
2017). Traditional methods of obtaining crop type information involve series imagery allows the detection of vegetation phenological charac­
field surveys and census, which require sophisticated survey infra­ teristics and thus better determination of crop types (Hao et al., 2020).
structure and often introduce time delays and subjectivity (Kuenzer and However, coarse spatial resolution limits its applicability to croplands
Knauer, 2013; Pott et al., 2021). Inter-annual variation of crop types characterized by small patches and high spatial complexity. Landsat
implies frequent agricultural surveys, making large-area crop type satellite imagery has a relatively high spatial resolution, making it easier

* Corresponding author.
E-mail addresses: [email protected] (Y. Wang), [email protected] (L. Feng), [email protected] (Z. Zhang), [email protected] (F. Tian).

https://doi.org/10.1016/j.isprsjprs.2023.04.002
Received 20 October 2022; Received in revised form 16 March 2023; Accepted 1 April 2023
Available online 12 April 2023
0924-2716/© 2023 International Society for Photogrammetry and Remote Sensing, Inc. (ISPRS). Published by Elsevier B.V. All rights reserved.
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

to capture details of crop distribution. However, the 16-day repeat cycle, Since model building does not involve samples from the target domain,
coupled with cloud contamination, may result in reduced data avail­ classification accuracy is more susceptible to domain shift when the
ability during critical periods of crop growth (Jönsson et al., 2018). fitted model is directly transferred and applied to a new region or year.
Sentinel-2 images have demonstrated great potential for crop type Fine-tuning is a common transfer learning technique to address the
mapping, which can be attributed to the finer spectral, spatial and issue of domain shift. It uses a few samples from the target region to
temporal resolutions (Forkuor et al., 2018). Especially, Sentinel-2 in­ adjust the whole or part of the pre-trained model to adapt it to a new
corporates three red-edge spectral bands centered at 705, 740, and 783 task. For example, fine-tuning technique was adopted to tune the pa­
nm, the superiority of which has been emphasized by studies related to rameters of the pre-trained VGG16 and GoogLeNet, and it was demon­
the detection of biophysical parameters such as green leaf area index strated to be effective in improving the accuracy of crop classification
(Delegido et al., 2011), Chlorophyll (Bramich et al., 2021) and nitrogen (Nowakowski et al., 2021). However, overfitting can occur during fine-
content (Clevers and Gitelson, 2013). Additionally, multi-temporal tuning process, especially when only a small dataset is available in the
Sentinel-2 data have alleviated the difficulty in determining the target region. Therefore, data augmentation techniques have been used
optimal crop identification period (Vuolo et al., 2018), and effectively to expand the size of the data by adding modified or synthetic versions of
improved the accuracy of cropland extent identification (Watkins and the existing data. For example, six pre-trained CNNs were employed to
Van Niekerk, 2019) and vegetation classification (Wakulińska and identify crop diseases, and a data augmentation technique was utilized
Marcinkowska-Ochtyra, 2020). to improve generalization and prevent overfitting (Aravind and Raja,
The construction of the relationship between spectral properties and 2020). Compared with the aforementioned supervised methods, unsu­
crop types is the core process of remote sensing-based crop type map­ pervised domain adaptation (UDA) methods can transfer knowledge
ping, and the performance of the algorithm that implements this process learned from a source domain with a large number of labeled training
directly affects the final mapping accuracy. In this context, various samples to a target domain with only unlabeled data. As a UDA frame­
machine learning methods like support vector machines (SVM) and work, a deep adaptation network (DAN) was proposed by Long et al.
random forest (RF) have been applied for crop classification (Saini and (2015), which can alleviate the negative impact of domain shift by
Ghosh, 2018). Recently, deep learning methods characterized by deeper aligning feature distributions between source and target domains. Given
and more complex structures have brought revolutionary advances in the effectiveness of DAN in transferable representation learning, its
various research fields including agricultural applications (Feng et al., potential in crop type mapping deserves to be further explored.
2021b; Ma et al., 2021). Research on crop type mapping particularly In this study, we developed a deep adaptation crop classification
benefit from deep learning because of its ability to perform automatic network (DACCN) based on DAN framework to address the issue of
high-level representations from data (Kussul et al., 2017). For example, missing labels of target regions and years during crop type mapping.
long short-term memory (LSTM) network was employed to extract Specifically, four states in the U.S. Corn Belt and two provinces in
temporal characteristics from a sequence of Sentinel-2 observations to northeastern China were used as study areas, where samples used for
enhance the accuracy of crop classification (Rußwurm and Korner, model building and accuracy evaluation were collected based on time-
2017). Another study showed that convolution neural network (CNN) series Sentinel-2 imagery and reference maps in 2018 and 2019. Three
outperformed XGBoost, RF, and SVM, with an overall accuracy of experiments were then designed to verify the effect of model transfer in
85.54% in summer crop classification (Zhong et al., 2019). Despite the space, time, and space–time respectively. In each experiment, the pro­
success of these models, they require extensive agricultural ground truth posed DACCN was compared to two machine learning methods, RF and
to generate predictive function mapping from data to labels, and ground SVM, and a deep learning method, deep crop classification network
samples are usually collected through field surveys which are labor- (DCCN), which is structurally similar to DACCN but has no domain
intensive and time-consuming. Especially, it is difficult to conduct adaptation mechanism. Finally, the established models were used for
large-scale field surveys and to make findings public in many parts of the crop type mapping in multiple regions, and the superiority of DACCN
world due to economic, administrative, or other reasons. was further interpreted.
Transfer learning is a machine learning method that uses knowledge
gained from a previous task to enhance the learning of a related task. In 2. Materials
the last few years, it has been introduced into remote sensing-based crop
type mapping to address insufficient ground samples, thereby reducing 2.1. Study areas
labeling efforts (Hao et al., 2020; Nowakowski et al., 2021). Directly
applying the models established in regions with abundant samples to The U.S. and China are the leading agricultural producers in the
other regions has been considered as a simple and effective transfer world, and both have a significant impact on global food security.
method. The continental United States (U.S.) is generally regarded as a Therefore, this study selected four states in the U.S. (Fig. 1a) and two
sample-rich region due to the publicly available and widely used provinces in China (Fig. 1b) as the experimental areas. The four U.S.
Cropland Data Layer (CDL). For example, high confidence pixels in the states are Minnesota (MN), Nebraska (NE), Wisconsin (WI), and Iowa
CDL and corresponding satellite based Normalized Difference Vegeta­ (IA). They are all located in the Corn Belt, an agriculturally productive
tion Index (NDVI) time series were selected to train the RF classification region that benefits from fertile soils, level land, and temperate climate.
models, which were then applied to identify crop types in three target In 2018 and 2019, the study areas in the U.S. had an average temper­
regions (Hao et al., 2020). In addition to cross-regional transfer, inter- ature of about 7 ◦ C and an average monthly precipitation of about 84
annual transfer is also observed in crop type mapping related studies. mm according to the fifth generation European Centre for Medium-
For example, RF classifiers developed using the crop reference samples Range Weather Forecasts atmospheric reanalysis dataset (ERA5). The
and Sentinel-1/2 data obtained in 2020 were successfully applied in selected Chinese provinces are Heilongjiang (HLJ), and Jilin (JL), both
2019 to discriminate crop types in Hetao Irrigation District of China, located in Northeast China. The heartland of the region is the large­
with an overall accuracy of 0.92 (Hu et al., 2022). The intuition behind st plain in China, the Northeast Plain, which has extensive amounts of
direct transfer is that crops of the same type have similar temporal arable land widely covered with chernozem. According to the ERA5
growth patterns. However, due to meteorology-induced phenological dataset, the average temperature in the Chinese study areas in 2018 and
and crop calendar differences, spectral information for crops of the same 2019 was about 4 ◦ C, and the average monthly precipitation was about
type may vary across regions and years. The difference between source 63 mm. Among the various crops, corn and soybean are the dominant
and target data distribution is known as domain shift, which can explain crops in the Corn Belt and Northeast Plain. Therefore, we focused on the
the performance degradation in the application of the model from the identification of corn and soybean in this study.
source domain to the target domain (Gadiraju and Vatsavai, 2020).

103
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 1. Study areas in (a) the U.S. and (b) China.

2.2. Satellite imagery 2.4. Data preprocessing

Sentinel-2 is an earth observation mission that currently includes two The CDLs and crop type maps for 2018 and 2019 were first collected
satellites, Sentinel-2A and Sentinel-2B, which were launched in 2015 to extract the labels of the samples. The low-confidence pixels in the
and 2017, respectively. Each satellite provides global coverage of CDLs were filtered out by thresholding the confidence layer at 90%. The
Earth’s surface every 10 days, resulting in a combined satellite CDLs were upsampled to 10 m to get the same resolution as the crop type
constellation revisit of 5 days. Moreover, each carries a multi-spectral maps. Since we only focus on the classification of corn and soybean,
instrument that provides 13 spectral bands spanning from the visible other categories are grouped into “others”. For each of the three cate­
and near infrared to the shortwave infrared at a spatial resolution of up gories, 3000 pixels were randomly selected in each study area.
to 10 m. Top-of-atmosphere (TOA) reflectance images (level-1C) in 2018 Following You et al. (2021), we then preprocessed the satellite imagery.
and 2019 were included in this study because the reliability of TOA has Specifically, satellite observations corresponding to the selected labels
been demonstrated in previous research (d’Andrimont et al., 2020; Ni were extracted from the beginning of April to the end of October based
et al., 2021). Specifically, 10 bands were used as spectral features, on the growth calendar for corn and soybean, and cloudy observations
including Red, Blue, Green, Red-edge1, Red-edge2, Red-edge3, Near- were removed using a cloud masking algorithm (Oreopoulos et al.,
Infrared (NIR), Red-edge4, Short wave infrared (SWIR) 1, and SWIR2. 2011). 10-day composites were generated using the median values of the
Using the Red, NIR, and SWIR1 bands, two vegetation indices including valid observations, with data gaps filled by linear interpolation. Further,
NDVI and Normalized Difference Water Index were calculated as com­ Savitzky-Golay algorithm was adopted to smooth the resulting time
plementary features with the formulas shown in Eq. (1)-(2). series data by fitting successive seven adjacent data points with a
quadratic polynomial (Feng et al., 2021a). Finally, a total of 252 features
NIR − Red
NDVI = (1) were adopted for modeling, including 12 features and 21 temporal ob­
NIR + Red
servations for each. The above data preprocessing steps were performed
NIR − SWIR1 on Google Earth Engine platform.
NDWI = (2)
NIR + SWIR1
3. Methods

2.3. Reference data 3.1. Deep adaptation crop classification network (DACCN)

This study included two types of reference maps as alternatives to Traditional deep neural network (DNN) associates input samples x ∈
ground truth data, namely CDL released by the U.S. Department of X with labels y ∈ Y, where X and Y are input feature space and output
Agriculture (USDA) and crop type maps published by You et al. (2021). space, respectively. Here in crop classification, x are spectral bands and
The USDA CDL is one of the most successful crop-specific land cover VIs and y are crop type labels. Given a source domain D s = {(xsi , ysi )}ni=1
s

maps in terms of annual update frequency, national coverage, 30 m with ns labeled samples and a target domain of D t = {(xti , yti )}ni=1
t
with nt
spatial resolution and high classification accuracy, and is therefore unlabeled samples were characterized by probability distributions p and
widely used as reference data in agricultural applications (Boryan et al., q respectively. Generally, p is usually different from q due to
2011; Shrestha et al., 2017). It contains information on the distribution meteorology-induced phenological difference. Consequently, a model
of over 100 major crops grown in the U.S. The production of CDL in­ trained with labeled samples from D s usually has unsatisfactory per­
tegrates multi-source data, including various satellite imagery, elevation formance when directly applied to D t .
data, agricultural census data, etc. More than 100 crop categories can be To deal with the issue, DAN framework was introduced to reduce the
identified from CDL. The Confidence Layer associated with CDL is also impact of domain shift. The framework is mainly composed of two parts,
provided to represent the predicted confidence of each pixel based on including a feature extractor Gf and a domain aligner Gd . Data samples
the ruleset used for classification. Since there is no official CDL released
from D s and D t are first fed to the feature extractor Gf . Feature
in China, the regional mapping product published by You et al. (2021)
extractor Gf has a flexible structure and is used to convert the original
covering Northeast China was used as an alternative. The annual 10 m
input features to high-level representations. Then, the extracted features
crop type maps from 2017 to 2019 was generated using Sentinel-2 time
are forwarded into domain aligner Gd to predict the crop type labels.
series data and abundant ground truth. Three major crops including
Domain aligner Gd is the part that distinguishes DAN from traditional
corn, soybean, and rice can be identified with an overall classification
DNN. Gd is generally composed of L fully connected layers, and the
accuracy of over 80%. Moreover, the crop area estimates derived from
discrepancy between source and target distributions is measured on
the crop type maps were found to be in good agreement with the sta­
each of them except the last. Multiple kernel variant of maximum mean
tistics of most prefecture-level cities.

104
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

discrepancy (MK-MMD) can be used to estimate the distribution differ­

ence. As a kernel two sample test, MK-MMD accepts or rejects the null
hypothesis p = q, given samples generated respectively from p and q.
The formula for MK-MMD is defined in Eq. (3).

(3)
2
dk (p, q)≜‖Ep [ϕ(xs )] − Eq [ϕ(xt )]‖H k

where E[•] represents the mathematical expectation, H k denotes the

reproducing kernel Hilbert space (RKHS) endowed with a characteristic
kernel k. Kernel k is defined as a combination of m kernels, and it as­
sociates with the feature map ϕ(•) which is used to map the original
samples to RKHS.
DAN reduces domain shift and makes precise classification by
simultaneously minimizing the classification loss and domain adapta­
tion loss (Long et al., 2015). Specifically, the classification loss part is
generally used for measuring the performance of a classification model,
and cross entropy loss is the most popular in classification problems. The
domain adaptation loss part is added to make the distributions of the
source and target closer under the hidden representations of the fully
connected layers, which can be realized by MK-MMD. Therefore, the
objective of DAN can be formulated as:

1 ∑ ns
( ( ) ) ∑ ( )
min J θ xsi , ysi + λ ̂ d k pl , ql (4)
Θ ns i=0 l∈L

where Θ represents the operation of DAN, θ(xsi ) is the conditional

probability that DAN assigns xsi to label ysi , and λ is a trade-off parameter
that controls the relative importance of classification loss and domain
adaptation loss.
In this study, we proposed DACCN based on the DAN framework.
Specifically, a simplified ResNet-50 is adopted as the feature extractor.
ResNet, short for residual network, overcomes the vanishing gradient
problem of traditional CNNs and allows for training of extremely deep
networks by introducing shortcut connections (He et al., 2016). As a Fig. 2. The structure of (a) ResNet 50, (b) identity block, and (c) convolu­
popular variant of ResNet, ResNet-50 is consist of five stages (Fig. 2a). At tional block.
the first stage, the feature map size is downsampled through a con­
volutional layer with stride 2, followed by batch normalization, ReLU the trade-off parameter λ in the objective of DACCN was set as 0.5.
activation, and max pooling operations. The second to the fifth stages Stochastic gradient descent was used as the optimization algorithm to
consist of 3, 4, 6, and 3 bottleneck blocks, respectively. Each bottleneck update trainable parameters with an initial learning rate of 0.001 and a
block is composed of three convolutional layers with filter sizes of 1 × 1, decay pattern as shown in Eq. (5).
3 × 3, and 1 × 1, respectively. Each convolutional layer is followed by a [ ]0.75
batch normalization and a ReLU activation function. Also, a shortcut lri = 1 +
10 × (i − 1)
(5)
connection bypass the operations and add the original input to the nmax
output of the bottleneck block before the last activation function. There
are two types of shortcuts, identity shortcut and projection shortcut. The where lri represents the learning rate of the ith epoch and nmax denotes
identity shortcut corresponds to the case where the input and output are the predefined maximum epoch. The model was trained using an early
of the same dimensions, otherwise, a projection shortcut is adopted, stopping trick with the patience and maximum of 50 and 500 epochs
which uses a 1 × 1 convolution layer to match the dimensions. Identity respectively.
shortcuts and projection shortcuts further differentiate blocks into
identity blocks (Fig. 2b) and convolutional blocks (Fig. 2c). After stage
3.2. Experimental setup
five, the model outputs the final classification result through an average
pooling layer and a fully connected layer. Since the input size in this
Three experiments were designed to evaluate the generalization and
study is 12 × 21, which is much smaller than the default size that
transferability of the proposed model across space, time, and space­
ResNet-50 takes, we dropped the part after stage 3 to reduce the number
–time, respectively. In the first experiment, models built on MN, and NE
of down-sampling operators. Moreover, for simplicity, domain aligner
were first transferred to WI, and IA respectively, and then to HLJ, and JL
was designed to have two fully connected layers, on the first of which the
respectively. During the process, the years of the source and target do­
discrepancy between the distribution of source and target representa­
mains remained the same. In the second experiment, models were built
tions is measured. The multi-kernel used in discrepancy measurement
separately in the six regions, and each model was transferred to a
MK-MMD was constructed by five Gaussian kernels.
different year in the same region. The design of the third experiment is
The proposed DACCN was finalized with its structure shown in Fig. 3.
similar to that of the first experiment, except that the years of the source
It starts with a convolution layer and a max pooling layer with filters of
and target domains are different. In all experiments, samples from each
size 7 × 7 and 3 × 3 respectively followed by seven bottleneck blocks.
source domain were randomly divided into two parts, two thirds were
The hidden representations extracted from the above feature extractor
used for training and the remaining one third are used for validation.
are flattened to one dimension before being forwarded into the domain
Samples on the target domain were used to test the performance of the
aligner, which contains a hidden layer with 256 neurons followed by an
final models. Therefore, a total of 18,000 samples were involved in each
output layer. In the training process, the batch size was set as 512 and
transfer case, including 6000 training samples, 3000 validation samples,

105
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 3. The structure of deep adaptation crop classification network.

and 9000 testing samples. To verify the superiority of DACCN, we 4. Results

implemented three models for comparison: DCCN, RF and SVM. For a
fair comparison, DCCN was designed to have a similar architecture as 4.1. Spectral characteristics of crops
DACCN but without discrepancy measurement between source and
target domains, or domain adaptation loss part in the objective function. To clarify the similarities and differences of the spectral character­
All models were run on a computing platform featuring an Intel Xeon istics of crops in each region and year, we presented the NDVI temporal
Gold 6330 CPU, an RTX 3090 GPU with 24 GB memory, 32 GB RAM and profiles of corn, soybean, and “others” in Fig. 4. The mean profiles of
1 TB storage. The deep learning models were developed with Pytorch “others” are significantly different from those of corn and soybean, and
1.10, and the machine learning models were implemented with Scikit- the standard deviation (std) of “others” is much larger at each time
learn 1.2.1. A confusion matrix was utilized to describe the distribu­ point. This is mainly because “others” includes various land use types,
tion of all the predicted responses, with each row representing an actual each with their own spectral characteristics. The NDVI profiles of corn
class and each column representing a predicted class. Overall accuracy and soybean are of a highly similar trend in the same region and the
(OA), macro-averaged F1 score (F1), and Cohen’s kappa coefficient same year. Specifically, the mean NDVI values remain stable or increase
(Kappa) were adopted to represent the overall performance of the slightly before ascending steeply from about DOY 150. After that, the
models, with their formulas shown in Eq. (6)-(8). values peak between DOY 200–250 and then decrease gradually before
finally stabilizing between 0 and 0.2. The temporal profiles of NDVI are
1 ∑n
OA = xii (6) consistent with the phenology of the crops. Corn and soybean are usually
N i=1 planted in spring. Seedlings emerge within days to weeks after planting,
followed by rapid vegetative and reproductive phases (Hanway, 1966;
2∑ n
Pi Ri Purcell et al., 2014). When crops reach full maturity, green foliage turns
F1 = (7)
n i=1 Pi + Ri yellow, and harvesting is then carried out. However, the temporal NDVI
∑n ∑n profiles of crops in MN and WI exhibit an abnormal small peak before
N i=1 xii − x x DOY 150 in 2018 (Fig. 4 (a1) and (a2)). A similar phenomenon has also
Kappa = ∑n i=1 +i i+ (8)
2
N − i=1 x+i xi+ been observed by Gao et al. (2020). This can be explained by the
apparent green-up of weeds induced by favorable weather in May and
where N is the number of samples, n is the number of classes, xii denotes sufficient soil moisture provided by early spring snowfall (USDA, 2018).
the ith diagonal value of the confusion matrix, x+i and xi+ represent the There are also slight differences between the NDVI profiles of corn and
sum of the values in the ith column and ith row of the confusion matrix, soybean. The green-up of soybean is slower than for corn while the
respectively. Pi and Ri are the precision and recall of class i, and they can drying and leaf fall are more rapid, which results in a slower increase
be calculated using Eq. (9)-(10). and a faster decrease in the NDVI profiles of soybean.
xii Distinct differences were discernible between NDVI temporal pro­
Pi = (9)
x+i files of crops across regions and years in terms of the timing of green-up,
peak greenness, and senescence. For example, crop growth progress in
Ri =
xii
(10) the four U.S. states remained ahead of the two provinces in China
xi+ throughout the year 2018, which is closely related to the differences in
geographical environment and management practices between the two
To further provide an insight into the effects of DACCN’s domain
countries. There are also differences in crop phenology between states/
adaptation, the feature distributions were visualized using t-distributed
provinces, but not as pronounced as differences between countries.
Stochastic Neighbor Embedding (t-SNE) algorithm. The intuition behind
Moreover, the four U.S. states experienced a later planting of corn and
t-SNE is to map the relationship between high-dimensional data to a low
soybean in 2019 due to an exceptionally wet spring, which can be easily
dimensional space, that is, for data that are far apart in high-dimensional
detected by comparing the NDVI profiles for each state in 2018 and
space, the distance will always be larger after dimensionality reduction
2019. It is the differences in crop phenology between regions and years
(Van der Maaten and Hinton, 2008). Using t-SNE, we visualized and
that lead to the problem of domain shift. Therefore, great importance
compared the feature distributions before and after domain adaptation
should be attached to narrow down the discrepancy between source and
during the transfer across space–time (from 2018 to 2019). The feature
target distributions.
distributions before and after domain adaptation were represented by
the feature distributions of the input of DACCN and the output of the
penultimate fully connected layer of the model.

106
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 4. The Normalized Difference Vegetation Index (NDVI) temporal profiles of crops in (1) Minnesota, (2) Wisconsin, (3) Heilongjiang, (4) Nebraska, (5) Iowa, and
(6)Jilin in (a) 2018 and (b) 2019.

4.2. Model performance and comparison However, it was observed that the performance of DCCN, RF, and SVM
degraded in the cases of cross-country transfer due to domain shift. The
4.2.1. Transfer across space DACCN reduced the domain discrepancy by narrowing the distance
The first experiment was carried out to evaluate the models’ trans­ between hidden representations of the source and target, and thereby
ferability across regions. In each testing year, models trained in two U.S. significantly outperformed other models. For example, in 2019, the
states (MN and NE) were first transferred to the other two U.S. states (WI DACCN made accurate predictions in the transfer case of NE to JL with
and IA), respectively, and then to two Chinese provinces (HLJ and JL), an OA of 0.855, which is more than 7.6% higher than other models.
respectively. The results of model training, validation, and testing in The confusion matrix was calculated to further show the distribution
2018 and 2019 are reported in Table S1, Table S2, and Table 1, of predictions for each class (Fig. 5). The diagonal and off-diagonal el­
respectively. We focus on the model performance on the test data ements of the confusion matrix represent correctly classified and mis­
because it represents the ability of the models to transfer across space. classified outcomes, respectively. Comparing the four models, the
The proposed DACCN model performed best in each testing year, with DACCN exhibited higher accuracy and stability in differentiating the
OA values exceeding 0.85. In the domestic transfer cases, all models three classes. In particular, the DACCN maintained satisfactory perfor­
made satisfactory predictions with insignificant gaps in accuracy. mance in the cases of cross-country transfer, which significantly

Table 1
Classification accuracies of deep adaptation crop classification network (DACCN), deep crop classification network (DCCN), random forest (RF), and support vector
machines (SVM) in transfer cases across space.
Source Target Source Target DACCN DCCN RF SVM
year year region region
OA F1 Kappa OA F1 Kappa OA F1 Kappa OA F1 Kappa

2018 2018 MN WI 0.870 0.870 0.805 0.848 0.847 0.771 0.847 0.845 0.771 0.848 0.847 0.772
NE IA 0.884 0.884 0.826 0.870 0.870 0.806 0.883 0.883 0.824 0.881 0.881 0.821
MN HLJ 0.866 0.867 0.800 0.808 0.809 0.711 0.843 0.844 0.765 0.840 0.840 0.760
NE JL 0.855 0.855 0.783 0.750 0.749 0.626 0.809 0.812 0.714 0.828 0.830 0.742
2019 2019 MN WI 0.867 0.866 0.800 0.852 0.851 0.779 0.851 0.849 0.776 0.839 0.839 0.759
NE IA 0.905 0.906 0.858 0.885 0.885 0.828 0.905 0.905 0.857 0.902 0.902 0.853
MN HLJ 0.864 0.863 0.795 0.791 0.789 0.686 0.835 0.829 0.752 0.812 0.809 0.718
NE JL 0.855 0.854 0.783 0.738 0.731 0.607 0.788 0.780 0.682 0.795 0.788 0.692

107
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 5. The confusion matrices of (a) deep adaptation crop classification network, (b) deep crop classification network, (c) random forest, and (d) support vector
machines in the cross-space transfer cases of (1) Minnesota → Wisconsin, (2) Nebraska → Iowa, (3) Minnesota → Heilongjiang, and (4) Nebraska → Jilin.

degraded the performance of other models. DCCN, RF, and SVM always annual transfer in JL province. In the two provinces in China, all
failed to accurately predict a certain crop type in cross-country transfer models performed well in cross-year transfer, indicating that the climate
cases. For example, in the transfer case from NE to JL in 2019, models change in Northeast China was insignificant from 2018 to 2019 and thus
other than DACCN had problems identifying soybean, with more than a the effect of domain shift was not severe. On the other hand, the DACCN
third of samples labeled soybean being misclassified as corn. significantly outperformed other models in inter-annual transfer in the
U.S. states, especially in IA. This can be explained by the difference in
4.2.2. Transfer across time the temporal spectral profiles of the Corn Belt region between 2018 and
The second experiment was designed to evaluate the models’ efficacy 2019, as shown in Fig. 4.
in inter-annual transfer. Specifically, the models built on each of the six We employed a number of confusion matrices to further demonstrate
states/provinces were transferred to a different year in the same region, the classification performance of the models (Fig. 6). The proposed
with the results of model training, validation, and testing shown in DACCN model showed the best performance, with over 80% correct
Table S3, Table S4, and Table 2. The model performance on the test set predictions for each crop type in all cases, followed by DCCN. Addi­
indicates the transferability of models across time. The DACCN per­ tionally, the DACCN performed stably in all cases of inter-annual
formed better than other models in most cases except for the inter- transfer, while other models performed better in the two northeastern

108
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Table 2
Classification accuracies of deep adaptation crop classification network (DACCN), deep crop classification network (DCCN), random forest (RF), and support vector
machines (SVM) in transfer cases across time.
Source Target Source Target DACCN DCCN RF SVM
year year region region
OA F1 Kappa OA F1 Kappa OA F1 Kappa OA F1 Kappa

2018 2019 MN MN 0.879 0.879 0.819 0.858 0.858 0.788 0.825 0.822 0.738 0.841 0.838 0.762
NE NE 0.922 0.922 0.883 0.904 0.904 0.856 0.900 0.899 0.850 0.899 0.898 0.848
WI WI 0.860 0.860 0.790 0.853 0.853 0.780 0.839 0.836 0.758 0.816 0.810 0.724
IA IA 0.886 0.886 0.829 0.822 0.821 0.733 0.797 0.788 0.696 0.786 0.774 0.680
HLJ HLJ 0.922 0.922 0.883 0.914 0.914 0.871 0.915 0.915 0.872 0.908 0.907 0.861
JL JL 0.901 0.901 0.851 0.889 0.889 0.834 0.901 0.901 0.851 0.896 0.896 0.844
2019 2018 MN MN 0.893 0.892 0.839 0.832 0.831 0.748 0.834 0.833 0.751 0.759 0.755 0.639
NE NE 0.924 0.924 0.886 0.903 0.903 0.855 0.868 0.866 0.801 0.862 0.861 0.794
WI WI 0.845 0.845 0.767 0.797 0.797 0.695 0.827 0.825 0.741 0.780 0.778 0.670
IA IA 0.878 0.879 0.818 0.838 0.839 0.757 0.835 0.834 0.753 0.811 0.811 0.717
HLJ HLJ 0.923 0.923 0.885 0.916 0.917 0.875 0.899 0.900 0.849 0.908 0.908 0.862
JL JL 0.906 0.907 0.859 0.904 0.905 0.856 0.911 0.911 0.866 0.899 0.900 0.849

Chinese provinces than in the U.S. Corn Belt, which was consistent with adaptation than before, especially for groups that transfer across coun­
the results shown in Table 2. tries. The above two visualization experiments intuitively demonstrate
the superiority of DACCN in alleviating domain shift and improving the
4.2.3. Transfer across space–time feature discrimination.
The third experiment aimed to evaluate the performance of the
models when transferred simultaneously across regions and years.
4.4. Crop type mapping
Specifically, the models trained using samples collected from MN and
NE in year A were first transferred to WI and IA in year B, and then to
We further carried out spatially continuous crop type mapping.
HLJ and JL in year B, where A and B are different. Table S5 and Table S6
Specifically, a county was first randomly selected from each of the four
show the results of model training and validation. Table 3 shows the
target regions, resulting in four selected counties of Marquette in WI,
model performance in the transfer cases across space–time. The DACCN
Marshall in IA, Keshan in HLJ, and Yushu in JL. Then, the transfer
significantly outperformed other models in all the cases (Table 3), which
models cross space–time (from 2018 to 2019) trained in Section 4.2.3
further demonstrated the superiority of DACCN in alleviating domain
were used for crop type mapping in the selected counties. The reference
shift.
maps of U.S. counties were produced based on CDL which combines
The confusion matrices shown in Fig. 7 present the distribution of the
multi-source satellite images and extensive agricultural ground truth.
predicted responses for the four models. The accuracy and robustness of
The CDL was first created in 1971, and it has been continuously updated
DACCN were further demonstrated by the high proportion of correct
(Craig, 2010) and widely used in agricultural applications (Ray et al.,
predictions for the three classes in each case. The other three models
2018; Sakamoto et al., 2014). Due to the unavailability of the official
were inferior to DACCN in all transfer cases, which illustrated the
CDL in China, the crop type maps produced in You et al. (2021) were
importance of including the mechanism of domain adaptation.
used as the reference maps for counties in China. In contrast, the data
may not be as reliable as CDL, because few studies have conducted in­
4.3. Interpretation of DACCN dependent validation of the data.
Marquette is located in the south-central part of Wisconsin, with an
To investigate the effects of DACCN’s domain adaptation, we visu­ area of about 1200 km2 . It has the lowest proportion of corn and soybean
alized and compared the distributions of original spectral features planting areas among the six counties, and the planting areas of the two
(Fig. 8) and the distributions of the extracted features after domain crops are scattered throughout the county. Marshall is a county located
adaptation using DACCN (Fig. 9) during the transfer across space–time in central Iowa and covers an area of about 1480 km2 . Corn and soybean
(from 2018 to 2019) in Section 4.2.3. From the first two columns of are widely grown in this county, except along the Iowa River, especially
Fig. 8, we can see that the original features of corn and soybean from the Marshalltown. The counties selected from China’s Heilongjiang and
two domains are clearly separated, indicating the existence of distri­ Jilin provinces are Keshan and Yushu, which are larger than the counties
bution discrepancy between the source and target domains. The sepa­ selected from the U.S. In Keshan, corn and soybean are extensively
ration is caused by a combination of regional differences in natural distributed and the soybean planting area is larger. The main food crop
conditions and farming management practices, as well as inter-annual in Yushu is corn, whose planting area is significantly larger than that of
climate variability, and is the reason for the performance degradation soybean. All four counties have a humid continental climate charac­
when the trained models are directly applied to new domains. Unlike terized by four distinct seasons and large seasonal temperature differ­
corn and soybean, the “others” category includes a variety of land use ences. Marquette and Keshan are located at higher latitudes and
types such as developed land, water body, and vegetation. Their spectral therefore have colder and longer winters and milder summers.
characteristics exhibit different degrees of homogeneity and heteroge­ We compared the differences between the reference maps and the
neity in space and time, making the original data of “others” from the models’ predictions shown in Fig. 11. The crop maps generated by
two domains less separated. After the domain adaptation, the distribu­ DACCN were most consistent with the reference maps, especially in
tions of the extracted feature from the source and target domains Marshall, where the predicted map and the reference map are basically
became closer, indicating that the proposed DACCN could effectively the same. All models were able to distinguish corn and soybean from
reduce the distribution discrepancy and overcome domain shift. “others” in the crop type mapping for Marquette, but all tended to
We further visualized the feature distributions for the three cate­ misclassify corn as soybean. Among the models, the DACCN has the
gories before and after domain adaptation (Fig. 10). The three categories weakest misclassification tendency. Additionally, all models had a ten­
are color-coded, with green, orange, and blue representing corn, soy­ dency to predict soybean as corn in Keshan. The DACCN showed the
bean, and “others”, respectively. It was notable that the feature lowest degree of misclassification. In the crop type mapping of Yushu,
discrimination for the three categories is much larger after domain the DACCN roughly identified the Lalin River running through the

109
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 6. The confusion matrices of (a) deep adaptation crop classification network, (b) deep crop classification network, (c) random forest, and (d) support vector
machines in the cross-time transfer cases in (1) Minnesota, (2) Nebraska, (3) Wisconsin, (4) Iowa, (5) Heilongjiang, and (6) Jilin.

110
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Table 3
Classification accuracies of deep adaptation crop classification network (DACCN), deep crop classification network (DCCN), random forest (RF), and support vector
machines (SVM) in transfer cases across space–time.
Source Target Source Target DACCN DCCN RF SVM
year year region region
OA F1 Kappa OA F1 Kappa OA F1 Kappa OA F1 Kappa

2018 2019 MN WI 0.837 0.834 0.755 0.788 0.782 0.682 0.755 0.735 0.633 0.740 0.721 0.610
NE IA 0.895 0.895 0.843 0.848 0.848 0.772 0.834 0.832 0.752 0.843 0.842 0.765
MN HLJ 0.835 0.835 0.752 0.796 0.797 0.694 0.831 0.831 0.747 0.833 0.832 0.750
NE JL 0.835 0.833 0.752 0.745 0.743 0.618 0.796 0.796 0.694 0.800 0.802 0.701
2019 2018 MN WI 0.849 0.848 0.773 0.757 0.752 0.635 0.745 0.740 0.617 0.649 0.640 0.474
NE IA 0.883 0.883 0.824 0.839 0.840 0.759 0.818 0.817 0.727 0.789 0.786 0.684
MN HLJ 0.870 0.870 0.804 0.805 0.806 0.708 0.850 0.847 0.776 0.848 0.847 0.772
NE JL 0.847 0.846 0.771 0.751 0.749 0.626 0.803 0.801 0.705 0.819 0.817 0.728

Fig. 7. The confusion matrices of (a) deep adaptation crop classification network, (b) deep crop classification network, (c) random forest, and (d) support vector
machines in the transfer cases across space–time: (1) Minnesota → Wisconsin, (2) Nebraska → Iowa, (3) Minnesota → Heilongjiang, and (4) Nebraska → Jilin.

111
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 8. The original feature distributions of (a) corn, (b) soybean, and (c) others in the source and target domains in the transfer cases of (1) 2018 Minnesota → 2019
Wisconsin, (2) 2018 Nebraska → 2019 Iowa, (3) 2018 Minnesota → 2019 Heilongjiang, and (4) 2018 Nebraska → 2019 Jilin.

county and the Songhua River in the southwest. But other models, domain shift. Despite some misclassifications, the DACCN model per­
especially DCCN and SVM, misclassified a large portion of the water as formed satisfactorily in Marshall, where the planting rates of corn and
corn. soybean are high and similar. In addition to corn and soybean, land use
Based on the reference and predicted crop type maps, we further types such as alfalfa, grassland, forest, and wetlands are also widely
obtained the classification accuracy of the four models for the four present in Marquette and Marshall. The DACCN predicted them well
counties, which was measured by OA, F1, and Kappa (Table 4). The with only a few misclassifications. Due to the absence or low rate of rice
DACCN outperformed the other models in all four counties, with the planting in the two source U.S. states, MN and NE, few samples labeled
largest OA, F1, and Kappa values. Especially in Marshall, the DACCN as rice were selected into the training dataset. Therefore, it can be
demonstrated superior performance, with all three metrics above 0.8. observed that rice misclassification rates are high in the two target
The superiority of DACCN was also evident in the crop type mapping of counties in China. In addition to rice, crops such as sorghum and po­
Yushu, where the OA, F1, and Kappa were more than 3.4%, 9.1%, and tatoes that are widely grown in target counties also have low planting
18.3% higher than those of the other models, respectively. The results of rates in the two source U.S. states, resulting in very limited corre­
quantitative statistics are consistent with those presented in Fig. 11. sponding training samples. This could explain the high misclassification
Furthermore, to investigate the mapping ability of DACCN for the rates in “others” category in the two target counties in China.
four counties, we presented the flow from reference values to predicted
values using Sankey diagrams (Fig. 12). For the two counties in the U.S., 5. Discussion
we selected the nine most common land cover types from the county’s
corresponding CDL in 2019, combined the other categories into one 5.1. Advantages of DACCN
category, and used these 10 categories as the source nodes of the Sankey
diagram. For the two counties in China, due to the limited crop types in Most existing crop type mapping related studies rely heavily on a
the reference maps, the Sankey diagrams contained only four source large number of local labeled data for modeling and making predictions
nodes, namely corn, soybean, rice, and “others”. For each county, the (Feng et al., 2019; Prins and Van Niekerk, 2021), and thus tedious and
target nodes of the Sankey diagram are corn, soybean, and “others”. costly sample collection needs to be carried out extensively and
From Fig. 12a, we can see that about one-third of corn in Marquette frequently. The sharing of collected labeled samples is an effective way
is mistakenly divided into soybean, which may be related to several to address the dilemma of ground truth sampling. However, due to the

112
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 9. The extracted feature distributions of (a) corn, (b) soybean, and (c) others in the source and target domains in the transfer cases of (1) 2018 Minnesota →
2019 Wisconsin, (2) 2018 Nebraska → 2019 Iowa, (3) 2018 Minnesota → 2019 Heilongjiang, and (4) 2018 Nebraska → 2019 Jilin.

differences in climate conditions across regions and years, trained crop of the DACCN model in transferring and utilizing samples from different
classification models may lose their validity when applied to new do­ regions and years for crop type mapping.
mains. Therefore, the DACCN was developed to address the distribution
discrepancy existed between the source and target domains, that is,
domain shift. 5.2. Analysis of crop type mapping
To fully verify the superiority of the DACCN model, we designed
three transfer experiments across space, across time, and across space­ In this study, spatial continuous mapping was performed to test the
–time. The proposed model outperformed other models in most cases of usability of the proposed DACCN model in practical applications. One
the transfer experiments with OAs ranging from 0.835 to 0.922. Espe­ county was selected for each target region to make the evaluation results
cially, the DACCN achieved 0.09 higher OA than other models in the more comprehensive. The mapping results demonstrated the superiority
transfer case from 2019 MN to 2018 WI. The feature distributions before of DACCN over other models. Especially in Marshall, the crop map
and after domain adaptation were further visualized using t-SNE. It was generated by DACCN was in good agreement with the reference map
found that DACCN effectively reduced the distribution discrepancy be­ (Fig. 11). A quantitative evaluation of the mapping results was further
tween source and target domains, and improved the separability among carried out. Again, the DACCN model performed best among all the
different crops. models, with the OA values ranging from 0.763 to 0.893. Therefore, the
Different from traditional models, the DACCN model not only uses proposed DACCN model performed satisfactorily not only in sample-
training samples but also includes unlabeled samples from the target based evaluation, but also in mapping-based evaluation.
domain during model training. The distances between feature distribu­ Sankey diagram was then adopted to present the flow of the
tions of source and target domains were measured using MK-MMD in the DACCN’s predictions for various crop types. The misclassification of
hidden layer of DACCN. The domain adaptation is finally achieved by corn as soybean was severe in Marquette. The phenomenon is likely
extending the loss function with MK-MMD estimates. Therefore, in the related to severe domain shift, that is, the spectral characteristics of corn
process of error backpropagation, the classification accuracy of the are significantly different between the source domain (Fig. 4 (a1)) and
model is improved, and the discrepancy in feature distributions between target domain (Fig. 4 (b2)). Specifically, due to warmer temperature,
the source and target domains is also narrowed at the same time. corn plants in MN experienced faster growth until mid-July 2018
Experimental and visualization results also demonstrated the superiority (Grassini et al., 2018), characterized by a steeper rise in NDVI profiles
(Fig. 4 (a1)). Additionally, WI observed a late start and delayed

113
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 10. The feature distributions of (a) corn, (b) soybean, and (c) others before and after domain adaptation in the transfer cases of (1) 2018 Minnesota → 2019
Wisconsin, (2) 2018 Nebraska → 2019 Iowa, (3) 2018 Minnesota → 2019 Heilongjiang, and (4) 2018 Nebraska → 2019 Jilin.

development of corn in 2019 due to prolonged cool and wet spring 5.3. Limitations and future work
(Schnepf, 2020). The misclassification also indicates the limitations of
DACCN in overcoming severe domain shift problem and the need for Despite progress, this study still has some limitations that can be
further improvements. The DACCN model accurately predicted the land improved in the future. First, the source and target domains were chosen
use types such as alfalfa, grassland, forest, and wetland with only a few at relatively close latitudes, where corn and soybean have similar
misclassifications in Marquette and Marshall, suggesting that the spec­ growing seasons. For future studies, in order to complete more chal­
tral characteristics of these objects are quite different from those of corn lenging domain transfer cases, such as trans-hemispheric transfer, the
and soybean. It should be mentioned that the proposed DACCN model is acquisition time of time series features needs to be adjusted from ab­
not necessarily responsible for all misclassifications, since the labels are solute time to relative time of crop growing season. Second, this study
not ground truth but sampled from the reference maps considering the only focused on the identification of corn and soybean, and classified
feasibility of data labeling. Moreover, the spatial resolution mismatch of other land use types as “others”. However, the spectral characteristics of
CDLs and Sentinel-2 imagery could also introduce some inaccurate some crops included in “others” were more similar to those of corn and
labeled samples. In the crop type mapping for the two counties in China, soybean than to other land use types in this category, which could
clear misclassifications could be observed in rice and other categories. explain some of the misclassifications. Therefore, it is necessary to
This can be attributed to the absence or low planting rates of crops such further refine the “others” category. Third, the target domains are
as rice, sorghum, and potatoes in the two source U.S. regions. To deal widely planted with crops that are rarely or not grown in the source
with the problem, the source region should be appropriately expanded domains, resulting in the model lacking the ability to identify such
to include a wider range of crops, and more samples corresponding to crops. Properly expanding the source domains and increasing the pro­
crops with low planting rates need to be included during the process of portion of low-frequency crops in the training dataset are effective so­
training sample acquisition. lutions. Finally, considering the superiority of machine learning in

114
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Fig. 11. The crop maps of (a) Marquette in Wisconsin, (b) Marshall in Iowa, (c) Keshan in Heilongjiang, and (d) Yushu in Jilin from (1) references maps, (2) deep
adaptation crop classification network, (3) deep crop classification network, (4) random forest, and (5) support vector machines.

115
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

Table 4
Classification accuracies of deep adaptation crop classification network (DACCN), deep crop classification network (DCCN), random forest (RF), and support vector
machines (SVM) in the crop type mapping for Marquette, Marshall, Keshan, and Yushu.
Source Target Source Target DACCN DCCN RF SVM
year year region region
OA F1 Kappa OA F1 Kappa OA F1 Kappa OA F1 Kappa

2018 2019 MN Marquette 0.882 0.669 0.625 0.860 0.624 0.570 0.870 0.604 0.575 0.877 0.627 0.581
NE Marshall 0.893 0.889 0.835 0.831 0.833 0.743 0.759 0.769 0.640 0.838 0.839 0.753
MN Keshan 0.763 0.755 0.634 0.713 0.706 0.562 0.754 0.755 0.624 0.751 0.753 0.622
NE Yushu 0.878 0.645 0.586 0.844 0.569 0.450 0.849 0.591 0.495 0.844 0.573 0.441

Fig. 12. The prediction flow of deep adaptation crop classification network for (a) Marquette in Wisconsin, (b) Marshall in Iowa, (c) Keshan in Heilongjiang, and (d)
Yushu in Jilin.

handling multi-dimensional data, we did not perform feature selection most transfer cases with higher classification accuracy. Moreover, the t-
in this study, resulting in a long time spent in the process of crop type SNE based model interpretation further demonstrated the effectiveness
mapping. In the future, the features and model structure will be of DACCN in simultaneously narrowing distribution discrepancy and
simplified to improve mapping efficiency. improving crop separability. The method proposed in this study pro­
vides a new solution for crop mapping in sample-missing regions.
6. Conclusions Meanwhile, it effectively alleviate the issue of domain shift and thus
enhance the transferability of deep learning models. Further study will
In this paper, we proposed an UDA model DACCN for crop type focus on simplifying the structure of the model to improve the efficiency
mapping. Compared with traditional deep learning models that heavily of crop type mapping in large areas, and improving the model’s appli­
rely on local labeled samples, DACCN is able to solve the label missing cability to more challenging transfer scenarios, such as trans-
problem by learning knowledge from labeled samples in other domains hemispheric transfer.
and adaptively transferring the knowledge to the target domains. Three
experiments were designed to evaluate the transferability of the pro­ Funding
posed model across space, time, and space–time, respectively. It was
found that DACCN had better performance than RF, SVM, and DCCN in This work was supported by the National Natural Science Foundation

116
Y. Wang et al. ISPRS Journal of Photogrammetry and Remote Sensing 199 (2023) 102–117

of China (No. 42201354, No. 42001299), the Zhejiang Provincial Nat­ He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition,
pp. 770-778.
ural Science Foundation of China (No. LQ22D010007), the Public Pro­
Hu, Y., Zeng, H., Tian, F., Zhang, M., Wu, B., Gilliams, S., Li, S., Li, Y., Lu, Y., Yang, H.,
jects of Ningbo City (No. 2022S101), Ningbo Science and Technology 2022. An interannual transfer learning approach for crop classification in the Hetao
Innovation 2025 Major Special Project (No. 2021Z107), the Open Fund Irrigation district, China. Remote Sensing 14, 1208.
of Key Laboratory of Urban Land Resources Monitoring and Simulation Jönsson, P., Cai, Z., Melaas, E., Friedl, M.A., Eklundh, L., 2018. A method for robust
estimation of vegetation seasonality from Landsat and Sentinel-2 time series data.
(No. KF-2021-06-089), and the Seed Fund Program for Sino-Foreign Remote Sens. (Basel) 10, 635.
Joint Scientific Research Platform of Wuhan University (No. Kuenzer, C., Knauer, K., 2013. Remote sensing of rice crop areas. Int. J. Remote Sens. 34,
WHUZZJJ202205). 2101–2139.
Kussul, N., Lavreniuk, M., Skakun, S., Shelestov, A., 2017. Deep learning classification of
land cover and crop types using remote sensing data. IEEE Geosci. Remote Sens. Lett.
14, 778–782.
Declaration of Competing Interest Long, M., Cao, Y., Wang, J., Jordan, M., 2015. Learning transferable features with deep
adaptation networks. PMLR 97–105.
The authors declare that they have no known competing financial Ma, Y., Zhang, Z., Yang, H.L., Yang, Z., 2021. An adaptive adversarial domain adaptation
approach for corn yield prediction. Comput. Electron. Agric. 187, 106314.
interests or personal relationships that could have appeared to influence Ni, R., Tian, J., Li, X., Yin, D., Li, J., Gong, H., Zhang, J., Zhu, L., Wu, D., 2021. An
the work reported in this paper. enhanced pixel-based phenological feature for accurate paddy rice mapping with
Sentinel-2 imagery in Google Earth Engine. ISPRS J. Photogramm. Remote Sens.
178, 282–296.
Appendix A. Supplementary data Nowakowski, A., Mrziglod, J., Spiller, D., Bonifacio, R., Ferrari, I., Mathieu, P.P., Garcia-
Herranz, M., Kim, D.-H., 2021. Crop type mapping by using transfer learning. Int. J.
Supplementary data to this article can be found online at https://doi. Appl. Earth Obs. Geoinf. 98, 102313.
Oreopoulos, L., Wilson, M.J., Várnai, T., 2011. Implementation on Landsat data of a
org/10.1016/j.isprsjprs.2023.04.002. simple cloud-mask algorithm developed for MODIS land bands. IEEE Geosci. Remote
Sens. Lett. 8, 597–601.
References Pott, L.P., Amado, T.J.C., Schwalbert, R.A., Corassa, G.M., Ciampitti, I.A., 2021. Satellite-
based data fusion crop type classification and mapping in Rio Grande do Sul, Brazil.
ISPRS J. Photogramm. Remote Sens. 176, 196–210.
Aravind, K.R., Raja, P., 2020. Automated disease classification in (Selected) agricultural Prins, A.J., Van Niekerk, A., 2021. Crop type mapping using LiDAR, Sentinel-2 and aerial
crops using transfer learning. Automatika: J. Control Measurement Electron. imagery with machine learning algorithms. Geo-spatial Information Sci. 24,
Comput. Commun. 61. 215–227.
Boryan, C., Yang, Z., Mueller, R., Craig, M., 2011. Monitoring US agriculture: the US Purcell, L.C., Salmeron, M., Ashlock, L., 2014. Soybean growth and development.
department of agriculture, national agricultural statistics service, cropland data layer Arkansas soybean production handbook 197, 1-8.
program. Geocarto Int. 26, 341–358. Ray, R.L., Fares, A., Risch, E., 2018. Effects of drought on crop production and cropping
Bramich, J., Bolch, C.J.S., Fischer, A., 2021. Improved red-edge chlorophyll-a detection areas in Texas. Agric. Environ. Lett. 3, 170037.
for Sentinel 2. Ecol. Ind. 120, 106876. Rußwurm, M., Korner, M., 2017. Temporal vegetation modelling using long short-term
Clevers, J.G.P.W., Gitelson, A.A., 2013. Remote estimation of crop and grass chlorophyll memory networks for crop identification from medium-resolution multi-spectral
and nitrogen content using red-edge bands on Sentinel-2 and-3. Int. J. Appl. Earth satellite images, pp. 11-19.
Obs. Geoinf. 23, 344–351. Saini, R., Ghosh, S.K., 2018. Crop classification on single date sentinel-2 imagery using
Craig, M., 2010. A History of the Cropland Data Layer at NASS. Available online: https:// random forest and suppor vector machine. Int. Arch. Photogrammetry Remote Sens.
www.nass.usda.gov/Research_and_Science/Cropland/CDL_History_MEC.pdf Spatial Information Sci. 42, 683–688.
(accessed on Oct, 10, 2022). Sakamoto, T., Gitelson, A.A., Arkebauer, T.J., 2014. Near real-time prediction of US corn
d’Andrimont, R., Taymans, M., Lemoine, G., Ceglar, A., Yordanov, M., van der Velde, M., yields based on time-series MODIS data. Remote Sens. Environ. 147, 219–231.
2020. Detecting flowering phenology in oil seed rape parcels with Sentinel-1 and-2 Schnepf, R., 2020. Federal Crop Insurance: Record Prevent Plant (PPL) Acres and
time series. Remote Sens. Environ. 239, 111660. Payments in 2019. CRS Report. Washington, DC: Congressional Research Service.
Delegido, J., Verrelst, J., Alonso, L., Moreno, J., 2011. Evaluation of sentinel-2 red-edge Shrestha, R., Di, L., Eugene, G.Y., Kang, L., Shao, Y.-Z., Bai, Y.-Q., 2017. Regression
bands for empirical estimation of green LAI and chlorophyll content. Sensors 11, model to estimate flood impact on corn yield using MODIS NDVI and USDA cropland
7063–7081. data layer. J. Integr. Agric. 16, 398–407.
Feng, L., Wang, Y., Zhang, Z., Du, Q., 2021a. Geographically and temporally weighted USDA, 2018. 2018 WISCONSIN CROP PROGRESS REVIEW. Available online: https
neural network for winter wheat yield prediction. Remote Sens. Environ. 262, ://www.nass.usda.gov/Statistics_by_State/Wisconsin/Publications/Crop_Progress
112514. _&_Condition/2018/WI-Annual-Crop-Progress-2018.pdf (accessed on Oct, 10,
Feng, L., Zhang, Z., Ma, Y., Sun, Y., Du, Q., Williams, P., Drewry, J., Luck, B., 2021b. 2022).
Multitask Learning of Alfalfa Nutritive Value From UAV-Based Hyperspectral Van der Maaten, L., Hinton, G., 2008. Visualizing data using t-SNE. J. Mach. Learn. Res.
Images. IEEE Geosci. Remote Sens. Lett. 19, 1–5. 9.
Feng, S., Zhao, J., Liu, T., Zhang, H., Zhang, Z., Guo, X., 2019. Crop type identification Vuolo, F., Neuwirth, M., Immitzer, M., Atzberger, C., Ng, W.-T., 2018. How much does
and mapping using machine learning algorithms and sentinel-2 time series data. multi-temporal Sentinel-2 data improve crop type classification? Int. J. Appl. Earth
IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 12, 3295–3306. Obs. Geoinf. 72, 122–130.
Forkuor, G., Dimobe, K., Serme, I., Tondoh, J.E., 2018. Landsat-8 vs. Sentinel-2: Wakulińska, M., Marcinkowska-Ochtyra, A., 2020. Multi-temporal sentinel-2 data in
examining the added value of sentinel-2’s red-edge bands to land-use and land-cover classification of mountain vegetation. Remote Sens. (Basel) 12, 2696.
mapping in Burkina Faso. GIScience & Remote Sens. 55, 331–354. Wardlow, B.D., Egbert, S.L., 2008. Large-area crop mapping using time-series MODIS
Gadiraju, K.K., Vatsavai, R.R., 2020. Comparative analysis of deep transfer learning 250 m NDVI data: An assessment for the US Central Great Plains. Remote Sens.
performance on crop classification, pp. 1-8. Environ. 112, 1096–1116.
Gao, F., Anderson, M., Daughtry, C., Karnieli, A., Hively, D., Kustas, W., 2020. A within- Watkins, B., Van Niekerk, A., 2019. Automating field boundary delineation with multi-
season approach for detecting early growth stages in corn and soybean using high temporal Sentinel-2 imagery. Comput. Electron. Agric. 167, 105078.
temporal and spatial resolution imagery. Remote Sens. Environ. 242, 111752. You, N., Dong, J., Huang, J., Du, G., Zhang, G., He, Y., Yang, T., Di, Y., Xiao, X., 2021.
Grassini, P., Edreira, J.I.R., Rizzo, G., Yang, H., 2018. 2018 Corn Yield Forecasts as of The 10-m crop type maps in Northeast China during 2017–2019. Sci. Data 8, 1–11.
July 11: Higher Temperature Led To Faster Corn Development. Available online: Zhong, L., Hu, L., Zhou, H., 2019. Deep learning based multi-temporal crop classification.
https://cropwatch.unl.edu/corn-yield-forecast-7-11-18 (accessed on Oct, 10, 2022). Remote Sens. Environ. 221, 430–443.
Hanway, J.J., 1966. How a corn plant develops. Zhu, L., Radeloff, V.C., Ives, A.R., 2017. Improving the mapping of crop types in the
Hao, P., Di, L., Zhang, C., Guo, L., 2020. Transfer Learning for Crop classification with Midwestern US by fusing Landsat and MODIS satellite data. Int. J. Appl. Earth Obs.
Cropland Data Layer data (CDL) as training samples. Sci. Total Environ. 733, Geoinf. 58, 1–11.
138869.

117