IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL.

16, 2023 5247

FPGA Accelerator for Meta-Recognition Anomaly

Detection: Case of Burned Area Detection
Mihai Coca and Mihai Datcu , Fellow, IEEE

Abstract—Optical remote sensing instruments accumulate abun- absence of labeled data to characterize the anomalousness in
dant data from across all of the earth’s land surfaces, making real scenarios, a dominant solution for detecting anomalies is to
it possible both to understand the effects of climate change and learn a model from normal data and try to deflect anomalies.
to monitor, investigate, and manage ground-level events in detail.
Processing data using resources located near on-board satellite Anomaly detection techniques dealing with satellite imagery
sensors can bring major benefits in terms of minimizing analysis are either concerned with any changes in an image over time,
time and quickly initiating active actions in critical situations. i.e., change detection [5], or in areas that appear abnormal on the
In satellite missions, long-term production on-board algorithms stationary image [6]. Frequently, anomalies are identified with
may encounter unexplored samples, i.e., abnormal ground-level man-made targets. However, the anomalies produced by natural
events, and need to be able to discriminate and take the correct
action. In this matter, the authors present a field programmable disasters on the earth’s surface are of more interest due to their
gate array (FPGA)-based solution for natural anomaly detection high-scale damaging potential, e.g., fire spot in a forest, oil spill
in multispectral imagery using deep convolutional neural networks. in the sea [7].
The effects of weather-induced hazards and natural disasters, Event detection in earth science is often critical for im-
considered anomalies in this sense, are discovered by modeling an mediately addressing negative impacts on natural resources,
anomaly detector on a hybrid system that is hardware efficient.
The proposed approach is assembled on a Xilinx Zynq UltraScale+ e.g., drought-related vegetation disturbances [5], devastating
XCZU9EG multiprocessor system-on-chip (MPSoC) device, where floods [8], active fire detection [9]. Wildfire is the most extreme
a deep convolutional model is scaled into the FPGA logic, followed natural hazard that has caused serious damages to human safety
by a downstream statistical meta-recognition predictor. The pro- and natural ecosystems in recent years, i.e., Australian areas
posed anomaly detection accelerator has produced notable results
affected by wildfire in 2019 [10], ravage outbreak of wildfires in
in identifying a contemporary natural hazard, i.e., burned areas, in
scenes acquired by Sentinel-2 over Europe, i.e., Spain and France. Bolivia in 2019 [11], large fire events in California in 2020 [12].
The implemented algorithm achieved on the FPGA accelerator an This disastrous event has an annual cycle in predisposed places,
equivalent speedup of 4.46× and 4.5× lower power consumption but new places are constantly appearing around the world due
than the equivalent implementation on the Tesla K80 GPU. to climate change. Accurate location at an early stage is of great
Index Terms—Anomaly detection, burned area detection, field importance in fire-fighting strategies. Global coverage charac-
programmable gate array (FPGA), on-board processing, remote
sensing.
teristics of satellite imagery combined with computer vision
mapping techniques represent one of the preferred alternatives
I. INTRODUCTION for operational wildfire surveillance. High-resolution sensors
NOMALIES are rare patterns in data observations that placed on-board a satellite provide huge amounts of informa-
A deviate from a perception of normal and predicted be-
havior. Classical concepts for unsupervised anomaly detection,
tion that are impossible to transmit wholly and in real-time to
the ground station. Automatic data processing near the sensor,
extended across multiple disciplines (e.g., statistics, medicine, immediately after acquisition, can reduce the information flow
engineering, natural sciences), consists of principal compo- and promote early detection of anomalous events.
nent analysis (PCA) [1], nearest neighbor algorithms [2], the Space-based missions generate complex computing needs
one-class support vector machine (OCSVM) [3], and support because limited resources must be used in harsh environments
vector data description (SVDD) [4], among others. Due to the with big temperature ranges, radiation, vacuum and vibration.
The environmental conditions as well as the limited power
Manuscript received 12 December 2022; revised 7 March 2023; accepted 19 generation capabilities on-board restrict the hardware resources
April 2023. Date of publication 5 May 2023; date of current version 14 June used and decrease the capability and complexity of the algo-
2023. This work was supported by the Romanian Ministry of Education and rithms implemented on-board. Small-satellite (SmallSat) mis-
Research, CNCS - UEFISCDI, under Project PN-III-P4-ID-PCE-2020-2120,
within PNCDI III. (Corresponding authors: Mihai Coca; Mihai Datcu.) sions are low-cost platforms that provide programmatic flex-
Mihai Coca is with the Military Technical Academy “Ferdinand I”, 050141 ibility, monitoring strategy in constellations, and distributed
Bucharest, Romania, and also with the Research Center for Spatial Information capacities, all composed in a scalable architecture [13]. Field-
(CEOSpaceTech), University Politehnica Bucharest (UPB), 060042 Bucharest,
Romania (e-mail: [email protected]). programmable gate arrays (FPGAs) are adaptive devices related
Mihai Datcu is with the Earth Observation Center (EOC), German Aerospace to reconfigurable computing, highly desirable for space appli-
Center (DLR), 82234 Wessling, Germany, and also with the Research Center for cations, e.g., image processing and image compression [14].
Spatial Information (CEOSpaceTech), University Politehnica Bucharest (UPB),
060042 Bucharest, Romania (e-mail: [email protected]). FPGAs empower the utilization of algorithmic parallelism in
Digital Object Identifier 10.1109/JSTARS.2023.3273309 application-specific architectures and provide multiple high-

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
5248 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 16, 2023

bandwidth input interfaces at a more energy-efficient cost term to be matched [20]. The most popular anomaly detection
than a general-purpose central processing unit (CPU). Hybrid algorithm is Reed-Xiaoli (RX) [20] that computes a background
architectures, e.g., system-on-chip (SoC) devices, pursue mixing covariance matrix and its inverse, followed by a distance mea-
diverse computing technologies in order to achieve cumula- surement. This algorithm concentrates on detecting pixels whose
tive improvements by balancing all incorporated benefits. SoC spectral signatures are in contrast with their background by using
devices integrate certain computing architectures by combining Mahalanobis distance. Due to the parallel and distributed nature
general-purpose blocks, i.e., CPU, with specialized blocks, i.e., of RX algorithm, a recursive variant was deployed on FPGA
FPGA, graphic processing unit (GPU), digital signal processor fabric [21] for on-board processing convenience. The authors
(DSP). These attractive combinations in the architecture of a proposed a framework that used an off-chip memory for data
system make it easy to split algorithms into subroutines run caching and a processor for two tasks, updating the inverse
on the most suitable execution blocks, all with performance matrix R−1 and filtering for anomaly detection.
gains. Xilinx’s UltraScale architecture broadens FPGA capa- Progressive line processing methods have shown a growing
bility for space applications, enabling high-throughput satellite interest in anomaly detection applications under real-time con-
services [15]. straints. In [22], [23], and [24], the authors proposed a line-
On-board processing for space applications is currently lim- by-line fast anomaly detector for hyperspectral imagery (LbL-
ited due to development and qualification cycles. Real-time data FAD) that uses an orthogonal projection strategy to separate the
processing is achieved through hybrid processing systems such background distribution. This hardware-friendly anomaly de-
as SoC devices, which are essential for small-satellite missions. tector variant computes an orthogonal subspace on background
PhiSat-1 nanosatellite mission was the first earth observation samples in order to better detect anomalous pixels. In [23], the
satellite with AI on-board, using a CNN model to detect clouds LbL-AD algorithm is evaluated on real images acquired by a
in images and filter out unusable data to maximize relevant push-broom sensor mounted on an unmanned aerial vehicle
information downlinked to the ground segment [16], [17]. (UAV). Further, in [24], the LbL-AD algorithm proved an attrac-
The challenges of processing data on embedded devices are tive tradeoff between time performance, energy consumption,
significant and require innovative solutions. This article presents and cost of processing a hyperspectral image of 825 × 1024
a novel anomaly detection architecture based on the concept × 160 pixels. Zhang et al. [25] suggested a real-time causal
of meta-recognition [18]. By incorporating a postrecognition linewise progressive anomaly detection (RCLP-AD) using fast
classifier that makes predictions on the last activation layer of a Cholesky decomposition. Their work improved real-time com-
trained DCNN model, the proposed anomaly detector can effec- putational performance and contributed to numerical stability in
tively discover anomalies in multispectral scenes. The proposed the background suppression process.
model is assembled as software for an integrated multiprocessor The current state of RS imaging processing technology
system-on-chip (MPSoC) device, suitable for small-satellite increasingly includes AI components that demand more compu-
missions. tational resources due to increased complexity. Recent advances
The main contributions of this work are briefly described as in microelectronics offer hardware accelerators with an opti-
follows: mised computation-to-power ratio, allowing AI algorithms to be
1) codesign of a hardware-software architecture for natural deployed close to the sensor. PhiSat-1 mission [16] proved the
anomaly detection in multispectral imagery; utility of AI as a robust and efficient method for accurate cloud
2) verification of the implemented hardware-software parti- detection on-board. This demonstration mission emphasized the
tioned anomaly detection system on a Zynq UltraScale+ robustness of the on-board Intel Movidius Myriad 2 hardware
MPSoC ZCU102 Evaluation Kit for two real multispectral accelerator and the accuracy of the CloudScout convolutional
datasets with scenes affected by forest fires. neural network [17]. The CloudScout classifier achieved a 96%
The rest of this article is organized as follows. First, a accuracy when performing cloud detection on live images on-
comprehensive review of related works is provided in Section satellite.
II. Sections III and IV introduce the reader to the proposed In real-time detection of anomalous targets from on-board
methodology and datasets used in evaluation stage. Results on multispectral imagery, the processing speed and the accuracy
the presented datasets are provided in Section V. Finally, Section are equally important but difficult to achieve simultaneously. Ma
VI concludes this article. et al. [26] proposed a pruning-quantization-anomaly-detector
(P-Q-AD) that balances accuracy and throughput by shrinking a
II. RELATED WORK neural model by removing the redundant neurons. Lei et al. [27]
The traditional on-board real-time classification and detection proposed a fast spectral-spatial anomaly detection algorithm
methods for remote sensing (RS) imagery are almost entirely called Fast-MGD, which uses morphological reconstruction and
based on the computation
 of the data correlation matrix R or a simplified guided filter to achieve high detection accuracy
covariance matrix and their inverses, with the aim to contin- with simple filtering techniques. The algorithm utilizes a deeply
uously update the inverses as pixels are being received [19]. For pipelined acceleration scheme designed for hardware implemen-
a known target to be detected d, common features are computed tation on FPGAs that handle hyperspectral images.
using R−1 for data whitening and matched filter dT for detection Pixel-based anomaly detection algorithms, i.e., RX Detector
or classification. In the case of anomaly detection when the [20], extract targets that are spectrally distinct from the image
required target x is unknown, the pixel itself xT serves as the background with good performance. In order to be effective,
COCA AND DATCU: FPGA ACCELERATOR FOR META-RECOGNITION ANOMALY DETECTION: CASE OF BURNED AREA DETECTION 5249

Fig. 1. Host application for anomaly detection. After training a deep convolutional neural network, the same training set, i.e., observed dataset, is used to obtain
the features of correctly classified samples. The features are further used to compute an activation vector per class containing the mean of the activation vectors
(MAV) of the samples belonging to that class. Next, a distance vector per class is calculated which contains the distances between all samples of that class and the
previously calculated mean, i.e., distances between vectors. Subsequently, the tails of the distance distributions are modeled by Weibull distributions. In the testing
phase, the characteristics of a query sample determine its position in the distance distributions of all classes encountered in the observed dataset. Depending on the
result of the Weibull estimation on the test point, i.e., the query sample, the α top features of the test point are calibrated resulting in a new scalar value, i.e., the
equivalent weight for the anomaly class, in the component of the initial feature vector. The new feature vector generates via a normalized exponential function a
probability distribution. Finally, based on a threshold applied to the probability distribution, a binary decision is obtained, i.e., whether it is an anomaly or not.

the anomalous targets must be sufficiently narrow relative to Consequently, it was proposed an algorithm in which the deep
the background and expected to follow a Gaussian distribution. latent values extracted from an observed dataset are extended to
However, in complex multispectral images with convoluted estimate the probability that a given input is associated with an
feature relations, the Gaussian assumption fails in providing abnormal class that was not seen in the training phase. In this
an accurate model for the real data, particularly at the tails context, it was defined as a framework that involves two consec-
of the distribution. With this in mind, in the proposed method, utive stages, the first for implementing, training, and testing the
open and broad patch-level anomalies were searched for in large algorithm on a GPU cluster, followed by an optimization and
heterogeneous multispectral scenes. deployment phase of the algorithm on a low-resource platform.
For the first stage, Fig. 1 depicts the proposed method as a
host application that uses the deep latent characteristics of a
III. PROPOSED METHODOLOGY test sample to determine if it belongs to the abnormal class.
In most situations, the trained classifiers are implemented The abnormal class, a new class introduced in the nomenclature
under closed set conditions, since the classes encountered dur- of training classes, characterizes all samples that deviate from
ing implementation are known and are exactly the same as the observed categories, and finally considers anomalies. For the
during training. In uncontrolled real-world environments, it second stage, Fig. 2 describes the flow used to build the proposed
is very likely to encounter instances of classes that have not algorithm as an anomaly detection accelerator and deploy it on
been covered by the training data [28]. In remote sensing (RS) an embedded MPSoC device equipped with an FPGA device. In
scenes, anomaly detection algorithms address a discovery task of this stage, the host application block incorporates the algorithm
small-scale portions that do not harmonize with the background. described in Fig. 1.
Considering a collection of observations from a number of The proposed anomaly detection algorithm contains three
normal classes, e.g., usual land cover classes, a query sample steps, i.e., data preprocessing, background modeling, and simi-
that does not belong to one of those current classes would be larity measurement. In the preprocessing step, data are prepared
identified with a new abnormal class, e.g., burned areas. by standardization [29] for use as input to a DCNN classifier.
5250 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 16, 2023

Fig. 2. General flowchart for the optimization and deployment of the on-board anomaly detection accelerator. Training a model on a host GPU device generates a
feature extractor working with FP32 data. Quantifying the feature extraction graph in INT8 data and compiling it into a specific instruction set compress the model
for the target platform, i.e., ZCU102. The host application described in Fig. 1 is integrated with the compiled feature extractor and a suite of AI libraries to get the
target application running on the target platform. The hardware component running on the FPGA is obtained by configuring a DPU instance. On the embedded
platform, the proposed algorithm detects anomalies from a set of test samples.

Afterward, the classifier is trained on observed data and used calibration subset of the observed dataset to evaluate the distri-
as a feature extractor, e.g., data reduction. In the second step, bution of activations and to mitigate the accuracy degradation.
background statistical features are approximated through fitting Next, a compiler translated from Tensorflow specific format to
Weibull distributions on each observed class in the training DPU engine specific format. Next, the software build step, i.e.,
dataset (see Fig. 1). Finally, a distance metric is utilized for Build SW, generated for the target platform an inference-only
the similarity metric, e.g., Euclidean and Cosine distances, and model that was called by the host application. Thereafter, the
a threshold is selected for a binary decision. target application effectively used the hardware-built DPU via
A difficult problem in implementing a DCNN on the FPGA a specific library, i.e., AI Library, and AI Runtime, resulting in
is the long design time as well as the substantial effort in- an executable file for a specific FPGA target board.
volved. Xilinx, Inc. has released a group of parameterizable The approach proposed in this article considers a method-
intellectual property cores called deep learning processing unit ology that adapts DCNNs for open set recognition [31]. This
(DPU) [30], specialized in efficiently implementing various methodology introduced a new model layer, i.e., OpenMax,
DCNN architectures on FPGA by supporting diverse deep which estimates the probability of an input belonging to an un-
learning operations. The DPU has a specialized instruction set known class. The OpenMax model is based on meta-recognition
utilized in the DL algorithm implementation. The acceleration concepts adapted to the activation patterns in the penultimate
development flow involved a unified software stack called Vitis layer of a classifier. The OpenMax tier is an extension of the last
AI Framework, provided by Xilinx, to convert, i.e., quantizing activation function, e.g., SoftMax, of a DCNN that normalizes
and compiling, a Tensorflow DCNN model into a compatible the output of a network to a probability distribution over the
format supported by the DPU engine. Moreover, the Vitis soft- initial predicted output classes plus an anomalous class. The
ware permitted concurrent development and test of both the resulted likelihood was used to estimate the probability of a given
software component (i.e., the target application), and the hard- input belonging to an anomalous class. Further, the activation
ware component (i.e., the kernel), contained in a heterogeneous vectors from the penultimate layer were utilized to estimate if
application. the test input is out of distribution from known training data.
In the optimization and deployment phase of the algorithm
(see Fig. 2), a DCNN model learned with the Tensorflow frame-
work was converted from a floating-point format to a single- A. Meta-Recognition
precision format using a quantizer. In addition to the frozen Meta-recognition [18] is a prediction method that uses sta-
graph of the DCNN model, the quantizer needed as input a tistical extreme value theory (EVT) for postrecognition score
COCA AND DATCU: FPGA ACCELERATOR FOR META-RECOGNITION ANOMALY DETECTION: CASE OF BURNED AREA DETECTION 5251

analysis. In particular, after a system produces a series of dis- values (see Fig. 1). With Sn was noted the set of indexes, in de-
tances or similarity scores for a particular test sample, a predictor scending order, of the highest n probabilities generated by a final
will produce a decision of recognition success or failure. Dis- Sigmoid layer on activation vector v. Next, for each class Ci ,
tributional modeling and machine learning, among others, may a Weibull model ρCi = (τCi , λCi , κCi ), 1 ≤ i ≤ k with three
be the basis for a postrecognition classifier. Scheirer et al. [18] parameters, data shifting τ , Weibull scale λ, respectively Weibull
proposed a statistical theory of postrecognition score analysis, shape κ, is computed. Parameter τ is dynamic and depends on
i.e., meta-recognition, derived from the extreme value theory. the data itself, i.e., is the smallest score (distance) on the tested
They developed a statistical classifier based on the Weibull activation vector v, aiming to shift v in zero (1a). Model ρCi (x)
distribution and demonstrated that a statistical meta-recognition provides meta-recognition estimated probability that determines
is substantially better than a standard threshold test over the if query sample x is anomalous or not.
original score data. In the second phase, the activations of a query sample x adjust
Multiclass classifiers in deep learning applications are usually α top activations, i.e., α top probabilities given by a Sigmoid
trained on a closed-set dataset. In contrast to the closed set layer, by approximating the Weibull distribution function (1a)
nature, recognition may be assimilated with the open set attribute  
x−τ
 κi 
− λ i
of a real world classifier that rejects unseen classes at query wi (x) = 1 − f (i) 1 − e i , 1≤i≤k (1a)
time. A DCNN classifier applies on the last activation layer a 
normalized exponential function, i.e., SoftMax, which generates α+1−ranki (Sα (v))
α if i ∈ Sα (v)
a probability distribution over a known and fixed number of class f (i) = . (1b)
0 otherwise
labels, i.e., k. At test time, this classifier trained only on the
closed set samples may erroneously label with high certainty After computing Weibull CDF on the distance between query
an anomalous sample belonging to one of the classes in the sample x and MAVs of α top activations, revised activation
closed set. For an unseen input, it is unlikely to produce such vector is computed (2a), where operator ◦ is used for the scalar
low probabilities for all known classes that a threshold would product between two vector.
reject the input and classify it as an anomaly. In practice, the
v̂(x) = v(x) ◦ w(x) (2a)
deep network activations are not bounded, ensuing that SoftMax
cannot regulate the open space recognition [32]. Starting from k

this hypothesis, the meta-recognition task was involved in this v̂k+1 (x) = vi (x)(1 − wi (x)). (2b)
work to inspect the scores and identify when a DCNN classifier i=1
is possibly incorrect in evaluation. Afterward, a pseudoactivation for the unseen class Ck+1 is
Extreme value theory holds that the extreme values of a computed while preserving the total activation level constant
continuous distribution accept only three types of parametric (2b). Finally, the rejection is decided on the revised probabilities
forms, specifically Gumbel, Frechet, reversed Weibull distri- of normal classes Ci , 1 ≤ i ≤ k, with regard to the anomalous
butions [18]. Minimum pairwise distance of a test sample to class Ck+1 (3)
the closest sample in the closed set is proved through EVT
that follows a Weibull distribution. The argumentation is that ev̂i (x)
pi (x) = k+1 , 1≤i≤k+1 (3)
extreme features in contrast to average ones are the most v̂j (x)
j=1 e
essential for discriminating between different objects [33]. This
where x is anomalous if pk+1 is the highest in pi , 1 ≤ i ≤ k + 1
facilitates the development of an inclusion map to decide if
or higher than a threshold η.
a test sample is in class with the minimal pairwise distance
or is classified as an anomaly. In this article, building on the
multiclass meta-recognition concept proposed in [31], the mean B. Model Quantization and Compilation
scores from the per-class estimation layer, i.e., logit layer, of a Quantization scales down the bit width of the parameters and
trained deep network classifier is used to predict if an input activation values processed in a DCNN. Smaller data types are
sample is separated from the known trained dataset. In the appealing for energy efficiency and storage cost area because
first stage, EVT is used to fit the postrecognition activations, they produce a reduced model size with minor deterioration in
i.e., mean activation vectors (MAVs) of the logit layer, of the accuracy [34]. Another reason for using fixed-point represented
known classes to Weibull distributions. For Weibull fitting, data is that the DSP unit in the FPGA is better at handling
the FitHigh function is used, which is available in the libMR fixed-point numbers. In this work, the quantizer is used to
library [18]. convert numerical values of model weights from 32-b floating
Remote sensing images need a rich characterization through point to 8-b fixed point precision, reducing computational
multilabeling since they contain a huge diversity of semantically complexity. The quantization step included a posttraining
complex content. Starting from this position, a convolutional process in which a small calibration subset (see Fig. 2) of
classifier must use a sigmoid layer as the last activation layer training images, i.e., 1000 calibration images, was used to
in order to classify multilabel samples. A multilabel feature analyze the distribution of activations and to limit the accuracy
extractor is trained on normal classes Ci , 1 ≤ i ≤ k to compute degradation. Data-free quantization (DFQ) algorithm [35] was
for each class the corresponding MAV, μi = mean(vij ), 1 ≤ used in posttraining process to equalize the weight ranges and to
i ≤ k, 1 ≤ j ≤ |Ci | using the activation vector v, i.e., the logit correct biases in the errors introduced during quantization. After
5252 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 16, 2023

model quantization, each layer is converted by an integrated quadratic, with a size directly proportional to the semantic
Vitis AI compiler into a sequence of associated supported diversification of the original scene. In this work, the patch size
instructions that actuate the DPU. considered was 120 × 120 pixels with ten spectral bands.

C. Data Processing A. BigEarthNet Dataset

For continuous land scanning, the carpet mapping imagery The BigEarthNet [39], [40] is a large-scale Sentinel-2 bench-
requires a large amount of storage space resulting from the mark archive, consisting of 590 326 Sentinel-2 image patches.
combination of swath width, spectral channels count, and spatial The image patch size on the ground is 1.2 × 1.2 km2 with
resolution of the sensor. For example, Sentinel-2 sensor acquires variable image size depending on the channel resolution. This is
1.6 TB of useful data per orbit, resulted from a 290 km swath with a multilabel dataset with 43 imbalanced labels. Each Sentinel-2
13 spectral channels at a spatial resolution of 10 m. For long-term image from BigEarthNet dataset was associated with one or
acquisitions, imagery is too large to be processed efficiently in more class labels extracted from the CORINE land cover map of
memory by convolutional operations. Thus, a significant latency 2018, with an accuracy of around 85% [41]. The solar radiation
between acquisition and result output is generated. To overcome reflected from the earth’s surface to the Sentinel-2 sensor is
this problem, the band interleaved by lines (BIL) [36] format is quantified in bands at different wavelengths [42]. As band B10
used for real-time on-board applications. does not include surface information, it was not included in the
The line-by-line technique reduces the on-board memory BigEarthNet dataset nor in the proposed datasets.
footprint and output latency making it ideal for satellites
equipped with push-broom sensors. In the BIL data format B. Proposed Datasets
approach, combined with push-broom optical sensors such as The Copernicus Open Access Hub1 provides complete, free,
those mounted on Sentinel-2 or Landsat 8 satellites, N lines of and open access to Sentinel-1, Sentinel-2, Sentinel-3, and
pixels need to be received in order to prepare a collection of Sentinel-5P user products. From the Copernicus portal, two
quadratic samples. Additionally, super-resolution methods that separate Sentinel-2 products were downloaded, named Zamora
enhance spectral measurement by tilting an area sensor instead and Bordeaux due to the fact that the products were acquired in
of shifting a linear sensor can be adjusted to maximize the close proximity to these cities. A Sentinel-2 product includes a
data throughput for our proposed patch-level method [37]. It is multispectral scene with a size of 10980 × 10980 pixels, with
worth noting that although the proposed method is demonstrated reference to 10 m bands [42]. For each package, a region of
just on data acquired by Sentinel-2, the same approach can be interest (ROI) was chosen based on in situ information. The
extended to other optical remote sensing products that acquire SeNtinel Application Platform (SNAP)2 was used to extract the
data in the visible, near-infrared and shortwave infrared parts of ROI from each product. To successfully crop the ROI, the bicubic
the spectrum. method was used on all bands with resolution of 20 and 60 m to
up-sample to 10 m. This step was applied to all the data evaluated
IV. STUDY AREAS AND DATASETS because, in the context of deep learning, it is advisable to use a
Fire detection systems combine data from RGB cameras, homogeneous collection of data to reduce the sensitivity of the
by using color information, and InfraRed (IR) cameras, by model to variations in the distribution of the input data [43].
measuring the thermal radiation, to achieve a consistent level Next, each scene derived from each ROI was split into
of accuracy. Preparing multispectral data for training and test- nonoverlapping patches, generating a dataset. The image patch
ing the proposed algorithm requires building and annotating size was identical to that in the BigEarthNet dataset, i.e., size
datasets based on off-the-self datasets, such as the Sentinel-2 on the ground of 1.2 × 1.2 km2 , for compatibility reasons in
archive. MultiSpectral Instrument (MSI) mounted on Sentinel-2 using the model originally trained on the BigEarthNet dataset.
satellite measures the earth’s reflected radiance in 13 spectral Within the proposed datasets, vegetation areas were deemed
bands, i.e., B01, B02, B03, B04, B05, B06, B07, B08, B8A, as the background of the dataset while forest fires affected
B09, B10, B11, B12, from visible (RGB) to short-wave IR areas were considered as the anomalies. To generate the ground
(SWIR) [38]. In this study, three datasets from data acquired truth (GT) map, i.e., to identify burned and unburned patches,
by the Sentinel-2 mission were studied, named BigEarthNet, necessary to evaluate the proposed method, a classical spectral
Zamora, and Bordeaux. BigEarthNet was operated as a genesis indices method was utilized. The way the classical spectral index
dataset, in different versions, to train a feature extractor model method is applied is described in [10]. The proposed datasets
capable of distill multispectral images. Forwards, two proposed were described below. The total number of samples, the number
datasets, i.e., Zamora and Bordeaux, were employed in the of burned samples, and the ratio of burned samples in the entire
conducted experiments. dataset were included in Table I. In these datasets, an anomalous
Remote sensing images generally incorporate areas with great sample contains active fire fronts or vegetation already burned
semantically complex diversity that usually must be expressed across the entire footprint, while a normal sample can consist
through multiple class labels. In order to maintain a high in-
terclass variance, a conglomerate scene produces a substantial 1 https://scihub.copernicus.eu/dhus/#/home

collection of samples, i.e., patches. Usually, the samples are 2 https://step.esa.int/main/toolboxes/snap/

COCA AND DATCU: FPGA ACCELERATOR FOR META-RECOGNITION ANOMALY DETECTION: CASE OF BURNED AREA DETECTION 5253

Fig. 3. Zamora scene (Spain, 2022). In (a) is displayed in TCI format (R: Band 4; G: Band 3; B: Band 2) and in (b) is shown in FCI format (R: Band 12; G: Band
11; B: Band 8 A). The burned areas (anomalous samples) are highlighted in brown color in (b).

TABLE I and August 2022, in Gironde department, near Bordeaux city.

SUMMARY OF THE USED DATASETS
This scene was recorded on July 17 with clear skies and thin
smoke generated by the still-burning canopy. An ROI of 4800 ×
9600 pixels (with reference to 10 m bands) was cropped using
in situ information [45]. After performing a nonoverlapping
cropping, based on a 120 × 120 pixel patch, a collection of 3200
samples was obtained. Fig. 4(a) and (b) depicts a TCI, respec-
of any combination of normal classes, i.e., common land cover tively, an FCI for the selected ROI. The composition of SWIR
classes. bands in Fig. 4(b) provides a picture of the burned areas and
1) Zamora: The Zamora scene contains disastrous wildfires reveals areas of ongoing fire where smoke is not too opaque, as
that took place in Spain during June 2022, in the northwest in the highlighted sample. Besides the metropolitan area of Bor-
province of Zamora, mountain range Sierra de la Culebra. This deaux, the terrestrial space scene contains hills, pastures, forests,
scene was recorded on June 28, when over 30 000 hectares of orchards, vineyards, all interspersed with hilltop villages.
a highly biodiverse ecosystem were already burnt [44]. During
recording, the scene was completely cloud free. A ROI of 7200 ×
V. EXPERIMENTS
7200 pixels (with reference to 10 m bands) was cropped using in
situ information. Using an image patch size of 120 × 120 pixels Deep learning models are powerful feature extraction meth-
(with reference to 10 m bands), it results a cardinality of 3600 ods capable of extracting more abstract and salient features
samples in this dataset. Fig. 3(a) exhibits a true-color image from data, compared to traditional approaches. To obtain a
(TCI), i.e., a combination of three channels that are sensitive powerful spectral-spatial feature extractor, a ResNet model [46]
to the red, green, and blue visible light, for the selected ROI with 50 convolutional layers was trained on the BigEarthNet
in order to facilitate a quick delineation of surface variety and dataset. Subsets of the BigEarthNet dataset were considered
atmospheric characteristics. In Fig. 3(b) is shown a false-color depending on the specifics of the downstream task. Hereinafter,
image (FCI) that renders nonvisible parts of the electromagnetic the ResNet50 model is denoted as a multiclass classifier used
spectrum, i.e., bands B12, B11, B8A from SWIR range, aiming to obtain a feature extractor. ResNet-based networks preserve
to visually analyze different enhanced features in the landscape, representations from degrading toward the end of the network
e.g., burned areas in brown color. The scene contains areas of thanks to the incorporated residual units. The DCNN model
wheat fields, pastures, vineyards, and wooded hills. was implemented in the Tensorflow framework due to high
2) Bordeaux: The Bordeaux scene contains devastating integration with the software stack responsible for quantizing
blazes that took place in southwest France during June, July, and compiling the model into DPU instructions. As proposed
5254 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 16, 2023

Fig. 4. Bordeaux scene (France, 2022). In (a) is displayed in TCI format (R: Band 4; G: Band 3; B: Band 2) and in (b) is shown in FCI format (R: Band 12; G:
Band 11; B: Band 8 A). The burned areas (anomalous samples) are highlighted in brown color in (b). In the highlighted sample, the fire fronts are visible due to
the long wavelength bands used in the FCI composite.

in [39], the images from the BigEarthNet dataset that were com- resulting in a 42 class-nomenclature [40]. Afterward, the rule
pletely covered by seasonal snow, clouds, and cloud shadows 60:20:20 for randomly choosing training, validation, and test
were removed, resulting in a clean version of BigEarthNet. Here datasets was adopted. During the training, the network received
and throughout, training datasets were derived starting from the a frame of 120 × 120 × 10 pixels at the input, considering
clean version. In the proposed experiments, the studied areas (see only the bands that originally had resolutions of 10 and 20 m.
Section IV-B) contain one specific anomaly, i.e., burned areas, The GT map used in the training phase was provided by the
in Zamora and Bordeaux. To detect anomalous burned areas creators of the BigEarthNet dataset [39]. The adaptive moment
inside complex land cover multispectral scenes, class Burnt estimation method, i.e., Adam, was applied to optimize the
areas was filtered out from the clean version of BigEarthNet, trainable network, with a learning rate of 1e − 3, 100 epochs, and
COCA AND DATCU: FPGA ACCELERATOR FOR META-RECOGNITION ANOMALY DETECTION: CASE OF BURNED AREA DETECTION 5255

a batch size of 300 samples. The ResNet50 model was trained TABLE II
ACCURACY EVOLUTION OF MULTICLASS CLASSIFICATION MODEL ON 1000
on a workstation machine equipped with Intel(R) Xeon(R) CPU TEST SAMPLES
[email protected] GHz and Tesla K80 12 GB GPU. For a dataset
of 254 812 samples, the training process took approximately
48 h. The loss function used was a sigmoid cross-ntropy loss
function, obtaining a minimum value of 0.0132.
After training the multiclass classifier, the MAVs were calcu-
lated on the training dataset, for each class, taking into account TABLE III
DETECTION TIME OF THE PROPOSED ANOMALY DETECTOR FOR A TEST
all samples that were correctly classified (see Fig. 1). Working COLLECTION OF 60 SAMPLES
with a multilabel dataset, a correctly classified sample can be
included in the MAV calculation of multiple classes, depending
on the number of labels assigned to that sample. After finding the
MAVs, the distance vectors were estimated with cosine distance
as follows:
A·B
cos(θ) = (4)
A2 B2
where A is the activation vector of an observed sample and B application processing unit. The loading time of the locally
is the corresponding MAV of the class. Further, in the Weibull saved Weibull model is fixed and is performed only once in
tail fitting step, the parameter α was fixed to five since 96.80% the initialization stage.
of images in BigEarthNet dataset are not associated with more At inference time, DPU processes work with the input and
than five labels [40]. output of fixed-point data. To deploy the GPU-trained model on
In on-board implementation, a programmable engine opti- an edge hardware, i.e., FPGA platform, for inference, a reduction
mized for DNN, i.e., DPU, provided by the Vitis AI development in computing complexity was performed. Thus, 32-b full preci-
environment for AI inference on Xilinx hardware platforms was sion (FP32) weights and activation values were quantizated to
used. DPU component uses a specialized tensor-level instruction 8-b fixed point (INT8) format. Reduction of bit lengths through
set that efficiently assists the implementation of DL networks quantization, i.e., from 32-b float to 8-b fixed, decreases the
and accelerates the computing workloads of DL inference al- memory usage four times. Fixed-point network model required
gorithms. The DL accelerator is placed on FPGA fabric. In the less memory bandwidth, providing higher speed and power
conducted experiments, the target platform is a Xilinx Zynq Ul- efficiency than the floating-point model, but with performance
traScale+ MPSoC ZCU102 evaluation board3 which combines reduction due to quantization noise. Table II shows the decrease
a powerful processing system (PS) and a user-programmable in accuracy of the multiclass classifier following quantization.
logic (PL) into a powerful MPSoC, i.e., Zynq UltraScale+ There is a small decrease of 0.068 on the F1-score metric.
XCZU9EG-2FFVB1156E MPSoC. This MPSoC runs a custom Fig. 5 depicts four lines of 60 samples from a line-by-line data
light-weight Linux-based operating system, i.e., Petalinux,4 on processing flow. One line is evaluated on the Zynq UltraScale+
the ARM Cortex A53 64-b quad-core multiprocessing CPU. XCZU9EG-2FFVB1156E MPSoC device in 0.7 s. Table III
Additional, this MPSoC incorporates also an ARM Cortex-R5 shows the evolution of the processing time when testing the
32-b dual-core real-time processing unit. This CPU-FPGA hy- anomaly detection algorithm on a collection of 60 samples.
brid system was programmed on each part, the CPU with logic The first column identifies the device on which the algorithm
for input and output processing, and the FPGA with execution is running. The second column, i.e., the number of threads,
of the DCNN architecture. identifies the degree of parallelization on the target application
The target application is run on the CPU with Xilinx runtime run on the processor and the third column, i.e., the FPS, indicates
application interface calls to manage runtime interaction with the number of frames per second. Since most spaceborne and
accelerator. During the on-board deployment phase, the precom- airborne spectral systems capture data line-by-line, in the fourth
puted MAVs and distance vectors are downloaded together with column, i.e., the run time, is shown the time needed to evaluate a
the host application to the target platform and saved on disk (see line of 60 samples, e.g., one line of patches in the Zamora dataset
Fig. 2). MAVs have a static storage size of approximately 17 included in the experiments (see Section IV-B). The values in the
KBytes, while the distance vectors storage size vary according third and fourth columns, i.e., mean and standard deviation, are
to the cardinality of the observed dataset, i.e., in these experi- obtained over 20 runs. The best time performance is achieved
ments, the size is approximately 24 MBytes. Also, the libMR when the anomaly detection algorithm is run on XCZU9EG MP-
library [18] was cross-compiled for Arm v8 architecture in order SoC device with 8 threads on the PS side, achieving a processing
to be used on the host application running on the Cortex-A53 speed of 81.018 ns/pixel. Based on the best processing speed
obtained in Table III, the Zamora and Bordeaux datasets are
3 https://docs.xilinx.com/v/u/en-US/ug1182-zcu102-eval-bd processed in 42 and 37.3 s, respectively. The running time on the
4 https://docs.xilinx.com/r/en-US/ug1144-petalinux-tools-reference- K80 GPU device is an average of 3.128 s, with standard deviation
guide/Overview of 0.744 s over 20 runs. On the Tesla K80 GPU device, the
5256 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 16, 2023

Fig. 5. Processing Zamora scene in band interleaved by lines format (BIL). Each line contains 60 samples shown in FCI format (R: Band 12; G: Band 11; B:
Band 8 A) extracted from scene displayed in Fig. 3(b). Each sample has a dimension of 120 × 120 pixels and the color label indicates if it is anomalous, i.e., red
color, or not, i.e., white color.

TABLE IV TABLE V
COMPUTATIONAL RESOURCES UTILIZATION BY THE PROPOSED ANOMALY ACCURACY ASSESSMENT OF ANOMALY DETECTOR ON TEST DATASETS
DETECTOR ON THE ZCU102 PLATFORM

running time is about 4.46× slower than the best time achieved sample, the highest probability is that of the unknown class, or
on the FPGA device. Furthermore, the power consumed by the if the probability of the unknown class is greater than η, then
Tesla K80 GPU device alone is 135 W, compared to that of the the sample is considered anomalous. Independent of the running
entire Zynq UltraScale+ board which is 30 W, resulting in an device, the proposed algorithm obtained for the Zamora dataset
efficiency factor of 4.5 for the latter device. an F1-score value of 0.784 and for the Bordeaux dataset an
In Table IV is presented the resource utilization of a F1-score value of 0.772. The main parameter that determines
DPU (DPUCZDX8G) single core integrated into the proposed the performance obtained by the proposed algorithm is the level
anomaly detection accelerator. The B4096 architecture variant of false positive (FP) rate, incorporated in Precision metric. The
was utilized for the DPU configuration to obtain the maxi- FP rate, i.e., images that essentially contain no anomalies but are
mum scaling factor in terms of logic resource utilization and detected as anomalies, is inversely proportional to the Precision
parallelism, i.e., total number of MACs per DPU clock cycle. metric. The Precision metric obtained by running the algorithm
Although the model used is complex and resource-consuming, on the FPGA device was 0.727 for the Zamora dataset and 0.787
it can be limited to using a maximum of one third of the target for the Bordeaux dataset, respectively. In the case of the Zamora
board resources. dataset, few burnt areas in hilly areas were included in a known
In the context of binary classification, due to the high im- class, i.e., Peatbogs class. In the case of the Bordeaux dataset,
balance in the considered datasets (see Table I), i.e., number of false alarms occurred mostly in drought areas. The outlined
samples in the normal class versus the number of samples in sample in Fig. 4(b) is a genuine anomaly because it contains
anomalous class differ by more than one order of magnitude, a class that was not seen by the model in the training phase,
three accuracy metrics were studied, i.e., Precision, Recall, and i.e., fire fronts class. This sample was evaluated by the proposed
F1-score. The Precision measure is the rate of predicted anoma- model, returning an unknown class probability greater than 0.9.
lous patches that are actual anomalous, while the Recall measure From the results of the proposed experiments, one can see
is the ratio of actual anomalous patches correctly identified. that the proposed algorithm provides a high precision in all
The F1-score is a harmonic mean between Precision and Recall. cases, keeping false alarms at a low rate. In terms of sensitivity,
Performance assessment is performed by knowing the GT map i.e., recall, the proposed algorithm needs to be improved in
of the anomalous samples within the scene under analysis. background modeling phase. This is a challenging task due
Table V depicts the accuracy of the proposed method on to immense spatial coverage with deeply heterogeneous back-
datasets described in Section IV-B by running the method on ground. There is the well-known problem of label noise from
different devices. Due to quantization, there is a small decrease in machine learning in remote sensing, which is correlated with the
the F1-score metric when the algorithm is run on the embedded complexity of different applications. Generating perfect labels in
platform, i.e., 0.061 for the Zamora dataset and 0.041 for the remote sensing is impossible, so a trade-off between quantity and
Bordeaux dataset. Results in Table V are obtained by setting quality will always remain in place. One idea for enhancement is
the anomaly threshold η to 0.5. This means that if, for a query to increase the class nomenclature of the training set resulting in
COCA AND DATCU: FPGA ACCELERATOR FOR META-RECOGNITION ANOMALY DETECTION: CASE OF BURNED AREA DETECTION 5257

a proportional decrease in intraclass variation, thus increasing TABLE VI

COMPUTATIONAL RESOURCES CONSUMPTION BY DIFFERENT ANOMALY
the classification accuracy. This improvement, specific to the DETECTORS ON THE ZCU102 PLATFORM
proposed algorithm, may increase its sensitivity to anomalies in
general.

A. Comparison
In order to effectively evaluate the performance of the pro-
posed method, we considered two performance criteria, the
hardware resources used, respectively precision and recall met-
rics. Del Rosso et al. [47] proposed a prototype of an on-board of 81.018 ns/pixel on the same hardware platform, resulting in
convolutional model for volcanic eruption detection to generate an increased speed factor of 4.16×. A resource consumption
immediate alerts. To demonstrate the method, the authors used is evaluated in Table VI. Due to the reduced data precision,
an experimental hardware, namely, a drone with a payload with fewer bits per data, P-Q-AD consumes fewer BRAMs,
composed of a Raspberry Pi 3 Model B processing unit, an while more DSPs are employed for 16 b operations and a
Intel Movidius Neural Compute Stick coprocessor unit and a large quantity of LUTs for network implementation. Resource
Raspberry Pi camera module (RPi-Movidius system). Multi- utilization shows that our implementation confers higher space
spectral data from Sentinel-2 and Landsat-7 satellites were used availability for increased parallelization.
in the experiments and each scene was divided into 512 × For comparison in terms of precision and recall indicators,
512 pixel patches with three spectral bands. Starting from an we chose a state-of-the-art anomaly detection algorithm [48]
unbalanced dataset, since the hazard of volcanic eruptions is that detects postdisaster building damage in different natural
rare, the augmentation of anomalous samples was used to reach disaster contexts, including wildfires. The results of the two
a balanced dataset, following a binary classification. Ultimately, methods have a common denominator for comparison because
the experiments involved flying a drone over a print made with the model used for comparison is strongly influenced by the
Sentinel-2 data of an erupting volcano. type and context of the disaster event. In the case of wildfires,
The method proposed in this article obtained better results the burned building surroundings had a higher discriminative
than the method run on the RPi-Movidius system [47] in terms power to derive correct classifications for larger patches due to
of hardware resources usage and processing speed, while de- high anomaly scores, whereas the buildings itself obtained lower
creasing in F1-score by 0.099. The FPGA method used less than scores. Therefore, the Skip-GANomaly location-based model
30% of the hardware resources, while the comparative method obtained an F1-score of 0.807 on the Santa-Rosa dataset after
made limited use of the resources provided by the RPi-Movidius strict preprocessing, i.e., no vegetation, no shadows, with a patch
system. The RPi-Movidius method achieved a processing speed size of 32 × 32. Independent of the implementation device, our
of seven samples (one sample was 512 × 512 × 3 pixels) per method obtained comparable results, achieving an F1-score of
second, while our proposed FPGA method attained a speed 0.784 for the Zamora dataset. However, our results were obtained
of 60 samples (one sample was 120 × 120 × 10 pixels) in on a larger patch size, i.e., 120 × 120, which implies a larger
0.7 seconds, resulting in an increased speed factor of 2.24×. visual variety and a shorter processing time for the same scene.
Considering a tradeoff between performance and speed, the
proposed method has a high anomaly detection capability in
VI. CONCLUSION
skewed data collections.
Further, we made a comparison with an anomaly detec- This work proposed a meta-recognition hardware accelerator
tion system built specifically for an FPGA platform. In [26], for on-board anomaly detection in multispectral imaging mis-
a lightweight hyperspectral image anomaly detector for real- sions. The experimental results have shown that the proposed
time missions was proposed. The authors implemented a accelerator achieved notable processing speed, i.e., 4.46× in-
pruning–quantization–anomaly–detector (P-Q-AD) for hyper- crease, with minimal decrease in accuracy, i.e., 0.041 decrease
spectral datasets containing anomalies such as ships and planes, in F1-score, compared to the equivalent GPU implementation.
and tested it by running it on the same hardware device used Integration of AI algorithms next to the sensor significantly re-
in this article, namely, the ZCU102 platform. This allows us to duces the time between image acquisition and image analysis by
make accurate comparisons in terms of hardware resources used. transmitting only relevant and informative data to the end user,
Since the complexity of the anomalies in the datasets treated making it possible to produce early warnings and interventions
by the two papers, i.e., ours and [26], differ considerably, we when dangerous events are about to occur.
compared the two methods in terms of detection time per pixel A future direction for improvement of the proposed solution
and the amount of hardware resources consumed. We considered is to increase the utilization of the FPGA component by running
the best detection time obtained in [26] for the Sandiego dataset. multiple instances of the same kernel, i.e., several instances of
For a size of 100 × 100 × 166 pixels, the P-Q-AD method re- the feature extractor that load a set of images simultaneously
quired a detection time of 0.56 s, resulting in a processing speed or at different timings. In this case, a higher throughput can
of 337.34 ns/pixel. Our method obtained a processing speed be achieved. At the same time, selecting a reduced number of
5258 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 16, 2023

multispectral bands used for on-board inference can decrease [20] I. Reed and X. Yu, “Adaptive multiple-band CFAR detection of an optical
the computational resources required while maintaining or even pattern with unknown spectral distribution,” IEEE Trans. Acoust., Speech,
Signal Process., vol. 38, no. 10, pp. 1760–1770, Oct. 1990.
improving model accuracy. [21] Q. Du, B. Tang, W. Xie, and W. Li, “Parallel and distributed computing
The European Space Agency has taken the first steps in for anomaly detection from hyperspectral remote sensing imagery,” Proc.
deploying AI on-board via the PhiSat-1 nanosatellite mission. IEEE, vol. 109, no. 8, pp. 1306–1319, Aug. 2021.
[22] M. Díaz, R. Guerra, P. Horstrand, S. López, and R. Sarmiento, “A line-by-
Therefore, an on-ground trained model can be deployed and line fast anomaly detector for hyperspectral imagery,” IEEE Trans. Geosci.
engaged on intelligent systems promoting power-efficient pro- Remote Sens., vol. 57, no. 11, pp. 8968–8982, Nov. 2019.
cessing platforms. This proposal focused entirely on the im- [23] P. Horstrand, M. Díaz, R. Guerra, S. López, and J. F. López,
“A novel hyperspectral anomaly detection algorithm for real-time
plementation of an anomaly detection application running on applications with push-broom sensors,” IEEE J. Sel. Topics Appl.
an embedded device, which could be relevant for future PhiSat Earth Observ. Remote Sens., vol. 12, no. 12, pp. 4787–4797, Dec.
missions. 2019.
[24] J. Caba, M. Díaz, J. Barba, R. Guerra, S. Escolar, and S. López, “Low-
power hyperspectral anomaly detector implementation in cost-optimized
FPGA devices,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens.,
vol. 15, pp. 2379–2393, Mar. 2022, doi: 10.1109/JSTARS.2022.3157740.
REFERENCES [25] L. Zhang et al., “Fast real-time causal linewise progressive hyperspec-
tral anomaly detection via Cholesky decomposition,” IEEE J. Sel. Top-
[1] H. Hoffmann, “Kernel pca for novelty detection,” Pattern Recognit., ics Appl. Earth Observ. Remote Sens., vol. 10, no. 10, pp. 4614–4629,
vol. 40, no. 3, pp. 863–874, 2007. Oct. 2017.
[2] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, “LOF: Identifying [26] N. Ma, X. Yu, Y. Peng, and S. Wang, “A lightweight hyperspectral image
density-based local outliers,” in Proc. ACM SIGMOD Int. Conf. Manage. anomaly detector for real-time mission,” Remote Sens., vol. 11, no. 13,
Data, 2000, pp. 93–104. 2019, Art. no. 1622.
[3] B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. [27] J. Lei, G. Yang, W. Xie, Y. Li, and X. Jia, “A low-complexity hyperspectral
Williamson, “Estimating the support of a high-dimensional distribution,” anomaly detection algorithm and its FPGA implementation,” IEEE J. Sel.
Neural Comput., vol. 13, no. 7, pp. 1443–1471, 2001. Topics Appl. Earth Observ. Remote Sens., vol. 14, pp. 907–921, Oct. 2021,
[4] D. M. Tax and R. P. Duin, “Support vector data description,” Mach. Learn., doi: 10.1109/JSTARS.2020.3034060.
vol. 54, no. 1, pp. 45–66, 2004. [28] W. J. Scheirer, A. de Rezende Rocha, A. Sapkota, and T. E. Boult, “Toward
[5] J. Verbesselt, A. Zeileis, and M. Herold, “Near real-time disturbance de- open set recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35,
tection using satellite image time series,” Remote Sens. Environ., vol. 123, no. 7, pp. 1757–1772, Jul. 2013.
pp. 98–108, 2012. [29] C. M. Bishop et al., Neural Networks for Pattern Recognition. London,
[6] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly detection: A survey,” U.K.: Oxford Univ. Press, 1995.
ACM Comput. Surv., vol. 41, no. 3, pp. 1–58, 2009. [30] Xilinx, “DPUCZDX8G for zynq UltraScale MPSoCs product guide
[7] S. Matteoli, M. Diani, and G. Corsini, “A tutorial overview of anomaly (PG338),” Jun. 24, 2022. Accessed: Nov. 22, 2022. [Online]. Available:
detection in hyperspectral images,” IEEE Aerosp. Electron. Syst. Mag., https://docs.xilinx.com/r/en-US/pg338-dpu/Introduction?tocId=
vol. 25, no. 7, pp. 5–28, Jul. 2010. Bd4R4bhnWgMYE6wUISXDLw
[8] M. Coca, A. Anghel, and M. Datcu, “Unbiased seamless SAR image [31] A. Bendale and T. Boult, “Towards open set deep networks,” in Proc. IEEE
change detection based on normalized compression distance,” IEEE J. Sel. Conf. Comput. Vis. Pattern Recognit., 2016, pp. 1563–1572.
Topics Appl. Earth Observ. Remote Sens., vol. 12, no. 7, pp. 2088–2096, [32] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily
Jul. 2019. fooled: High confidence predictions for unrecognizable images,” in Proc.
[9] A. Rostami, R. Shah-Hosseini, S. Asgari, A. Zarei, M. Aghdami-Nia, and IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 427–436.
S. Homayouni, “Active fire detection from landsat-8 imagery using deep [33] W. J. Scheirer, Ed., Extreme Value Theory-Based Methods for Visual
multiple kernel learning,” Remote Sens., vol. 14, no. 4, 2022, Art. no. 992. Recognition, Ser. Synthesis Lectures on Computer Vision. Cham, Switzer-
[10] M. Coca, I. C. Neagoe, and M. Datcu, “Hybrid DNN-Dirichlet land: Springer, 2017.
anomaly detection and ranking: Case of burned areas discovery,” IEEE [34] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient processing of
Trans. Geosci. Remote Sens., vol. 60, Sep. 2022, Art. no. 4414116, deep neural networks: A tutorial and survey,” Proc. IEEE, vol. 105, no. 12,
doi: 10.1109/TGRS.2022.3207311. pp. 2295–2329, Dec. 2017.
[11] M. Coca and M. Datcu, “Anomaly detection in post fire assessment,” in [35] M. Nagel, M. van Baalen, T. Blankevoort, and M. Welling, “Data-free
Proc. IEEE Int. Geosci. Remote Sens. Symp., 2021, pp. 8620–8623. quantization through weight equalization and bias correction,” 2019. [On-
[12] J. Keeley and A. Syphard, “Large california wildfires: 2020 fires in line]. Available: https://arxiv.org/abs/1906.04721
historical context,” Fire Ecol., vol. 17, 2021, Art. no. 22. [36] J. R. Jensen et al., Introductory Digital Image Processing: A Remote
[13] A. D. George and C. M. Wilson, “Onboard processing with hybrid and Sensing Perspective. Englewood Cliffs, NJ, USA: Prentice-Hall, 1996.
reconfigurable computing on small satellites,” Proc. IEEE, vol. 106, no. 3, [37] K. Kitano et al., “Super-resolution for a dispersive spectrometer using a
pp. 458–470, Mar. 2018. tilted area sensor and spectrally varying blur kernel interpolation,” Opt.
[14] D. Rudolph et al., “CSP: A multifaceted hybrid architecture for space Exp., vol. 29, no. 2, pp. 2809–2818, Jan. 2021.
computing,” in Proc. 28th Annu. AIAA/USU Conf. Small Satellites, 2014, [38] E. S.A. (ESA), “MultiSpectral instrument (MSI) overview,” 2016. Ac-
pp. 1–7. cessed: Nov. 11, 2022. [Online]. Available: https://sentinels.copernicus.
[15] RT Kintex UltraScale FPGAs for Ultra High Throughput and High Band- eu/web/sentinel/technical-guides/sentinel-2-msi/msi-instrument
width Applications (WP523), Xilinx, May 19, 2020, [Online]. Available: [39] G. Sumbul, M. Charfuelan, B. Demir, and V. Markl, “Bigearthnet:
https://docs.xilinx.com/v/u/en-US/wp523-xqrku060 A. large-scale benchmark archive for remote sensing image under-
[16] G. Giuffrida et al., “The phi-sat-1 mission: The first on-board deep neural standing,” in Proc. IEEE Int. Geosci. Remote Sens. Symp., 2019,
network demonstrator for satellite earth observation,” IEEE Trans. Geosci. pp. 5901–5904.
Remote Sens., vol. 60, pp. 1–14, 2022. [40] G. Sumbul et al., “BigEarthNet-MM: A. large scale multi-modal multi-
[17] G. Giuffrida et al., “Cloudscout: A deep neural network for on-board cloud label benchmark archive for remote sensing image classification and
detection on hyperspectral images,” Remote Sens., vol. 12, no. 14, 2020, retrieval,” IEEE Geosci. Remote Sens. Mag., vol. 9, no. 3, pp. 174–180,
Art. no. 2205. Sep. 2021.
[18] W. J. Scheirer, A. Rocha, R. J. Micheals, and T. E. Boult, “Meta- [41] G. Zeug, S. Tumampos, M. Buchauer, A. Pennec, and H. Dufour-
recognition: The theory and practice of recognition score analysis,” mont, “HRL grassland 2018 validation report,” 2020. Accessed: Nov.
IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1689–1695, 11, 2022. [Online]. Available: https://land.copernicus.eu/user-corner/
Aug. 2011. technical-library/clms_hrl_gra_validation_report_sc04_v1_5.pdf
[19] Q. Du and R. Nekovei, “Fast real-time onboard processing of hyperspectral [42] European Space Agency (ESA), “Resolution and swath,” 2014. Accessed:
imagery for detection and classification,” J. Real-Time Image Process., Aug. 5, 2022. [Online]. Available: https://sentinel.esa.int/web/sentinel/
vol. 4, no. 3, pp. 273–286, 2009. missions/sentinel-2/instrument-payload/resolution-and-swath
COCA AND DATCU: FPGA ACCELERATOR FOR META-RECOGNITION ANOMALY DETECTION: CASE OF BURNED AREA DETECTION 5259

[43] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, Mihai Datcu (Fellow, IEEE) received the M.S. and
MA, USA: MIT Press, 2016. Ph.D. degrees in electronics and telecommunications
[44] E. M. F. (EFIMED), “Local initiatives to mitigate wildfire risk in Spain,” from the University Politehnica Bucharest (UPB),
2022. Accessed: Nov. 15, 2022. [Online]. Available: https://medforest.net/ Bucharest, Romania, in 1978 and 1986, respectively,
2022/06/28/local-initiatives-to-mitigate-wildfire-risk-in-spain/ and the habilitation a Diriger Des Recherches degree
[45] BBC, “France firefighters battle ‘monster’ wildfire near bordeaux,” 2022. in computer science from the University Louis Pas-
Accessed: Nov. 15, 2022. [Online]. Available: https://www.bbc.com/ teur, Strasbourg, France, in 1999.
news/world-europe-62503775 From 1992 to 2002, he had a longer Invited Pro-
[46] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image fessor Assignment with the Swiss Federal Institute
recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, of Technology (ETH Zurich), Zurich, Switzerland.
pp. 770–778. He was a Visiting Professor with the University of
[47] M. P. Del Rosso, A. Sebastianelli, D. Spiller, P. P. Mathieu, and S. L. Oviedo, Oviedo, Spain; University Louis Pasteur, Strasbourg, France; Interna-
Ullo, “On-board volcanic eruption detection through CNNs and satellite tional Space University, Strasbourg, France; the University of Siegen, Siegen,
multispectral imagery,” Remote Sens., vol. 13, no. 17, 2021, Art. no. 3479. Germany; University of Innsbruck, Innsbruck, Austria; University of Alcala,
[48] S. Tilon, F. Nex, N. Kerle, and G. Vosselman, “Post-disaster building Alcala de Henares, Spain; University Tor Vergata, Rome, Italy; the Universidad
damage detection from earth observation imagery using unsupervised and Pontificia de Salamanca, Madrid, Spain; University of Camerino, Camerino,
transferable anomaly detecting generative adversarial networks,” Remote Italy; and the Swiss Center for Scientific Computing, Manno, Switzerland.
Sens., vol. 12, no. 24, 2020, Art. no. 4193. Since 1981, he has been a Professor with the Department of Applied Electronics
and Information Engineering, Faculty of Electronics, Telecommunications and
Information Technology, UPB. Since 1993, he has been a Scientist with the
German Aerospace Center (DLR), Wessling, Germany. He is developing algo-
rithms for model-based information retrieval from high-complexity signals and
methods for scene understanding from very-high-resolution synthetic aperture
radar (SAR) and interferometric SAR data. Since 2011, he has been leading the
Immersive Visual Information Mining Research Laboratory, Munich Aerospace
Faculty, and he is the Director of the Research Center for Spatial Information,
UPB. Since 2001, he had been initiating and leading the Competence Center on
Information Extraction and Image Understanding for Earth Observation, Paris-
Tech, Paris Institute of Technology, Telecom Paris, Paris, France, a collaboration
of DLR with the French Space Agency (CNES). He has been a Professor with the
DLR-CNES Chair, ParisTech, Paris Institute of Technology, Telecom Paris. He
has initiated the European frame of projects for image information mining (IIM)
and is involved in research programs for information extraction, data mining
and knowledge discovery, and data understanding with the European Space
Agency (ESA), NASA, and in a variety of national and European projects. He
and his team have developed and are developing the operational IIM processor
in the Payload Ground Segment systems for the German missions, TerraSAR-X,
Mihai Coca received the B.S. and M.S. degrees TanDEM-X, and the ESA Sentinel-1 and Sentinel-2. He is a Senior Scientist and
in computer science and information security from the Data Intelligence and Knowledge Discovery Research Group Leader with the
the Military Technical Academy “Ferdinand I” Remote Sensing Technology Institute, DLR and involved in the DLRONERA
Bucharest, Romania, in 2015 and 2017, respec- Joint Virtual Center for AI in Aerospace. His research interests include explain-
tively, and the Ph.D. degree in electronic engi- able and physics aware artificial intelligence, smart sensors design, and quantum
neering, telecommunication, and information tech- machine learning with applications in earth observation.
nologies from the University Politehnica Bucharest Prof. Datcu is a Member of the ESA Working Group Big Data from Space
(UPB), Bucharest, Romania, in 2023. and Visiting Professor with the ESA’s Φ-Lab. He was the recipient of the Best
He joined CEOSpaceTech as a Research Assistant, Paper Award and the IEEE Geoscience and Remote Sensing Society Prize, in
in 2017. Since 2015, he has been an Associate Teacher 2006, the National Order of Merit with the rank of Knight, for outstanding
with the Military Technical Academy “Ferdinand I” international research results, awarded by the President of Romania, in 2008, and
covering subjects such as architecture of computer systems and microprocessor the Romanian Academy Prize Traian Vuia for the development of the SAADI
systems. In 2018, he performed a three-month internship with the German image analysis system and his activity in image processing, in 1987. He was
Aerospace Center (DLR), Oberpfaffenhofen, Germany, as a visiting Ph.D. also the recipient of the Chaire d’excellence internationale Blaise Pascal 2017
student with the Remote Sensing Technology Institute (IMF). In the same for international recognition in the field of data science in earth observation. He
year, he participated on BigSkyEarth Training School, which took place at the has served as a co-organizer for international conferences and workshops and
Vicomtech Research Center, San Sebastian, Spain. His current research interests as Guest Editor for a special issue on AI and Big Data of the IEEE and other
include on-ground and on-board artificial intelligence methods for detecting journals. He is a representative of Romanian in the Earth Observation Program
changes and anomalies in multispectral and SAR imagery. Board.