Sentinel-2 and MODIS Data Fusion for Generation of Daily

Cloud-Free Images at the Sentinel Resolution Level

Natalya Ivanchuk1,2, Peter Kogut3,4 and Petro Martyniuk1,5
1
EOS Data Analytics Ukraine, Desyatynny lane, 5, Kyiv, 01001, Ukraine
2
Department of Computer Sciences and Applied Mathematics, National University of Water and Environmental
Engineering, Soborna str., 11, Rivne, 33028, Ukraine
3
Department of Mathematical Analysis and Optimization, Oles Honchar Dnipro National University, Gagarin av., 72,
Dnipro, 49010, Ukraine
4
EOS Data Analytics Ukraine, Gagarin av., 103a, Dnipro, 49010, Ukraine
5
Institute of Automation, Cybernetics and Computer Engineering, National University of Water and Environmental
Engineering, Soborna str., 11, Rivne, 33028, Ukraine

Abstract
In this paperwe discuss a newvariational approach to the Date Fusion problem of multi-spectral
satellite images from Sentinel-2 and MODIS that have been captureed at different resolution level
and, arguably, on different days. The crucial assumption to our approach is that the MODIS image
has to be cloud-free whereas the images from Sentinel-2 can be corrupted by clouds or noise. We
formulate the data fusion problem as the two-level optimization problem. We discuss the well
thoroughness and consistency of the proposed variational models. We also derive some optimality
conditions and supply our approach by results of numerical simulations with the real satellite
images.

Keywords
Data Fusion, Variational Approach, Image Reconstruction, Optimization Problems, Image
restoration

1. Introduction
It is well-known that the data fusion problem is often exacerbated by cloud contamination. In some
cloudy areas, researchers are fortunate to get 2–3 cloud-free satellite scenes per year, what is insufficient for
many applications that require dense temporal information, such as crop condition monitoring and
phenology studies. In view of this, we can indicate the following general requirements for the satellite
image fusion process: (i) The fused image should preserve all relevant information from the input images;
(ii) The image fusion should not introduce artifacts which can lead to wrong inferences.
In spite of the fact that the first requirement (item (i)) sounds rather vague, we give a precise treatment
for it in Section 5, making use of a collection of special constrained minimization problems (see (22)). As
for the second item, it is important to emphasize that we are mainly interesting by satellite images that can
be useful from agricultural point of view (land cover change mapping, crop condition monitoring, yield
estimation, and many others).

_______________________________
COLINS-2023: 7th International Conference on Computational Linguistics and Intelligent Systems, April 20–21, 2023, Kharkiv, Ukraine
EMAIL: [email protected] (N. Ivanchuk); [email protected] (P. Kogut); [email protected] (P. Martyniuk)
ORCID: 0000-0002-6824-9406 (N. Ivanchuk); 0000-0003-1593-0510 (P. Kogut); 0000-0002-2750-2508 (P. Martyniuk)
© 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
Because of this an important option in the image data fusion is to preserve the precise geo-location of the
existing crop fields and avoid an appearance of the so-called false contours and pseudo-boundaries on a
given territory.
In this paper we mainly focus on the image fusion problem coming from two satellites — Setnitel-2 and
Moderate Resolution Imaging Spectroradiometer (MODIS) [1]. Since each band (spectral channel) in
Sentinel images has 10, 20, or 60 meters in pixel size, it gives an ideal spatial resolution for vegetation
mapping at the field scale. Moreover, taking into account that Sentinel-2 has 3–5 revisit cycle over the
same territory, it makes its usage for studying global biophysical processes, which allows to evolve rapidly
during the growing season, essentially important. The unique problem that drastically restricts its practical
implementation, is the fact that the images from Sentinel-2, as a rule, are often contaminated by clouds,
shadows, dust, and other atmospheric artifacts.
One of possible solutions for practical applications is to make use of frequent coarse-resolution data of
the MODIS. Taking into account that the MODIS data can be delivered with the daily repeat cycle and
500-m surface reflectance, the core idea is to use the Sentinel and MODIS data to generate synthetic ’daily’
surface reflectance products at Sentinel spatial resolution [2].
The problem we consider in this paper can be briefly described as follows. We have a collection of
multi-band images { , , … , : → ℝ } from Sentinel-2 that were captured at some time instances
{ , , … , }, respectively, and we have a MODIS image : → ℝ from some day . It is assumed
that all of these images are well co-registered with respect to the unique geographic location. We also
suppose that the MODIS image is cloud-free and the day may does not coincide with any of time
instances { , , … , }. Meanwhile, the Sentinel images { , , … , : → ℝ } can be corrupted by
clouds. The main question is how to generate a new synthetic ’daily’ multi-band image of the same territory
from the day at the Sentinel-2 spatial resolution , utilizing for that the above mentioned data.
In principle, this problem is not new in the literature [3, 4, 5]. For nowadays the spatial and temporal
adaptive reflectance fusion model (STARFM) is one of the most popular model where the idea to generate
a new synthetic ’daily’ satellite images at high resolution level has been realized [6, 7]. However, its
performance essentially depends on the characteristic patch size of the landscape and degrades somewhat
when used on extremely heterogeneous fine-grained landscapes [3].
Instead of this, we mainly focus on the variational approach to the satellite image data fusion (see [8, 9,
10]). We formulate the data fusion problem as the two-level optimization problem. At the first level,
following a simple iterative procedure, we generate the so-called structural prototype for a synthetic
Sentinel image from the given day . The main characteristic feature of this prototype is the fact that, it
must have a similar geometrical structure (namely, precise location of contours and field boundaries) to the
nearest in time ’visible’ Sentinel images, albeit they may have rather different intensities in all bands. Since
the revisit cycle of Sentinel-2 is 2–3 days, such prototype can be easily generated. We consider the above
mentioned structural prototype as a reasonable input data for ’daily’ prediction problem that we formulate a
special constrained minimization problem, where the cost functional has a nonstandard growth and the edge
information for restoration of MODIS cloud-free images at the Sentinel resolution is accumulated both in
the variable exponent of nonlinearity and in the directional image gradients which we derive from the
predicted structural prototype. It is worth to emphasize that our model is considerably different from the
variational model for P+XS image fusion that was proposed in [11].

2. Non-Formal Statement of the Problem

Let Ω ⊂ ℝ be a bounded connected open set with a sufficiently smooth boundary Ω and nonzero
Lebesgue measure. In majority cases Ω can be interpreted as a rectangle domain. Let and be two
sample grids on Ω.
Let [0, ] be a given time interval. Normally, by we mean a number of days. Let and { } be
moments in time (particular days) such that 0 ≤ < < ⋯ < ≤ and < < . Let
{ , ,…, : → ℝ } be a collection of multispectral images of some territory, delivered from
Sentinel-2, that were taken at time instances , , … , , respectively. Let : → ℝ be a MODIS
image of the same territory and this image has been captured at time = . It is assumed that:
1. The Sentinel-2 images { } can be corrupted by some noise, clouds and blur, whereas is
a cloud-free image;
2. We divide the set of all bands for Sentinel images onto two parts and such that each
spectral channel of the MODIS image has the similar spectral characteristics to some channel of
-group;
3. The MODIS image : → ℝ is not corrupted by clouds or its damage zone can be
neglected;
4. The MODIS image and the images { , , … , } from Sentinel-2 are rigidly co-registered
[12, 13]. This means that the MODIS image after arguably some affine transformation and each Sentinel
images after the resampling to the grid with low resolution , could be successfully matched according to
the unique geographic location [14, 15].

2.1. Functional Spaces

Let us recall some useful notations. For given 1 ≤ ≤ +∞, the space (Ω; ℝ ) is defined by
(Ω; ℝ ) = : Ω → ℝ ∶ ∥ ∥ ( ;ℝ )< +∞ ,
/
where ∥ ∥ ( ;ℝ )= ∫ | ( )| for 1 ≤ < +∞.
Given a real Banach space , we will denote by ([0, ]; ) the space of all continuous functions from
[0, ] into . We recall that, for 1 ≤ < ∞, (0, ; ) is the space of all measurable functions
: [0, ] → such that [16, 17]
∥ ∥ ( , ; )= ∫ ∥ ( ) ∥ < ∞,
while (0, ; ) is the space of measurable functions such that ∥ ∥ ( , ; ) = sup ∈[ , ] ∥
( ) ∥ < ∞. For more detailed presentation of the theory of these spaces and the Sobolev spaces with
variable exponents, we refer to [18, 19, 20, 21, 22, 23].

2.2. Topographic Maps and Geometry of Satillite Multispectral Images

Following the main principle of the Mathematical Morphology, a scalar image : Ω → ℝ is a
representative of an equivalence class of images obtained from via a contrast change, i.e., = ( ),
where is a continuous strictly increasing function. Under this assumption, a scalar image can be
characterized by its upper level sets ( ) = { ∈ Ω ∶ ( ) ≥ } . Moreover, each image can be
recovered from its level sets by the reconstruction formula ( ) = sup{ ∶ ∈ ( )}. Thus, according
to the Mathematical Morphology Doctrine, we can suppose that the reliable geometric information about a
scalar image is contained in those level sets.
In order to describe the level sets by their boundaries, ( ), we assume that ∈ , (Ω), where
, (Ω) stands for the standard Sobolev space of all functions ∈ (Ω) with respect to the norm
,
∥ ∥ , ( )=∥ ∥ ( ) +∥ ∇ ∥ ( ) . Then at almost all points of almost all level sets of ∈ (Ω)
we may define a unit normal vector ( ) [24]. This vector field formally satisfies the following relations
( , ∇ ) = |∇ | and | | ≤ 1 a. e. in. In the sequel, we will refer to the vector field as the vector field
of unit normals to the topographic map of a function . So, we can associate with the geometry of the
scalar image [25, 26].

∇ ( ,⋅)
Remark 2.1 In practice, a better choice for ( , ) would be to compute it as the ration for
|∇ ( ,⋅)|
some small value of > 0, where ( , , ) is a solution (for small value of > 0) of the following
initial-boundary value problem with 1 -Laplace operator in the principle part
∇
= div , ∈ (0, +∞), ( , ) ∈ Ω, (1)
|∇ |
(0, , ) = ( , ), ( , ) ∈ Ω, (2)
( , , )
= 0, ∈ (0, +∞), ( , ) ∈ Ω. (3)

2.3. Texture Index of a Gray-Scale Image

Let ∈ ([0, ]; (Ω)) be a given function. For each ∈ [0, ], we associate the real-valued
mapping ( ,⋅): Ω ↦ ℝ with a gray-scale image, and the mapping : (0, ) × Ω → ℝ with an optical
flow.
Definition 2.1 We say that a function : (0, ) × → ℝ is the texture index of a given optical
flow : (0, ) × → ℝ if it is defined by the rule
( , ): = 1 + ∫ |(∇ ∗ ( ,⋅))( )| , (4)
for all ( , ) ∈ , where denotes zero extension of from = (0, ) × Ω to ℝ , stands for
the two-dimentional Gaussian of width (standard deviation) >0 : [0, ∞) → (0, ∞) is the
edge-stopping function which we take in the form of the Cauchy law ( ) = with > 0 small
enough, and ℎ > 0 is a small positive value.

Since ∈ (ℝ ) , it follows from (4) that 1 < ( , ) ≤ 2 in and

∈ ([0, ]; (ℝ )) even if is just an absolutely integrable function in . Moreover, for each
∈ [0, ], ( , ) ≈ 1 in those places of Ω where some edges or discontinuities are present in the image
( ,⋅), and ( , ) ≈ 2 in places where ( , ) is smooth or contains homogeneous features. In view of
this, ( , ) can be interpreted as a characteristic of the sparse texture of the function that can change
with time.
Lemma 2.1 Let ∈ ([0, ]; ( )) be a measurable function extended by zero outside of .
Let be the corresponding texture index. Then there exists a constant > 0 depending on , , and
∥ ∥ ([ , ]; ( )) such that
: = 1 + ≤ ( , ) ≤ : = 2, ∀ ( , ) ∈ , (5)
∈ , ( ), (6)
where = ℎ ℎ+∥ ∥ ( )
|Ω| ∥ ∥ ( , ; ( )) .

3. Data Fusion Problem. Main Requirements to the Formal Statement

Let { , , … , : → ℝ } be a collection of multispectral images of some territory from Sentinel-2
that were taken at time instances { , , … , } ⊂ [0, ], respectively. Let { , , … , } ⊂ 2 be a
collection of damage regions for the corresponding Sentinel-images. So, in fact, we deal with the set of
images { : \ → ℝ }, = 1, . . . , . Let : → ℝ be a MODIS image of the same territory and
this image has been captured at time = ∈ ( , ).
We begin with the following assumption: = ∅ and the damage zones for the rest images from
Sentinel-2 are such that each , = 2, … , , is a measurable closed subset of Ω with property ℒ ( ) ≤
0.6 ℒ (Ω), where ℒ ( ) stands for the 2-D Lebesgue measure of .
We say that multi-band images : →ℝ are structural prototypes of the corresponding
cloud-corrupted ones { : \ → ℝ } if they are defined as follows:
 , ( ) = , ( ),
, ( ), ∈ \ , (7)
, ( ) = ,
, , ( ), ∈ ∩ ,
[ , ] \
where = 2, … , , = 1, … , , , = , and
[ , ] \

[ , ] \ := ∑ ∈ \ , ( ).
ℒ ( \ )
Moreover, each structural prototype : →ℝ is rigidly related to the corresponding day when
the image : \ → ℝ had been taken.

Remark 3.1 As follows from the rule (7), this iterative procedure should be applied to each spectral
channel of all multi-band images from Sentinel-2. Since the revisit time for Sentinel-2 is 3 − 5 days and
the collection of images { } is rigidly co-registered, it follows from (7) that the structural prototypes
are also well co-registered and they have the similar topographic maps with respect to their precise
space location, albeit some false contours can appear along the boundaries of the damage zones . In
fact, in order to avoid the appearance of the false contours, the weight coefficients , have been
introduced.

Since the MODIS image has been taken at a time instance ∈ ( , ), we can have three possible
cases:
(А1) there exists an index ∗ ∈ {1, . . . , } such that = ∗;
∗
(А2) there exists an index ∈ {1, . . . , − 1} such that ∗ < < ∗ ;
(А3) < < .

In view of this, we will distinguish three different statements of the data fusion problem:
(Restoration Problem) The problem (A1) consists in restoration of the damaged multi-band optical
image ∗ : \ ∗ → ℝ using result of its fusion with the cloud-free MODIS image : → ℝ of the
same territory. It means that, we have to create a new image ∗ : → ℝ , which would be well defined
on the entire grid , such that
∗ ( ) = ∗ ( ), ∀ = ( , ) ∈ \ ∗ , (8)
∑ ∈ ∩ ∗ ( ∗ , ( )− ( )) = inf ∑ ∈ ∩ ∗ (( ∗ )( ) − ( )) , (9)
∈ℐ ∀ ∈ ,
( ) = ∗ , ( ), ∀ ∈ , ∀ ∈ .
∗, (10)
The precise description of the class of admissible (or feasible) images ℐ will be given in the next
section.
(Interpolation Problem) The problem (A2) consists in generation of a new multi-band optical image
: → ℝ at the Sentinel-level of resolution using result of the fusion of cloud-free MODIS image
: → ℝ with the predicted structural prototype : → ℝ from the given day . In fact, in this
case we deal with the two-level problem. At the first level, having the collection of structural prototypes
: →ℝ which is associated with the time instances { , , … , } ⊂ [0, ], we create a new
’intermediate’ image : → ℝ that can be considered as daily prediction of the topographical map of
a given territory from the day . Then, at the second level, we realize the fusion procedure of this
predicted image with the cloud-free MODIS image : → ℝ of the same territory.
(Extrapolation Problem) The problem (A3) consists in generation of a new multi-band optical image
: → ℝ using result of the data assimilation from the cloud-free MODIS image : → ℝ into
the structural prototype : → ℝ of the Sentinel-image : \ → ℝ .
4. The Model for Prediction of Structural Prototypes

Let ∗ , and ∗ , be structural prototypes of the corresponding images from given days ∗ and
∗ . Since ∗ , and ∗ , are well co-registered images, it is reasonable to assume that they have the
similar geometric structure albeit they may have very different intensities.
The main question we are going to discuss in this section is: how to correctly define the ’intermediate’
image : → ℝ that can be considered as daily prediction of the topographical map of a given
territory from the day . With that in mind, for each spectral channel ∈ {1,2, … , }, we make use of the
following model
− div |∇ | ( , ) ∇ = in ( ∗ , ∗ ) × Ω, (11)
= 0 on ( ∗ , ∗ ) × Ω, (12)
( ∗ ,⋅) = ∗ , (⋅) in Ω, (13)
where ( , ) stands for the texture index of the scalar image (see Definition 2.1), and ∈
( ∗ , ∗ ; (Ω)) is an unknown source term that should be defined in the way to guarantee the
fulfillment (with some accuracy) of the relation
( ∗ ,⋅) ≈ ∗ , (⋅) in Ω. (14)
Here, in (13) and (14), by default, we assume that the images ∗ , and ∗ , are well defined onto
the entire domain Ω.

Remark 4.1 The main characteristic feature of the proposed initial-boundary value problem (IBVP) is
the fact that the exponent depend not only on ( , ) but also on ( , ). It is well-known that the
variable character of exponent causes a gap between the monotonicity and coercivity conditions.
Because of this gap, equations of the type (11) can be termed equations with nonstandard growth
conditions. So, in fact, we deal with the Cauchy-Neumann IBVP for parabolic equation of =
( , , )-Laplacian type with variable exponent of nonlinearity. It was recently shown that the model (11)–
(13) naturally appears as the Euler-Lagrange equation in the problem of restoration of cloud contaminated
satellite optical images [27]. In particular, this model has been proposed in [2] in order to avoid the
blurring of edges and other localization problems presented by linear diffusion models in images
processing.

We note that the distributed control in the right hand side of (11) describes the fictitious sources or
sinks of the intensity that may have a tendency to change at most pixels even for co-registered structural
prototypes ∗ , (⋅) and ∗ , (⋅).

Definition 4.1 We say that, for given ∈ ( ) and ∗ , ∈ ( ), a function is a weak solution
to the problem (11)–(13) if, for a.a. ∈ [ ∗ , ∗ ],
, ∗
∈ ( ∗ , ∗ ; (Ω)), ( ,⋅) ∈ (Ω) , ∫ ∗ ∫ |∇ | ( , ) < +∞, (15)
and the integral identity
∗
∫ ∗
∫ − + (|∇ | ∇ , ∇ )
∗
=∫ ∗
∫ +∫ ∗, | ∗ (16)
holds true for any function ∈ Φ, where Φ = ∈ ([ ∗ , ∗ ] × Ω) ∶ | ∗ =0 .

Utilizing the perturbation technique and the classical fixed point theorem of Schauder, it has been
recently proven the following existence result.
 Theorem 4.1 [28] Let ∈ ( ) and ∗ , ∈ ( ) be given distributions. Then initial-boundary
value problem (11)–(13) admits at least one weak solution = ( , ) with the following higher
inegrability properties
,
∈ ( ∗ , ∗ ; (Ω)), ∈ (( ∗ , ∗ ) × Ω), ∈ ( ∗, ∗ ; (Ω)), (17)
where the exponent is given by the rule
= .
∥ ∥ | | ∥ ∥ ( )
∥ ∗, ∥ ( )
( )

In order to satisfy the condition (14) and define an appropriate source term = ( , ), we utilize some
issues coming from the well-known method of Horn and Schunck that has been developed in order to
compute optical flow velocity from spatiotemporal derivatives of image intensity. Following this approach,
we define the function ∗ as a solution of the problem
∫ − div (|∇ | ∇ )| − + ∫ |∇ | → inf , (18)
∈ ( )
where ̂ = ( ∗ + ∗ )/2, > 0 is tuning parameter (for numerical simulations we take = 0.5),
and the spatiotemporal derivatives are computed by the rules
= ∗ ∗
∗ , − ∗, ,
∗, ∗
div (|∇ | ∇ )| = [div |∇ ∗, | ∇ ∗, + div |∇ ∗ , | , ∇ ∗ , ].

It is clear that a minimum point ∗ ∈ (Ω) to unconstrained minimization problem (18) is unique and
satisfies necessarily the Euler-Lagrange equation
Δ ∗+ − div (|∇ | ∇ )| − ∗ = 0 (19)
∗
with the Nuemann boundary condition = 0 on Ω.
∗
Setting = in (11), we can define a function ∗ = ∗ ( , ) as the weak solution of the IBVP (11)–
(13). Numerical experiments show that, following this way, we obtain a function ∗ with properties (15)
and (17) such that ∗ ( ∗ , ) = ∗ , ( ) and ∗ ( ∗ , ) ≈ ∗ , ( ) in Ω , where the peak
signal-to-noise ratio (PSNR) between images ∗ ( ∗ , ) and ∗ , ( ) is sufficiently large, >
46.
This observation leads us to the following conclusion: the ’intermediate’ image : → ℝ can be
defined as follow:
( ) = ∗ ( , ), ∀ ∈ . (20)

5. Variational Statements of the Data Fusion Problems (A1)–(A3)

Coming back to the principle cases (A1)–(A3), that have been described in Section 3, we can suppose
that a structural prototype from the given day is well defined. As it was pointed out in Section 3,
this prototype coincides either with one of the images in cases (A1) and (A3), or it should be defined
using the solutions of the problem (11)–(13), (19) for each = 1, . . . , in the case (A2)-problem (see the
rule (20)).
Let ∈ {1, . . . , } be a fixed index value (the number of spectral channel). Let : Ω → ℝ be the
texture index of the -th band for the structural prototype : → ℝ . Let ∈ (0,1) be a given
threshold. Let = [ , be a vector field such that | ( )|ℝ ≤ 1 and
( ), ∇ , ( ) = |∇ , ( )|ℝ a. e. inΩ.
ℝ
We define the linear operator , : ℝ → ℝ as follows
, (Ω).
, ∇ := ∇ − ,∇ ℝ , ∀ ∈ (21)
 It is clear that , ∇ reduces to (1 − )∇ in regions where the gradient ∇ is colinear to , and
to ∇ , where ∇ is orthogonal to . It is important to note, that this operator does not enforce gradients in
the direction .
Let ( , ) be the Dirac’s delta at the point ( , ). Then Π = ∑( , )∈ ( , ) stands for the
Dirac’s comb defined by the sample grid .
We are now in a position to give a precise meaning of the solutions to the Problems (A1)–(A3). We say
that:
1. A multi-band image ∗ : → ℝ , where ∗ = , is a solution of the Restoration Problem,
if it is given by the rule
∗, ( ) , ∀ ∈ \ ∗,
∗, ( ) = ∀ ∈ ,
( )+ ,∀ ∈ ∩ ∗,
∗, ( ) = , ( ), ∀ ∈ , ∀ ∈ .
Here, is the weight coefficient and we define it as follows
= ∫ \ ∗
| ∗, ( )| − ∫ \ ∗
| , ( )|
and is a solutions of the following constrained minimization problem
( )
ℱ( )=∫ | , ∇ ( )| + ∫ ∇ ( )−∇ , ( )
( )

+ ∫ Π ( ∗ − , ) → inf , (22)
∈
, (⋅)
Ξ = ∈ (Ω) ∶ 0 ≤ ( ) ≤ a. e. in Ω stands for the set of feasible solutions, and
, (⋅)
(Ω) denotes the Sobolev space with variable exponent. As for the constants , their choice
depends on the format of signed integer numbers in which the corresponding intensities , ( ) are
represented. In particular, it can be = 2 − 1, = 2 − 1, and so on.

2. A multi-band image : → ℝ , with ∗ < < ∗ , is a solution of the Interpolation

Problem, if it is given by the rule
( )+ , ∀ ∈ ,
, ( )= ∀ ∈ ,
, ( ) , ∀ ∈ ,
where
= ∫ | , ( )| − ∫ | , ( )| (23)
and is a solutions of the constrained minimization problem (22).

3. A multi-band image : → ℝ , with < < , is a solution of the Extrapolation

Problem, if it is given by the rule
( )+ , ∀ ∈ ,
, ( )= ∀ ∈ ,
( ) , ∀ ∈ , ,
where is a solutions of the constrained minimization problem (22), and is defined as in (23).

Let us briefly discuss the relevance of the proposed minimization problem. The first term in (22) can be
, (⋅)
considered as the regularization in the Sobolev-Orlicz space (Ω) [29]. As for the second term in
(22), it reflects the fact that the topographic map of the retrieved image should be as close as possible to the
topographic map of predicted structural prototype : → ℝ . We interpret this closedness in its
simplified form, namely, in the sense of -norm of the difference of the corresponding gradients [6, 30,
31]. The last term in (22) represents an -distortion between a -th spectral channel in the MODIS image
and the corresponding channel of the retrieved image which is resampled to the grid of low resolution
 .

6. Existence Result and Optimality Conditions for Constrained Minimization

Problem ( )
Following in many aspects the recent studies [28], we give the following existence result.

Theorem 6.1 Let : →ℝ be a given structural prototype for unknown image from
Sentinel-2. Then for any given ∈ , > 0, > 0, and ∈ (0,1), the minimization problem (22) admits
a unique solution ∈ .

In order to derive some optimality conditions to the problem (22) and characterize its solution
, (⋅)
∈ (Ω), we note that the cost functional ℱ : Ξ → ℝ is Gâteaux differentiable. Since Ξ is a
, (⋅)
nonempty convex subset of (Ω) ∩ (Ω) and the objective functional ℱ : Ξ → ℝ is strictly
convex, the well known classical approach leads us to the following conclusion.

Theorem 6.2 Let : → ℝ be a given structural prototype for unknown image from
Sentinel-2. Let : → ℝ be a given MODIS image. Let stands for the texture index of the -th band
for the predicted structural prototype . Then the unique minimizer ∈ to the minimization
problem ℱ ( ) is characterized by the following variational inequality
∈
( )
∫ | , ∇ ( )| ∇ ( ), , , ∇ ( )− , ∇ ( )
+ ∫ ∇ ( )−∇ , ( ), ∇ ( )
∗
+ ∫ Π ∗ ∗ − , ≥ 0, ∀ ∈ Ξ .

Remark 6.1 In practical implementation, it is reasonable to define an optimal solution ∈ using a

’gradient descent’ strategy. Indeed, following the standard procedure and starting from the initial image
, , we can pass to the following initial value problem for the quasi-linear parabolic equations with
Nuemann boundary conditions
( )
− div | , ∇ ( )| , ∇ ( )
( )
= − div | , ∇ ( )| , ∇ ( ),
+ div ∇ ( ) − ∇ , ( ) − Π ∗∗ ∗ − ,
,
( )
| , ∇ ( )| , ∇ , = 0 on Ω,
0≤ ( )≤ a. a. in Ω,
(0, ) = , ( ), ∀ ∈ Ω. (24)
In principle, instead of the initial condition (24) we may consider other image that can be generated
from , and the bicubic interpolation of the MODIS band , onto the entire domain Ω following
one of well-known simple data fusion methods [32].

7. Numerical Experiments
In order to illustrate the proposed approach for the restoration of satellite multi-spectral images we have
used two Sentinel-2 images (610 × 699 in pixels) over the Dnipro Airport area, Ukraine (see Fig. 1) that
were captured at different time instances. What is the most important, it was a growing season when the
global biophysical processes are rapid enough. This region represents a typical agricultural area with
medium sides fields of various shapes.

Figure 1: Given images from Sentinel-2. Date of generation: (left) — 2019/06/15, (right) —
2019/07/17

Figure 2: MODIS image with the date of generation 2019/07/01

Figure 3: (left) Result of MODIS interpolation to the Sentinel resolution, (right) Its re-normalization
to the spectral characteristics of Sentinel-2

We also have a cloud-free MODIS image (67 × 77 in pixels) from 2019/07/01 (see Fig. 2). We solve
the interpolation problem (A2). As a result, a new image from the date 2019/07/01 at the Sentinel-level of
resolution is depicted in Fig. 3. These numerical simulations have been supplied by a detailed analysis of
the obtained results using the special validation metrics.

8. References
[1] M. Joshi, K. Upla, Multi-resolution Image Fusion in Remote Sensing, Cambridge University Press,
2019.
[2] P. Kogut, O. Kupenko, N. Uvarov, On increasing of resolution of satellite images via their fusion with
imagery at higher resolution, JODEA 28 (1) (2021) 54–78. doi:https:
//doi.org/10.1007/s10851-022-01131-w.
[3] F. Gao, J. Masek, M. Schwaller, F. Hall, On the blending of the landsat and modis surface reflectance:
Predicting daily landsat surface reflectance, IEEE Tran. Geosci. Remote Sens. 44 (5) (2006) 2207–
2218.
[4] J. Ju, D. Roy, The availability of cloud-free landsat etm+ date over the conterminous united states and
globally, Remote Sens. Environ. 112 (3) (2008) 1196–1211.
[5] P. Wang, F. Gao, J. Masek, Operational data fusion framework for building frequent landsat-like
imagery, IEEE Transactions on Geoscience and Remote Sensing 52 (11) (2014) 7353–7365.
[6] L. Bungert, D. Coomes, M. Ehrhardt, J. Rasch, R. Reisenhofer, C.-B. Schönlieb, Blind image fusion
for hyperspectral imaging with the directional total variation, Inverse Problems 34 (4) (2018) Article
044003.
[7] L. Loncan, L. De Almeida, J. Bioucas-Dias, X. Briottet, J. Chanussot, N. Dobigeon, S. Fabre, W. Liao,
G. Licciardi, M. Simoes, J. Tourneret, M. Veganzones, G. Vivone, Q. Wei, N. Yokoya, Hyperspectral
pansharpening: a review, IEEE Geoscience and Remote Sensing Magazine 3 (3) (2015) 27–46.
[8] C.D’Apice, P. Kogut, O. Kupenko, R. Manzo, On variational problem with nonstandard growth
functional and its applications to image processing, J Math Imaging Vis. (2022) 1–20.
doi:https://doi.org/10.1007/s10851-022-01131-w.
[9] P. Kogut, O. Kupenko, R. Manzo, C. Pipino, On variational model in sobolev-orlicz spaces for
spatiotemporal interpolation of multi-spectral satellite images, in: IEEE Int. Conf. System Anal.
Intelligent Comp., 2022, pp. 134–139.
[10] P. Khanenko, P. Kogut, M. Uvarov, On variational problem with nonstandard growth condi-tions for
the restoration of clouds corrupted satellite images, in: CEUR Workshop Proceed-ings, the 2nd
International Workshop on Computational and Information Technologies for Risk-Informed Systems,
CITRisk-2021, September 16-17, 2021, Kherson, Ukraine, volume 3101, 2021, pp. 6–25.
[11] C. Ballester, V. Caselles, L. Igual, J. Verdera, B. Rougé, A variational model for p+xs image fusion,
Int. J. Comp. Vision 69 (2006) 43–58.
[12] D. Roy, J. Li, H. Zhang, L. Yan, Best practices for the reprojection and resampling of sentinel-2 multi
spectral instrument level 1c data, Remote Sens. Lett. 7 (2016) 1023–1032.
[13] L. Yan, D. Roy, H. Zhang, J. Li, H. Huang, An automated approach for sub-pixel registration of
landsat-8 operational land imager (oli) and sentinel-2 multi spectral instrument (msi) imagery, Remote
Sens. 8 (2016) Article 520.
[14] D. Frantz, Landsat + sentinel-2 analysis ready data and beyond, Remote Sens. 11 (2019) Article 1124.
[15] D. Roy, H. Huang, L. Boschetti, L. Giglio, H. Zhang, J. Li, Landsat-8 and sentinel-2 burned area
mapping — a combined sensor multi-temporal change detection approach, Remote Sens. Environ. 231
(2019) Article 111254.
[16] R. Dautray, J. Lions, Mathematical Analysis and Numerical Methods for Science and Technology,
volume 5, Springer-Verlag, 1985.
[17] J.-L. Lions, Optimal Control of Systems Governed by Partial Differential Equations, Springer, 1971.
[18] Y. Chen, S. Levine, M. Rao, Variable exponent, linear growth functionals in image restora-tion, SIAM
Journal Appl. Math. 66 (4) (2006) 1383–1406.
[19] D. Cruz-Uribe, A. Fiorenza, Variable Lebesgue Spaces: Foundations and Harmonic Analysis,
Birkhäuser, 2013.
[20] L. Diening, P. Harjulehto, P. Hästö, M. R ̇ uˆ ziˆ cka, Lebesgue and Sobolev Spaces with Variable
Exponents, Springer, 2011.
[21] L. Nirenberg, Topics in Nonlinear Analysis, Lecture Notes, New York University, 1974.
[22] V. Zhikov, Solvability of the three-dimensional thermistor problem, Proceedings of the Steklov
Institute of Mathematics 281 (2008) 98–111.
[23] V. Zhikov, On variational problems and nonlinear elliptic equations with nonstandard growth
conditions, Journal of Mathematical Sciences 173 (5) (2011) 463–570.
[24] L. Ambrosio, V. Caselles, S. Masnou, J. M. Morel, The connected components of sets of finite
perimeter, European Journal of Math. 3 (2001) 39–92.
[25] V. Caselles, B. Coll, J. Morel, Topographic maps and local contrast changes in natural images, IEEE
Transactions on Image Processing 10 (8) (1999) 5–27.
[26] V. Caselles, B. Coll, J. Morel, Geometry and color in natural images, J. Math. Imaging and Vision 16
(2002) 89–107.
[27] C. D’Apice, P. Kogut, R. Manzo, M. Uvarov, Variational model with nonstandard growth conditions
for restoration of satellite optical images using synthetic aperture radar, Eur. J. Appl. Math. (2022).
doi:https://doi.org/10.1017/S0956792522000031.
[28] P. Kogut, Y. Kohut, N. Parfinovych, Solvability issues for some noncoercive and nonmono-tone
parabolic equations arising in the image denoising problems, JODEA 30 (2) (2022) 19–48.
[29] P. Blomgren, Total variation methods for restoration of vector valued images, Ph.D. Thesis (1998)
384–387.
[30] L. Bungert, M. Ehrhardt, Robust image reconstruction with misaligned structural informa-tion, IEEE
Access 8 (2020) 222944–222955.
[31] P. Blomgren, T. Chan, P. Mulet, C. Wong, Total variation image restoration: Numerical methods and
extensions, Proceedings of the 1997 IEEE International Conference on Image Processing III (1997)
384–387.
[32] K. Rani, R. Sharma, Study of different image fusion algorithm, International Journal of Emerging
Technology and Advanced Engineering 3 (5) (2013) 288–290.