See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/369519110

Joint Demosaicing and Fusion of Multiresolution Coded Acquisitions: A Unified

Image Formation and Reconstruction Method

Article in IEEE Transactions on Computational Imaging · March 2023

DOI: 10.1109/TCI.2023.3261503

CITATIONS READS

2 43

3 authors:

Daniele Picone Mauro Dalla Mura

Grenoble Institute of Technology Grenoble Institute of Technology
28 PUBLICATIONS 96 CITATIONS 239 PUBLICATIONS 7,955 CITATIONS

SEE PROFILE SEE PROFILE

Laurent Condat
King Abdullah University of Science and Technology
127 PUBLICATIONS 3,384 CITATIONS

SEE PROFILE

All content following this page was uploaded by Daniele Picone on 11 April 2023.

The user has requested enhancement of the downloaded file.

 Joint Demosaicing and Fusion of Multiresolution
Coded Acquisitions: A Unified Image Formation
and Reconstruction Method
Daniele Picone , Member, IEEE, Mauro Dalla Mura , Senior Member, IEEE,
and Laurent Condat , Senior Member, IEEE

Abstract—Novel optical imaging devices allow for hybrid ac- full scanning of the datacube. We in particular focus our
quisition modalities such as compressed acquisitions with locally attention to coded acquisitions, for which the pixel on the
different spatial and spectral resolutions captured by a single focal plane can be described as a particular linear combination
focal plane array. In this work, we propose to model the imaging
camera system for a multiresolution coded acquisition (MRCA) of the samples of the original datacube; this is for example
in a unified framework, which includes conventional devices such the case of acquisitions captured through a color filter array
as those based on spectral/color filter arrays, compressed coded (CFA), or where the input image is masked through any coded
apertures, and multiresolution sensing. We also propose a model- aperture before combining on the sensing system. As coded
based image reconstruction algorithm performing a joint demo- acquisitions [7], [8] are not captured in the end user desired
saicing and fusion (JoDeFu) of any acquisition modeled in the
MRCA framework. The JoDeFu reconstruction algorithm solves domain, a computational phase is needed to retrieve a datacube
an inverse problem with a proximal splitting technique and is able that is intelligible to the final user. In this work, we refer to
to reconstruct an uncompressed image datacube at the highest such acquisition techniques as image formation methods, and
available spatial and spectral resolution. An implementation of to the required processing algorithms that recover the desired
the code is available at https://github.com/danaroth83/jodefu. datacube as image reconstruction methods.
Index Terms—Color filter array, compressed acquisitions, pan- A wide selection of snapshot image formation methods are
sharpening, data fusion, demosaicing, multiresolution sensors,
nonconventional optical devices. available in the literature [9], [10], which demands adequate
techniques to process the acquisitions in a prompt and flexible
manner. As the manufacture of many of such prototypes
I. I NTRODUCTION can be seen as a combination of elemental components, this
ONVENTIONAL cameras acquire images that are im-
C mediately exploitable by the end user with little or no
processing of the raw acquisition. When the acquisition relies
work proposes to describe them under a unified mathematical
reconstruction framework. In the proposed framework, the
image formation step is described as a composition of basic
on a spectral or spatial scanning of the scene, these cameras operators which emulates the behavior of the elementary
provide the image as a datacube with spatial and spectral camera components. In particular, the work is focused on two
dimensions [1], [2]. For example, this is the case of red- classic acquisition scenarios, the multiresolution sensing and
green-blue (RGB), multispectral (MS) and hyperspectral (HS) the mosaicing, which are described below:
imaging systems, where the datacube has either three, up to a
few tens, or more channels, respectively. • The multiresolution sensing, shown in Fig.1a, is an
A different acquisition approach, following the computa- acquisition setup where different sensor technologies are
tional imaging paradigm [3]–[6], is based on compressed employed to provide complementary information of the
acquisitions. In some cases, this allows to perform acquisi- same scene to be fused in the processing stage. This
tions that are instantaneous (i.e., snapshot) or with a lower setup addresses the technical constraint of single sensors
number of acquired samples with respect to a conventional which are not capable of simultaneously achieving the
spatial and spatial resolution desired by the final user.
D. Picone is with Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, Most commonly, the product is available as a bundle
GIPSA-lab, 38000 Grenoble, France. D. Picone is also with Univ. Greno-
ble Alpes, CNRS, Grenoble INP, IPAG, 38000 Grenoble, France (e-mail: of two images: the high resolution image (HRI), with
[email protected]). high spatial and low spectral resolution, and the low
M. Dalla Mura is with Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, resolution image (LRI), with low spatial and high spectral
GIPSA-lab, 38000 Grenoble, France. M. Dalla Mura is also with Institut
Universitaire de France (IUF), 75005 Paris, France (e-mail: mauro.dalla- resolution. In the data fusion phase, known as sharpening,
[email protected]). the target is to produce a synthetic image with the highest
L. Condat is with King Abdullah University of Science and available resolutions both in the spectral and spatial
Technology (KAUST), 23955 Thuwal, Saudi Arabia (e-mail:
[email protected]). domain. This is a more generic formulation of the pan-
This work is partly supported by grant ANR FuMultiSPOC (ANR-20- sharpening problem [11], [12], where the target is to fuse
ASTR-0006). a monochromatic acquisition, known as panchromatic
This paper has supplementary downloadable material available at
http://doi.org/10.1109/TCI.2023.3261503, provided by the authors. (Corre- (PAN), and a MS (multispectral) image.
sponding author: Mauro Dalla Mura) • The mosaicing, shown in Fig. 1b, is an acquisition

© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media,
including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to
servers or lists, or reuse of any copyrighted component of this work in other works.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 2

Image formation Image reconstruction

tions. Specifically, the effectiveness of the encoding is often
LRI
[m]
Ad : Spa. degr. measured through associated metrics, and among those, the
B
Decimation
restricted isometry property has proven to be a common choice
M
to provide theoretical limits of the quality of the reconstruc-
Ideal
product ∗
Estimated
product
tion. For the CASSI, this analysis has been performed in [17]
and later reinterpreted in [18] in terms of the coherence of the
f
M Sharpening
sensing matrix. While we consider this topic out of scope in
HRI this work, the interested reader may refer to the vast literature
X
× +
b
X on the field for further information [19], [20].
∗ = Spatial
convolution
More recently, both the scientific community and device
{w:k }k=1,...,Nk P
[p]
Ad : Spe. degr. × =
Elementwise
product
manufacturers are showing interest for the design of hybrid
Ad : Multires.
sensing + =
Sum over systems, where compressed acquisitions might have different
channels
spatial/spectral resolutions. Specifically, for image formation
(a) Degradation and sharpening
methods, a series of RGBW patterns were proposed where
Image formation Image reconstruction
H[m]
a set of wideband pixels are interleaved to a more classic
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
RGB pattern. This is the case, for example, of the Onyx
Ideal
product
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
Y
Y

R
R

Y
Y

R
R
B
B

G
G

B
B

G
G
Estimated
product
device by Teledyne e2v [21] and some patents deposed by
Acquisition Kodak [22]. A similar effect could be obtained with the
× + Demosaicing COLOR SHADES [23], a technology which allows for a fully
[m]
A⊡ : Masking
[m]
A+ customizable spectral response for each filter on the FPA. We
y[x]
X
[m]
Ac : Mosaicing b
X can interpret these acquisition as hybrid, since the wideband
(b) Mosaicing and demosaicing pixels can be seen as a binning in the spectral domain, and
consequently have a different spectral resolution with respect
Fig. 1: Classic formulation of the sharpening and demosaicing to the color ones.
problem. The red, green, blue, and yellow slices represent each Modern commercial CFA patterns, such as the Quad Bayer,
of the available channels of the image to reconstruct. are also starting to implement mechanism of spatial binning
across adjacent pixels, in order to improve the SNR of the
detected photons in conditions of low illumination [24]. More
technique where the captured samples are the output of a recently, a novel technology was proposed to focus the incident
set of image sensors distributed over a focal plane array light rays over photodiodes through customly manufactured
(FPA) and overlaid with an array of filters, known as microlenses [25]. As a consequence, the resulting samples are
CFA or multispectral filter array (MSFA) [13]. As an effectively at a lower spatial resolution with respect to non-
effect of filtering, each captured image pixel is associated binned or less focused alternatives.
to a given color/channel component and the full raw Espitia et al. [26] suggested to capture both the LRI and
acquisition is composed by a mosaic of such components. the HRI as separate CASSI acquisitions, introducing a joint
For example, the reader may be familiar with CFA Bayesian framework for the reconstruction of the full reso-
designs such as the Bayer pattern [14], where the filters lution image. An optical testbed of manufacturing feasibility
are arranged in periodic 2 × 2 RGB squares with 2 for such device is reported in [27]. Fu et al. [28] proposed
repeated green filters placed on the opposite vertices. The instead a reconstruction algorithm where a HS image is fused
associated image reconstruction method, known as demo- with a MS mosaiced image obtained with a CFA acquisition
saicing, consists in recovering the full spectral component system. Takeyama and Ono [29] address the compressed
of the image at each available position in the FPA. pansharpening, where the quality of a noisy HRI is restored
In more sophisticated mosaicing-based compressed acquisi- with the help of an associated compressed acquisition of a
tion systems, a given pixel can also be expressed as a coding LRI.
and multiplexing of a portion of the datacube; in other words, Additionally, such hybrid devices are a reasonable proof
each pixel represents an encoding of the input signal as it that, in the near future, multiresolution sensors could po-
captures a generic linear combination of samples associated tentially be accommodated over the same focal plane. Such
to different channels. For example, this is the case of the unconventional coded instruments can bring benefits in terms
compressive coded aperture spectral imaging (CASSI) [15], for of reduced size, cost, and acquisition time with respect to
which each available channel is firstly masked with a digital traditional hyperspectral camera. In the field of remote sensing,
micromirror device and shifted horizontally over the focal most high-end commercial satellites are equipped with scan-
plane before recombination. More advanced variants of this ners (e.g., pushbroom). However, commercial off-the-shelf
original design have also been proposed, i.e. by introducing components such as consumer cameras based on masking are
colored coded apertures in place of the traditional block- often used with drones or smallsats (e.g., AmicalSat [30]), and
ing/unblocking mechanisms [15] or by combining the spectral make up for an ideal application field for our work.
mosaicing with the shearing in the sensing mechanism [16]. Armed with this knowledge, the main aim of this work is
For such acquisitions, many authors have shown that coding to develop a unified framework for both the image formation
optimization is an important step to provide quality acquisi- and its reconstruction which includes all the previously cited
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 3

examples. For this reason, in the context of image formation • vectors with lowercase bold letters (e.g. u);
methods, we propose the multiresolution coded acquisition • matrices with uppercase bold letters (e.g. U);
(MRCA) framework, a formalization of the multiresolution • tensors (that is, arrays with more than two dimensions)
acquisition model with compressed acquisitions. This model with bold italic fonts (e.g. U ).
can formalize the design of an optical device based on A detailed description of the variables used in this paper is
the assumption that sensors with different characteristics can given in TABLE I. In this work, the image samples are either
be accommodated on the same FPA and that the resulting organized in their classic form, as a 3-way tensor whose
compressed acquisition contains partial information both from dimensions represent the rows, columns and channels (e.g.,
the LRI and the HRI, whose physical implementation was U [x] ∈ RNi ×Nj ×Nk ), or in lexicographic order, where the
presented in our previous work [31]. An example of the first two dimensions of the classic form are concatenated into
acquisitions of such device is shown in Fig. 2, where the one. As both representations contain the same samples, we can
MRCA models the acquisition of a monochromatic raw image switch from the first to the second form without any loss of
which includes samples from both a PAN and a MS, according information, and we denote this operation with matr(·) (e.g.,
to arrangement shown in Fig. 3b. X = matr U [x] , where X ∈ RNi Nj ×Nk ).
Since we have a unified acquisition model, we also propose
When we select a generic k-th slice of the image, this
the joint demosaicing and fusion (JoDeFu), a generic image
is denoted with a subscribed index k, while the non-sliced
reconstruction algorithm that addresses both the demosaicing [x]
dimensions are denoted with a colon (e.g., U::k denotes the
and the multiresolution fusion, extending the definition from [x]
k-th band of U ). When confusion may arise, the subscribed
our previous work [32]. The proposed algorithm both recovers [x]
indices are separated by a comma (e.g., U:,:,k is equivalent
the missing information of the compressed acquisition and [x]
fuses the multiresolution samples to reach the maximum to U::k ). Once again, a more detailed description of such
available spatial and spectral resolution. The algorithm, which operations is shown in TABLE I.
makes use of a Bayesian framework, is not exclusively a Finally, ∥ · ∥2 and ∥ · ∥F denote the ℓ2 and Frobenius norm,
demosaicing-style image reconstruction, as we aim to re- respectively.
construct a fused product, nor it is a simple fusion, as the
acquisition is not given by well-distinguished multimodal III. P ROPOSED IMAGE FORMATION MODEL
sources, but rather by a lossy compressed combination of the In this section, we introduce the mathematical model of the
two. MRCA (Section III-C) and its properties (Section III-D). As
In short, the novel contributions of this work include: a preliminary step, we firstly define the two main component
• the definition of the MRCA, a flexible model for multires- of our image formation models: the multiresolution sensing in
olution sensors sharing a common focal plane, which in- Section III-A and the mosaicing in Section III-B.
cludes a series of well-known image formation methods,
such as the CFA [33] acquisitions and the multiresolution A. Multiresolution sensing
sensing;
• the derivation of some properties of the direct model
When multiresolution acquisitions are involved, each of the
operator associated to the MRCA, which enable its use sensor technologies is characterized by a limited spatial and
with proximal algorithms; spectral resolution. Therefore, in the most general case, the
• the definition of the JoDeFu, an image reconstruction
multiresolution sensing setup is expressed as a set of multiple
framework capable of simultaneously addressing the acquisitions, which present a certain spatial and/or spectral
problem of demosaicing and the fusion of partial mul- degradation with respect to the ideal datacube to reconstruct
tiresolution acquisition; X. For the sake of exposition but without loss of general-
• a comparison of the performances of the JoDeFu with
ization, we limit our analysis to the most common scenario
respect to classic image reconstruction methods for com- in the literature [11], [12], [34], in which the acquisition is
pressed acquisitions; we also analyze the reconstructed composed by an HRI and an LRI. The HRI P and the LRI
f are respectively obtained as a degradation in the spectral
M
products taken with the proposed MRCA, when they
embed a different amount of MS bands, and compare and spatial domain, so that:
(
the JoDeFu with alternative reconstruction approaches [p]
P = Ad (X) ,
obtained by cascading a set of classical algorithms. (1)
f = A[m] (X) .
M d
The paper is organized as follows: Section II introduces the
notation, Section III describes the MRCA, highlighting how In the previous equation:
[p]
it expands on classic image formation methods; Section IV • the spectral degradation operation P = Ad (X) is
presents the JoDeFu algorithm, and Section V provides the described by a linear combination in the form:
related experiments. Nk
X
p:j = wjk x:k , ∀j ∈ [1, ... , Np ] (2)
II. N OTATION k=1
In this paper, we denote: where {wjk }j∈[1,... ,Np ], k∈[1,... ,Nk ] are the weight coeffi-
• scalars with lowercase non-bold letters (e.g. u); cients associated to the spectral responses of the sensors.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 4

TABLE I
N OTATION FOR THE MULTIDIMENSIONAL ARRAYS DEFINED IN THIS PAPER . E ACH VARIABLE CAN BE EXPRESSED EITHER IN THEIR CLASSICAL ORDER OR
BY ITS LEXICOGRAPHIC ORDER , BY RESHAPING THE SPATIAL DIMENSIONS OF THE FORMER INTO A SINGLE DIMENSION .

Classic representation Lexicographic order representation

Dimensions k-th (i, j)-th (i, j, k)-th Dimensions k-th i-th (i, k)-th
Variable Symbol Symbol
(Row × col. × band) band pixel element (Pixel × band) band pixel element
[x] [x] [x]
Reference U [x] Ni × Nj × Nk U::k uij: uijk X Ni Nj × Nk x:k xi: xik
[p] [p] [p]
HRI U [p] Ni × Nj × Np U::k uij: uijk P Ni Nj × Nk p:k pi: pik
Ni Nj [m] [m] [m] Ni Nj
LRI U [m] ρ
× ρ
× Nk U::k uijk uijk M ρ2
× Nk m:k mi: mik
[m] [m] [m]
Upscaled LRI U [m] Ni × Nj × Nk U::k uij: uijk M Ni Nj × Nk m m m
e e e e f e :k e i: e ik
Acquisition Y Ni × Nj × 1 - yij yij y Ni Nj × 1 - yi yi
[b
x] [b
x] [b
x]
Estimated product U [b
x]
Ni × Nj × Nk U::k uij: uijk X
b Ni Nj × Nk x
b:k x
bi: x
bik
[b] [b] [b]
Blurring kernels U [b] Nb × Nb × Nk U::k uij: uijk B Nb2 × Nk b:k bi: bik
[p] [p] [p]
HRI mask - - - - - H[p] Ni Nj × Np h:k hi: hik
[m] [m] [m]
LRI mask - - - - - H[m] Ni Nj × Nk h:k hi: hik

[m]
This is a widespread choice in the literature, as the sen- • Sum over channels y = A+ (X⊡ ): where the final
sors perform an integration of the incoming radiance that observation is obtained by summing along the spectral
is modulated by the spectral response of the filters [1]. dimension, returning:
• the spatial degradation operation M f = matr(U [m] e
)= Nk Nk
[m] [b] X X
Ad (X) is a convolution by a set of filters U ∈ y= x⊡
:k = x:k ⊙ h:k . (5)
RNb ×Nb ×Nk . Specifically: k=1 k=1

[m] [x] [b]

In some more advanced acquisition devices, some optical
U::k = U::k ∗ U::k , ∀k ∈ [1, ... , Nk ] , (3)
e
elements allow to shift the captured light rays of a given
channel over the FPA. To model such effect, we introduce
where ∗ denotes a spatial convolution operator. Once an additional shifting operator A[m] → in between the previ-
again, this is a widely employed model in the literature, ously presented ones, so that the full operation of mosaicing
derived from the assumption that the blurring effect of the becomes:
sensors can be described by a linear translation-invariant   
[m] [m]
operator, and hence uniquely defined by its modulation y = A[m]
c (X) = A + A
[m]
→ A ⊡ (X) . (6)
transfer function (MTF) [34]. The generic element sijk of the shifted image S in the
Finally, a decimation of Mf by a scale factor ρ (i.e., taking operation matr(S) = A[m]
→ (X) is given by:
every ρ samples both in the vertical and horizontal direction) [x]
sijk = ur(i,j,k) , (7)
produces the actual LRI M.
where r(i, j, k) : N3 → N3 is a one-to-one vector function
which defines the transformation from a source to a target
B. Mosaicing position.
For example, in the case of the single dispersion
In the classic formulation [13] shown in Fig. 1b, the CASSI [15], the focal plane associated to each channel can be
CFA/MSFA-based mosaicing is modeled as an element-wise rigidly translated in the horizontal direction through a diffrac-
multiplication of the input by a mask H ∈ RNi Nj ×Nk . Among tion prism. In the problem defined by the original authors,
those, binary masks define a special case of H whose elements this is a shift by one pixel between adjacent channels, which
can only be either zeros or ones; in other terms, each pixel we reformulated within the MRCA framework by defining a
of the mask identifies a specific channel, which is transferred shifted input S ∈ RNi ×(Nj +Nk −1)×Nk such that:
on the focal plane. This allows them to be represented by a
color-coded matrix, such as those shown in Fig. 3. r(i, j, k) = (i, j + k − 1, k) . (8)
[m]
In this work, we model the acquisition y = Ac (X) due to
the mosaicing as the cascade of the following two operations: C. Multiresolution coded acquisition (MRCA)
Masking X⊡ =
[m] The main target of the proposed MRCA is to allow for
• A⊡ (X):
where an element-wise mul-
tiplication (denoted by ⊙) is applied independently on multiresolution images to be stored over the same focal plane.
each band, yielding: The proposed model y = A(X) for the acquisition system is
shown in Fig. 2, and is expressed as:
 
x⊡
:k = x:k ⊙ h:k , ∀k ∈ [1, ... , Nk ] . (4) [p] [p] [m] [m]
A(X) = A+ Ab (Ac (Ad (X))) , Ac (Ad (X)) . (9)

If the mask is binary, the variable x⊡ The operator A is given by the following cascade of
:k is commonly known
as sparse channel in the demosaicing literature [35], operations:
since the element-wise multiplication sets most of its • A multiresolution sensing operator Ad , composed by
[m] [p]
values to zero. the operators Ad from eq. (3) and Ad from eq. (2) of
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 5

Image formation Image reconstruction

[m]
[m]
Ad : Spa. degr. H[m] Ac : LRI mosaicing
LRI
[m]
B Y
Y B
B Y
Y B
B A+
Shifting A[m] +
R
R G
G R
R G
G

Decimation →
Y
Y B
B Y
Y B
B

R
R G
G R
R G
G LRI sparse
M channels y[m]

Ideal Estimated
product ∗ × product
[m]
A⊡ : LRI masking Acquisition
Joint
f
M
demosaicing
A+ +
and fusion
HRI (JoDeFu)
[p] y
A⊡ : HRI masking
X b
X
× + ∗ ×

∗ = Spatial
convolution
P H[p] y[p]
P
P P
P P
P P
P P
P P
P P
P P
P Elementwise
{w:k }k=1,...,Nk P
P P
P P
P P
P × =
P
P P
P P
P P
P P
P P
P P
P P
P product
P
P P
P P
P P
P

P
P P
P P
P P
P P
P P
P P
P P
P

[p]
Ad : Spe. degr. B[p] Sum over
+ =
P
P P
P P
P P
P

P P P P P P P P

channels
P P P P P P P P

[p]
P
P P
P P
P P
P
Ac : HRI mosaicing
Ad : Multires.
sensing Ab : Blur Ac : Mosaicing + =
Elementwise
sum
A: Multiresolution coded acquisition (MRCA)

Fig. 2: Representative scheme of the image formation model for the proposed MRCA framework. The multiresolution
configuration is represented by a PAN and by a color coded 4-band MS; the white spots in the masks denote zero pixels
[p]
in all bands. The HRI shifting A[p]
→ and summing A+ operators are not depicted in the figure.

TABLE II
Section III-A, which generate the LRI and HRI branch, D ESCRIPTION OF CLASSICAL IMAGE FORMATION METHODS UNDER THE
respectively; PROPOSED MRCA FRAMEWORK .
[m]
• A mosaicing operator Ac : where the operator Ac of Multires. HRI mosaicing LRI mosaicing
[p] [p] [p] [m] [m]
eq.(6) is applied on the LRI branch and Ac , identical to Image formation method [p]
Ad
[m]
Ad A⊡ A→
[p]
A+ A⊡
[m]
A→ A+ A+
[m] MRCA ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Ac except for acting over Np channels instead of Nk , Multiresolution sensing ✓ ✓ ✗ ✗ ✗ ✓1 ✗ ✗ ✗
and applied to the HRI branch. CFA mosaicing ✓2 ✗ ✗ ✗ ✓ ✓ ✗ ✓ ✓
CASSI acquisition [15] ✓2 ✗ ✗ ✗ ✓ ✓ ✓3 ✓ ✓
• A blur operator Ab , to adjust its scale ratio of the HRI Lu et al. [38] ✓ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗
with respect to the reference and modeled as a spatial Takeyama and Ono [29] ✓ ✓ ✗ ✗ ✗ ✓4 ✗
convolution.
1 where H[m] is a binary mask, with zeros at the interleaved positions of the LRI;
2 with wkl = 0 , ∀ k ∈ [1, ... , Np ] , l ∈ [1, ... , Nk ] to suppress HRI samples;
• A sum operator A+ (y[m] , y[p] ), which sums the LRI 3 with the condition of eq. (8);
4 with any kind of linear compression is allowed for the LRI.
mosaic y[m] and HRI mosaic y[m] over the same focal In this table, ✗ marks linear operators of the MRCA being substituted by an identity.
plane:
 
[m] [p]
A+ y , y = y[m] + y[p] . (10) is also capable to model some more advanced sensing mech-
anism inspired by the CASSI, such as the one with colored
The reader may have noticed that the upscaled LRI M f is coded apertures proposed in [37]. For such case, as the spectral
not decimated in the MRCA pipeline; this step is unneces- bands are combined according to the spectral response of each
sary in our framework as the mask H[m] can automatically pixel of the coded aperture, the coefficients of the mask are
suppress the pixel values that would be eliminated during the non binary. By inverting the shearing and mosaicing operators,
decimation process. the design can also be extended to the colored CASSI [16].
The MRCA can model (assuming that no blur is introduced
by Ab ): D. Properties of the direct model operators
• multiresolution sensing, if both the operators Ac and A+ We investigate here two key properties of the operator A,
are substituted with an identity, as eq. (9) reduces to the adjoint operator and the operator norm, that are necessary
eq. (1); to define the generic image reconstruction algorithm to be
[m]
• mosaicing, if both operators in Ad is an identity and y[p] presented in Section IV. Specifically:
is an all-zero matrix, for which eq. (9) reduces instead to ∗
• the adjoint operator A is the one that verifies the
eq. (6). condition:
Therefore, the MRCA framework is a general formation
⟨A(X), y⟩ = ⟨X, A∗ (y)⟩ , (11)
method for acquisition systems that involves multiresolution
sensing and mosaicing. A more detailed description of the for all X ∈ RNi Nj ×Nk and y ∈ RNi Nj . Here, ⟨·, ·⟩
operators for such special cases, which also includes the on the left and right side of the equation are the scalar
frameworks proposed by Li et al. [36] and by Takeyama and products of the spaces of y and X, respectively. If A
Ono [29], is given in TABLE II. Additionally, the framework defines a matrix multiplication applied over vectorized
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 6

inputs, A∗ is equivalent of applying the Hermitian of the • Shifting: As shifting is assumed to be a one-to-one oper-
same matrix; ation, the intensity values of each sample are unchanged,
• the operator norm ∥ A ∥op is defined [39] as the smallest hence the operator norm is unitary;
scalar γ such that: • Element-wise product by H: As pixels are scaled by
Ni Nj ×Nk the intensity value of H, the operator norm is equal to
∥ A(X)∥2 ≤ γ∥X∥F , ∀X ∈ R . (12)
the largest value of H (i.e., it is equal to one if H is a
If A defines a matrix multiplication over vectorized non-degenerate binary mask).
inputs, its operator norm is the largest singular value of
the matrix itself [39]. IV. P ROPOSED IMAGE RECONSTRUCTION ALGORITHM
For composed operators, their properties can be dynamically A. Problem statement
obtained by combining the elementary building blocks associ-
This section presents the proposed algorithm to produce an
ated to the simple operations defined in the previous sections, b of the target image. Given the observation y of
estimation X
in order to produce the image formation model under test. b as close as possible to
the optical device, we aim to make X
Specifically, the adjoint of a composed operation A(X) =
the ideal (and typically unknown) reference X which generates
Ac (Ad (X)) is equal to applying the individual adjoint opera-
it. We operate under the assumption that the observations are
tors in reverse order:
affected by noise modeled as an additive independent and
∗ ∗ ∗
A (y) = Ad (Ac (y)) . (13) identically distributed (i.i.d.) Gaussian distribution with zero
∗
The only requirement to evaluate A then simplifies to deriving mean. While this hypothesis is common for previous Bayesian
the adjoint operator of each elementary component of the formulations of similar problems in the literature [12], [41],
MRCA separately. These components can be split into the its validity is currently a point of contention in the scientific
following categories: community, but is at least reasonable under sufficient high
[m] illumination [42].
• Spatial convolution (Operators Ad , and Ab ): the ad-
The proposed JoDeFu reconstruction algorithm is based on
joint of a convolution by a given kernel is a correlation
the following Bayesian formulation of the inverse problem:
by the same kernel;
b = arg min 1 ∥A(X) − y∥2 + f (X) ,
[m] [p]
• Sum over channels (Operators A+ , A+ , and A+ ): X (16)
X 2 2
the adjoint is equivalent to replicating a monochromatic
image across all bands; where the first term of the right side is the maximum a
[m] [p]
• Shifting (Operators A→ , and A→ ): the adjoint of shift- posteriori estimation, also known as data fidelity term, and
ing a sample to a new position is a shift back to its f (X) : RNi Nj ×Nk → R+ is a regularization function, which
original one. we can customize according to our prior knowledge on the re-
[p] [p] [m]
• Element-wise product (Operators Ad , A⊡ , and A⊡ ): sult to reconstruct and is used to counteract the ill-conditioned
the adjoint operator is itself, as this operation is self- nature of the formulation [43]. We focus our attention on the
adjoint. expression of the data fidelity term in Section IV-B, on the
We follow a similar approach for the operator norm. Specif- regularizer in Section IV-C, and on the algorithm for solving
ically, we apply the Cauchy inequality to a composed operator this problem in Section IV-D.
A(x) = Ac (Ad (X)) to identify an upper limit for ∥ A ∥op :
∥ A ∥op ≤ ∥ Ac ∥op ∥ Ad ∥op . (14) B. Data fidelity term
The JoDeFu algorithm can be applied to the observation
This inequality can be substituted with a strict equality as of any device that can be described within the MRCA image
the reconstruction algorithms we employ only require upper formation model. That is, other than the general MRCA itself,
bounds for the operator norm. Nonetheless, we can once it may also approach the problem of sharpening (if A models
again separate the problem into evaluating the operator norms a multiresolution sensing), that of demosaicing (if A models
individually: a mosaicing), and so on. While we specialize here on the
• Spatial convolution of X by B: for each band, the
MRCA framework, the proposed algorithm admits a solution
convolution by the k-th band b:k can be rewritten as a for any image formation model, as long as we can define the
multiplication by a circulant matrix. Its singular values properties of the operator A as a combination of those defined
are hence defined as the sum of the coefficients b:k in Section III-D.
weighted by the complex roots of unity [40]. A conser- Special care has to be taken in the sharpening scenario (and
[m]
vative estimate for this operator norm (e.g., for Ad ) is in general, for every setup in which A+ is an identity operator).
then given by: As the observation is made of two separate acquisitions, the
v
u N2 reconstruction problem is equivalent to:
uX b

= max t
[m]
Ad b2ik ; (15)
op k∈[1,... ,Nk ]
i=1
b = arg min 1 A[p] (X) − P
X d
•
√ over Nk channels: the operator norm upper bound
Sum X 2 F
is Nk , as a result of the triangular inequality applied 1 [m] f
+ Ad (X) − M + f (X) , (17)
over every pixel; 2 F
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 7

which is very similar to the Bayesian formulation of the Nm = 2, the gradients are taken along the horizontal and
sharpening problem proposed in [12], but assumes the same vertical spatial dimensions, and the elements wijkm of W
weight for the data fidelity term associated to the LRI and to (assuming that the elements out of range in U [x] are zero) are
the HRI. defined as follows:
Previous works [44] have shown that the algorithms that (
[x] [x]
uijk − ui−1,j,k for m = 1
were proposed to solve image reconstruction problems based wijkm = [x] [x] . (19)
on multiresolution acquisition (e.g., pansharpening), are not uijk − ui,j−1,k for m = 2
well suited for the case of reconstruction of missing acquisi- Similar four way tensors can be also defined for alterna-
tions (e.g., inpainting), which is also the case of the demo- tive TV-based operators, such as the upwind total variation
saicing as binary masks technically perform a subsampling. (UTV) [48], and the Shannon total variation (STV) [49].
This motivates the need of an ad-hoc algorithm where these To define the metric function g(·), we took inspiration from
problems can be solved jointly. the framework of the collaborative total variation (CTV) [50],
To deal with such scenarios, the proposed blur filter Ab [51], where g(·) is defined as a set of norms applied sequen-
is used to adjust the results even when the HRI is at the tially over different dimensions.
same scale of the reference. Its inclusion allows to recast our The most relevant that are also employed in this work, are
demosaicing problem from pure inpainting to an hybrid of defined below:
inpainting/magnification. The formulation implies that some • g(W) = ∥W∥p1 p2 p3 : this stands for the ℓp1 , ℓp2 , and ℓp3 -
information from the suppressed pixels is contained in adjacent norm applied, in this order, respectively to dimension of
pixels, but this comes at the cost of reducing the spatial the gradient, that of the channels, and that of the pixels.
resolution of the final product. Among those, the most widespread are the ∥W∥221 ,
which is used in the vector total variation [52] and whose
C. Regularization mathematical expression is:
v
We want to setup here a proximal algorithm. This class Nj u Nk Nm
Ni X
X uX X
of algorithms works in very general conditions, allowing for ∥W∥ = t w2 (20)
221 ijkm
some nonsmooth real-valued constraint on the cost function, i=1 j=1 k=1 m=1
and is relatively fast with respect to other alternatives [45]. To
this end, we propose a regularization function in the form: and the ∥W∥111 norm, known as the least absolute
shrinkage and selection operator (LASSO) [53] and em-
f (X) = λ g(L(X)) , (18) ployed for the classic inversion of the CASSI acquisi-
tions [15].
where we denote:
• g(W) = ∥W∥Sp ℓq : this stands for the Shatten p-
• a linear operator L(·) : R i j
N N ×Nk
→ El , which de- norm firstly applied on both the gradient and the bands’
scribes X within a sparsity-inducing transformed normed dimensions, and then the ℓq -norm applied over the pixels;
space El ; particularly good performances can be obtained with
• a metric function g(·) : El → R , for which it is
+
∥W∥S1 ℓ1 , defined as:
possible to define a proximal operator proxγg (·) scaled v
by a generic positive scalar γ; Ni XNj u Nr
X uX
• a regularization parameter λ ∈ R , used to weight
+ ∥W∥S1 ℓ1 = t 2 (W
ξm ij:: ) , (21)
the contribute of the regularization with respect to the i=1 j=1 m=1
data term in the cost function; we sometimes refer to this where ξm (Wij:: ) is the m-th singular value and Nr is
term in its normalized form λ = λ/ρy , where ρy is the the total amount of singular values of Wij:: .
dynamic range of the observation y (e.g., ρy = 255 for
8 bits images).
D. Implementation details
For the linear transformation W = L(X), the CASSI
authors proposed to use a symlet-8 discrete wavelet transform The proposed JoDeFu image reconstruction framework can
and a discrete cosine transform (DCT) transform in the spatial be summarized as follows:
and spectral domain, respectively [15]. In this work, we b = arg min 1 ∥ A(X) − y∥2 + λ g(L(X)) ,
X (22)
2
propose instead an approach based on the total variation (TV), X 2

a regularizing transformation that acts as a discrete repre- which is the minimization of a cost function composed by
sentation the Rudin-Osher-Fatemi model [46]. In its modern a differentiable data fidelity term and a regularization term
interpretation, the TV is often seen as a sparsity-inducing whose metric function g(·) is a lower semi-continuous convex
operator working in the domain of image gradients. This function. As long as the adjoint operators for A and L are
favors piecewise constant images with sparse edges, which known, and it is possible to define a proximal operator for g(·),
are typically a better representation of natural images [44]. a variety of algorithms are available that iteratively converge
Following the generic mathematical description of the third to the desired solution X. b Those are known as proximal
author [47], the TV-based expression of W is a 4-way tensor algorithms in the literature [45]. Among those, the Chambolle-
W ∈ RNi ×Nj ×Nk ×Nm , whose forth dimension is made up of Pock solver [54] is the most widespread, but we prefer
the gradients of U [x] . For the classic TV in particular, where here to employ instead the Loris-Verhoeven algorithm [55],
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 8

which simplifies the choice of the convergence parameters (as Algorithm 1: JoDeFu algorithm, based on the Loris-
reported in [56]). Verhoeven solver [55] with over-relaxation [56].
The full procedure, described by the Algorithm 1, requires Result: Estimated product X b
the definition of: Input:
∗
• the adjoint operator A and the operator norm ∥ A ∥op of • Acquisition: y ∈ R i j (with the LRI samples
N N
A, which were described in Section III-D; histogram matched to the HRI)
∗
• the adjoint operator L and the operator norm ∥ L ∥op of • Direct model operator A(·), according to eq. (9), with
L: for orthogonal operators (such as the DCT and some A∗ and ∥ A ∥op defined in Section III-D;
wavelets), their operator norm is unitary. For the classic • Domain transformation operator L(·), e.g. a TV
TV the generic vijk element of V ∈ RNi ×Nj ×Nk , of according to eq. (19), an UTV, or a STV, with L∗ and
matr(V) = L∗ (W) is: ∥ L ∥op defined in Section IV-D;
• Proximal operator proxλg ⋆ (·), e.g. from eq. (24);
vijk = (wi,j,k,1 − wi−1,j,k,1 ) + (wi,j,k,1 − wi,j−1,k,2 ) , −3
• Regularization parameter: λ (default: 10 ρy , where ρy
(23)
where we assume once again that the elements out of is the dynamic range of y);
• Over-relaxation parameter: ρo (default: 1.9);
range
√ of W are null. Its operator norm is ∥ L ∥op = [max]
• Maximum number of iterations: q (default: 250);
8 [47]. For the other TV-like operators, we redirect
the reader towards the related articles [48], [49]. Initialization:
2
• the scaled proximal operator proxλg⋆ (·) of the Fenchel • First convergence parameter: τ = 0.99/∥ A ∥op
2
conjugate of g(·): a summarizing table of its expression • Second convergence parameter: σ = 1/(τ ∥ L ∥op )
(0) ∗
is provided in [51]. We just remind here that, for the • Primal variable: X = A (y)

∥W∥221 norm, this is equal to: (0)
• Dual variable: W = L X(0)
W • Iteration: q = 0
proxλg⋆ (y) = s ! , (24) while q < q [max] do

Pk N
N Pm

max 1
λ W::km , 1 V(q) = A∗ A X(q) − y  
X(q+ 2 ) = X(q) − τ V(q) + L∗ W (q)
k=1 m=1 1

  
where max(X, 1) is an operator substituting with 1 all
W (q+ 2 ) = proxλg⋆ W (q) + σ L X(q+ 2 )
1 1

elements of X that are superior to 1, and the fraction   

X(q+1) = X(q) − ρo τ V(q) + L∗ W (q+ 2 )
1
stands for an element-wise division broadcasted over 4
 
dimensions. 1
W (q+1) = W (q) + ρo W (q+ 2 ) − W (q)
The JoDeFu framework allows for multiple ways to con-
q ←q+1
struct a custom cost function, which include the choice for
the regularization parameter λ, for the metric function g(·), b = X(q[max] )
return X
and the linear transformation operator L. We propose here two
possible solutions, whose specifics are shown in table III: the
TABLE III
JoDeFu v1 provides reasonable performances while keeping S UGGESTED SETUPS FOR MRCA IMAGE RECONSTRUCTION ALGORITHMS .
the computation time relatively short, while the JoDeFu v2 JoDeFu setups
variant produces optimal performances if there is no constraint g(W) L(·) Ab (·)
on time. JoDeFu v1 ∥W∥221 [50] TV [46] Identity
JoDeFu v2 ∥W∥S2 ℓ1 [50] UTV [48] or STV [49] ρb = 1–1.5 px
Since the proposed algorithm is iterative in nature, the Cascaded classic algorithms setups
PAN interpolation Demosaicing Sharpening
computational burden is surely higher than the classic one-shot Classic v1 TPS-RBF [58] ARI [59] MTF-GLP-HPM [60]
algorithms, which may impact the scalability of the proposed Classic v2 TPS-RBF [58] ID [35] GSA [61]
algorithm. In terms of computational complexity, the linear op- The first half refers to the JoDeFu algorithm, while the rows labeled with classic v1 and
v2 refer to the alternative algorithms described in Section V-C.
erators A and L used in the proposed algorithm scale linearly
with the number of samples Ns = Ni Nj Nk , being composed
of convolutions with a spatial kernel or of weighted element-
acquisition under test, we compare the reconstructed
wise products. The proximal operator is the main bottleneck
products obtained both with a representative state-of-the-
for the v2 version of the algorithm, since the computational
art classic reconstruction algorithm and with the proposed
complexity of eq. (21) is O(Nk Nm min(Nm , Nk )) for each
JoDeFu algorithm. This experiment aims to show the
pixel, as it involves a singular value decomposition [57], while
effectiveness of the MRCA in modeling a wide variety
the v1 is limited to a O(Nk Nm ) complexity from eq. (20).
of capturing techniques, by verifying that the quality of
the reconstructed image meets the standards set up by the
V. E XPERIMENTS
previous literature when this model is employed within
In this section, the proposed image formation/reconstruction the proposed reconstruction algorithm;
framework is tested under different viewpoints: • in Section V-C, the image formation is fixed to the
• In Section V-B, we analyze the acquisition obtained proposed MRCA and we inspect a variety of solutions for
with a variety of image formation methods. For each the image reconstruction, by comparing the results when
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 9

TABLE IV
P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P
C HARACTERISTICS OF THE GT OF THE DATASETS EMPLOYED IN THE TESTS
P
P G
G P
P B
B P
P G
G P
P B
B P
P Y
Y P
P B
B P
P Y
Y P
P B
B P
P Y
Y P
P D
D P
P O
O P
P B
B
OF S ECTION V.
P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P

Label Scene Sensor GSD Sizes [px] P

P R
R P
P G
G P
P R
R P
P G
G P
P R
R P
P G
G P
P R
R P
P G
G P
P SS P
P G
G P
P R
R P
P C
C

P P P P P P P P P P P P P P P P P
P P
P P
P P
P P
P P
P P
P P
P
Beijing Bird’s nest, China WV2 1.6 m 512 × 512 P P P P P P P P P P P P P P P P

P
P G
G P
P B
B P
P G
G P
P B
B P
P Y
Y P
P B
B P
P Y
Y P
P B
B P
P O
O P
P B
B P
P Y
Y P
P D
D
Janeiro Bay area, Brazil WV3 1.2 m 512 × 512
P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P P
P

Washington Capitol building, U.S. WV3 1.6 m 512 × 512 P

P R
R P
P G
G P
P R
R P
P G
G P
P R
R P
P G
G P
P R
R P
P G
G P
P R
R P
P C
C P
P SS P
P G
G

Stockholm Central city, Sweden WV3 1.2 m 512 × 512

(a) Bayer (3 (b) Periodic (4 (c) Uniform Binary
Fields Mineral mountain, China IKONOS 3.2 m 256 × 256 Bands) [14] Bands) [66] Tree (8 bands) [66]
The ground sample distance (GSD) refers to the spatial resolution of the GT.
Fig. 3: Masks used for the experiments of Section V-C. The
masks are obtained by mosaicing a set of HRI sensors (in gray)
a different amount of LRI channels are encoded in the to classical literature designs for LRI masks, color coded with
acquisition. This experiments aims to show the capability their characteristic channel. The dashed outline denotes the
of the JoDeFu in recovering the relevant information periodicity of each mask.
embedded in the compressed acquisition under different
conditions.
• in Section V-D, we analyze the effect of the parameters of spectral degradation of the GT, in order to follow the Wald’s
associated to the JoDeFu algorithm. The experiment aims protocol for reduced resolution validation [63].
[p]
to test the robustness of the proposed reconstruction algo- In the spectral degradation model Ad , the weighting co-
rithm with respect to deviations from an ideal parametric efficients are always set as equal to w1,k = 1/Nk , for all
setup. k ∈ [1, ... , Nk ], so that the HRI is modeled as the average
To introduce the experiments, a description of the employed of the channels of the GT. In the spatial degradation model
[p]
datasets, of the experimental setup, and of the validation Ac , the blurring kernels B are Gaussian functions whose
method is given in Section V-A. The experiments provided cutoff frequency matches the one of the MTF of the sensors.
in this section are fully reproducible with the MATLAB The Ab (·) is implemented by a infinite impulse response
implementation of the algorithm1 and additional results are (IIR) Butterworth filter whose bilateral cutoff frequency is
available in the supplementary materials. 1/ρb , with ρb denoting the diameter of the blurring filter and
expressed in pixels (px). All JoDeFu algorithms are run for
250 iterations. In the JoDeFu preprocessing stage, the overall
A. Experimental setup
mean and standard deviation of the LRI samples are equalized
Our validation setup consists of four steps: to that of HRI samples.
• Reference choice: where we select a high resolution For the comparison step, we employ as quality indices the
image as reference X ∈ RNi Nj ×Nk , that is referred as peak signal to noise ratio (PSNR), the spectral angle mapper
ground truth (GT); (SAM) [64], and the structural similarity (SSIM) [65], which
• Simulation: where the acquisition y = A(X) is evalu- is given as average over all bands.
ated from the direct model A under test (e.g., with the
architecture in Fig. 2 for the MRCA); B. Image formation
• Testing: where the reconstruction algorithm under test
is applied to the observation, in order to achieve an We consider here a selection of image formation methods
estimation Xb of X; and compare the obtained quality of the estimated product,
• Comparison: where the estimated product X b and the produced both with a carefully selected classic image recon-
reference X are compared by evaluating a series of struction algorithm and with the JoDeFu. For completeness,
quality indices. this approach is also applied to an observation obtained with
the complete declination of the MRCA method, using the mask
Each reference dataset is composed of a HRI/LRI image
of Fig. 3b.
bundle acquired almost simultaneously, originally featuring
Since formation methods can be interpreted as compressed
a scale ratio of 1 : 4, although the tests are performed at
acquisitions, the final observation generally contains less
reduced resolution with a scale ratio of ρ = 2. The HRI is
samples than the datacube to reconstruct. Therefore, their
monochromatic (i.e. a PAN) and the LRI has up to 8 channels.
compression ratio ρc can be defined as the ratio between the
The bundles were acquired by the IKONOS, WorldView-2
amount of samples of the observation y and the reconstructed
(WV2) and WorldView-3 (WV3) satellites, and are available b For example, if the dynamic range of all samples
product X.
for download on the MAXAR Technologies website [62].
is the same, the compression ratio of the full MRCA model
Their characteristics are shown in TABLE IV. Additional
is equal to:
experiments over different datasets are available in the sup- Ni Nj 1
plementary materials. ρc = = , (25)
Ni Nj Nk Nk
For the simulation step, when applicable, the HRI is given
as a spatial degradation of a HRI at higher resolution, instead as the multiresolution setup compresses the acquisition over
a single FPA matching the spatial dimensions of the recon-
1 [Online]. Available: https://github.com/danaroth83/jodefu structed product.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 10

TABLE V
R ESULTS OF CLASSIC AND PROPOSED RECONSTRUCTION ALGORITHMS FOR DIFFERENT METHODS OF IMAGE FORMATION WITH THE 4- BANDS
“WASHINGTON ” AND “F IELDS ” DATASET.

Washington Fields
Image formation ρc Image reconstruction
λ SSIM PSNR SAM λ SSIM PSNR SAM
Reference (GT) 1.000 - - 1 ∞ 0 - 1 ∞ 0
Pansharpening [67] - 0.9772 31.14 3.751 - 0.9724 41.42 2.483
Multiresolution sensing 0.500
JoDeFu v1 1 × 10−3 0.9868 33.22 2.797 2 × 10−4 0.9744 42.10 2.313
Demosaicing [68] - 0.9613 30.38 4.361 - 0.9513 38.92 3.465
Mosaicing 0.250
JoDeFu v1 2 × 10−3 0.9312 28.36 4.371 8 × 10−4 0.9450 38.54 3.276
CASSI reconstruction [15] 3× 10−3 0.7645 24.48 9.750 2 × 10−3 0.8717 33.75 7.795
CASSI acquisition [15] 0.251
JoDeFu v1 2 × 10−3 0.8600 26.89 5.954 1 × 10−3 0.9147 36.66 4.569
Classic v2 (TABLE III) - 0.9156 28.13 6.246 - 0.9517 39.54 3.503
MRCA 0.250 JoDeFu v1 1× 10−3 0.9446 29.34 4.553 1× 10−3 0.9442 38.09 3.652
JoDeFu v2 with UTV (ρb = 1.4) 1 × 10−3 0.9560 29.79 3.841 4 × 10−4 0.9541 39.19 3.035
Best results for each image formation method are shown in bold.

(a) Reference (GT) (b) Multires. sens. - Class. [60] (c) Mosaicing - Classic [35] (d) CASSI - Classic [15]

(e) MRCA - JoDeFu v1 (f) Multires. sens. - JoDeFu v1 (g) Mosaicing - JoDeFu v1 (h) CASSI - JoDeFu v1

Fig. 4: Visual comparison for different image formation methods, comparing the reconstructed product obtained with the
proposed JoDeFu algorithm and with classic solutions available in the literature. The images show a 256 × 256 cropped area
of the 4-band “Washington” dataset. Zoomed areas are provided for a detailed comparison.

We investigate the follow image formation methods: construction [15].

• Multiresolution sensing: with the same scale ratio of The validation procedure is applied to the “Washington” and
the MRCA, this is compared with the MTF-matched “Fields” datasets and compared to the baseline JoDeFu, with
generalized Laplacian pyramid with high pass modula- the results shown in TABLE V and a visual comparisons in
tion injection (MTF-GLP-HPM) algorithm [67], the best Fig. 4.
performing one for this dataset among the classic ones The JoDeFu algorithm achieves results which are at least
that were tested; comparable to the state-of-the-art. The benefit of the TV-
• Mosaicing: applying a 4-band mask with period 2 × 2, style regularizer is immediately evident for the CASSI re-
which also reaches the same compression ratio of the construction (Fig. 4h). For the pansharpening problem, the
MRCA, which is compared with the intensity difference final product obtained with the JoDeFu (Fig. 4f) shows subtler
(ID) demosaicing algorithm; improvements, as the final product achieves slightly more
• CASSI: using the model proposed by the authors em- accurate color quality. For the demosaicing problem, however,
ploying one acquisition in their single dispersion variant there are still margins of improvement (Fig. 4g). The proposed
and comparing with the proposed algorithm for the re- algorithm is not fully capable of eliminating some texture
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11

TABLE VI
R ESULTS OF MRCA IMAGE RECONSTRUCTION WITH DIFFERENT AMOUNT OF MOSAICED BANDS FOR THE “JANEIRO ” AND “S TOCKHOLM ” DATASET.

3 bands (RGB) 4 bands (RGB + NIR) 8 bands (All VIS/NIR)

SSIM PSNR SAM SSIM PSNR SAM SSIM PSNR SAM
Reference (GT) 1 ∞ 0 1 ∞ 0 1 ∞ 0
Classic v1 0.9634 32.30 2.502 - - - - - -
Classic v2 0.8680 28.77 5.253 0.9049 28.11 6.463 0.8917 28.50 9.166
Janeiro

JoDeFu v1 (λ = 2 × 10−3 ) 0.8803 28.78 4.988 0.9159 28.38 5.218 0.9080 29.08 7.220
JoDeFu v2 with UTV 0.9050 29.57 3.901 0.9264 28.58 4.891 0.9247 29.63 6.693
JoDeFu v2 options λ = 2 × 10−3 , ρb = 1 λ = 1 × 10−3 , ρb = 1.4 λ = 1 × 10−3 , ρb = 1.4
Classic v1 0.9765 32.33 2.552 - - - - - -
Stockholm

Classic v2 0.9064 29.41 4.530 0.9240 28.77 7.512 0.8838 27.32 11.08
JoDeFu v1 (λ = 2 × 10−3 ) 0.9050 29.05 4.196 0.9249 28.54 6.075 0.8905 27.47 8.749
JoDeFu v2 with UTV 0.9241 29.75 3.255 0.9414 29.30 5.536 0.9105 28.17 8.181
JoDeFu v2 options λ=2× 10−3 , ρb = 1.4 λ=2× 10−3 , ρb = 1.4 λ=1× 10−3 , ρb = 1.4
The image construction is obtained with the MRCA model using the masks shown in Fig. 3. Best results are marked in bold fonts.

(a) Reference (GT) (b) 3 bands - JoDeFu v2 (c) 4 bands - JoDeFu v2 (d) 8 bands - JoDeFu v2

(e) 3 bands - Classic v1 (f) 3 bands - Classic v2 (g) 4 bands - Classic v2 (h) 8 bands - Classic v2

Fig. 5: Visual comparison of different MRCA image reconstruction algorithms, employing a different amount of embedded
bands from the “Janeiro” dataset. In all cases, we visualize the RGB bands of 256 × 256 px cropped area with zoomed details.

effects; this is a known weakness of TV-based regularizer, achieved with different methods, starting from an observa-
which are not well suited for the reconstruction of thread-like tion acquired with the full version of the proposed MRCA
structures [44]. In the supplementary materials, we also present architecture. Our main target is to show the robustness of the
a visual comparison for the “Fields” dataset, where the terrain proposed algorithm to recover the desired information when
has a smoother structure compared to the more piecewise ap- a sufficiently large amount of channels are embedded in the
pearence of the “Washington” one. The MRCA reconstruction observation.
results are also compared to the image quality obtained with
The analysis is carried out by simulating the observation
standard software compression algorithms aimed at matching
starting from the “Janeiro” and “Stockholm” dataset bundle.
the compression ratio achieved by the MRCA [69].
For the LRI, we select either 3, 4, or 8 channels, by either
only choosing the RGB in the first case, adding a near infrared
C. Image reconstruction response (NIR) in the second one, or selecting all the visible
In this section, we shift the focus on image reconstruction (VIS) and NIR channels in the last case. The MRCA model
methods, and compare the quality of the estimated products employs the periodic masks shown in Fig. 3.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

0.940 175.00 0.940 175.00 0.940 175.00 0.940 175.00

0.935 174.75 0.935 174.75 0.935 174.75 0.935 174.75

0.930 174.50 0.930 174.50 0.930 174.50 0.930 174.50

0.925 174.25 0.925 174.25 0.925 174.25 0.925 174.25

180 - SAM

180 - SAM

180 - SAM

180 - SAM
SSIM

SSIM

SSIM

SSIM
0.920 174.00 0.920 174.00 0.920 174.00 0.920 174.00

0.915 173.75 0.915 173.75 0.915 173.75 0.915 173.75

0.910 173.50 0.910 173.50 0.910 173.50 0.910 173.50

0.905 173.25 0.905 173.25 0.905 173.25 0.905 173.25
0.900 0.0 173.00 0.900 173.00 0.900 173.00 0.900 1.0
0.5 1.0 1.5 2.0 2.5 3.0 S 1 221 111 S1 1 11 211 STV3 TV STV2 UTV 1.2 1.4 1.6 1.8 2.0 173.00
Normalized regularization parameter (×10 3) Metric function norm g( ) Total variation operator ( ) b( ) blur diameter b

(a) Regularization parameter λ (b) Metric operator norm g(·) (c) Total variation L(·) (d) Blur diameter ρb of Ab (·)

Fig. 6: Quality indices results obtained with different setups of the JoDeFu image reconstruction algorithm applied to the
4-band “Beijing” dataset. For each figure, we vary the indicated parameter with respect to the baseline JoDeFu v1 setup. Better
reconstruction performances are associated to higher values of the “SSIM” (in blue) and of the “180◦ − SAM” (in orange).

Other than with JoDeFu, this specific problem can be also obtained with the mask of Fig. 3b. For this test, we firstly
approached with a custom image formation method, composed define as baseline the JoDeFu v1 with λ = 1 × 10−3 (whose
of the following three-step procedure: specifics are given in TABLE III) and evaluate its estimated
• HRI interpolation: recovering the sparse channel associ- product. Then, for each parameter under test, we estimate
[p] the reconstruction results obtained by solely varying a single
ated to the HRI (p⊡ :,1 = y⊙h:,1 ) and estimate the missing
HRI samples with a multivariate interpolation. parameter from the baseline setup. There is no guarantee on
• Demosaicing: obtaining the LRI mosaic by decimating the combined effect of varying multiple parameter, but we
the observation y. Furthermore, apply any classic demo- empirically experienced that optimizing each parameter sepa-
saicing algorithm to estimate all the channels of the LRI; rately still returns performances within a reasonable ballpark
• Sharpening: performing a fusion on the reconstructed of the overall best optimization. The tests are applied to the 4-
HRI and LRI from the previous two steps. band “Beijing” dataset, and a summary of the measured quality
In our tests, we employ a thin plate spline radial basis indices is given in Fig. 6, with an associated visual comparison
function (TPS-RBF) [58] for the HRI interpolation. For a in Fig. 7.
comprehensive comparison, we isolated two viable config- A more in-depth discussion for each of the parameters under
urations for the demosaicing and pansharpening algorithms, test is given in the following list:
whose specifics are given in TABLE III: • Regularization Parameter λ: as a rule of thumb, λ =
• The classic v1 setup is optimized for the RGB setup
10−3 is a good compromise in most scenarios and can
and employs the adaptative residual interpolation (ARI) be used as starting test to further refine the parameters
demosaicing method [59], which is only applicable to if higher quality is required. In Fig. 7i and 7l, some
Bayer masks. reconstructed products are shown for implausibly low and
• The classic v2 method can be applied to all cases,
high values of λ, respectively; some inbetween choices
and employs the Gram-Schmidt adaptive (GSA) fusion are also shown for the sake of completion. If λ is too
method [61], as it provides more robust results for larger low, we impose no structure of the final image, and most
amount of bands. texture effects from the mosaicing are not flattened. If λ
The analysis of TABLE VI and its associated visual com- is too high, the smoothing effect applies to relevant image
parison of Fig. 5 shows that the JoDeFu yields sharper results features;
than the classic v2, regardless of band setup. However, it does • Metric function norm g(·): the quantitative verification
not provide a consistent reconstruction of larger scale homo- shows that the ℓ221 norm is the best compromise between
geneous zones, such as the swimming pool in Fig. 5c. This is quality of the reconstructed product and computational
most likely due to the choice the regularization parameter λ, speed. Among the remaining choices, better performances
which is not properly set to control the smoothing of zones are only achieved with the S1 ℓ1 , due to the noise whiten-
with differents spatial scales, that would require to change the ing effect that this constraint imposes across different
value of λ locally. Some additional effects may come into bands. The visual analysis shows some spectral spot-
play which are due to the nature of the datasets employed in shaped spectral distortions in Fig. 7f;
this work, as remote sensing images are particularly sensitive • Linear operator L(·): for this configuration we tested
to aliasing effects [70]. Additionally, it is likely that a more the classic TV, the UTV [48] and the STV [49] with an
accurate choice of the encoder may yield to more accurate upscaling factor of 2 and 3, but no noticeable differences
reconstructions, as the regularity of the residual textures may were found;
be due to a sub-optimal choice of the mask coefficients. • Diameter ρb of the blurring operator Ab (·): in our tests,
the optimal value of the blur diameter was found to be
D. Setting the parameters in the range 1.3 − 1.5 px for a scale ratio ρ = 2. This
We test here various possible parameters for the optimiza- optimal value has to be chosen as a trade-off between a
tion of the JoDeFu algorithm applied to the MRCA acquisition more accurate recovery of the HRI samples and avoiding
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13

(a) Reference (GT) (b) g(W) = ∥W∥S1 ℓ1 (c) L(·): STV (upsc. ratio 2) (d) ρb = 1.4 px

(e) JoDeFu v1 (baseline) (f) g(W) = ∥W∥111 (g) L(·): UTV (h) ρb = 2.0 px

(i) λ = 1 × 10−4 (j) λ = 6 × 10−4 (k) λ = 7 × 10−3 (l) λ = 1 × 10−1

Fig. 7: Visual comparison of the effects of changing parameters in the baseline setup of the JoDeFu image reconstruction
applied to the MRCA acquisition of the 4-band “Beijing” dataset (256 × 256 px cropped area).

out-of-focus effects in the final product. product, and can be declined into two different variants with
respect of the requirements in term of computation time. The
VI. C ONCLUSION proposed method does not necessarily match the state-of-
In this paper, we proposed the MRCA, a novel multireso- the-art for every available image formation setup, but some
lution compressed acquisition system with enough flexibility alternatives were analyzed for particularly simple ones, which
to model both classic image formation methods, such as the employ a cascade of classic techniques.
CFA/MSFA-based mosaicing, the HRI/LRI image bundles, A possible extension of this work may involve supervised
and some nonconventional acquisition system, such as the learning [71], [72] to fine tune the parameters of the JoDeFu
CASSI, and various hybrid methods. The proposed design algorithm. Additionally, improved robustness to a higher num-
aims to intercept the future trends for unconventional optical ber of channels can be obtained by capturing samples directly
devices, and allows for an immediately available procedure to in a sparse domain, and the reconstructed products can achieve
model the physics of the image formation. higher quality with suitable mask designs based on compressed
We also proposed the JoDeFu, a Bayesian solver for image sensing, expanding the results of our previous works [69], [73].
reconstruction applicable to any available variant of MRCA-
based acquisitions, and specialized on data with a strong
R EFERENCES
LRI component. The proposed algorithm jointly addresses the
problem of image fusion and reconstruction of compressed [1] M. T. Eismann, Hyperspectral remote sensing, ser. Press Monographs.
data, exploits the CTV regularization to recover the desired Society of Photo-Optical Instrumentation Engineers, 2012.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14

[2] D. Manolakis, R. Lockwood, and T. Cooley, Hyperspectral imaging [27] H. Rueda-Chacon, F. Rojas, and H. Arguello, “Compressive spectral
remote sensing: physics, sensors, and algorithms. Cambridge University image fusion via a single aperture high throughput imaging system,”
Press, 2016. Scientific Reports, vol. 11, no. 1, May 2021.
[3] D. G. Stork, “Rewriting the rules of imaging design in the new era of [28] Y. Fu, Y. Zheng, H. Huang, I. Sato, and Y. Sato, “Hyperspectral image
electro-optics,” in Frontiers in Optics 2009/Laser Science XXV/Fall 2009 super-resolution with a mosaic RGB image,” IEEE Transactions on
OSA Optics & Photonics Technical Digest, Optical Society of America. Image Processing, vol. 27, no. 11, pp. 5539–5552, Nov. 2018.
OSA, 2009. [29] S. Takeyama and S. Ono, “Compressed hyperspectral pansharpening,”
[4] T. Mirani, D. Rajan, M. P. Christensen, S. C. Douglas, and S. L. in 2020 IEEE International Conference on Image Processing (ICIP).
Wood, “Computational imaging systems: joint design and end-to-end IEEE, Oct. 2020.
optimality,” Applied Optics, vol. 47, no. 10, pp. B86–B103, Mar. 2008. [30] Université Grenoble Alpes. (2022) AmicalSat project. [Online].
[5] O. Cossairt, M. Gupta, and S. K. Nayar, “When does computational Available: http://amicalsat.univ-grenoble-alpes.fr/
imaging improve performance?” IEEE Transactions on Image Process- [31] D. Picone, M. Dalla Mura, and L. Condat, “Pansharpening of images
ing, vol. 22, no. 2, pp. 447–458, Feb. 2013. acquired with color filter arrays,” in Unconventional Optical Imaging,
[6] K. Mitra, O. S. Cossairt, and A. Veeraraghavan, “A framework for C. Fournier, M. P. Georges, and G. Popescu, Eds., International Society
analysis of computational imaging systems: role of signal prior, sensor for Optics and Photonics. SPIE, May 2018.
noise and multiplexing,” IEEE Transactions on Pattern Analysis and [32] D. Picone, L. Condat, F. Cotte, and M. Dalla Mura, “Image fusion and
Machine Intelligence, vol. 36, no. 10, pp. 1909–1921, Oct. 2014. reconstruction of compressed data: A joint approach,” in 2018 25th IEEE
[7] D. J. Brady, Optical imaging and spectroscopy. Wiley & Sons, International Conference on Image Processing (ICIP). IEEE, Oct. 2018.
Incorporated, John, 2009. [33] J. Nakamura, Image sensors and signal processing for digital still
[8] A. Stern, Optical compressive imaging. CRC Press, 2016. cameras (Optical science and engineering). CRC Press, 2005.
[9] X. Yuan, D. J. Brady, and A. K. Katsaggelos, “Snapshot compressive [34] L. Alparone, B. Aiazzi, S. Baronti, and A. Garzelli, Remote sensing
imaging: Theory, algorithms, and applications,” IEEE Signal Processing image fusion. CRC Press, 2015.
Magazine, vol. 38, no. 2, pp. 65–88, Mar. 2021. [35] S. Mihoubi, O. Losson, B. Mathon, and L. Macaire, “Multispectral
[10] C. Zhou and S. K. Nayar, “Computational cameras: convergence of demosaicing using pseudo-panchromatic image,” IEEE Transactions on
optics and processing,” IEEE Transactions on Image Processing, vol. 20, Computational Imaging, vol. 3, no. 4, pp. 982–995, Dec. 2017.
no. 12, pp. 3322–3340, Dec. 2011. [36] M. Li, P. Tu, and W. Heidrich, “Robust joint image reconstruction from
[11] G. Vivone, L. Alparone, J. Chanussot, M. Dalla Mura, A. Garzelli, color and monochrome cameras,” in British Machine Vision Conference
G. A. Licciardi, R. Restaino, and L. Wald, “A critical comparison (BMVC). BMVC, Sep. 2019.
among pansharpening algorithms,” IEEE Transactions on Geoscience [37] H. Arguello and G. R. Arce, “Colored coded aperture design by concen-
and Remote Sensing, vol. 53, no. 5, pp. 2565–2586, May 2015. tration of measure in compressive spectral imaging,” IEEE Transactions
[12] L. Loncan, L. B. de Almeida, J. M. Bioucas-Dias, X. Briottet, J. Chanus- on Image Processing, vol. 23, no. 4, pp. 1896–1908, Apr. 2014.
sot, N. Dobigeon, S. Fabre, W. Liao, G. A. Licciardi, M. Simoes, J.- [38] Y. M. Lu, C. Fredembach, M. Vetterli, and S. Susstrunk, “Designing
Y. Tourneret, M. A. Veganzones, G. Vivone, Q. Wei, and N. Yokoya, color filter arrays for the joint capture of visible and near-infrared im-
“Hyperspectral pansharpening: A review,” IEEE Geoscience and Remote ages,” in 2009 16th IEEE International Conference on Image Processing
Sensing Magazine, vol. 3, no. 3, pp. 27–46, Sep. 2015. (ICIP). IEEE, Nov. 2009.
[13] P.-J. Lapray, X. Wang, J.-B. Thomas, and P. Gouton, “Multispectral filter [39] W. Rudin, Functional analysis. McGraw-Hill, 1991.
arrays: Recent advances and practical implementation,” Sensors, vol. 14, [40] P. J. Davis, Circulant matrices. New York: Wiley, 1979.
no. 11, pp. 21 626–21 659, Nov. 2014. [41] H. Tan, X. Zeng, S. Lai, Y. Liu, and M. Zhang, “Joint demosaicing
[14] B. E. Bayer, “Color imaging array,” Jul. 1976, US Patent 3,971,065. and denoising of noisy bayer images with ADMM,” in 2017 IEEE
[15] G. R. Arce, D. J. Brady, L. Carin, H. Arguello, and D. S. Kittle, International Conference on Image Processing (ICIP). IEEE, Sep.
“Compressive coded aperture spectral imaging: An introduction,” IEEE 2017.
Signal Processing Magazine, vol. 31, no. 1, pp. 105–115, Jan. 2014. [42] J. D. Cohn, “Power spectrum and correlation function errors: Poisson
[16] C. V. Correa, H. Arguello, and G. R. Arce, “Snapshot colored com- vs. Gaussian shot noise,” New Astronomy, vol. 11, no. 4, pp. 226–239,
pressive spectral imager,” Journal of the Optical Society of America A, Jan. 2006.
vol. 32, no. 10, p. 1754, Sep. 2015. [43] J. Hadamard, “Sur les problèmes aux dérivées partielles et leur signifi-
[17] H. Arguello and G. R. Arce, “Restricted isometry property in coded cation physique,” Princeton university bulletin, pp. 49–52, 1902.
aperture compressive spectral imaging,” in 2012 IEEE Statistical Signal [44] A. Chambolle and T. Pock, “An introduction to continuous optimization
Processing Workshop (SSP). IEEE, Aug. 2012. for imaging,” Acta Numerica, vol. 25, pp. 161–319, May 2016.
[18] A. Parada-Mayorga and G. R. Arce, “Colored coded aperture design in [45] N. Parikh and S. Boyd, “Proximal algorithms,” Foundations and Trends®
compressive spectral imaging via minimum coherence,” IEEE Transac- in Optimization, vol. 1, no. 3, pp. 127–239, 2014.
tions on Computational Imaging, vol. 3, no. 2, pp. 202–216, Jun. 2017. [46] L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based
[19] S. Jalali and X. Yuan, “Snapshot compressed sensing: Performance noise removal algorithms,” Physica D: Nonlinear Phenomena, vol. 60,
bounds and algorithms,” IEEE Transactions on Information Theory, no. 1-4, pp. 259–268, 1992.
vol. 65, no. 12, pp. 8005–8024, Dec. 2019. [47] L. Condat, “Discrete total variation: New definition and minimization,”
[20] E. J. Candès, “The restricted isometry property and its implications for SIAM Journal on Imaging Sciences, vol. 10, no. 3, pp. 1258–1290, Jan.
compressed sensing,” Comptes Rendus Mathematique, vol. 346, no. 9- 2017.
10, pp. 589–592, May 2008. [48] A. Chambolle, S. E. Levine, and B. J. Lucier, “An upwind finite-
[21] Teledyne e2v. (2020) Onyx 1.3m - EV76C664 - CMOS image difference method for total variation–based image smoothing,” SIAM
sensor. [Online]. Available: https://imaging.teledyne-e2v.com/products/ Journal on Imaging Sciences, vol. 4, no. 1, pp. 277–299, Jan. 2011.
standard-image-sensors/cmos-standard-image-sensors/onyx/ [49] R. Abergel and L. Moisan, “The Shannon total variation,” Journal of
[22] C. Kwan, B. Chou, L.-Y. M. Kwan, and B. Budavari, “Debayering Mathematical Imaging and Vision, vol. 59, no. 2, pp. 341–370, May
RGBW color filter arrays: A pansharpening approach,” in 2017 IEEE 8th 2017.
Annual Ubiquitous Computing, Electronics and Mobile Communication [50] J. Duran, M. Moeller, C. Sbert, and D. Cremers, “Collaborative total
Conference (UEMCON). IEEE, Oct. 2017. variation: A general framework for vectorial TV models,” SIAM Journal
[23] Silios Technologies. (2020) COLOR SHADES by Silios. [Online]. on Imaging Sciences, vol. 9, no. 1, pp. 116–151, 2016.
Available: https://www.silios.com/multispectral-imaging [51] ——, “On the implementation of collaborative TV regularization: Ap-
[24] G. A. Agranov, C. Molgaard, A. Bahukhandi, , C. Lee, and X. Li, “Pixel plication to cartoon+ texture decomposition,” IPOL Journal. Image
binning in an image sensor,” Jul. 2017, US Patent 9,971,065. Processing Online, vol. 6, pp. 27–74, 2016.
[25] A. Lakcher, L. Bidault, J. Ducoté, E. Mortini, A. Ostrovsky, B. Le- [52] X. Bresson and T. F. Chan, “Fast dual minimization of the vectorial
Gratiet, L. Berthier, C. Jamin-Mornet, and M. Besacier, “SEM contour total variation norm and applications to color image processing,” Inverse
based metrology for microlens process studies in CMOS image sensor Problems and Imaging, vol. 2, no. 4, pp. 455–484, 2008.
technologies,” in Optical Microlithography XXXI, J. Kye and S. Owa, [53] R. Tibshirani, “Regression shrinkage and selection via the lasso: A ret-
Eds. SPIE, Mar. 2018. rospective,” Journal of the Royal Statistical Society: Series B (Statistical
[26] Ó. Espitia, S. Castillo, and H. Arguello, “Compressive hyperspectral Methodology), vol. 73, no. 3, pp. 273–282, Apr. 2011.
and multispectral imaging fusion,” in Algorithms and Technologies for [54] A. Chambolle and T. Pock, “A first-order primal-dual algorithm for
Multispectral, Hyperspectral, and Ultraspectral Imagery XXII, M. Velez- convex problems with applications to imaging,” Journal of Mathematical
Reyes and D. W. Messinger, Eds. SPIE, May 2016. Imaging and Vision, vol. 40, no. 1, pp. 120–145, Dec. 2010.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15

[55] I. Loris and C. Verhoeven, “On a generalization of the iterative soft- Daniele Picone (Member, IEEE) received the B.Sc.
thresholding algorithm for the case of non-separable penalty,” Inverse and M.Sc. degrees in electronic engineering from
Problems, vol. 27, no. 12, p. 125007, Nov. 2011. the University of Salerno, Salerno, Italy, in 2008
[56] L. Condat, D. Kitahara, A. Contreras, and A. Hirabayashi, “Proximal and 2016, respectively, and the Ph.D degree in
splitting algorithms: A tour of recent advances, with new twists,” SIAM Signal and Image processing form the University
Review, 2022, to appear. of Grenoble Alpes, Grenoble, France, in 2021. In
[57] G. H. Golub and C. F. Van Loan, Matrix computations. JHU press, 2016, he was a Research fellow for 4 months at
2013. University of Salerno. In 2019, he was a visiting
[58] M. D. Buhmann, Radial basis functions: theory and implementations. researcher at the Tokyo Insitute of Technology, Japan
Cambridge university press, 2003, vol. 12. for three months. Since December 2021, he was a
[59] Y. Monno, D. Kiku, M. Tanaka, and M. Okutomi, “Adaptive residual Postdoctoral fellow for one year with the University
interpolation for color and multispectral image demosaicking,” Sensors, of Grenoble Alpes, Grenoble, France, and then with the Grenoble Institute of
vol. 17, no. 12, p. 2787, Dec. 2017. Technology (Grenoble-INP), Grenoble, France, starting from January 2023.
[60] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, “MTF- He is conducting his research at the Grenoble Images Speech Signals and
tailored multiscale fusion of high-resolution MS and pan imagery,” Automatics Laboratory (GIPSA-Lab). His main research activities are in the
Photogrammetric Engineering and Remote Sensing, vol. 72, no. 5, pp. fields of image processing and remote sensing, with applications mainly
591–596, May 2006. involving computational imaging, optimization, data fusion, and hyperspectral
[61] B. Aiazzi, S. Baronti, and M. Selva, “Improving component substitution data processing.
pansharpening through multivariate regression of MS + Pan data,” IEEE
Transactions on Geoscience and Remote Sensing, vol. 45, no. 10, pp.
3230–3239, Oct. 2007.
[62] MAXAR Technologies. (2020) MAXAR product samples. [Online].
Available: https://www.maxar.com/product-samples
[63] L. Wald, T. Ranchin, and M. Mangolini, “Fusion of satellite images of Mauro Dalla Mura (Senior Member, IEEE) re-
different spatial resolutions: Assessing the quality of resulting images,” ceived the B.Sc. and M.Sc. degrees in Telecommu-
Photogrammetric engineering and Remote Sensing, vol. 63, no. 6, pp. nication Engineering from the University of Trento,
691–699, Jun. 1997. Italy in 2005 and 2007, respectively. He obtained
[64] R. H. Yuhas, A. F. H. Goetz, and J. W. Boardman, “Discrimination in 2011 a joint Ph.D. degree in Information and
among semi-arid landscape endmembers using the spectral angle mapper Communication Technologies (Telecommunications
(SAM) algorithm,” in Proc. Summaries 3rd Annu. JPL Airborne Geosci. Area) from the University of Trento, Italy and in
Workshop, vol. 1, 1992, pp. 147–149. Electrical and Computer Engineering from the Uni-
[65] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image versity of Iceland, Iceland.
quality assessment: from error visibility to structural similarity,” IEEE In 2011 he was a Research fellow at Fondazione
Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, Apr. Bruno Kessler, Trento, Italy, conducting research on
2004. computer vision. He is currently an Assistant Professor at Grenoble Institute of
[66] L. Miao and H. Qi, “The design and evaluation of a generic method for Technology (Grenoble INP), France since 2012. He is conducting his research
generating mosaicked multispectral filter arrays,” IEEE Transactions on at the Grenoble Images Speech Signals and Automatics Laboratory (GIPSA-
Image Processing, vol. 15, no. 9, pp. 2780–2791, Sep. 2006. Lab). He is a Junior member of the Institut Universitaire de France (2021-
[67] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, “Ad- 2026). Dr. Dalla Mura has been appointed ”Specially Appointed Associate
vantages of Laplacian pyramids over ''à trous'' wavelet transforms for Professor” at the School of Computing, Tokyo Institute of Technology, Japan
pansharpening of multispectral images,” in Image and Signal Processing for 2019-2022. His main research activities are in the fields of remote sensing,
for Remote Sensing XVIII, L. Bruzzone, Ed. SPIE, Nov. 2012. computational imaging, image and signal processing.
[68] S. Mihoubi, O. Losson, B. Mathon, and L. Macaire, “Multispectral Dr. Dalla Mura was the recipient of the IEEE GRSS Second Prize in the
demosaicing using intensity-based spectral correlation,” in 2015 Interna- Student Paper Competition of the 2011 IEEE IGARSS 2011 and co-recipient
tional Conference on Image Processing Theory, Tools and Applications of the Best Paper Award of the International Journal of Image and Data
(IPTA). IEEE, Nov. 2015. Fusion for the year 2012-2013 and the Symposium Paper Award for IEEE
[69] D. Picone, L. Condat, and M. Dalla Mura, “Analysis of masks for IGARSS 2014. Dr. Dalla Mura was the IEEE GRSS Chapter’s Committee
compressed acquisitions in variational-based pansharpening,” in Work- Chair for 2020-2021. He was President of the IEEE GRSS French Chapter
shop on Compressed Sensing applied to Radar, Multimodal Sensing,and 2016-2020 (he previously served as Secretary 2013-2016). In 2017 the IEEE
Imaging (CoSeRa). Citeseer, Sep. 2018. GRSS French Chapter was the recipient of the IEEE GRSS Chapter Award
[70] R. A. Schowengerdt, “Aliasing in remote sensing imagery,” in SPIE and the “Chapter of the year 2017” from the IEEE French Section. He is on
Proceedings, Z. ur Rahman, R. A. Schowengerdt, and S. E. Reichenbach, the Editorial Board of the IEEE Journal of Selected Topics in Applied Earth
Eds. SPIE, Jul. 2002. Observations and Remote Sensing (J-STARS) since 2016.
[71] R. Laumont, V. De Bortoli, A. Almansa, J. Delon, A. Durmus, and
M. Pereyra, “Bayesian imaging using plug & play priors: when Langevin
meets Tweedie,” SIAM Journal on Imaging Sciences, vol. 15, no. 2, pp.
701–737, May 2022.
[72] V. Monga, Y. Li, and Y. C. Eldar, “Algorithm unrolling: Interpretable, Laurent Condat (Senior Member, IEEE) received
efficient deep learning for signal and image processing,” IEEE Signal two Master’s degrees in 2003, in computer science
Processing Magazine, vol. 38, no. 2, pp. 18–44, Mar. 2021. and applied mathematics, and a PhD in applied
[73] D. Picone, “Model based signal processing techniques for mathematics in 2006 from Grenoble Institute of
nonconventional optical imaging systems,” Theses, Université Grenoble Technology, Grenoble, France. After 2 years as
Alpes, Nov. 2021. [Online]. Available: https://tel.archives-ouvertes.fr/ a postdoc in the Helmholtz Zentrum Muenchen,
tel-03596486 Munich, Germany, he was hired as a permanent
researcher by the French National Center for Sci-
entific Research (CNRS). Since Nov. 2019, he is
on leave from the CNRS and a Senior Research
Scientist in King Abdullah University of Science and
Technology (KAUST), Saudi Arabia.
Dr. Condat’s area of interest spans optimization, signal and image pro-
cessing, inverse problems, and machine learning. He has co-authored more
than 100 articles on these topics. He is a senior member of the IEEE and an
associate editor of IEEE Transactions on Signal Processing (TSP). He received
a best student paper award at the conference IEEE ICIP 2005, a best PhD
award from Grenoble Institute of Technology in 2007, and several meritorious
reviewer awards.

View publication stats