3230 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 45, NO.

10, OCTOBER 2007

Improving Component Substitution Pansharpening

Through Multivariate Regression of MS+Pan Data
Bruno Aiazzi, Stefano Baronti, Member, IEEE, and Massimo Selva

Abstract—In this paper, multivariate regression is adopted to exactly three MS bands are concerned since the IHS transform
improve spectral quality, without diminishing spatial quality, in is defined for three components only. First, the IHS transform
image fusion methods based on the well-established component is applied to the spectral bands, once they have been registered
substitution (CS) approach. A general scheme that is capable of
modeling any CS image fusion method is presented and discussed. to the Pan image. Then, the intensity component I is replaced
According to this scheme, a generalized intensity component is with the high-resolution Pan P. The fused bands are finally
defined as the weighted average of the multispectral (MS) bands. obtained via an inverse IHS. Usually, P is histogram-matched,
The weights are obtained as regression coefficients between the i.e., radiometrically transformed by a constant gain and bias
MS bands and the spatially degraded panchromatic (Pan) image, in such a way that it exhibits mean and variance that are the
with the aim of capturing the spectral responses of the sensors.
Once it has been integrated into the Gram–Schmidt spectral- same as I, before substitution is carried out. However, since
sharpening method, which is implemented in Environment for the histogram-matched Pan and the intensity component do not
Visualizing Images (ENVI) program, and into the generalized generally have the same radiometry, i.e., local mean, when the
intensity-hue-saturation fusion method, the proposed preprocess- fusion result is displayed in color composition, large spectral
ing module allows the production of fused images of the same spa- distortions may be noticed as color changes. This effect occurs
tial sharpness but of increased spectral quality with respect to the
standard implementations. In addition, quantitative scores carried because the spectral response of I, as synthesized by means
out on spatially degraded data clearly confirm the superiority of of the MS bands, may be far different from that of Pan. Thus,
the enhanced methods over their baselines. also slowly space-varying radiance offsets, and not only spatial
Index Terms—Component substitution (CS) pansharpening, details, are locally injected.
Gram–Schmidt (GS) spectral sharpening, IKONOS satellite data, When more than three bands are available, a viable solution
image fusion, intensity-hue-saturation (IHS) transform, multi- is used to define a generalized IHS (GIHS) transform by
spectral (MS) imagery, multivariate regression, QuickBird images. including the response of the near-infrared (NIR) band into the
intensity component [9]. In this case, I is obtained by weighting
I. I NTRODUCTION each band with a set of coefficients. The choice of these weights
can be related to the spectral responses of the Pan and MS bands
R EMOTE sensing image fusion techniques aim at inte-
grating the information conveyed by the data acquired
with different spatial and spectral resolutions from satellite or
by considering the spectral characteristics of the sensors, as in
[9] and [10], where prefixed values are adopted.
aerial platforms in order to obtain images of higher quality. An alternative to the IHS techniques is principal component
Generally, a gold reference is not available to evaluate the analysis (PCA). Being analogous to the IHS scheme, the Pan
efficacy of a fusion method. Thus, quality evaluation should image is substituted to the first principal component (PC1).
refer to the aims of the fusion process [1]. The most straight- Histogram matching of Pan to PC1 is mandatory before sub-
forward goal is photo-analysis, but also such automated tasks stitution because the mean and variance of PC1 are generally
as features extraction and segmentation/classification have been far greater than those of Pan. It is well established that PCA
found to benefit from the fusion [2]–[4]. A variety of im- performances are better than those of IHS [5] and, in particular,
age fusion techniques is devoted to merge multispectral (MS) that the spectral distortion in the fused bands is usually less
and panchromatic (Pan) images, which exhibit complemen- noticeable, even if it cannot completely be avoided. Generally
tary characteristics of spatial and spectral resolutions [5], [6]. speaking, if the spectral responses of the MS bands are not
Among these techniques, component substitution (CS) methods perfectly overlapped with the bandwidth of Pan, as it happens
[7] are attractive because they are fast and easy to implement with the most advanced very high resolution imagers, namely,
and they allow user’s expectations to be fulfilled most of IKONOS and QuickBird, IHS- and PCA-based methods may
the time. yield poor results in terms of spectral fidelity [11].
A forerunner for CS techniques is the intensity-hue- Another CS technique reported in the literature is
saturation (IHS) method [8]. This algorithm is suitable when Gram–Schmidt (GS) spectral sharpening, which was invented
by Laben and Brower in 1998 and patented by Eastman Kodak
[12]. The GS method is widely used since it has been im-
Manuscript received October 18, 2006; revised March 30, 2007.
The authors are with the Institute of Applied Physics “Nello Carrara” plemented in the Environment for Visualizing Images (ENVI)
CNR Area di Ricerca di Firenze (IFAC-CNR), 50019 Florence, Italy (e-mail: program package. It has two operational modes, depending on
[email protected]; [email protected]; [email protected]). how the low-resolution version of the Pan image that is used
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org. in the forward GS transformation is defined. In the first case,
Digital Object Identifier 10.1109/TGRS.2007.901007 which will be referred to as “mode 1” or GS1, the low-pass

0196-2892/$25.00 © 2007 IEEE

AIAZZI et al.: IMPROVING CS PANSHARPENING THROUGH MULTIVARIATE REGRESSION 3231

CS methods is reported in Section III, while the proposed

enhancement is motivated and described in Section IV for both
the GS and GIHS fusion schemes. Quantitative and qualitative
results are reported and discussed in Section V. Eventually, the
conclusion is drawn in Section VI.

II. GS S PECTRAL S HARPENING

In the GS method, as described by its inventors [12], the
spatial resolution of the MS image is enhanced by merging the
high-resolution Pan image with the low spatial resolution MS
bands. According to the authors’ description, the main steps of
this method are the following.
1) A lower spatial resolution Pan image is simulated at the
same scale of Pan.
Fig. 1. General scheme of fusion methods based on CS. The aim of weights 2) The GS transformation is performed on the simulated
wi is to synthesize an intensity image (I) similar to the original Pan image. The
weights gi determine the amounts of detail that are added to the expanded MS lower spatial resolution Pan image together with the
bands. plurality of lower spatial resolution spectral band images
resampled at the same scale of Pan. The simulated lower
approximation is computed by the ENVI program as the aver- spatial resolution Pan image is employed as the first band
age of the MS bands, which are given as input to the procedure. in the GS transformation.
In the other case, which will be referred to as “mode 2” or 3) The statistics of the higher spatial resolution Pan image
GS2, the approximation is preliminarily provided by the user is adjusted to match the statistics of the first transform
by means of a low-pass filtering and a decimation of the Pan band that results from the GS transformation to produce
image which is entered into the procedure with the MS bands. a modified higher spatial resolution Pan image.
Although the spectral quality of CS fusion results may be 4) The modified higher spatial resolution Pan image is sub-
sufficient for most applications and users, methods based on stituted for the first transform band that results from the
injecting zero-mean high-pass spatial details, which are taken GS transformation to produce a new set of transform
from the Pan image without resorting to any transformation, bands.
have been extensively studied to definitely overcome the incon- 5) The inverse GS transformation is performed on the new
venience of spectral distortion. Multiresolution analysis (MRA) set of transform bands to produce the enhanced spatial-
provides effective tools, like wavelets and Laplacian pyramids resolution MS image.
[13]–[17], to help carry out the data fusion tasks. However, The presence of some freedom in step 1) determines the
in the case of high-pass detail injection, spatial distortions two options of the ENVI package that have been described in
may occur in fusion products, which are typically caused by Section I as GS1 and GS2.
ringing or aliasing effects, originating shifts, or blur of contours In GS1, the ENVI procedure is responsible in computing
and textures. These drawbacks, which may be as annoying the simulated low-resolution Pan image that is obtained as the
as spectral distortions, are emphasized by a misregistration pixel average of the MS bands. In GS2, it is the user’s task to
between MS and Pan data, especially if the MRA underlying produce the synthetic Pan image that is usually obtained by
detail injection is not shift-invariant [15], [18]. preliminarily low-pass filtering and, then, decimating the Pan
Due to their impressive spatial quality and to their low image. The major difference in the results between these two
computational cost, CS techniques are still investigated by modalities, which is mostly noticeable in true-color display, is
the research community, with the objective of upgrading their that GS1 exhibits an outstanding spatial quality, but spectral
spectral accuracy. In this perspective, the contribution of this distortions with respect the resampled MS image may occur
paper is to define and discuss a general scheme that is capable because the average of the MS spectral bands is not likely to
of modeling any CS-based method, and thus, it is suitable to have the same radiometry as the Pan image. Instead, GS2 is
improve its performances. In particular, two CS fusion methods, unaffected by spectral distortion but generally suffers from a
namely, the fast GIHS with spectral adjustment [9] and the lower sharpness and spatial enhancement, which is possibly
GS spectral sharpening of ENVI, are considered. The general due to improper digital filtering of the Pan image [17] or
scheme reported in Fig. 1 defines a generalized intensity com- subpixel misregistration. This effect is noticeable for small
ponent as the weighted average of the MS bands. By applying objects appearing on a quasi-constant background.
an optimization method such as a standard linear regression
algorithm in calculating the weights, an evident improvement
III. G ENERAL CS S CHEME
with respect to the performances of standard GIHS and GS
methods is obtained. The computed coefficients are satisfacto- Let us consider the fusion scheme in Fig. 1. The original MS
rily capable of matching the spectral responses of MS and Pan. bands B, G, R, and NIR are preliminarily expanded to the
This paper is organized as follows. GS spectral sharpening same spatial scale of the full-resolution Pan image P to obtain
is briefly reviewed in Section II. The general scheme of the a new data set B̃, G̃, R̃, and NĨR. The expanded MS bands
3232 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 45, NO. 10, OCTOBER 2007

TABLE I
EXPRESSIONS FOR THE WEIGHTS OF THE SCHEME REPORTED IN
FIG. 1 RELATIVE TO THE VARIOUS CS-BASED METHODS

the average of four MS bands, while for the GIHS with fixed
weights, which is introduced in [9] (GIHSF), I is the weighted
average of the four MS bands.
With regard to GIHS, an exhaustive mathematical formula-
Fig. 2. IKONOS spectral responses.
tion is reported in [9]. Its description and implementation is
immediate from the scheme of Fig. 1.
Being analogous to GIHS, for GS1, I is the average of the
are weighted by the coefficients {wi } to generate a synthetic four MS bands. A variant of this scheme that is considered in
intensity component I this paper (GSF hereinafter) is obtained by synthesizing I with
the set of {wi } reported in [9], which is the same as GIHSF.
I = w1 · B̃ + w2 · G̃ + w3 · R̃ + w4 · NĨR + b. (1) GS2 is obtained in the scheme of Fig. 1 when a low-pass-
filtered version of P is given instead of I. It is to be noted
The constant parameter b is an offset image introduced in the that GS2 is not strictly a CS method since only P is utilized
model to consider the additive component that may arise from in producing spatial details, analogously to the MRA-based
the acquisition process and from the sensor spectral responses methods. Thus, {wi } is not defined in this mode.
reported in Fig. 2 for IKONOS imager. In fact, the Pan sensor In any case, for all the GS-based methods, the ith coeffi-
collects photons in a spectral band that is larger than that of cient gi is proportional to the covariance value between the
the MS channels, and spectral gaps exist between MS spectral synthesized intensity and the expanded ith MS band. The
responses. Taking into account the offset brings a stabilization demonstration of this relation is reported in the Appendix.
of the weights {wi } at different spatial scales. With regard to PCA, it can easily be shown that, if Pan
Afterward, I is subtracted from P to produce a detail image is histogram-matched to PC1, wi = gi , with both being the
δ, i.e., ith component of the first eigenvector x1 that constitutes the
δ = P − I. (2) unitary transformation matrix.
The two variants that are proposed in this paper and are
A second set of coefficients {gi } is adopted to modulate the described in Section IV, namely, GS adaptive (GSA) and GIHS
detail image δ before its addition to the expanded data set in adaptive (GIHSA), are also reported in Table I.
order to yield the final fused bands, i.e., B̂, Ĝ, R̂, and NÎR.
A histogram matching is usually performed on P to match
IV. E NHANCED GS AND GIHS I MAGE F USION
its mean and standard deviation to those of the synthesized
I component. This procedure has the objective of reducing To understand the influence of spectral response on the fusion
spectral distortions that are caused by the spectral mismatch procedure for high spatial resolution sensors, as IKONOS and
between I and P images, as discussed in Section II. QuickBird, the relative spectral response plots of the IKONOS
The scheme described in Fig. 1 is general and is suitable spectral channels, which are reported in Fig. 2, have been
in describing any CS-based fusion algorithm, depending on considered. Ideally, the MS bands (B, G, R, and NIR) should
the values of the two sets of weights {wi } and {gi }. This is be disjoint and should entirely fall within the bandwidth of Pan.
evidenced in the following discussion, where b is assumed to From Fig. 2, however, it appears that the G and B bands are
be equal to zero, without losing generality. largely overlapped and that the B band mostly falls outside the
The expressions of the weights wi and gi for the considered 3-dB cutoff of the Pan spectral band. Furthermore, the spectral
CS-based methods are summarized in Table I. response of Pan is extended beyond the NIR band. The color
Actually, the IHS procedure is equivalent to the injection, i.e., distortion problem in fusion stems from such mismatches and,
addition, of the difference between P and I into the resampled in particular, from the fact that the synthetic intensity of the
MS bands [19]. Consequently, IHS fusion method can easily be methods with equal weights (IHS, GIHS, and GS1) does not
described by the scheme when the weights are taken as w1 = reflect the spectral response of Fig. 2 since it is obtained as a
w2 = w3 = 1/3, g1 = g2 = g3 = 1, and w4 = g4 = 0. plain average of MS bands. For example, vegetation appears
As previously stated, the IHS method can be generalized. of relatively high reflectance in the NIR band and Pan image,
Hereinafter, GIHS will denote a case in which I is simply while its reflectance is low in the visible (R, G, and B) bands.
AIAZZI et al.: IMPROVING CS PANSHARPENING THROUGH MULTIVARIATE REGRESSION 3233

As a consequence, the radiometric values of vegetation areas is V. R ESULTS

likely to be much smaller in the synthetic intensity than in the
Results have been assessed on two data sets acquired by
true Pan. This effect causes the injection in the fused MS bands
IKONOS and QuickBird satellites. Both qualitative and quanti-
of a radiance offset, which may give rise to color distortion.
tative assessments have been performed.
In order to avoid this drawback, the idea reported in [20] and
Evaluations are not oriented to a specific application task but
[21] is to generate the synthetic intensity in such a way that
rather to photo-analysis. However, to give an idea on how the
the spectral response of the sensor is considered, by differently
methods perform on different landscapes, quantitative scores
weighting the contributions coming from the MS spectral
are also reported and discussed for three classes of land-use.
channels.
Quantitative assessments are obtained by degrading the spa-
These methods, however, only consider the nominal spec-
tial resolution of all original images by a factor of four and
tral responses when available. Actually, the influence of other
by performing fusion on such images [23]. The comparison
phenomena, such as on-orbit working conditions, variability
of fused original images allows such global synthetic scores
of the observed scene, and postprocessing effects, can signif-
as Q4 [24], spectral angle mapper (SAM), and erreur relative
icantly modify the nominal spectral response. In particular,
globale adimensionnelle de synthése (ERGAS) [14], [16] to be
atmospheric influence depends on the viewing angle since the
computed.
scattering effect is related to the wavelength and on atmospheric
SAM and ERGAS are distortion figures and should take
conditions.
values as low as possible (ideally zero). ERGAS is mainly re-
A viable solution is to perform a linear regression between
lated to radiometric distortion, while SAM is related to angular
Pan and MS bands. Thus, in the proposed scheme, a synthetic
distortion.
intensity, having a minimum mean-square error (mse) with
Conversely, Q4 represents a quality index and should be
respect to the reduced Pan, is computed. The steps of the
as high as possible (ideally one). Q4 takes into account the
procedure are the following.
contrast, mean bias, and similarity between images [24].
1) Reduce the original full-resolution Pan image to the spa- It should be noted that, since the regression strategy mini-
tial scale of the MS image by means of low-pass filtering mizes the mse between P and I images, improvements should
and decimation. Let us denote this image as P̆. be expected, in particular, for ERGAS.
2) Assume that Quantitative scores have been reported on the whole
processed images for both IKONOS and QuickBird test images.
For the IKONOS test image, in order to verify the behavior of
Ĭ = w1 · B + w2 · G + w3 · R + w4 · NIR + b (3) the fusion algorithms on different landscapes, scores have been
computed by identifying three different land classes: urban,
and estimate the set of coefficients ŵ1 , ŵ2 , ŵ3 , ŵ4 , and b̂ vegetated, and homogeneous areas.
by means of a linear regression algorithm [22] in order to The next two sections present and discuss the results.
minimize the mse between P̆ and Ĭ.
3) Calculate intensity I at the scale of P by utilizing the set
of coefficients determined at step 2) A. IKONOS
Fig. 3 refers to the IKONOS data set and shows a 512 × 512
I = ŵ1 · B̃ + ŵ2 · G̃ + ŵ3 · R̃ + ŵ4 · NĨR + b̂. (4) detail of the whole 2048 × 2048 11-bit image of the area of
Toulouse, France, that has been processed. The full-resolution
Pan image is reported in Fig. 3(a), while the original true-color
The implicit assumption that is done in step 3) is that the MS image, which is expanded to the scale of Pan, is reported in
regression coefficients that are computed at the scale of the MS Fig. 3(b). Fig. 3(b) is assumed as the reference in evaluating
image are practically the same as those that would be computed color distortions. Fig. 3(c) shows the result of GS1, i.e., the
at the scale of the original Pan image, if MS observations were synthetic low-resolution Pan is obtained as the pixel average
available at full spatial resolution. This is to assume that the of the four MS bands. The spatial enhancement is impressive;
spectral responses of the data set are practically unaffected by however, some color distortion appears on the green areas on
the change of the spatial resolution. the left of the river and on the river itself. The green areas are
In this paper, the estimation of I using (4) is applied to GIHS, too bright while the color of the river is too dark.
and it is where the algorithm denoted as GIHSA originates. These spectral distortions are not present in Fig. 3(d), which
With regard to GS, since the ENVI procedure in “mode 2” shows the outcome of GS2, i.e., the synthetic Pan is obtained
accepts a synthetic intensity as input at the resolution of the MS as a low-pass-filtered version of the full-resolution Pan. As a
image, the intensity Ĭ that is estimated by (3) is provided to the counterpart, Fig. 3(d) is less sharp than Fig. 3(c). Fig. 3(e)
ENVI module for the implementation of the proposed regres- and (f) results from the proposed modified versions. Fig. 3(e)
sion strategy. This method is hereinafter referred to as GSA. has been obtained by synthesizing the low-resolution Pan as
In Section V, the proposed GSA and GIHSA methods are a weighted sum of all spectral bands (GSF), according to
assessed through comparisons with the other algorithms on two the fixed weights proposed by [9] and reported in Table I.
different high spatial resolution data sets acquired by IKONOS Fig. 3(e) appears as sharp as Fig. 3(c), but the spectral dis-
and QuickBird sensors. tortion is mitigated. The result obtained with the adaptive
3234 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 45, NO. 10, OCTOBER 2007

Fig. 3. Examples of a full-scale spatial enhancement of fusion algorithms displayed as 512 × 512 true-color compositions at a 1-m pixel spacing for the IKONOS
image. (a) Original Pan image. (b) Original MS bands (4 m) resampled to the scale of Pan image (1 m). (c) GS “mode 1” fusion (GS1). (d) GS “mode 2” (GS2).
(e) GS with fixed weights (GSF). (f) GS with adaptive weights (GSA). (g) GIHS with equal weights (GIHS). (h) GIHS with unequal fixed weights (GIHSF).
(i) GIHS with adaptive weights (GIHSA).

preprocessing (GSA) is shown in Fig. 3(f). The weights cal- TABLE II

REGRESSION WEIGHTS COMPUTED BETWEEN THE ORIGINAL 4-m MS
culated on the whole image are reported in Table II. The BANDS AND THE REDUCED 4-m PAN FOR IKONOS IMAGE
spectral fidelity with respect to the original MS data of Fig. 3(b)
is impressive, while spatial details appear as sharp as those
in Fig. 3(c).
Fig. 3(g) reports the result obtained by applying GIHS,
while Fig. 3(h) refers to GIHSF. Both methods have been in both images. Fig. 3(i) shows the result obtained by the
implemented in the same way as described in [9]. Spatial proposed GIHSA. Spectral fidelity is very high and comparable
details are very sharp, but spectral distortions are evident to that of GS2 and the proposed GSA.
AIAZZI et al.: IMPROVING CS PANSHARPENING THROUGH MULTIVARIATE REGRESSION 3235

Fig. 4. IKONOS. Results of fusion algorithms displayed as 128 × 128 true-color compositions at a 1-m pixel spacing. (a) Original Pan image. (b) Original MS
bands (4 m) resampled to the scale of Pan image (1 m). (c) GS1. (d) GS2. (e) GSF. (f) GSA. (g) GIHS. (h) GIHSF. (i) GIHSA.

Small 128 × 128 details produced by all the methods, which TABLE III
IKONOS: AVERAGE CUMULATIVE QUALITY INDEXES BETWEEN
are extracted from the image shown in Fig. 3, are displayed in THE O RIGINAL 4-m MS AND THE F USED I MAGES
Fig. 4; the disposition of the images is the same as in Fig. 3. OBTAINED FROM 16-m MS AND 4-m PAN
Spectral distortions are noticeable as changes in color hue of
the river with respect to the resampled low-resolution original
MS of Fig. 4(b). The images of Fig. 4(f) and (i), which are for
GSA and GIHSA, respectively, exhibit high spectral and spatial
quality. All fused images show impressive spatial details since
all are derived from CS methods. Spatial distortions are only
noticeable in Fig. 4(d) (GS2), especially for the cars on the regarding ERGAS. Since GIHSA and GSA utilize the same I
bridge. However, GS2 de facto belongs to the MRA methods component, this is evidently due to the {gi } weights that are
because the synthetic intensity component is obtained by low- different according to Table I. For the IKONOS test image,
pass filtering and decimating the original Pan image. GIHSA weights are more efficient than those of GSA.
Visual judgement is corroborated by quantitative assess- The only slight exception in Table III is represented by
ments. Score values are reported in Table III for the whole SAM of GIHSF and GIHSA. This is however compensated
2048 × 2048 image and show that the proposed regression- by the significant improvement of ERGAS. Actually, the mse
based enhancement strategy improves both GS and GIHS minimization, in this case, has produced regression coefficients
fusion algorithms with respect to all other nonadaptive imple- that are not the best for SAM. It was verified during the test
mentations. In particular, GSA results better than GIHSA when phase that a not negligible improvement of SAM of GIHSA
considering Q4 and SAM, while GIHSA is superior to GSA is obtained by just slightly decreasing ŵ1 and correspondingly
3236 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 45, NO. 10, OCTOBER 2007

TABLE IV a problem for GSA and GIHSA since these algorithms estimate
IKONOS: AVERAGE CUMULATIVE QUALITY INDEXES BETWEEN THE
ORIGINAL 4-m MS AND THE FUSED IMAGES ON URBAN AREAS their weights from the data. With regard to GSF and GIHSF,
performances are computed with the fixed weights reported in
[9], which have been found suitable for IKONOS and consid-
ered reasonable for QuickBird.
Fig. 5 reports the results of the algorithms on a 128 × 128
detail extracted from the selected 2048 × 2048 11-bit test
TABLE V image. The disposition of the images of Fig. 5 is the same as
IKONOS: AVERAGE CUMULATIVE QUALITY INDEXES BETWEEN THE
ORIGINAL 4-m MS AND THE FUSED IMAGES ON VEGETATED AREAS
in Figs. 3 and 4. As a general consideration, differences in the
results among methods of the same family are less apparent for
QuickBird than for IKONOS. Conversely, differences among
GS- and IHS-based methods are more apparent in QuickBird
since all IHS methods have the tendency to inject too much
blue on the vegetated areas. The correction introduced by the
TABLE VI regression algorithm is consistent, but not sufficient to com-
IKONOS: AVERAGE CUMULATIVE QUALITY INDEXES BETWEEN THE
ORIGINAL 4-m MS AND THE FUSED IMAGES ON HOMOGENEOUS AREAS
pletely avoid this inconvenience. The image produced by GSA
and GIHSA appear to be of great quality. GSA colors [Fig. 5(f)]
appear consistent with Fig. 5(b), while GIHSA [Fig. 5(i)]
appears a little bit bluish, but of great contrast.
The weights calculated from the whole image of GSA and
GIHSA are shown in Table VII. If these weights are compared
increasing ŵ2 . As expected, the improvement of SAM is paid with those of IKONOS that are shown in Table II, the two sets
in terms of ERGAS. of coefficients have the same trend since they are monotonic
Tables IV–VI report the quality indices that have been evalu- in both cases along the wavelength. The B weight (ŵ1 ) for
ated on the three classes of urban, vegetated, and homogeneous QuickBird is rather low, but this agrees with the data distri-
areas, respectively. The values of these indices depend on the bution in the B channel, which is very narrow (low standard
quantity of details that is present on the landscape, as one might deviation). This also explains why images that are produced
expect. Urban class figures are the worst, while those of the by the IHS methods are bluish. In fact, images fused by such
homogeneous class are the best. GSA and GIHSA are always methods suffer from an over-injection into the blue band.
better than their nonadaptive counterparts, with only few minor Table VIII reports the quantitative scores for the selected
exceptions; also in the other cases, their score is quite near to 2048 × 2048 11-bit images. All the GS-based methods perform
the best one. In any case, their ERGAS is always the lowest. better than the IHS-based algorithms. GS2, GSF, and GSA are
Another general consideration is that the fixed weights that particularly effective on QuickBird and obtain the best scores.
are defined in [9] do not perform better than the baseline in Among the IHS-based algorithms, GIHSF performances are
which all the weights are equal. This is a firm indication that the worst, thus confirming again that the regression strategy of
fixed coefficients are not suitable in coping with such vari- GIHSA is superior to the fixed weight choice.
abilities as those introduced by scene variations, atmospheric
conditions, and attitude or aging of the acquisition platform.
VI. C ONCLUDING R EMARKS
Conversely, regression strategy guarantees a greater flexibility
and more steady performances. The superiority of GSA and Multivariate regression brings advantages in methods based
GIHSA is evident on the vegetated areas, on which they largely on CS with respect to all the preprocessing methods that have
outperform the other schemes, demonstrating that modeling of been considered to create the synthetic low-resolution-intensity
spectral response is effective on vegetation. The performance images. The proposed enhanced strategy is effective in improv-
of constant weight methods on homogeneous areas (mainly ing the quality of the images fused with GS spectral sharpening
extracted from the river appearing in Fig. 3) is disappointing. and with GIHS and requires only a negligible extra computation
Since homogeneous areas are dealt with, the only explanation is with respect to that of the standard implementations. A general
that they are not able to efficiently model the local average value model is derived for the CS schemes that allows, in particular,
of the synthetic intensity image. The ERGAS values are par- an easy comparison between different fusion strategies. In this
ticularly low for GSA, GS2, and GIHSA, demonstrating their framework, regression optimization can be implemented as a
effectiveness. preprocessing block, which can easily be introduced in the
fusion algorithms to compute the generalized intensity. With
regard to GS spectral sharpening, the block can be written
B. QuickBird
in image display language (IDL) and integrated within the
In order to assess the fusion algorithms for a different sensor, ENVI module. Future work will explore different types of
also a QuickBird data set has been considered. Conversely, optimization strategies that are made possible in the general
from IKONOS, for which spectral responses that are reported in scheme for CS modeling, with the objective of reducing the gap
Fig. 2 are accessible, spectral responses of QuickBird have not in performance that presently exists between fusion methods
been made available to our knowledge. This does not represent based on CS and the ones based on MRA.
AIAZZI et al.: IMPROVING CS PANSHARPENING THROUGH MULTIVARIATE REGRESSION 3237

Fig. 5. QuickBird. Results of fusion algorithms displayed as 128 × 128 true-color compositions at a 0.7-m pixel spacing. (a) Original Pan image. (b) Original
MS bands (2.8 m) resampled to the scale of Pan image (0.7 m). (c) GS1. (d) GS2. (e) GSF. (f) GSA. (g) GIHS. (h) GIHSF. (i) GIHSA.

TABLE VII TABLE VIII

REGRESSION WEIGHTS COMPUTED BETWEEN THE ORIGINAL 2.8-m MS QUICKBIRD: AVERAGE CUMULATIVE INDEXES BETWEEN THE
BANDS AND THE REDUCED 2.8-m PAN FOR QUICKBIRD IMAGE ORIGINAL 2.8-m MS AND THE FUSED IMAGES OBTAINED
FROM 11.2-m MS AND 2.8-m PAN

A PPENDIX
Let
The relation between U and V is expressed by the forward
V = (I B̃ G̃ R̃ NĨR) GS transform
be the matrix given as input to the GS transform, where the U=V·C (5)
first column of V is the simulated intensity I, and B̃, G̃, R̃,
and NĨR are the MS bands. All these images are reported at
where C is a 5 × 5 upper unit triangular matrix, where the
the scale of the high-resolution panchromatic P and arranged
nonzero elements, apart from the diagonal ones, are obtained
as vectors, which are hereinafter supposed to be zero-mean in
by the GS orthogonalization procedure, as shown in [25].
order to simplify the notation without losing generality.
In this Appendix, the focus is not in evidencing the direct
Let
transformation, which is not difficult but formally complicated,
U = (GS1 GS2 GS3 GS4 GS5 ) but rather in the inverse transform.
From (5), the inverse GS transform is obviously
also be the matrix that is obtained as an output of the GS
transform applied to V. V = U · C−1 (6)
3238 IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 45, NO. 10, OCTOBER 2007

where the inverse matrix C−1 is again a 5 × 5 upper unit Eventually, from (9), the set {gi } that is determined by (15) is
triangular matrix [25], whose nonzero elements are being easily expressed by the following relations for the B, G, R, and NIR
expressed by bands, respectively:

C −1 (i, j) = cij = projui vj , i≤j (7) cov(I, B̃)

g1 =
var(I)
with projui vj as the orthogonal projection of vector vj onto
vector ui , which is given by cov(I, G̃)
g2 =
var(I)
ui , vj 
projui vj = (8) cov(I, R̃)
ui 2 g3 =
var(I)
with ui , vj  as the scalar product between ui and vj . Since all
cov(I, NĨR)
vectors are zero-mean, the following relation holds: g4 = . (16)
var(I)
cov(ui , vj )
projui vj = . (9)
var(ui ) ACKNOWLEDGMENT
Note that, from the definition of C (upper unit triangular The authors would like to thank Prof. L. Alparone for his
matrix), it follows that GS1 coincides with I. invaluable contribution and stimulating discussions.
The GS fusion algorithms substitute GS1 with the high-
resolution panchromatic P after it has been histogram-matched
to I in the matrix U, thus obtaining a new matrix U , and then R EFERENCES
perform the inverse transform. Therefore, the output matrix V̂ [1] L. Wald, “Some terms of reference in data fusion,” IEEE Trans. Geosci.
is given by Remote Sens., vol. 37, no. 3, pp. 1190–1193, May 1999.
[2] J. Wahlen, “Comparison of standard and image-filter fusion techniques,”
in Data Mining III, A. Zanasi, C. A. Brebbia, N. F. F. Ebecken, and

= U · C−1 .
V (10) P. Melli, Eds. Southampton, U.K.: WIT Press, 2002.
[3] Y. Zang and R. Wang, “Multi-resolution and multi-spectral image
fusion for urban object extraction,” in Proc. 20th ISPRS Congr., 2004,
The fusion products, i.e., the high-resolution MS bands, apart pp. 960–966.
from the average values, are simply obtained starting from the [4] R. Colditz, T. Wehrmann, M. Bachmann, K. Steinnocher, M. Schmidt,
second column in the matrix V̂ G. Strunz, and S. Dech, “Influence of image fusion approaches on clas-
sification accuracy: A case study,” Int. J. Remote Sens., vol. 27, no. 15,
pp. 3311–3335, Aug. 2006.

= (P
V B̂ Ĝ R̂ NÎR). [5] P. S. Chavez, Jr., S. C. Sides, and J. A. Anderson, “Comparison of
three different methods to merge multiresolution and multispectral data:
Landsat TM and SPOT panchromatic,” Photogramm. Eng. Remote Sens.,
If δ denotes the detail image to be injected, we can express vol. 57, no. 3, pp. 295–303, 1991.
P as [6] Z. Wang, D. Ziou, C. Armenakis, D. Li, and Q. Li, “A comparative
analysis of image fusion methods,” IEEE Trans. Geosci. Remote Sens.,
vol. 43, no. 6, pp. 1391–1402, Jun. 2005.
P = P − I + I = I + δ. (11) [7] V. K. Shettigara, “A generalized component substitution technique for
spatial enhancement of multispectral images using a higher resolution
data set,” Photogramm. Eng. Remote Sens., vol. 58, no. 5, pp. 561–567,
Consequently, the matrix U can be written as May 1992.
[8] W. Carper, T. Lillesand, and R. Kiefer, “The use of intensity-hue-
saturation transformations for merging SPOT panchromatic and multi-
spectral image data,” Photogramm. Eng. Remote Sens., vol. 56, no. 4,
U = U + (δ 0 0 0 0) (12) pp. 459–467, 1990.
[9] T.-M. Tu, P. S. Huang, C.-L. Hung, and C.-P. Chang, “A fast intensity-
and by applying (6) hue-saturation fusion technique with spectral adjustment for IKONOS
imagery,” IEEE Geosci. Remote Sens. Lett., vol. 1, no. 4, pp. 309–312,
Oct. 2004.

= V + (δ
V 0 0 0 0) · C−1 . (13) [10] M. Gonzáles Audícana, X. Otazu, O. Fors, and J. A. Alvarez-Mozos,
“A low computational-cost method to fuse IKONOS images using the
spectral response function of its sensors,” IEEE Trans. Geosci. Remote
The final expression for the output matrix V̂ by using (7) is Sens., vol. 44, no. 6, pp. 1683–1691, Jun. 2006.
[11] Y. Zhang, “Understanding image fusion,” Photogramm. Eng. Remote
the following: Sens., vol. 70, no. 6, pp. 657–661, Jun. 2004.
[12] C. A. Laben and B. V. Brower, “Process for enhancing the spatial
resolution of multispectral imagery using Pan-sharpening,” U.S. Patent

= V + (δ
V c12 δ c13 δ c14 δ c15 δ) (14) 6 011 875, Jan. 4, 2000. Tech. Rep., Eastman Kodak Company.
[13] J. Núñez, X. Otazu, O. Fors, A. Prades, V. Palà, and R. Arbiol,
“Multiresolution-based image fusion with additive wavelet decomposi-
and the coefficient set {gi } for the two GS methods, with tion,” IEEE Trans. Geosci. Remote Sens., vol. 37, no. 3, pp. 1204–1211,
reference to the scheme of Fig. 1, is given by May 1999.
[14] T. Ranchin and L. Wald, “Fusion of high spatial and spectral resolution
images: The ARSIS concept and its implementation,” Photogramm. Eng.
gi = c1,i+1 . (15) Remote Sens., vol. 66, no. 1, pp. 49–61, Jan. 2000.
AIAZZI et al.: IMPROVING CS PANSHARPENING THROUGH MULTIVARIATE REGRESSION 3239

[15] B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli, “Context-driven fu- Stefano Baronti (M’98) was born in Florence, Italy,
sion of high spatial and spectral resolution data based on oversampled in 1954. He received the Laurea degree in elec-
multiresolution analysis,” IEEE Trans. Geosci. Remote Sens., vol. 40, tronic engineering from the University of Firenze,
no. 10, pp. 2300–2312, Oct. 2002. Florence, in 1980.
[16] T. Ranchin, B. Aiazzi, L. Alparone, S. Baronti, and L. Wald, “Im- He was with the Italian Highway Company. He
age fusion—The ARSIS concept and some successful implementation joined the National Research Council of Italy in
schemes,” ISPRS J. Photogramm. Remote Sens., vol. 58, no. 1/2, pp. 4–18, 1985, as a Researcher of the Institute of Applied
Jun. 2003. Physics “Nello Carrara” CNR Area di Ricerca di
[17] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, “MTF- Firenze (formerly, IROE-CNR), Florence, where he
tailored multiscale fusion of high-resolution MS and Pan imagery,” is currently a Senior Researcher. After an initial
Photogramm. Eng. Remote Sens., vol. 72, no. 5, pp. 591–596, May 2006. period spent on computer vision applications in the
[18] M. Gonzáles Audícana, J. L. Saleta, R. García Catalán, and R. Garcia, framework of European Union (ESPRIT) and Italian projects, he moved toward
“Fusion of multispectral and panchromatic images using improved IHS remote sensing by participating in and as the Head of several projects funded
and PCA mergers based on wavelet decomposition,” IEEE Trans. Geosci. by the Italian, French, and European Space Agencies. His research interests
Remote Sens., vol. 42, no. 6, pp. 1291–1299, Jun. 2004. include computer vision applications, image compression, processing of optical
[19] T.-M. Tu, S.-C. Su, H.-C. Shyu, and P. S. Huang, “A new look at IHS-like and microwave remote sensing SAR images, and fusion and quality assessment
image fusion methods,” Inf. Fusion, vol. 2, no. 3, pp. 177–186, Sep. 2001. of remote sensing data. He is the coauthor of nearly 50 papers published in
[20] L. Alparone, S. Baronti, A. Garzelli, and F. Nencini, “Landsat ETM+ international peer-reviewed journals.
and SAR image fusion based on generalized intensity modulation,” IEEE Mr. Baronti is a member of the IEEE Geoscience and Remote Sensing
Trans Geosci. Remote Sens., vol. 42, no. 12, pp. 2832–2839, Dec. 2004. Society (GRSS) and the IEEE Signal Processing Society and participates in
[21] X. Otazu, M. Gonzáles Audícana, O. Fors, and J. Núñez, “Introduction the GRSS Technical Committees on Data Fusion and on Data Archiving and
of sensor spectral response into image fusion methods. Application to Distribution. He is the recipient of the IEEE GRSS 2005 Letter Prize Paper
wavelet-based methods,” IEEE Trans. Geosci. Remote Sens., vol. 43, Award and the IEEE Geoscience and Remote Sensing Society Certificate of
no. 10, pp. 2376–2385, Oct. 2005. Appreciation as the winner of the 2006 Data Fusion Contest, Fusion on Mul-
[22] S. M. Ross, Introduction to Probability and Statistics for Engineers and tispectral and Panchromatic Images, from the IEEE Geoscience and Remote
Scientists. Burlington, MA: Elsevier Academic, 2004. Sensing Society’s Data Fusion Technical Committee.
[23] L. Wald, T. Ranchin, and M. Mangolini, “Fusion of satellite images of
different spatial resolutions: Assessing the quality of resulting images,”
Photogramm. Eng. Remote Sens., vol. 63, no. 6, pp. 691–699, 1997. Massimo Selva was born in Florence, Italy in 1974.
[24] L. Alparone, S. Baronti, A. Garzelli, and F. Nencini, “A global quality He received the Laurea degree (summa cum laude)
measurement of Pan-sharpened multispectral imagery,” IEEE Geosci. in electronic engineering from the University of
Remote Sens. Lett., vol. 1, no. 4, pp. 313–317, Oct. 2004. Florence, Florence, in 2001.
[25] N. T. Lloyd and D. Bau, Numerical Linear Algebra, 3rd ed. Philadelphia, He is with the Institute of Applied Physics “Nello
PA: Soc. Ind. Appl. Math., 1997. Carrara” CNR Area di Ricerca di Firenze (IFAC-
CNR), Florence, working on remote sensing projects
funded by the Italian and European Space Agencies
Bruno Aiazzi was born in Florence, Italy, in 1961. and on activities for cultural heritage conservation as
He received the Laurea degree in electronic engi- well. His main scientific interests include multireso-
neering from the University of Florence, Florence, lution image analysis, data fusion, and image quality
in 1991. assessment. He is an expert of C++, MATLAB, ENVI-IDL, and algorithm
He was with the Italian Space Agency (ASI), implementation aspects. He is a supporter of the Google search engine para-
working on SAR image analysis and classification, digm for verifying research ideas, solving information technology problems,
and with the European Space Agency (ESA), Centre and collecting data. He also likes investigating whether Jungian Psychoanalysis
National d’Etudes Spatiales (CNES), and the ASI, could be useful in improving people’s research ability.
working on several international research projects on
remote sensing research topics such as image quality
definition and measurement, advanced methods for
lossless and near-lossless data compression, and data fusion algorithms for
environmental applications. After a period invested as a Research Assistant,
since 2001, he has been a Researcher with the Institute of Applied Physics
“Nello Carrara” CNR Area di Ricerca di Firenze (IFAC-CNR), Florence. He
is the coauthor of about 30 papers published in international peer-reviewed
journals.
Mr. Aiazzi received a Fellowship on Digital Image Processing and Compres-
sion, supported by the Italian National Research Council (CNR), in 1992 and
the IEEE Geoscience and Remote Sensing Society Certificate of Appreciation
as the winner of the 2006 Data Fusion Contest, Fusion on Multispectral
and Panchromatic Images, from the IEEE Geoscience and Remote Sensing
Society’s Data Fusion Technical Committee.