See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221912997

Survey of Multispectral Image Fusion Techniques in Remote

Sensing Applications

Chapter · June 2011

DOI: 10.5772/10548 · Source: InTech

CITATIONS READS

28 556

4 authors:

Dong Jiang Dafang Zhuang

Chinese Academy of Sciences Chinese Academy of Sciences
116 PUBLICATIONS   1,416 CITATIONS    137 PUBLICATIONS   4,281 CITATIONS   

SEE PROFILE SEE PROFILE

Yaohuan Huang Jingying Fu

State Key Laboratory of Resources and Environmental Informat… Chinese Academy of Sciences
42 PUBLICATIONS   799 CITATIONS    57 PUBLICATIONS   762 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Environmental Cancer View project

National Natural Science Foundation of China (41402288), the Special Fund for Basic Scientific Research of Central Colleges (NO.
2014G1271060), Chang’an University, China, and the High Resolution Earth Observation Systems of National Science and Technology
Major Project (Grant no. 30-Y30B13-9003-14/16-04). View project

All content following this page was uploaded by Dong Jiang on 04 June 2014.

The user has requested enhancement of the downloaded file.

 1

Survey of Multispectral Image Fusion

Techniques in Remote Sensing Applications
Dong Jiang, Dafang Zhuang, Yaohuan Huang and Jinying Fu
Data Center for Resources and Environmental Sciences,
State Key Lab of Resources and Environmental Information System,
Institute of Geographical Sciences and Natural Resources Research,
Chinese Academy of Sciences, Beijing,
China

1. Introduction
1.1 Definition of image fusion
With the development of multiple types of biosensors, chemical sensors, and remote
sensors on board satellites, more and more data have become available for scientific
researches. As the volume of data grows, so does the need to combine data gathered from
different sources to extract the most useful information. Different terms such as data
interpretation, combined analysis, data integrating have been used. Since early 1990’s,
“Data fusion” has been adopt and widely used. The definition of data fusion/image
fusion varies. For example:
- Data fusion is a process dealing with data and information from multiple sources to
achieve refined/improved information for decision making (Hall 1992)[1].
- Image fusion is the combination of two or more different images to form a new image by using a
certain algorithm (Genderen and Pohl 1994 )[2].
- Image fusion is the process of combining information from two or more images of a scene into a
single composite image that is more informative and is more suitable for visual perception or
computer processing. (Guest editorial of Information Fusion, 2007)[3].
- Image fusion is a process of combining images, obtained by sensors of different wavelengths
simultaneously viewing of the same scene, to form a composite image. The composite image is
formed to improve image content and to make it easier for the user to detect, recognize, and
identify targets and increase his situational awareness. 2010.
(http://www.hcltech.com/aerospace-and-defense/enhanced-vision-system/).
Generally speaking, in data fusion the information of a specific scene acquired by two or
more sensors at the same time or separate times is combined to generate an interpretation of
the scene not obtainable from a single sensor [4]. Image fusion is a component of data fusion
when data type is strict to image format (Figure 1). Image fusion is an effective way for
optimum utilization of large volumes of image from multiple sources. Multiple image
fusion seeks to combine information from multiple sources to achieve inferences that are not
feasible from a single sensor or source. It is the aim of image fusion to integrate different
data in order to obtain more information than can be derived from each of the single sensor
data alone (`1+1=3’)[4].
2 Image Fusion and Its Applications

Data fusion

Image fusion

Satellite
image fusion

Fig. 1. Illustration of relationship of data fusion and image fusion

The literature on data fusion in computer vision, machine intelligence and medical imaging
is substantial, but will not be discussed here. This chapter focused on multi-sensor data
fusion in satellite remote sensing area. The fusion of information from sensors with different
physical characteristics enhances the understanding of our surroundings and provides the
basis for planning, decision-making, and control of autonomous and intelligent machines
[1].

1.2 Advance of image fusion

In the past decades it has been applied to different fields such as pattern recognition,
visual enhancement, object detection and area surveillance [4].In 1997, Hall and Llinas
gave a general introduction to multi-sensor data fusion [1]. Another in-depth review
paper on multiple sensors data fusion techniques was published in 1998 [4]. This paper
explained the concepts, methods and applications of image fusion as a contribution to
multi-sensor integration oriented data processing. Since then, image fusion has received
increasing attention. Further scientific papers on image fusion have been published with
an emphasis on improving fusion quality and finding more application areas. As a case in
point, Simone et al. describe three typical applications of data fusion in remote sensing,
such as obtaining elevation maps from synthetic aperture radar (SAR) interferometers, the
fusion of multi-sensor and multi-temporal images, and the fusion of multi-frequency,
multi-polarization and multi-resolution SAR images [5]. Vijayaraj provided the concepts
of image fusion in remote sensing applications [6]. Quite a few survey papers have been
published recently, providing overviews of the history, developments, and the current
state of the art of image fusion in the image-based application fields [7-9], but recent
development of multi-sensor data fusion in remote sensing fields has not been discussed
in detail. The objectives of this paper are to present an overview of new advances in
multi-sensor satellite image fusion, focused on its main application fields in remote
sensing.
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 3

Data source Objective Authors Time

AutomaticOlivier Thepaut, Kidiyo Kpalma,
SPOT HRV & ERS SAR 1994
registration Joseph Ronsin [10]
Tamar Peli, Mon Young, Robert
Hyperspectral image & SAR Automatic target
Knox, Ken Ellis, Fredrick 1999
image cueing
Bennet[11]
G. Simone, A. Farina, F.C.
Multifrequency, Land use
Morabito, S.B. Serpico, L. 2001
multipolarization SAR images classification
Bruzzone[5]
Landsat ETM+ Pan band & Methods Marcia L.S. Aguena, Nelson D.A.
2006
CBERS-1 multiple spectral data comparison Mascarenhas[12]
Urban sprawl Ying Lei, Dong Jiang, and Xiaohuan
Landsat ETM+ & MODIS 2007
monitoring Yang[13]
AVIRIS and LIDAR Coastal mapping Ahmed F. Elaksher[14] 2008
Table 1. Examples of application of image fusion

1.3 Categorization of image fusion techniques

Image fusion can be performed roughly at four different stages: signal level, pixel level,
feature level, and decision level. Figure 2 illustrates of the concept of the four different
fusion levels [15].

Fig. 2. An overview of categorization of the fusion algorithms [15].

4 Image Fusion and Its Applications

1. Signal level fusion. In signal-based fusion, signals from different sensors are combined
to create a new signal with a better signal-to noise ratio than the original signals.
2. Pixel level fusion. Pixel-based fusion is performed on a pixel-by-pixel basis. It generates
a fused image in which information associated with each pixel is determined from a set
of pixels in source images to improve the performance of image processing tasks such
as segmentation
3. Feature level fusion. Feature-based fusion at feature level requires an extraction of
objects recognized in the various data sources. It requires the extraction of salient
features which are depending on their environment such as pixel intensities, edges or
textures. These similar features from input images are fused.
4. Decision-level fusion consists of merging information at a higher level of abstraction,
combines the results from multiple algorithms to yield a final fused decision. Input
images are processed individually for information extraction. The obtained information
is then combined applying decision rules to reinforce common interpretation.

2. Advance in image fusion techniques

During the past two decades, several fusion techniques have been proposed. Most of these
techniques are based on the compromise between the desired spatial enhancement and the
spectral consistency. Among the hundreds of variations of image fusion techniques, the
widely used methods include, but are not limited to, intensity-hue-saturation (IHS), high-
pass filtering, principal component analysis (PCA), different arithmetic combination(e.g.
Brovey transform), multi-resolution analysis-based methods (e.g. pyramid algorithm,
wavelet transform), and Artificial Neural Networks (ANNs), etc. The chapter will provide a
general introduction to those selected methods with emphases on new advances in the
remote sensing field.

2.1 Traditional fusion algorithms

The PCA transform converts inter-correlated multi-spectral (MS) bands into a new set of
uncorrelated components. To do this approach first we must get the principle components
of the MS image bands. After that, the first principle component which contains the most
information of the image is substituted by the panchromatic image. Finally the inverse
principal component transform is done to get the new RGB (Red, Green, and Blue) bands of
multi-spectral image from the principle components.
The intensity-hue-saturation (HIS) fusion converts a color MS image from the RGB space
into the IHS color space. The HIS components can be defined as follows:

I = ( R+ G+ B)/3 (1)

H= (B-R)/3(I-R), S=1-R/I, when R= Minimum (R, G, B) (2)

H= (R-G)/3(I-G), S=1-G/I, when G= Minimum (R, G, B) (3)

H= (G-B)/3(I-B), S=1-B/I, when B= Minimum (R, G, B) (4)

Were I,H,S stand for intensity, hue and saturation components respectively; R, G, B mean
Red, Green, and Blue bands of multi-spectral image.
Because the intensity (I) band resembles a panchromatic (PAN) image, it is replaced by a
high-resolution PAN image in the fusion. A reverse IHS transform is then performed on
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 5

the PAN together with the hue (H) and saturation (S) bands, resulting in an IHS fused
image.
Different arithmetic combinations have been developed for image fusion. The Brovey
transform, Synthetic Variable Ratio (SVR), and Ratio Enhancement (RE) techniques are some
successful examples [9]. The basic procedure of the Brovey transform first multiplies each
MS band by the high resolution PAN band, and then divides each product by the sum of the
MS bands. The algorithm is shown in equation (5) .

DNfused = DNpan× DNb1 / (DNb1+ DNb2+ DNb3) (5)

Where DNfused means the digital number(DN) of the resulting fused image; DNb1, DNb2 and
DNb3 stand for pixel values of three bands of multiple spectral image; DNpan stand for pixel
values of high resolution Pan band.
The SVR and RE techniques are similar, but involve more sophisticated calculations for the
MS sum for better fusion quality. For example (Fig.3 ), Spot 5 Pan band data with spatial
resolution of 2.5m of Yanqing city, Beijing China, in 2005 was fused with multiple spectral
bands of Landsat TM data (spatial resolution:30m) in 2007. A simple Brovey transformation
fusion method was used and the 3rd, 4th, 7th bands of TM were selected for calculation. The
building areas remained unchanged from 2005-2007 were grey-purple, meanwhile, the
newly established buildings were highlighted (lime color in Figure 3) in the composed
image and could be easily detected.

New
buildings

Old
buildings

Fig. 3. An example of Brovey transform based image fusion

Traditional fusion algorithms mentioned above have been widely used for relatively simple
and time efficient fusion schemes. However, several problems must be considered before
their application: (1) These fusion algorithms generate a fused image from a set of pixels in
the various sources. These pixel-level fusion methods are very sensitive to registration
accuracy, so that co-registration of input images at sub-pixel level is required; (2) One of the
main limitations of HIS and Brovey transform is that the number of input multiple spectral
bands should be equal or less than three at a time; (3) These image fusion methods are often
successful at improves the spatial resolution, however, they tend to distort the original
spectral signatures to some extent [16,17]. More recently new techniques such as the wavelet
6 Image Fusion and Its Applications

transform seem to reduce the color distortion problem and to keep the statistical parameters
invariable.

2.2 Multi-resolution analysis-based methods

Multi-resolution or multi-scale methods, such as pyramid transformation, have been
adopted for data fusion since the early 1980s [18]. The Pyramid-based image fusion
methods, including Laplacian pyramid transform, were all developed from Gaussian
pyramid transform, have been modified and widely used [19,20].
In 1989, Mallat put all the methods of wavelet construction into the framework of functional
analysis and described the fast wavelet transform algorithm and general method of
constructing wavelet orthonormal basis. On the basis, wavelet transform can be really
applied to image decomposition and reconstruction [21-23]. Wavelet transforms provide a
framework in which an image is decomposed, with each level corresponding to a coarser
resolution band. For example, in the case of fusing a MS image with a high-resolution PAN
image with wavelet fusion, the Pan image is first decomposed into a set of low-resolution
Pan images with corresponding wavelet coefficients (spatial details) for each level.
Individual bands of the MS image then replace the low-resolution Pan at the resolution level
of the original MS image. The high resolution spatial detail is injected into each MS band by
performing a reverse wavelet transform on each MS band together with the corresponding
wavelet coefficients (Figure 4).

Fig. 4. Generic flowchart of wavelet-based image fusion

In the wavelet-based fusion schemes, detail information is extracted from the PAN image
using wavelet transforms and injected into the MS image. Distortion of the spectral
information is minimized compared to the standard methods [24]. For example, CBERS
multiple spectral image (Figure 5, a) with spatial resolution of 19.2 m of Yiwu City, Zhejiang
Province, China, in 2007 was fused with CBERS-HR PAN image(Figure 5, b) with spatial
resolution of 2.4 m. Buildings and liner objects (roads,etc.) could be easily identified from
fused images(c).
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 7

(a) CBERS multiple spectral image (b) CBERS-HR PAN image

(c) Fused image

Fig. 5. Example of wavelet-based image fusion
In order to achieve optimum fusion results, various wavelet-based fusion schemes had been
tested by many researchers. Among these schemes several new concepts/algorithms were
presented and discussed. Candes provided a method for fusing SAR and visible MS images
using the Curvelet transformation. The method was proven to be more efficient for
detecting edge information and denoising than wavelet transformation [25]. Curvelet-based
image fusion has been used to merge a Landsat ETM+ panchromatic and multiple-spectral
image. The proposed method simultaneously provides richer information in the spatial and
spectral domains [26]. Donoho et al. presented a flexible multi-resolution, local, and
directional image expansion using contour segments, the Contourlet transform, to solve the
problem that wavelet transform could not efficiently represent the singularity of
linear/curve in image processing [27,28]. Contourlet transform provides flexible number of
directions and captures the intrinsic geometrical structure of images.
In general, as a typical feature level fusion method, wavelet-based fusion could evidently
perform better than convenient methods in terms of minimizing color distortion and
denoising effects. It has been one of the most popular fusion methods in remote sensing in
recent years, and has been standard module in many commercial image processing soft
wares, such as ENVI, PCI, ERDAS. Problems and limitations associated with them include:
(1) Its computational complexity compared to the standard methods; (2) Spectral content of
8 Image Fusion and Its Applications

small objects often lost in the fused images; (3) It often requires the user to determine
appropriate values for certain parameters (such as thresholds). The development of more
sophisticated wavelet-based fusion algorithm (such as Ridgelet, Curvelet, and Contourlet
transformation) could improve the performance results, but these new schemes may cause
greater complexity in the computation and setting of parameters.

2.3 Artificial neural network based fusion method

Artificial neural networks (ANNs) have proven to be a more powerful and self-adaptive
method of pattern recognition as compared to traditional linear and simple nonlinear
analyses [29,30]. The ANN-based method employs a nonlinear response function that
iterates many times in a special network structure in order to learn the complex functional
relationship between input and output training data. The general schematic diagram of the
ANN-based image fusion method can be seen in Figure 6.

Fig. 6. General schematic diagram of the ANN-based image fusion method.

The input layer has several neurons, which represent the feature factors extracted and
normalized from image A and image B. The function of each neuron is a sigmoid function
given by:

(6)

In Figure 6, the hidden layer has several neurons and the output layer has one neuron (or
more neuron). The ith neuron of the input layer connects with the jth neuron of the
hidden layer by weight Wij, and weight between the jth neuron of the hidden layer and
the tth neuron of output layer is Vjt (in this case t = 1). The weighting function is used to
simulate and recognize the response relationship between features of fused image and
corresponding feature from original images (image A and image B). The ANN model is
given as follows:
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 9

(7)

In equation (7), Y=pixel value of fused image exported from the neural network model, q =
number of nodes hidden (q~8 here), Vj=weight between jth hidden node and output node
(in this case, there is only one output node), γ=threshold of the output node, Hj=exported
values from the jth hidden node:

(8)

Where Wij=weight between ith input node and the jth hidden node, ai=values of the ith
input factor, n=number of nodes of input (n~5 here), θj=threshold of the jth hidden node.
As the first step of ANN-based data fusion, two registered images are decomposed into
several blocks with size of M and N (Figure 6). Then, features of the corresponding blocks in
the two original images are extracted, and the normalized feature vector incident to neural
networks can be constructed [31]. The features used here to evaluate the fusion effect are
normally spatial frequency, visibility, and edge. The next step is to select some vector
samples to train neural networks. An ANN is a universal function approximator that
directly adapts to any nonlinear function defined by a representative set of training data.
Once trained, the ANN model can remember a functional relationship and be used for
further calculations. For these reasons, the ANN concept has been adopted to develop
strongly nonlinear models for multiple sensors data fusion. Thomas et al. discussed the
optimal fusion method of TV and infrared images using artificial neural networks [32]. After
that, many neural network models have been proposed for image fusion such as BP, SOFM,
and ARTMAP neural networks. BP algorithm has been mostly used. However, the
convergence of BP networks is slow and the global minima of the error space may not be
always achieved [33]. As an unsupervised network, SOFM network clusters input sample
through competitive learning. But the number of output neurons should be set before
constructing neural networks model [34]. RBF neural network can approximate objective
function at any precise level if enough hidden units are provided. The advantages of RBF
network training include no iteration, few training parameters, high training speed, simply
process and memory functions [35]. Hong explored the way that using RBF neural networks
combined with nearest neighbor clustering method to cluster, and membership weighting is
used to fuse. Experiments show this method can obtain the better effect of cluster fusion
with proper width parameter [36].
Gail et al. used Adaptive Resonance Theory (ART) neural networks to form a new
framework for self-organizing information fusion. The ARTMAP neural network can act as
a self-organizing expert system to derive hierarchical knowledge structures from
inconsistent training data [37]. ARTMAP information fusion resolves apparent
contradictions in input pixel labels by assigning output classes to levels in a knowledge
hierarchy [38]. Rong et al. presented a feature-level image fusion method based on
segmentation region and neural networks. The results indicated that this combined fusion
scheme was more efficient than that of traditional methods [39].
The ANN-based fusion method exploits the pattern recognition capabilities of artificial
neural networks, and meanwhile, the learning capability of neural networks makes it
10 Image Fusion and Its Applications

feasible to customize the image fusion process. Many of applications indicated that the
ANN-based fusion methods had more advantages than traditional statistical methods,
especially when input multiple sensor data were incomplete or with much noises. It is often
served as an efficient decision level fusion tools for its self learning characters, especially in
land use/land cover classification. In addition, the multiple inputs − multiple outputs
framework make it to be a possible approach to fuse high dimension data, such as long-term
time-series data or hyper-spectral data.

2.4 Dempster-Shafer evidence theory based fusion method

Dempster-Shafer decision theory is considered a generalized Bayesian theory, used when
the data contributing to the determination of the analysis of the images is subject to
uncertainty. It allows distributing support for proposition not only to a proposition itself but
also to the union of propositions that include it. Huadong Wu et.al. presented a system
framework that manages information overlap and resolves conflicts, and the system
provides eneralizable architectural support that facilitates sensor fusion [40].
Compared with Bayesian theory, the Dempster-Shafer theory of evidence feels closer to our
human perception and reasoning processes. Its capability to assign uncertainty or ignorance
to propositions is a powerful tool for dealing with a large range of problems that otherwise
would seem intractable [40]. The Dempster-Shafer theory of evidence has been applied on
image fusion using SPOT/HRV image and NOAA/AVHRR series. The results show
unambiguously the major improvement brought by such a data fusion, and the performance
of the proposed method [41]. H. Borotschnig et.al. compared three frameworks for
information fusion and view-planning using different uncertainty calculi: probability
theory, possibility theory and Dempster-Shafer theory of evidence[42]. The results indicated
that Dempster-Shafer decision theory based sensor fusion method will achieve much higher
performance improvement, and it provides estimates of imprecision and uncertainty of the
information derived from different sources

2.5 Multiple algorithm fusion

As a coin has two sides, each fusion method has its own set of advantages and limitations. The
combination of several different fusion schemes has been approved to be the useful strategy
which may achieve better quality of results [16,24]. As a case in point, quite a few researchers
have focused on incorporating the traditional IHS method into wavelet transforms, since the
IHS fusion method performs well spatially while the wavelet methods perform well spectrally
[24,41]. However, selection and arrangement of those candidate fusion schemes are quite
arbitrary and often depends upon the user’s experience. Optimal combining strategy for
different fusion algorithms, in another word, ‘algorithm fusion’ strategy, is thus urgent
needed. Further investigations are necessary for the following aspects: 1)Design of a general
framework for combination of different fusion approaches; 2) Development of new
approaches which can combine aspects of pixel/feature/decision level image fusion;
3)Establishment of automatic quality assessment method for evaluation of fusion results.

3. Applications of image fusion

Remote sensing techniques have proven to be powerful tools for the monitoring of the
Earth’s surface and atmosphere on a global, regional, and even local scale, by providing
important coverage, mapping and classification of land cover features such as vegetation,
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 11

soil, water and forests [5] The volume of remote sensing images continues to grow at an
enormous rate due to advances in sensor technology for both high spatial and temporal
resolution systems. Consequently, an increasing quantity of image data from
airborne/satellite sensors have been available, including multi-resolution images, multi-
temporal images, multi-frequency/spectral bands images and multi-polarization image. The
goal of multiple sensor data fusion is to integrate complementary and redundant
information to provide a composite image which could be used to better understanding of
the entire scene. It has been widely used in many fields of remote sensing, such as object
identification, classification, and change detection. The following paragraphs describe the
recent achievements of image fusion in more detail.

3.1 Object identification

The feature enhancement capability of image fusion is visually apparent in VIR/VIR
combinations that often results in images that are superior to the original data. In order to
maximize the amount of information extracted from satellite image data useful products can
be found in fused images [4]. A Dempster-Shafer fusion method for urban building
detection was presented in 2004. First and last pulse of LIDAR data and multi-spectral aerial
imagery were used. Apart from buildings, the classes ‘tree’, ‘grass land’, and ‘bare soil’ are
also distinguished by a classification method based on the Dempster-Shafer theory of data
fusion. Identification of linear objects such as roads could also benefit from image fusion
techniques. An integrated system for automatic road mapping from high-resolution multi-
spectral satellite imagery by information fusion was discussed by Xiaoying et al. in 2005 [43].
Andrea presents a solution to enhance the spatial resolution of MS images with high-
resolution PAN data. The proposed method exploits the undecimated discrete wavelet
transform, and the vector multi-scale Kalman filter, which is used to model the injection
process of wavelet details. Fusion simulations on spatially degraded data and fusion tests at

Fig. 7. NDVI profile for different crop types.

12 Image Fusion and Its Applications

the full scale reveal that an accurate and reliable PAN-sharpening is achieved by the
proposed method [44]. A case study, which extract crop field using high spatial resolution
image and images with high time repetitiveness, was shown as follows.
Identification of crop types from satellite imagery is a challenging task. Here we present an
automatic approach for planting areas extracting in mixed planting regions around Beijing
city using MODIS data and Landsat TM data. Firstly, planting areas were distinguished
with non-crop areas from Landsat TM image using traditional supervised classifier. Then,
time series NDVI derived from MODIS data were used for indentifying different types of
crops. Because different crop has different growth stage, maximum or minimum value of
crop’s NDVI is not same and it appears in different date.
After investigating the planting structure of main crops and analyzing the NDVI value of
different crop from the middle of March to the middle of November 2002 in Beijing, planting
area of winter wheat, spring maize, summer maize and bean in Beijing has been extracted.

Fig. 8. Spatial distribution of main crops of Beijing in 2002

3.2 Classification
Classification is one of the key tasks of remote sensing applications. The classification accuracy
of remote sensing images is improved when multiple source image data are introduced to the
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 13

processing [4]. Images from microwave and optical sensors offer complementary information
that helps in discriminating the different classes. As discussed in the work of Wang et al., a
multi-sensor decision level image fusion algorithm based on fuzzy theory are used for
classification of each sensor image, and the classification results are fused by the fusion rule.
Interesting result was achieved mainly for the high speed classification and efficient fusion of
complementary information [45]. Land-use/land-cover classification had been improved using
data fusion techniques such as ANN and the Dempster-Shafer theory of evidence. The
experimental results show that the excellent performance of classification as compared to
existing classification techniques [46, 47]. Image fusion methods will lead to strong advances in
land use/land cover classifications by use of the complementary of the data presenting either
high spatial resolution or high time repetitiveness.
For example, Indian P5 Panchromatic image(Figure 9 b) with spatial resolution of 2.18 m of
Yiwu City, Southeast China, in 2007 was fused with multiple spectral bands of China-Brazil
CBERS data (spatial resolution: 19.2m) (Figure 9 a)in 2007. Brovey transformation fusion
method was used.

(a) CBERS multiple spectral image (b) P5 PAN image

(c) Fused image

Fig. 9. Result of image fusion: CBERS MS and P5 PAN
14 Image Fusion and Its Applications

(a) Land use classification based on (b) Land use classification based on
CBERS multiple spectral image fused image
Fig. 10. Land use classification of Yiwu city,2007
Results indicated that the accuracy of residential areas of Yiwu city derived from fused
image is much higher than result derived from CBERS multiple spectral image(Table 2).

Residential and build-up areasAccuracy

Data sources
（km2） （%）
CBERS 86 82
P5 + CBERS 67 92
Statistical data 73 -
Table 2. Comparison of land use classification results

3.3 Change detection

Change detection is the process of identifying differences in the state of an object or
phenomenon by observing it at different times [48]. Change detection is an important process
in monitoring and managing natural resources and urban development because it provides
quantitative analysis of the spatial distribution of the population of interest [49]. Image fusion
for change detection takes advantage of the different configurations of the platforms carrying
the sensors. The combination of these temporal images in same place enhances information on
changes that might have occurred in the area observed. Sensor image data with low temporal
resolution and high spatial resolution can be fused with high temporal resolution data to
enhance the changing information of certain ground objects. Madhavan et al. presented a
decision level fusion system that automatically performs fusion of information from multi-
spectral, multi-resolution, and multi-temporal high-resolution airborne data for a change-
detection analysis. Changes are automatically detected in buildings, building structures, roofs,
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 15

roof color, industrial structures, smaller vehicles, and vegetation [50]. Two examples of
Change detection using image fusion method are shown as follows.
1. Change detection using Landsat ETM+ and MODIS data
Recent study indicated that urban expansion could be efficiently monitored using satellite
images with multi-temporal and multi-spatial resolution. For example, Landsat ETM+
Panchromatic image(Figure 4 a) with spatial resolution of 10 m of Chongqing City,
Southwest China, in 2000 was fused with daily-received multiple spectral bands of MODIS
data (spatial resolution: 250m) (Figure 4 b)in 2006. Brovey transformation fusion method
was used. The building areas remained unchanged from 2000 to 2006 were in grey-pink.
Meanwhile, the newly established buildings were in dark red color in the composed image
(Figure 5) and could be easily identified.

a) ETM image, 2000 b) MODIS image, 2006

Fig. 4. Satellite images of Chongqing City

Fig. 5. Fusion result of multiple sources images of Chongqing City

16 Image Fusion and Its Applications

2. Change detection using former land-cover map and multiple spectral images
In the study area, Qingpu district of Shanghai City,China, two kinds of data were fused for
automatic urban sprawl monitoring, which include land cover map, multiple spectral image
of Environment Satellites1(HJ-1). The land cover map of 2005 was used as prior knowledge
for hyperspace analysis and segment. HJ-1 image of September 22, 2009 were geometric and
radiometric corrected. HJ-1 images consisted of four spectral bands, which are three visible
bands and a near infra-red (NIR) band.
Two data layers were overlapped and spectral DN value of the five kinds of land cover
types were extracted. The results in Figure 3 show that spectral DN value of the five land
cover types most clusters in relevant three-dimensional ellipsoid spaces. Outliers were
considered pixels with higher probability of changed area. Based on three-dimensional
feature space analysis, the map of urban expansion could be achieved.

Fig. 6. Three-dimensional scatter plots and feature space of five kinds of land cover types
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 17

In recent years, object-oriented processing techniques are becoming more popular,

compared to traditional pixel-based image analysis, object-oriented change information is
necessary in decision support systems and uncertainty management strategies. An in-depth
paper presented by Ruvimbo et al. introduced the concept and applications of object-
oriented change detection for urban areas [49]. In general, due to the extensive statistical and
derived information available with the object-oriented approach, a number of change
images can be presented depending on research objectives. In land use and land cover
analysis; this level of precision is valuable as analysis at the object level enables linkage with
other GIS databases or derived socio-economic attributes.

3.4 Maneuvering target tracking

Maneuvering target tracking is a fundamental task in intelligent vehicle research. With the
development of sensor techniques and signal/image processing methods, automatic
maneuvering targets tracking can be conducted operationally. Meanwhile, multi-sensor
fusion is found to be a powerful tool to improve tracking efficiency. The tracking of objects
using distributed multiple sensors is an important field of work in the application areas of
autonomous robotics, military applications, and mobile systems [51].
The numbers of the papers focused on the problem of fusion between radar and image
sensors in targets tracking have appeared in recent years [52,53]. Fusion of radar data and
infrared images could improve the positioning accuracy and narrow down the image
working area [54]. Vahdati-khajeh addressed the multi-target tracking problem for
maneuvering targets in cluttered environments. The multiple scan joint probabilistic data
association (MJPDA) algorithm was used for the sake of overcoming the problem of clutter
points and targets which have joint observation [55]. In order to overcome the defects of the
current statistical model on non-maneuvering target tracking, Chen et al. presented a novel
multi-sensor data fusion algorithm for tracking the large-scale maneuvering target. The
fuzzy adaptive Kalman filtering algorithm with maneuvering detection was used for large-
scale maneuvering target which extracts feature data from Kalman filtering processes to
estimate the magnitude and time of maneuvering. The simulation results showed that the
tracking system with active and passive radar has higher precision than those with a single
sensor for large-scale problems [52].

4. Discussion and conclusions

Multi-sensor image fusion seeks to combine information from different images to obtain
more inferences than can be derived from a single sensor. It is widely recognized as an
efficient tool for improving overall performance in image based application. The chapter
provides a state-of-art of multi-sensor image fusion in the field of remote sensing. Below are
some emerging challenges and recommendations.
1. Improvements of fusion algorithms.
Among the hundreds of variations of image fusion techniques, methods which had be
widely used including IHS, PCA, Brovey transform, wavelet transform, and Artificial
Neural Network (ANN). For methods like HIS, PCA and Brovey transform, which have
lower complexity and faster processing time, the most significant problem is color distortion
[16]. Wavelet-based schemes perform better than those methods in terms of minimizing
color distortion. The development of more sophisticated wavelet-based fusion algorithm
(such as Ridgelet, Curvelet, and Contourlet transformation) could evidently improve
18 Image Fusion and Its Applications

performance result, but they often cause greater complexity in computation and parameters
setting. Another challenge on existing fusion techniques will be the ability for processing
hyper-spectral satellite sensor data. Artificial neural network seem to be one possible
approach to handle the high dimension nature of hyper-spectral satellite sensor data.
2. Establishment of an automatic quality assessment scheme.
Automatic quality assessment is highly desirable to evaluate the possible benefits of fusion,
to determine an optimal setting of parameters for a certain fusion scheme, as well as to
compare results obtained with different algorithms [34]. Mathematical methods were used
to judge the quality of merged imagery in respect to their improvement of spatial resolution
while preserving the spectral content of the data. Statistical indices, such as cross entropy,
mean square error, signal-to-noise ratio, have been used for evaluation purpose. While
recently a few image fusion quality measures have been proposed, analytical studies of
these measures have been lacking. The work of Yin et al. focused on one popular mutual
information-based quality measure and weighted averaging image fusion [56]. Jiying
presented a new metric based on image phase congruency to assess the performance of the
image fusion algorithm [57]. However, in general, no automatic solution has been achieved
to consistently produce high quality fusion for different data sets [58]. It is expected that the
result of fusing data from multiple independent sensors will offer the potential for better
performance than can be achieved by either sensor, and will reduce vulnerability to sensor
specific countermeasures and deployment factors. We expect that future research will
address new performance assessment criteria and automatic quality assessment methods
[59].

5. References
Hall, L.; Llinas, J. (1997).An introduction to multisensor data fusion. Proc. IEEE. Vol.85,pp. 6-
23,ISSN 0018-9219
Genderen, J. L. van, and Pohl, C. Image fusion: Issues, techniques and applications.
Intelligent Image Fusion, Proceedings EARSeL Workshop, Strasbourg, France, 11
September 1994, edited by J. L. van Genderen and V. Cappellini (Enschede: ITC),
18- 26.
Guest editorial. (2007). Image fusion: Advances in the state of the art. Information Fusion.
Vol.8,pp.114-118, ISSN 1566-2535
Pohl, C.; Van Genderen, J.L. (1998). Multisensor image fusion in remote sensing: concepts,
methods and applications. Int. J. Remote Sens.Vol. 19,pp.823-854, ISSN 0143-1161
Simone, G.; Farina, A.; Morabito, F.C.; Serpico, S.B.; Bruzzone, L. (2002). Image fusion
techniques for remote sensing applications. Information Fusion. Vol.3,pp.3-15, ISSN
1566-2535
Vijayaraj, V.; Younan, N.; O’Hara, C.(2006). Concepts of image fusion in remote sensing
applications. In Proceedings of IEEE International Conference on Geoscience and Remote
Sensing Symposium, Denver, CO, USA, July 31–August 4, 2006, pp.3798-3801,
Dasarathy, B.V. (2007). A special issue on image fusion: advances in the state of the art.
Information Fusion. Vol.8,pp.113, ISSN 1566-2535
Smith, M.I.; Heather, J.P. (2005). Review of image fusion technology in 2005. In Proceedings of
Defense and Security Symposium, Orlando, FL, USA, 2005.
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 19

Blum, R.S.; Liu, Z. (2006). Multi-Sensor Image Fusion and Its Applications; special series on
Signal Processing and Communications; CRC Press: Boca Raton, FL, USA, 2006.
Olivier TheÂpaut, Kidiyo Kpalma, Joseph Ronsin. (2000). Automatic registration of ERS and
SPOT multisensory images in a data fusion context. Forest Ecology and Management .
Vol.123,pp.93-100, ISSN 0378-1127
Tamar Peli, Mon Young, Robert Knox, Kenneth K. Ellis and Frederick Bennett, Feature-level
sensor fusion, Proc. SPIE 3719, 1999,332, ISSN 0277-786X
Marcia L.S. Aguena, Nelson D.A. Mascarenhas.(2006). Multispectral image data fusion using
POCS and super-resolution. Computer Vision and Image Understanding
Vol.102,pp.178-187, ISSN 1077-3142
Ying Lei, Dong Jiang, and Xiaohuan Yang .(2007). Appllcation of image fusion in urban
expanding detection. Journal of Geomatics,vol.32,No.3,pp.4-5,ISSN 1007-3817
Ahmed F. Elaksher. (2008). Fusion of hyperspectral images and lidar-based dems for coastal
mapping. Optics and Lasers in Engineering Vol.46,pp.493-498,ISSN 0143-8166
Dai, X.; Khorram, S. (1999). Data fusion using artificial neural networks: a case study on
multitemporal change analysis. Comput. Environ. Urban Syst.Vol.23,pp.19-31,ISSN
0198-9715
Yun, Z. (2004). Understanding image fusion. Photogram. Eng. Remote Sens.Vol.6, pp.657-
661,ISSN 2702-4292
Pouran, B.(2005). Comparison between four methods for data fusion of ETM＋ multispectral
and pan images. Geo-spat. Inf. Sci.Vol.8,pp.112-122,ISSN
Adelson, C.H.; Bergen, J.R.(1984). Pyramid methods in image processing. RCA Eng.
Vol.29,pp.33-41,
Miao, Q.G.; Wang, B.S. (2007). Multi-sensor image fusion based on improved laplacian
pyramid transform. Acta Opti. Sin. Vol.27,pp.1605-1610,ISSN 1424-8220
Xiang, J.; Su, X. (2009). A pyramid transform of image denoising algorithm based on
morphology. Acta Photon. Sin.Vol.38,pp.89-103,ISSN 1000-7032
Mallat, S.G. (1989). A theory for multiresolution signal decomposition: the wavelet
representation. IEEE Trans. Pattern Anal. Mach. Intell.Vol.11,pp.674-693,ISSN 0162-
8828
Ganzalo, P.; Jesus, M.A. (2004). Wavelet-based image fusion tutorial. Pattern Recognit.
Vol.37,pp.1855-1872, ISSN 0031-3203
Ma, H.; Jia, C.Y.; Liu, S. (2005).Multisource image fusion based on wavelet transform. Int. J.
Inf. Technol. Vol. 11,pp 81-91,
Krista, A.; Yun, Z.; Peter, D. (2007).Wavelet based image fusion techniques — An
introduction, review and comparison. ISPRS J. Photogram. Remote Sens. Vol. 62,
pp.249-263.
Candes, E.J.; Donoho, D.L.(2000). Curvelets-A Surprisingly Effective Nonadaptive
Representation for Objects with Edges.Curves and Surfcaces; Vanderbilt University
Press: Nashville, TN, USA, pp.105-120,
Choi, M.; Kim, RY.; Nam, MR. Fusion of multi-spectral and panchromatic satellite images
using the Curvelet transform. IEEE Geosci. Remote Sens. Lett. Vol.2,pp. 136-140,ISSN
0196-2892
20 Image Fusion and Its Applications

Donoho, M.N.; Vetterli, M. (2002).Contourlets; Academic Press: New York, NY, USA, ISSN
0890-5401
Minh, N.; Martin, V.(2005). The contourlet transform: an efficient directional multiresolution
image representation. Available online:
http://lcavwww.epfl.ch/~vetterli/IP-4-2005.pdf (accessed June 29, 2009).
Louis, E.K.; Yan, X.H. (1998).A neural network model for estimating sea surface chlorophyll
and sediments from thematic mapper imagery. Remote Sens. Environ.Vol.,66, pp.153-
165,ISSN 0034-4257
Dong. J.; Yang, X.; Clinton, N.; Wang, N. (2004).An artificial neural network model for
estimating crop yields using remotely sensed information. Int. J. Remote Sens. Vol.
25,pp. 1723-1732,ISSN 0143-1161
Shutao, L.; Kwok, J.T.; Yaonan W.(2002). Multifocus image fusion using artificial neural
networks. Pattern Recognit. Lett. Vol. 23,pp. 985-997.,ISSN 0167-8655
Thomas, F.; Grzegorz, G. (1995).Optimal fusion of TV and infrared images using artificial
neural networks. In Proceedings of Applications and Science of Artificial Neural
Networks, Orlando, FL, USA, April 21, 1995;Vol. 2492, pp.919-925,
Huang, W.; Jing, Z.(2007). Multi-focus image fusion using pulse coupled neural network.
Pattern Recognit. Lett. Vol. 28,pp.1123-1132, ,ISSN 0167-8655
Wu, Y.; Yang, W. (2003).Image fusion based on wavelet decomposition and evolutionary
strategy. Acta Opt. Sin. Vol. 23,pp. 671-676, ISSN 0253-2239
Sun, Z.Z.; Fu, K.; Wu, Y.R. (2003).The high-resolution SAR image terrain classification
algorithm based on mixed double hint layers RBFN model. Acta Electron. Sin. Vol.,
31,pp. 2040-2044,
Zhang, H.; Sun, X.N.; Zhao, L.; Liu, L. (2008).Image fusion algorithm using RBF neural
networks. Lect. Notes Comput. Sci. Vol. 9,pp. 417-424,
Gail, A.; Siegfried, M.; Ogas, J.(2005). Self-organizing information fusion and hierarchical
knowledge discovery- a new framework using ARTMAP neural networks. Neural
Netw. Vol. 18, pp.287-295,
Gail, A.; Siegfried, M.; Ogas, J.(2004). Self-organizing hierarchical knowledge discovery by
an ARTMAP image fusion system. In Proceedings of the 7th International Conference
on Information Fusion, Stockholm, Sweden, 2004; pp. 235-242, ISSN 1210-0552
Wang, R.; Bu, F.L.; Jin, H.; Li, L.H.(2007). A feature-level image fusion algorithm based on
neural networks. Bioinf. Biomed. Eng. Vol. 7,pp. 821-824,
Huadong Wu;Mel Siegel; Rainer Stiefelhagen;Jie Yang.(2002).Sensor Fusion Using
Dempster-Shafer Theory ，IEEE Instrumentation and Measurement Technology
Conference Anchorage, AK, USA, 21-23 May 2002,
S. Le Hégarat-Mascle, D. Richard, C. (2003).Ottlé, Multi-scale data fusion using Dempster-
Shafer evidence theory, Integrated Computer-Aided Engineering, Vol.10,No.1,pp.9-22,
ISSN：1875-8835
H. Borotschnig, L. Paletta, M. Prantl, and A. Pinz, Graz. (1999).A Comparison of
Probabilistic, Possibilistic and Evidence Theoretic Fusion Schemes for Active Object
Recognition. Computing. Vol.62,pp. 293–319,
Survey of Multispectral Image Fusion Techniques in Remote Sensing Applications 21

Jin, X.Y.; Davis, C.H. (2005).An integrated system for automatic road mapping from high-
resolution multi-spectral satellite imagery by information fusion. Information Fusion.
Vol. 6, pp.257-273, ISSN 1566-2535
Garzelli, A.; Nencini, F. (2007).Panchromatic sharpening of remote sensing images using a
multiscale Kalman filter. Pattern Recognit.Vol. 40,pp. 3568-3577, ISSN: 0167-8655
Wu, Y.; Yang, W.(2003). Image fusion based on wavelet decomposition and evolutionary
strategy. Acta Opt. Sin.Vol. 23,pp.671-676, ISSN 0253-2239
Sarkar, A.; Banerjee, A.; Banerjee, N.; Brahma, S.; Kartikeyan, B.; Chakraborty, M.;
Majumder, K.L.(2005). Landcover classification in MRF context using Dempster-
Shafer fusion for multisensor imagery. IEEE Trans. Image Processing , Vol.14,pp.
634-645, ISSN : 1057-7149
Liu, C.P.; Ma, X.H.; Cui, Z.M.(2007). Multi-source remote sensing image fusion classification
based on DS evidence theory. In Proceedings of Conference on Remote Sensing and GIS
Data Processing and Applications; and Innovative Multispectral Technology and
Applications, Wuhan, China, November 15–17, 2007; Vol. 6790, part 2.
Rottensteiner, F.; Trinder, J.; Clode, S.; Kubik, K.; Lovell, B.(2004) .Building detection by
Dempster-Shafer fusion of LIDAR data and multispectral aerial imagery. In
Proceedings of the 17th International Conference on Pattern Recognition, Cambridge, UK,
August 23–26,2004; Vol. 2, pp. 339-342, ISSN: 1001-0920
Ruvimbo, G.;Philippe, D.; Morgan, D.(2009). Object-oriented change detection for the city of
Harare, Zimbabwe. Exp. Syst. Appl.Vol. 36, pp.571-588, ISSN 0013-8703
Madhavan, B.B.; Sasagawa, T.; Tachibana, K.; Mishra, K.(2005). A decision level fusion of
ADS-40, TABI and AISA data. Nippon Shashin Sokuryo Gakkai Gakujutsu Koenkai
Happyo Ronbunshu , Vol.2005, 163-166,
Duncan, S.; Sameer, S. (2006).Approaches to multisensor data fusion in target tracking: a
survey. IEEE Trans. Knowl. Data Eng. Vol. 18,pp. 1696-1710,ISSN 1041-4347
Chen, Y.; Han, C. (2005).Maneuvering vehicle tracking based on multi-sensor fusion. Acta
Autom. Sin.Vol. 31, pp.625-630,
Liu, C.; Feng, X.(2006). An algorithm of tracking a maneuvering target based on ir sensor
and radar in dense environment. J. Air Force Eng. Univ. Vol. 7,pp. 25-28,ISSN 1009-
3516
Zheng, M.; Zhao, Y.; Tian, H.(2006). Maneuvering target tracking based on fusion of multi-
sensor. J. Detect. Control, Vol. 28,pp. 43-45,
Vahdati-khajeh, E. (2004).Tracking the maneuvering targets using multiple scan joint
probabilistic data association algorithm. In Proceedings of the Target Tracking
2004: Algorithms and Applications, IEE, Sussex University, Brighton, UK, January
1, 2004. ISSN : 0537-9989
Chen, Y.; Xue, Z.Y.; Blum, R.S. (2008).Theoretical analysis of an information-based quality
measure for image fusion. Information Fusion ,Vol. 9,pp. 161-175, ISSN 0018-9251
Zhao, J.Y.; Laganiere, R.; Liu, Z.(2006). Image fusion algorithm assessment based on feature
measurement. In Proceedings of the 1st International Conference on Innovative
Computing, Information and Control, Beijing, China, August 30 – September 1, Vol.
2,pp. 701-704,
 22 Image Fusion and Its Applications

Goshtasby, A.; Nikolov, S.(2007). Image fusion: advances in the state of the art. Information
Fusion. Vol. 8, pp.114-118,ISSN 1566-2535
Dong Jiang;Dafang Zhuang,; Yaohuan Huang; Jingying Fu.(2009). Advances in multi-sensor
data fusion: algorithms and applications. Sensors, Vol.9,No.10,pp. 7771- 7784,ISSN
1424-8220

View publication stats