Fast Multispectral Imaging by Spatial Pixel-Binning

and Spectral Unmixing

Abstract:

Multispectral imaging system is of wide application in relevant fields for its capability
in acquiring spectral information of scenes. Its limitation is that, due to the large
number of spectral channels, the imaging process can be quite time consuming when
capturing high-resolution (HR) multispectral images. To resolve this limitation, this
system proposes a fast multispectral imaging framework based on the image sensor
pixel-binning and spectral un-mixing techniques. The framework comprises a fast
imaging stage and a computational reconstruction stage. In the imaging stage only a
few spectral images are acquired in HR, while most spectral images are acquired in
low resolution (LR). The LR images are captured by applying pixel binning on the
image sensor, such that the exposure time can be greatly reduced. In the
reconstruction stage an optimal number of basis spectra are computed and the signal-
dependent noise statistics are estimated. Then the unknown HR images are efficiently
reconstructed by solving a closed-form cost function that models the spatial and
spectral degradations. The effectiveness of the proposed framework is evaluated using
real scene multispectral images. Experimental results validate that, in general, the
method outperforms the state of the arts in terms of reconstruction accuracy, with
additional 20 or more improvement in computational efficiency.

Introduction

Multispectral imaging has attracted much interest because of its wide application in
relevant fields like biomedicine [1], remote sensing [2], color reproduction [3], and
etc.A a filter-based multispectral imaging system can acquire more spectral
information of scenes at the spatial resolution of camera sensor. In such a system, the
filters split the visible spectrum into many spectral channels, at which the camera
acquires images. The filter wheel is installed with a number of filters which are
sequentially rotated into the optical path in the imaging process. Compared with filter
wheel, the tunable filters have no moving parts and can provide rapid and vibration
less light transmission changes. A multispectral image is comprised of multiple
spectral images, which are acquired at different wavelengths. The framework
comprises the fast imaging stage and the computational reconstruction stage. In the
imaging stage, only a few spectral images are acquired at high resolution, and most
spectral images are captured at low-resolution (LR) based on the pixel-binning
operation incorporated in the image sensor. With such a treatment, the exposure time
can be greatly reduced. In the reconstruction stage, the unknown HR spectral images
corresponding to the LR ones are recovered according to the spectral unmixing
principle. Pixel-binning provides a way to improve imaging efficiency at the cost of
spatial resolution degradation, which is worthwhile in some practical applications. In
fact, pixelbinning has been implemented as a fundamental function in some scientific
and industrial cameras.1,2. A framework that employs the sensor pixel-binning and
spectral unmixing techniques is proposed for fast acquisition of HR multispectral
images.
EXISTING SYSTEM

Coded aperture snapshot spectral imager

The coded aperture snapshot spectral imager (CASSI) can acquire

multispectral images at a high frame rate, the image quality is not
satisfactory when the scene is not highly compressive . In the remote
sensing community, pan sharpening has been widely employed to improve the spatial
resolution of multispectral data. The resolution improvement is achieved by fusing the
LR spectral images with an HR panchromatic (PAN) image. With only one HR image
available, pan sharpening methods can produce obvious spectral distortion when
compensating spatial details. To deal with this problem, recent image fusion methods
employ multiple HR spectral images to improve both spatial and spectral resolutions

Disadvantages

Produces obvious spectral distortion when compensating spatial details

PROPOSED SYSTEM

Spatial Pixel-Binning and Spectral Unmixing

A fast multispectral imaging framework that consists of a fast imaging stage and a
computational reconstruction stage. In the imaging stage, the majority of spectral
images are acquired at low resolution such that the total exposure time can be greatly
reduced. In the reconstruction stage, an optimal number of basis spectra are
computed, based on which the problem of high-resolution image reconstruction is
well posed and can be solved in closed form. A framework that employs the sensor
pixel-binning and spectral un-mixing techniques is proposed for fast acquisition of
HR multispectral images. The signal-dependent noise statistics are incorporated in the
framework such that the reconstruction of multispectral images is robust to noise
corruption.

Advantages

Reconstruction accuracy is higher

Computational efficiency is higher

LITERATURE REVIEW

1. Multispectral imaging in biology and medicine: Slices of life

Multispectral imaging (MSI) is currently in a period of transition from its role as an

exotic technique to its being offered in one form or another by all the major
microscopy manufacturers. This is because it provides solutions to some of the major
challenges in fluorescence-based imaging, namely ameliorating the consequences of
the presence of auto fluorescence and the need to easily accommodate relatively high
levels of signal multiplexing. MSI, which spectrally characterizes and
computationally eliminates auto fluorescence, enhances the signal-to-background
dramatically, revealing otherwise obscured targets. While this article concentrates on
examples derived from liquid-crystal tunable filter-based technology, the intent is to
showcase the advantages of multispectral imaging in general. Some technologies used
to generate multispectral images are compatible with only particular optical
configurations, such as point-scanning laser confocal microscopy. Band-sequential
approaches, such as those afforded by liquid-crystal tunable filters (LCTFs), can be
conveniently coupled with a variety of imaging modalities, which, in addition to
fluorescence microscopy, include bright field (non fluorescent) microscopy as well as
small-animal, noninvasive in-vivo imaging. Bright field microscopy is the chosen
format for histopathology, which relies on immunohistochemistry to provide
molecularly resolved clinical information. However, in contrast to fluorescent labels,
multiple chromogens, if they spatially overlap, are much harder to separate and
quantitate, unless MSI approaches are used. In-vivo imaging is a rapidly growing field
with applications in basic biology, drug discovery, and clinical medicine. The
sensitivity of fluorescence-based in-vivo imaging, as with fluorescence microscopy,
can be limited by the presence of significant auto fluorescence, a limitation which can
be overcome through the utilization of MSI.

2. Spectral imaging for remote sensing

Spectral imaging for remote sensing of terrestrial features and objects arose as an
alternative to high-spatial-resolution, large-aperture satellite imaging systems. Early
applications of spectral imaging were oriented toward ground-cover classification,
mineral exploration, and agricultural assessment, employing a small number of
carefully chosen spectral bands spread across the visible and infrared regions of the
electromagnetic spectrum. Improved versions of these early multispectral imaging
sensors continue in use today. A new class of sensor, the hyperspectral imager, has
also emerged, employing hundreds of contiguous bands to detect and identify a
variety of natural and man-made materials. This overview article introduces the
fundamental elements of spectral imaging and discusses the historical evolution of
both the sensors and the target detection and classification applications.

3. Color-accurate image archives using spectral imaging

Digital imaging that includes spectral estimation can overcome limitations of typical
digital photography, such as limited color accuracy and constraints to a predefined
viewing condition or a specific output device. An example includes the use of ICC
color management to generate an archive of images rendered for a specific display or
for a specific printing technology. A spectral image offers enhanced opportunities for
image analysis, art conservation science, lighting design, and an archive that can be
used to relate back to an objects physical properties. The Munsell Color Science
Laboratory at Rochester Institute of Technology is involved in a joint research
program with the National Gallery of Art in Washington, D.C., and the Museum of
Modern Art in New York to develop a spectral-imaging system optimized for artwork
imaging, archiving, and reproduction. This paper summarizes the scientific approach.

4. Geometric calibration of lens and filter distortions for

multispectral filter-wheel cameras

High-fidelity color image acquisition with a multispectral camera utilizes optical

filters to separate the visible electromagnetic spectrum into several passbands. This is
often realized with a computer-controlled filter wheel, where each position is
equipped with an optical bandpass filter. For each filter wheel position, a grayscale
image is acquired and the passbands are finally combined to a multispectral image.
However, the different optical properties and non-coplanar alignment of the filters
cause image aberrations since the optical path is slightly different for each filter wheel
position. As in a normal camera system, the lens causes additional wavelength-
dependent image distortions called chromatic aberrations. When transforming the
multispectral image with these aberrations into an RGB image, color fringes appear,
and the image exhibits a pincushion or barrel distortion. In this paper, we address both
the distortions caused by the lens and by the filters. Based on a physical model of the
bandpass filters, we show that the aberrations caused by the filters can be modeled by
displaced image planes. The lens distortions are modeled by an extended pinhole
camera model, which results in a remaining mean calibration error of only 0.07 pixels.
Using an absolute calibration target, we then geometrically calibrate each passband
and compensate for both lens and filter distortions simultaneously. We show that both
types of aberrations can be compensated and present detailed results on the remaining
calibration errors.

5. Multispectral image out-of-focus deblurring using interchannel

correlation

Out-of-focus blur occurs frequently in multispectral imaging systems when the

camera is well focused at a specific (reference) imaging channel. As the effective
focal lengths of the lens are wavelength dependent, the blurriness levels of the images
at individual channels are different. This paper proposes a multispectral image
deblurring framework to restore out-offocus spectral images based on the
characteristic of interchannel correlation (ICC). The ICC is investigated based on the
fact that a high-dimensional color spectrum can be linearly approximated using rather
a few number of intrinsic spectra. In the method, the spectral images are classified
into an out-of-focus set and a well-focused set via blurriness computation. For each
out-offocus image, a guiding image is derived from the well-focused spectral images
and is used as the image prior in the deblurring framework. The out-of-focus blur is
modeled as a Gaussian point spread function (PSF), which is further employed as the
blur kernel prior. The regularization parameters in the image deblurring framework
are determined using generalized cross validation (GCV), and thus the proposed
method does not need any parameter tuning. Experimental results validate that the
method performs well on multispectral image deblurring and outperforms the state-of-
the-arts.

6. Multispectral color image capture using a liquid crystal tunable

filter

In this article we describe the experimental setup of a multispectral image acquisition

system consisting of a professional monochrome CCD camera and a tunable filter in
which the spectral transmittance can be controlled electronically. We have performed
a spectral characterisation of the acquisition system taking into account the
acquisition noise. To convert the camera output signals to device-independent data,
two main approaches are proposed and evaluated. One consists in applying regression
methods to convert from the K camera outputs to a device-independent colour space
such as CIEXYZ or CIELAB. Another method is based on a spectral model of the
acquisition system. By inverting the model using a Principal Eigenvector approach,
we estimate the spectral reflectance of each pixel of the imaged surface.

7. Multispectral imaging systems using acoustooptic tunable filter

This paper discusses recent activities of the Jet Propulsion Laboratory (JPL) in the
development of a new type of remote sensing multispectral imaging instrument using
acousto- optic tunable filter (AOTF) as a programmable bandpass filter. This remote
sensor filter provides real-time operation; observational flexibility; measurements of
spectral, spatial, and polarization information using a single instrument; and compact,
solid state structure without moving parts. An AOTF multispectral imaging prototype
system for outdoor field experiments was designed and assembled. Some preliminary
experimental results are reported. The field system is used to investigate spectral and
polarization signatures of natural and man-made objects for evaluation of the
technological feasibility for remote sensing applications. In addition, an airborne
prototype instrument is currently under development.

8. Compressive coded aperture spectral imaging: An introduction

Imaging spectroscopy involves the sensing of a large amount of spatial information

across a multitude of wavelengths. Conventional approaches to hyperspectral sensing
scan adjacent zones of the underlying spectral scene and merge the results to construct
a spectral data cube. Push broom spectral imaging sensors, for instance, capture a
spectral cube with one focal plane array (FPA) measurement per spatial line of the
scene [1], [2]. Spectrometers based on optical bandpass filters sequentially scan the
scene by tuning the bandpass filters in steps. The disadvantage of these techniques is
that they require scanning a number of zones linearly in proportion to the desired
spatial and spectral resolution. This article surveys compressive coded aperture
spectral imagers, also known as coded aperture snapshot spectral imagers (CASSI)
[1], [3], [4], which naturally embody the principles of compressive sensing (CS) [5],
[6]. The remarkable advantage of CASSI is that the entire data cube is sensed with
just a few FPA measurements and, in some cases, with as little as a single FPA shot.

9. Frame-transfer CMOS active pixel sensor with pixel binning

The first frame-transfer CMOS active pixel sensor (APS) is reported. The sensor
architecture integrates an array of active pixels with an array of passive memory cells.
Charge integration amplifer-based readout of the memory cells permits binning of
pixels for variable resolution imaging. A 32/spl times/32 element prototype sensor
with 24-/spl mu/m pixel pitch was fabricated in 1.2-/spl mu/m CMOS and
demonstrated.

10. Algorithmic stability and sanity-check bounds for leave-one-out

cross-validation

In this paper we prove sanity-check bounds for the error of the leave-one-out
crossvalidation estimate of the generalization error: that is, bounds showing that the
worst-case error of this estimate is not much worse than that of the training error
estimate. The name sanity-check refers to the fact that although we often expect the
leave-one-out estimate to perform considerably better than the training error estimate,
we are here only seeking assurance that its performance will not be considerably
worse. Perhaps surprisingly, such assurance has been given only for rather limited
cases in the prior literature on cross-validation. Any nontrivial bound on the error of
leave-one-out must rely on some notion of algorithmic stability. Previous bounds
relied on the rather strong notion of hypothesis stability, whose application was
primarily limited to nearest-neighbor and other local algorithms. Here we introduce
the new and weaker notion of error stability, and apply it to obtain sanity-check
bounds for leave-one-out for other classes of learning algorithms, including training
error minimization procedures and Bayesian algorithms. We also provide lower
bounds demonstrating the necessity of error stability for proving bounds on the error
of the leave-one-out estimate, and the fact that for training error minimization
algorithms, in the worst case such bounds must still depend on the Vapnik-
Chervonenkis dimension of the hypothesis class.
MODULES

Estimating Noise Statistics

Which assumes signal-independent additive white Gaussian noise in individual

spectral images -this work adopts the more reasonable assumption that image
noise is signal-dependent.

Solving Coefficient Matrix

The Sylvester matrix equation, which has also been employed in image fusion,
the noise statistics expressed in the covariance matrices are signal-dependent.
And by adopting an optimal number of basis spectra is always well-posed and a
closed-form solution can be obtained.

Determining Optimal Number of Basis Spectra

In this work the optimal value of K is determined by cross validation as

described one validation image which is a row vector of matrix Z, is selected
from the acquired l HR spectral images