Image

Processing
The Fundamentals
Image
Processing
The Fundamentals

Maria Petrou
University of SurreN Guildford, UK

Panagiota Bosdogianni
Technical Universify of Crete, Chania, Greece

JOHN WILEY & SONS, LTD

Chichester - New York WeinheimBrisbaneSingaporeToronto
Copyright 0 1999 by John Wiley & Sons Ltd,
Baffis Lane, Chichester,
West SussexP019 IUD, England
National 01243 779711
International (+44) 1243 779177
e-mail (for orders and customer service enquiries): cs-books@wiley.co.uk

Visit ourHome Page on http://www.wiley.co.uk or http://www.wiley.com

All Rights Reserved. No partof this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form orby any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the
terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright
Licensing Agency, 90 Tottenham Court Road, London, UK W1P 9HE, without the permission in writing of the
Publisher, with the exceptionof any material supplied specifically for the purpose of being entered and executed on
a computer system, for exclusive by usethe purchaserof the publication.

Designations used by companiesto distinguish their products are often claimed

as trademarks. In all instances
where John Wiley& Sons is aware of a claim, the product name appear in initial capital or all capital letters.
Readers, however, should contactthe appropriate companies for more complete information regarding trademarks
and registration.

Other Wiley EditorialOfices

John Wiley & Sons, Inc., 605 Third Avenue,

New York, NY10158-0012,USA

WLEY-VCH Verlag GmbH, Pappelallee 3,

D69469 Weinheim, Germany

Jacaranda Wiley Ltd,33 Park Road, Milton,

Queensland 4064, Australia

John Wiley & Sons (Asia) Pte Ltd, 2 ClementiLoop #02-01,

Jin Xing Distripark, Singapore 129809

John Wiley & Sons (Canada) Ltd, 22 Worcester Road

Rexdale, Ontario, M9W 1L1, Canada

Library of Congress Cataloging-in-PublicationData

Petrou,Maria
Image Processing : the fundamentals / Maria Petrou, Panagiota
Bosdogianni.
cm.
p.
Includes bibliographical references.
ISBN 0-471-99883-4 (alk. paper)
I. Image
processing-Digital techniques. 1. Bosdogianni,
Panagiota. II. Title.
TA1637.P48 1999
621.36 ' 7 4 ~ 2 1 99-32327
CIP

British Library Cataloguingin Publication Data

A catalogue record for
this book is available from the British Library
ISBN
0-471-99883-4
Produced from PostScript files suppliedby the authors.
Printed and bound in Great Britainby Bookcraft (Bath) Limited.
at least two
This book is printed onacid-free paper responsibly manufactured from sustainable forestry, in which
trees are planted for each one used for paper production.
To the Costases
in our lives
 Contents

Preface xv

List of Figures xvii

1 Introduction 1
Why do we process images? . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What is an image? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What is the brightness of an image at a pixel position? . . . . . . . . . . . . 2
Why are images often quoted as being 512 X 512. 256 X 256. 128 X 128 etc? 2
How many bits do we need to store an image? . . . . . . . . . . . . . . . . . 2
What is meant by image resolution? . . . . . . . . . . . . . . . . . . . . . . 2
How do we do Image Processing? . . . . . . . . . . . . . . . . . . . . . . . . 6
What is a linear operator? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
How are operators defined? . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
How does an operator transform an image? . . . . . . . . . . . . . . . . . . 6
What is the meaning of the point spread function? . . . . . . . . . . . . . . 7
How can we express in practice the effect of a linear operator on an image? 9
What is the implication of the separability assumption on the structure of
matrix H? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
How can a separable transform be written in matrix form? . . . . . . . . . 15
What is the meaning of the separability assumption? . . . . . . . . . . . . . 15
What is the “take home” message of this chapter? . . . . . . . . . . . . . . 18
What is the purpose of Image Processing? . . . . . . . . . . . . . . . . . . . 18
What is this book about? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

tions 2 Image
What is thischapterabout? . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
How can we define an elementaryimage? . . . . . . . . . . . . . . . . . . . 21
What is the outer product of two vectors? . . . . . . . . . . . . . . . . . . . 21
How can we expand an image in terms of vector outer products? . . . . . . 21
What is a unitarytransform? . . . . . . . . . . . . . . . . . . . . . . . . . . 23
What is a unitarymatrix? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
What is the inverse of a unitary transform? . . . . . . . . . . . . . . . . .. 24
How can we construct a unitary matrix? . . . . . . . . . . . . . . . . . . . . 24

vii
viii Image Processing:
FundamentalsThe

How should we choose matrices U and V so that g can be represented by

fewer bits than f ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
How can we diagonalize a matrix? . . . . . . . . . . . . . . . . . . . . . . . 24
How can we compute matrices U . V and A + needed for the image
diagonalization? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
What is the singular value decomposition of an image? . . . . . . . . . . . . 34
How can we approximate an image using SVD? . . . . . . . . . . . . . . . . 34
What is the error of the approximation of an image by SVD? . . . . . . . . 35
How can we minimize the error of the reconstruction? . . . . . . . . . . . . 36
What are the elementary images in terms of which SVD expands an image? 37
Are there any sets of elementary images in terms of which ANY image can
beexpanded? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
What is a complete and orthonormal set of functions? . . . . . . . . . . . . 45
Are there any complete sets of orthonormal discrete valued functions? . . . 46
How are the Haar functions defined? . . . . . . . . . . . . . . . . . . . . . . 46
How are the Walsh functions defined? . . . . . . . . . . . . . . . . . . . . . 47
How can we create the image transformation matrices from the Haar and
Walsh functions? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
What do the elementary images of the Haar transform look like? . . . . . . 51
Can we define an orthogonal matrix with entries only +l or - l ? . . . . . . 57
What do the basis images of the Hadamard/Walsh transform look like? . . 57
What are the advantages and disadvantages of the Walsh and the Haar
transforms? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
What is the Haar wavelet? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
What is the discrete version of the Fourier transform? . . . . . . . . . . . . 63
How can we write the discrete Fourier transform in matrix form? . . . . . . 65
Is matrix U used for DFT unitary? . . . . . . . . . . . . . . . . . . . . . . . 66
Which are the elementary images in terms of which DFT expands an image? 68
Why is the discrete Fourier transform more commonly used than the other
transforms? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
What does the convolution theorem state? . . . . . . . . . . . . . . . . . . . 72
How can we display the discrete Fourier transform of an image? . . . . . . . 79
What happens to the discrete Fourier transform of an image if the image is
rotated? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
What happens to the discrete Fourier transform of an image if the image is
shifted? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
What is the relationship between the average value of a function and its DFT? 82
What happens to the DFT of an image if the image is scaled? . . . . . . . . 83
What is the discrete cosine transform? . . . . . . . . . . . . . . . . . . . . . 86
What is the “take home” message of this chapter? . . . . . . . . . . . . . . 86

3 Statistical Description of Images 89

What is this chapter about? . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Why do we need the statistical description of images? . . . . . . . . . . . . 89
 Contents ix

Is there an image transformation that allows its representation in terms of

uncorrelated data that can be used to approximate the image in the
leastmeansquareerror sense? . . . . . . . . . . . . . . . . . . . . . . 89
What is a random field? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
What is a random variable? . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
How do we describerandom variables? . . . . . . . . . . . . . . . . . . . . . 90
What is the probability of an event? . . . . . . . . . . . . . . . . . . . . . . 90
What is the distribution function of a random variable? . . . . . . . . . . . 90
What is the probability of a random variable taking a specific value? . . . . 91
What is the probability density function of a random variable? . . . . . . . 91
How do we describemanyrandom variables? . . . . . . . . . . . . . . . . . 92
What relationships may n random variables have with each other? . . . . . 92
How do we then define a random field? . . . . . . . . . . . . . . . . . . . . . 93
How can we relate two random variables that appear in the same random
field? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
How can we relate two random variablesthat belong to two different random
fields? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Since we always have just one version of an image how do we calculate the
expectation values that appear in all previous definitions? . . . . . . . 96
When is a random fieldhomogeneous? . . . . . . . . . . . . . . . . . . . . . 96
How can we calculate the spatial statistics of a random field? . . . . . . . . 97
When is a random field ergodic? . . . . . . . . . . . . . . . . . . . . . . . . 97
When is a random field ergodic with respect to the mean? . . . . . . . . . . 97
When is a random field ergodic with respect to the autocorrelation
function? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
What is the implication of ergodicity? . . . . . . . . . . . . . . . . . . . . . 102
How can we exploit ergodicity to reduce the number of bits needed for
representing an image? . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
What is the form of the autocorrelation function of a random field with
uncorrelatedrandom variables? . . . . . . . . . . . . . . . . . . . . . . 103
How can we transform the image so that its autocorrelation matrix is
diagonal? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Is the assumption of ergodicity realistic? . . . . . . . . . . . . . . . . . . . . 104
How can we approximate an image using its K-L transform? . . . . . . . . . 110
What is the error with which we approximate an image when we truncate
its K-L expansion? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
What are thebasis images in termsof which the Karhunen-Loeve transform
expands an image? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
What is the “take home” message of this chapter? . . . . . . . . . . . . . . 124

t 4 Image 125
What is imageenhancement? . . . . . . . . . . . . . . . . . . . . . . . . . . 125
How can we enhance an image? . . . . . . . . . . . . . . . . . . . . . . . . . 125
Which methods of the image enhancement reason about the grey level
statistics of an image? . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
X Image Processing: The Fundamentals

What is the histogram of an image? . . . . . . . . . . . . . . . . . . . . . . 125

When is it necessary to modify the histogram of an image? . . . . . . . . . 126
How can we modify the histogram of an image? . . . . . . . . . . . . . . . . 126
What is histogram equalization? . . . . . . . . . . . . . . . . . . . . . . . . 127
Why do histogram equalization programs usually not produce images with
flat histograms? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Is it possible to enhance an image to have an absolutely flat histogram? . . 127
What ifwe do not wish to have an image with a flat histogram? . . . . . . 129
Why should one wish to perform something other than histogram
equalization? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
What if the image has inhomogeneous contrast? . . . . . . . . . . . . . . . 131
Is there an alternative to histogram manipulation? . . . . . . . . . . . . . . 132
How can we improve the contrast of a multispectral image? . . . . . . . . . 135
What is principal component analysis? . . . . . . . . . . . . . . . . . . . . . 136
What is the relationship of the Karhunen-Loeve transformation discussed
here and the one discussed in Chapter 3? . . . . . . . . . . . . . . . . 136
How can we perform principal component analysis? . . . . . . . . . . . . . . 137
What are the advantagesof using principal components to express an
image? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
What are the disadvantages of principal component analysis? . . . . . . . . 138
Some of the images with enhanced contrast appear very noisy . Can we do
anything about that? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
What are the types of noise present in an image? . . . . . . . . . . . . . . . 144
What is a rank order filter? . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
What is median filtering? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
What if the noise in an image is not impulse? . . . . . . . . . . . . . . . . . 146
Why does lowpass filtering reduce noise? . . . . . . . . . . . . . . . . . . . . 147
What ifwe are interested in the high frequencies of an image? . . . . . . . . 148
What is the ideal highpass filter? . . . . . . . . . . . . . . . . . . . . . . . . 148
How can we improve an image which suffers from variable illumination? . . 148
Can any of the objectives of image enhancement be achieved by the linear
methods we learned in Chapter 2? . . . . . . . . . . . . . . . . . . . . 151
What is the “take home” message of this chapter? . . . . . . . . . . . . . . 153

5 Two-Dimensional Filters 155

What is this chapter about? . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
How do we define a 2D filter? . . . . . . . . . . . . . . . . . . . . . . . . . . 155
How are the system function and the unit sample response of the filter
related? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Why are we interested in the filter function in the real domain? . . . . . . . 157
Are there any conditions which h(k.1 ) must fulfil so that it can be used as
a convolution filter? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
What is the relationship between the 1D and the 2D ideal lowpass filters? . 161
How can we implement a filter of infinite extent? . . . . . . . . . . . . . . . 161
How is the z-transform of a digital 1D filter defined? . . . . . . . . . . . . . 161
Contents xi

Why do we use z-transforms? . . . . . . . . . . . . . . . . . . . . . . . . . . 163

How is the z-transform defined in 2D? . . . . . . . . . . . . . . . . . . . . . 163
Is there any fundamental difference between 1D and 2D recursive filters? . . 170
How do we know that a filter does not amplify noise? . . . . . . . . . . . . 171
Is there an alternative to using infinite impulse response filters? . . . . . . . 171
Why do we need approximation theory? . . . . . . . . . . . . . . . . . . . . 171
How do we know how good an approximate filter is? . . . . . . . . . . . . . 171
What is the best approximation to an ideal given system function? . . . . . 171
Why do we judge an approximation accordingto the Chebyshev norm
instead of the square error? . . . . . . . . . . . . . . . . . . . . . . . . 172
How can we obtain an approximation to a system function? . . . . . . . . . 172
What is windowing? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
What is wrong with windowing? . . . . . . . . . . . . . . . . . . . . . . . . 173
How can we improve the result of the windowing process? . . . . . . . . . . 173
Can we make use of the windowing functions that have been developed for
1D signals. to define a windowing function for images? . . . . . . . . . 173
What is the formal definition of the approximation problem we are trying
tosolve? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
What is linear programming? . . . . . . . . . . . . . . . . . . . . . . . . . . 174
How can we formulate the filter design problem as a linear programming
problem? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Is there any way by which we can reduce the computational intensity of the
linear programming solution? . . . . . . . . . . . . . . . . . . . . . . . 179
What is the philosophy of the iterative approach? . . . . . . . . . . . . . . . 179
Are there any algorithms that work by decreasing the upper limit of the
fittingerror? ................................ 180
How does the maximizing algorithm work? . . . . . . . . . . . . . . . . . . 180
What is a limiting set of equations? . . . . . . . . . . . . . . . . . . . . . . 180
What does the La Vallee Poussin theorem say? . . . . . . . . . . . . . . . . 180
What is the proof of the La Vallee Poussin theorem? . . . . . . . . . . . . . 181
What are the stepsof the iterative algorithm? . . . . . . . . . . . . . . . . . 181
Can we approximate a filter by working fully in the frequency domain? . . . 182
How can we express the system function of a filter at some frequencies as a
function of its values at other frequencies? . . . . . . . . . . . . . . . . 182
What exactly are we trying to dowhen we design the filter in the frequency
domain only? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
How can we solve for the unknown values H ( k .Z)? . . . . . . . . . . . . . . 190
Does the frequency sampling method yield optimal solutions according to
the Chebyshev criterion? . . . . . . . . . . . . . . . . . . . . . . . . . . 190
What is the “take home” message of this chapter? . . . . . . . . . . . . . . 192

6 Image Restoration 193

What is image restoration? . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
What is the difference between image enhancement and image restoration? 193
Why may an image require restoration? . . . . . . . . . . . . . . . . . . . . 193
xii Image Processing: The Fundamentals

How may geometric distortion arise? . . . . . . . . . . . . . . . . . . . . . . 193

How can a geometrically distorted image be restored? . . . . . . . . . . . . 193
How do we perform the spatial transformation? . . . . . . . . . . . . . . . . 194
Why is grey level interpolation needed? . . . . . . . . . . . . . . . . . . . . 195
How does the degraded image depend on the undegraded image and the
point spread function of a linear degradation process? . . . . . . . . . 198
How does the degraded image depend on the undegraded image and the
point spread function of a linear shift invariant degradation process? . 198
What form does equation (6.5) take for the case of discrete images? . . . . 199
What is the problem of image restoration? . . . . . . . . . . . . . . . . . . . 199
How can the problem of image restoration be solved? . . . . . . . . . . . . . 199
How can we obtain information on the transfer function k ( u .v) of the
degradation process? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
If we know the transferfunction of the degradationprocess. isn’t the solution
to the problem of image restoration trivial? . . . . . . . . . . . . . . . 209
What happens at points ( U . v) where k ( u .v) = O? . . . . . . . . . . . . . . 209
Will the zeroes of k ( u .v) and G(u.v) always coincide? . . . . . . . . . . . . 209
How can we take noise into consideration when writing the linear degradation
equation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
How can we avoid the amplification of noise? . . . . . . . . . . . . . . . . . 210
How can we express the problem of image restoration in a formal way? . . . 217
What is the solution of equation (6.37)? . . . . . . . . . . . . . . . . . . . . 217
Can we find a linear solution to equation (6.37)? . . . . . . . . . . . . . . . 217
What is the linear least mean square error solution of the image restoration
problem? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
Since the original image f(r) is unknown. how can we use equation (6.41)
which relies on its cross-spectral density with the degraded image. to
derive the filter we need? . . . . . . . . . . . . . . . . . . . . . . . . . 218
How can we possibly use equation (6.47) ifwe know nothing about the
statistical properties of the unknown image f(r)? . . . . . . . . . . . . 219
What is the relationship of the Wiener filter (6.47) and the inverse filter of
equation (6.25)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Assuming that we know the statistical properties of the unknown imagef(r).
how can we determine the statistical propertiesof the noise expressed
by Svv(r)?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
If the degradation process is assumed linear. why don’t we solve a system of
linear equations to reverse its effect instead of invoking the convolution
theorem? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
Equation (6.76) seems pretty straightforward. why bother with any other
approach? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
Is there any waybywhich matrix H can be inverted? . . . . . . . . . . . . 232
When is a matrix block circulant? . . . . . . . . . . . . . . . . . . . . . . . 232
When is a matrix circulant? . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Why can block circulant matrices be inverted easily? . . . . . . . . . . . . . 233
Which are the eigenvalues and the eigenvectors of a circulant matrix? . . . 233
Contents xiii

How does the knowledge of the eigenvalues and the eigenvectors of a matrix
help in inverting the matrix? . . . . . . . . . . . . . . . . . . . . . . . 234
How do we know that matrixH that expresses the linear degradationprocess
is block circulant? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
How can we diagonalize a block circulant matrix? . . . . . . . . . . . . . . . 240
OK. now we know how to overcome the problem of inverting H ; however.
how can we overcome the extreme sensitivity of equation (6.76) to
noise? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
How can we incorporate the constraint in the inversion of the matrix? . . . 251
What is the relationship between the Wiener filter and the constrained
matrix inversion filter? . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
What is the “take home” message of this chapter? . . . . . . . . . . . . . . 262

7 Image Segmentation and Edge Detection 265

What is this chapter about? . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
What exactly is the purpose of image segmentation and edge detection? . . 265
How can we divide an image into uniform regions? . . . . . . . . . . . . . . 265
What do we mean by “labelling” an image? . . . . . . . . . . . . . . . . . . 266
What can we do if the valley in the histogram is not very sharply defined? . 266
How can we minimize the number of misclassified pixels? . . . . . . . . . . 267
How can we choose the minimum error threshold? . . . . . . . . . . . . . . 268
What is the minimum error threshold when object and background pixels
are normallydistributed? . . . . . . . . . . . . . . . . . . . . . . . . . 273
What is the meaning of the two solutions of (7.6)? . . . . . . . . . . . . . . 274
What are the drawbacks of the minimum error threshold method? . . . . . 278
Is there any method that does not depend on the availability of models for
the distributions of the object and the background pixels? . . . . . . . 278
Are there any drawbacks to Otsu’s method? . . . . . . . . . . . . . . . . . . 282
How can we threshold images obtained under variable illumination? . . . . 282
If we threshold the image according to the histogram of lnf(z.y). are we
thresholding it according to the reflectance properties of the imaged
surfaces? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Since straightforward thresholding methods breakdown under variable
illumination. how can we cope with it? . . . . . . . . . . . . . . . . . . 285
Are there any shortcomings of the thresholding methods? . . . . . . . . . . 287
How can we cope with images that contain regions that are not uniform but
they are perceived as uniform? . . . . . . . . . . . . . . . . . . . . . . . 288
Are there anysegmentation methods that take intoconsideration the spatial
proximity of pixels? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
How can one choose the seed pixels? . . . . . . . . . . . . . . . . . . . . . . 288
How does the split and merge method work? . . . . . . . . . . . . . . . . . 288
Is it possible to segment an image by considering the dissimilarities between
regions. as opposed to considering the similarities between pixels? . . . 289
How do we measure the dissimilarity between neighbouring pixels? . . . . . 289
What is the smallest possible window we can choose? . . . . . . . . . . . . 290
xiv Processing:
Image The Fundamentals

What happens when the image has noise? . . . . . . . . . . . . . . . . . . . 291

How can we choose the weights of a 3 X 3 mask for edge detection? . . . . . 294
What is the best value of parameter K? . . . . . . . . . . . . . . . . . . . . 296
In the general case. how do we decide whether a pixel is an edge pixel
ornot? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
Are Sobel masks appropriate for all images? . . . . . . . . . . . . . . . . . . 306
How can we choose the weights of the mask ifwe need a larger mask owing
to the presence of significant noise in the image? . . . . . . . . . . . . 306
Can we use the optimal filters for edges to detect lines in an image in an
optimal way? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
What is the fundamental difference between step edges and lines? . . . . . . 309
What is the “take home” message of this chapter? . . . . . . . . . . . . . . 322

Bibliography 325

Index 329
Preface

This book is the result of 11 years of teaching the subject to the students of Surrey
University studying for an MSc degree in Signal Processing and Machine Intelligence.
As the subject of Machine Intelligence has mushroomed in recent years, so it has
attracted the interest of researchers from as diverse fields as Psychology, Physiology,
Engineering and Mathematics. The problems of Machine Intelligence may be tackled
with various tools. However, when we have to perform a task using a computer, we
must have in mind the language a computer understands, and this language is the
language of arithmetic and by extension mathematics. So, the approach to solving
the problems of Machine Intelligence is largely mathematical. Image Processingis the
basic underlying infrastructure of all Vision and Image related Machine Intelligence
topics. Trying to perform Computer Vision and ignoring Image Processing islike
trying to build a house starting from the roof, and trying to do Image Processing
without mathematics is like trying to fly by flapping the arms!
The diversity of people who wish to work and contribute in the general area of
Machine Intelligence led me towards writing this book on twolevels. One level should
be easy to follow with a limited amount of mathematics. This is appropriate for new-
comers to thefield and undergraduate students. Thesecond level, more sophisticated,
going through the mathematical intricacies of the various methods and proofs, is ap-
propriate for the inquisitive student who wishes to know the “why” and the “how”
and get at the bottom of things in an uncompromising way. At the lower level, the
book can be followed with no reference made at all to the higher level. All mate-
rial referring to the higher level is presented inside grey boxes, and may be skipped.
The book contains numerous examples presented inside frames. Examples that refer
to boxed material are clearly marked with a B and they may be ignored alongside
the advanced material if so desired. The basic mathematical background required
by the reader is the knowledge ofhow to add and subtract matrices. Knowledge of
eigenvalue analysis of matrices is also important. However, there are several fully
worked examples, so that even if somebody is hardly familiar with the subject, they
can easily learn the nuts and boltsof it by working through this book. This approach
is also carried to the stochastic methods presented: one can start learning from the
basic concept of a random variable and reach the level of understanding and using
the concept of ergodicity.
I would like to take this opportunity to thank the numerous MSc students who
over the years helped shape this book, sometimes with their penetrating questions,

xv
xvi Image Processing: The Fundamentals

and sometimes with their seemingly naive(!) questions. However, there are no naive
questions when one is learning: the naivety is with thosewho do not askthe questions!
My students’ questions helped formulate the route to learning and gave me the idea
to present the material in the form of questions and answers.
Writing this book was a learning process for Panagiota and me too. We had
a lot of fun working through the example images and discovering the secrets of the
methods. One thing that struck us as most significant was the divergence betweenthe
continuous and the discrete methods. An analytically derived formula appropriate for
the continuous domainoften has very little todo with the formula onehas to program
into the computer in order to perform the task. This is very clearly exemplified in
Chapter 6 concerned with image restoration. That is the reason we demonstrate all
the methods we present using small, manageable discrete images, that allow us to
manipulate them “manually” andlearn what exactly the computer hasto do if a real
size image is to be used. When talking about real size images, we would like to thank
Constantinos Boukouvalaswho helped with the programming of some of the methods
presented.
Finally, I would also like to thank my colleagues in the Centre for Vision, Speech
and Signal Processing of Surrey University, and in particular the directorJosef Kittler
for all the opportunities and support he gave me, and our systems manager Graeme
Wilford for being always helpful and obliging.

Maria Petrou
List of Figures

1.1 Keeping m constant and decreasing the size of the image from 338 X 298
down to a quarter of that produces the checkerboard effect. . . . . . .
1.2 Keeping the size of the image constant (247 X 333) and reducing the
number of grey levels (= 2") produces false contouring. . . . . . . . .
1.3 When keeping the size of a detailed image constant (1022 X 677) and
reducing the number of grey levels, the result is not affected very much.
1.4 Top row: An original image and its enhanced version. Bottom row: An
original image and a version of it represented by 40% of the original
number of bits. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.5 Top row: A blurredoriginalimage and its restored version. Bottom
row: An original image and its edge map (indicating locations where
the brightness of the image changes abruptly). . . . . . . . . . . . . . 20

2.1 The original image and its five eigenimages, each scaled to have values
from 0 to 255. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.2 Image reconstruction using one, two,.. ., five eigenimages from top right
to bottom left sequentially, with the original image shown in (f). . . . 44
2.3 Haar transform basis images. In each image, grey means 0, black means
a negative and white means a positive number. Note that each image
has been scaled separately: black and white indicate different numbers
from one image to the next. . . . . . . . . . . . . . . . . . . . . . . . . 52
2.4 Hadamard/Walshtransform basis images. . . . . . . . . . . . . . . . . 58
2.5 Reconstructed imageswhen the basis imagesused are thosecreated
from the first one, two, three,.. ., eight Haar functions, from top left to
bottom right respectively. . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.6 Reconstructed imageswhen the basis imagesused are thosecreated
from the first one, two, three,. . ., eight Walsh functions, from top left
to bottom right respectively. . . . . . . . . . . . . . . . . . . . . . . . 61
2.7 Real part of the Fourier transform basis images. . . . . . . . . . . . . . 68
2.8 Imaginary part of the Fourier transform basis images. . . . . . . . . . 69
2.9 Reconstructed image when the basis images used are those createdfrom
the first one, two,.. ., eight lines of matrix U of Example 2.27, from top
left to bottom right respectively. . . . . . . . . . . . . . . . . . . . . . 71

xvii
xviii Image Processing: The Fundamentals

3.1 Ergodicity in a nutshell . . . . . . . . . . . . . . . . . . . . . . . . . . 103

3.2 The 64 eigenimages, each scaled separately to have values from 0 to 255 121
3.3 Reconstructed imagewhen the first 8,16,24,32,40,48,56and 64 eigen-
images shown in Figure 3.2 were used (from top left to bottom right
respectively). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122

4.1 (a) is the histogram of a “bad” image while (b) is the histogram of a
“good” image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
4.2 Enhancing the image of a bathtub cleaner by histogram equalization . 128
4.3 Histogram hyperbolization with a = 0.5 applied to theimage of Figure
4.2a . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
4.4 Enhancing the image of a young train driver. . . . . . . . . . . . . . . 133
4.5 Enhancing the image of the hanging train of Wupertal. . . . . . . . . . 134
4.6 Enhancing the image at the Karlstejn castle. . . . . . . . . . . . . . . 135
4.7 The pixels of a colour image form a cluster in the colour space. The
maximum spread of this cluster may be along a line not parallel with
any of the colour axes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4.8 Example of principal component analysis of a colour image. . . . . . . 139
4.9 Examples of filtering to remove noise. . . . . . . . . . . . . . . . . . . 145
4.10 Improving images4 . 4 and ~ 4 . 5 by~ using medianfiltering (median filter
size = 3 X 3). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
4.11 After a certain frequency, the spectrumof the noisy image is dominated
by the noise. Ideally the cutoff frequency of the lowpass filter should
be at rg = F. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
4.12 The ideal lowpass filter in 2D in the frequency domain. On the right a
+
6
cross-section of this filter with cutoff frequency rg ( r w i W”. . 148
4.13 The spectrum of the ideal highpass filter is 1 everywhere, except inside
a circle of radius rg in the frequency domain, where it is 0. On the
right, a cross-section of sucha filter. Here r +
6 W; w2. . . . . . . . 149
4.14 A cross-section of a homomorphic filter as a function of polar frequency,
r JW. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
4.15 Example of homomorphic filtering Haroula at the keyboard. . . . . . . 150
4.16 Homomorphic filtering Mrs Petrou at home. . . . . . . . . . . . . . . . 151

5.1 Top row: a signal andits Fourier transform. Middlerow: theunit

sample response function of a filter on the left, and the filter’s system
function on the right. Bottom row: On the left the filtered signal that
can be obtained by convolving the signal at the top with the filter in
the middle. On the right the Fourier transform of the filtered signal
obtained by multiplying the Fourier transform of the signal at the top,
with the Fourier transform (system function) of the filter in the middle. 156
5.2 The cross-section of the 2D ideal lowpass filter is similar to but different
from the cross-section of the 1D ideal lowpass filter. . . . . . . . . . . 162
5.3 The transformation of the area of summation. . . . . . . . . . . . . . . 165
List of Figures xix

5.4 Different recursive formulaeapply to different ranges of index values. 167

5.5 In the frequency domain, a good approximation is one that never de-
viates too muchaway from the desired response. . . . . . . . . . . . . 172
5.6 How to fit a surface gradually. . . . . . . . . . . . . . . . . . . . . . . . 180
5.7 Fixing some of the filter valuesin the frequency domain and letting
othersbe free. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

6.1 Examples of geometricdistortionscaused by the lens. . . . . . . . . . 194

6.2 In this figure the pixels correspond to the nodes of the grids. Pixel A of
the corrected grid corresponds to inter-pixel position A’ of the original
image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
6.3 The non-integer position of point A’ is surrounded by four pixels at
integer positions, with known grey values ([X]means integer part of X). 196
6.4 A test chart for the derivation of the point spread function of an imag-
ing device. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
6.5 Filters used to compute the derivative in 0,45,90 and 135 degrees. . . 208
6.6 The point spread function (PSF) of an imaging system, in two different
scales. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
6.7 Imagerestorationwith inverse filtering. . . . . . . . . . . . . . . . . . 215
6.8 Image restoration with inverse filtering in the presence of noise. . . . . 216
6.9 ImagerestorationwithWiener filtering. . . . . . . . . . . . . . . . . . 230
6.10 An original and a restored signal by direct matrix inversion. The noise
in the signal wasonly rounding error. . . . . . . . . . . . . . . . . . . 232
6.11 Restorationwithconstraintmatrix inversion. . . . . . . . . . . . . . . 262
6.12 (a) Original images. (b) Restoration by inverse filtering with omission
of all terms beyond the first zero of the filter transfer function. (c)
Restoration by Wiener filtering. (d) Restoration by constraint matrix
inversion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

7.1 The histogram of an image with a bright object on a dark background. 266
7.2 Simplethresholding versus hysteresis thresholding. . . . . . . . . . . . 267
7.3 The probability density functions of the grey values of the pixels that
make up the object @,(X)) and the background @ b ( ~ ) ) . Their sum,
normalized to integrate to 1, is what we obtain if we take thehistogram
of an image and normalize it. . . . . . . . . . . . . . . . . . . . . . . . 269
7.4 Optimalthresholding (t = 74). . . . . . . . . . . . . . . . . . . . . . . 276
7.5 Otsu’sthresholding (t = 84). . . . . . . . . . . . . . . . . . . . . . . . 282
7.6 z isless than U in the shadowed half plane. . . . . . . . . . . . . . . . 284
7.7 Global versus local thresholding for an image with variable illumination.286
7.8 Twovery different imageswith identical histograms. . . . . . . . . . . 287
7.9 The two identical histograms of the verydifferent images shown in
Figure 7.8. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
7.10 Imagesegmentation by splitting. . . . . . . . . . . . . . . . . . . . . . 289
7.11 Measuring the dissimilarity between two image regions using a sliding
widow.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
xx Image Processing: The Fundamentals

7.12 (a) Original image. (b) The image of the gradient magnitudes com-
puted by simple differencing without applying any smoothing. For dis-
playing purposes the gradient image has been subjected to histogram
equalization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
7.13 (a) Resultobtained by simplythresholdingthegradient values ob-
tainedwithoutanysmoothing.(b)Thesameas in (a) but using a
higher threshold, so some noise is removed. However, useful image in-
formation has also been removed. (c) Result obtained by smoothing
first along one direction and differentiating afterwards along the or-
thogonal direction, using a Sobel mask. (d) Result obtained using the
optimal filter of size 7 X 7. In all cases the same value of threshold was
used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
7.14 The left column shows a noisy signal and two smoothed versions of it.
On the right is the result of estimating thefirst derivative of the signal
by simple differencing. The edge manifests itself with a sharp peak.
However, the more the noise is left in the signal, the more secondary
peaks will be present, and the more the smoothing, the more blunt the
mainpeakbecomes. ............................ 295
7.15 Zooming into a 3 X 3 patch of an image around pixel ( i , j ) . . . . . . . 297
7.16 (a) Original image. (b) The gradient magnitude “image”: the brighter
the pixel, the higher the value of the gradient of the image at that point.
(c) After non-maxima suppression: only pixels with locally maximal
values of the gradient magnituderetaintheir values. Thegradient
values of all other pixels are set to zero. (d) Theedge map: the gradient
values in (c) are thresholded:all pixels with values above the threshold
are labelled 0 (black) and all pixels with values below the threshold are
labelled 255 (white). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
7.17 Trying to detect edges in a blurred and noisy image using the Sobel
edge detector may be very tricky. One can play with different threshold
values but the result is not satisfatory. The three results shown have
been obtainedby using different thresholds within the same edgedetec-
tion framework, and they were the best among many others obtained
for different threshold values. . . . . . . . . . . . . . . . . . . . . . . . 307
7.18 A noisy 1D edge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
7.19 The profile of a line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
7.20 Trying to deal with the high level of noise by using an optimal filter of
large size and playing with the thresholds may not help, if the edges
are blurred and resembleramps, and the “optimal” filter hasbeen
developed to be optimal for step edges. . . . . . . . . . . . . . . . . . . 312
7.21 Having chosen the right model for the featureswe wish to detect, using
the right filter size for the level of noise may prove crucial. . . . . . . . 313
7.22 Everyday life images are relatively clean and good results can be ob-
tained by using relatively small filters. Note that the smaller the filter,
the moredetailsare preserved. . . . . . . . . . . . . . . . . . . . . . . 314
Chapter 1

Introduction

Why do we process images?

Image Processing has been developed in response to three major problems concerned
with pictures:
0 Picture digitization and coding to facilitate transmission, printing and storage
of pictures.
0 Picture enhancement and restoration in order, for example, to interpret more
easily pictures of the surface of other planets taken by various probes.
Picture segmentation and description as an early stage in Machine Vision.

What is an image?

A monochrome image is a 2-dimensional light intensity function, f (x, y), where x and
y are spatial coordinates and the value off at (x, y) is proportional to the brightness of
the image at that point. If we have a multicolour image, f is a vector, each component
of which indicates the brightness of the image at point (x, y) at the corresponding
colour band
A digital image is an image f (x, y) that has been discretized both in spatial co-
ordinates and in brightness. It is represented by a 2-dimensional integer array, or
a series of Zdimensional arrays, one for each colour band. The digitized brightness
value is called the grey level value.
Each element of the array is called a pixel or a pel derived from the term "picture
element". Usually, the size of such an array is a few hundred pixels by a few hundred
pixels and there are several dozens of possible different grey levels. Thus, a digital
image looks like this:
2 Image Fundamentals
Processing: The

with 0 5 f(z,y) 5 G - 1 where usually N and G are expressed as integer powers of

2 ( N = 2n, G = 2m).

What is the brightness of an image at a pixel position?

Each pixel of an image corresponds to a part of a physical object in the 3D world.

This physical object is illuminated by some light which is partly reflected and partly
absorbed by it. Part of the reflected light reaches the sensor used to image the scene
and is responsible for the value recorded for the specific pixel. The recorded value
of course, depends on the type of sensor used to image the scene, and the way this
sensor responds to the spectrum of the reflected light. However, as a wholescene
is imaged by the same sensor, we usually ignore these details. What is important
to remember is that the brightness values of different pixels have significance only
relative to each other and they are meaningless in absolute terms. So, pixel values
between different images should only be compared if either care has been taken for
the physical processes used to form the two images to be identical, or the brightness
values of the two images have somehow been normalized so that the effects of the
different physical processes have been removed.

Why are images often quoted as being 512 X 512, 256 X 256, 128 X 128 etc?

Many image calculations with images are simplified when the size of the image is a
power of 2.

How many bits do we need to store an image?

The number of bits, b, we need to store an image of size N X N with 2m different grey
levels is:

So, for a typical 512 X 512 image with 256 grey levels ( m = 8) we need 2,097,152
bits or 262,144 8-bit bytes. That is why we often try to reduce m and N , without
significant loss in the quality of the picture.

What is meant by image resolution?

The resolution of an image expresses how much detail we can see in it and clearly
depends on both N and m.
Keeping m constant and decreasing N results in the checkerboard effect (Figure
1.1). Keeping N constant and reducing m results in false contouring (Figure 1.2).
Experiments have shown that the more detailed a picture is, the less it improves by
keeping N constant and increasing m. So, for a detailed picture, like a picture of
crowds (Figure 1.3), the number of grey levels we use does not matter much.
6 Image Fundamentals
Processing: The

How do we do Image Processing?

We perform Image Processing by using Image Transformations. Image Transforma-

tionsare performedusing Operators. An Operatortakesasinputanimageand
produces another image. In this book we shall concentrate mainly on a particular
class of operators, called Linear Operators.

What is a linear operator?

Consider 0 to be an operator which takes images into images. If f is an image, 0(f)

is the result of applying 0 to f . 0 is linear if

0b.f + bgl = a f l + bQ[gl (1.2)

for all images f and g and all scalars a and b.

How are operators defined?

Operators are defined in terms of their point spread functions. The point spread
function of an operatoris what we get out if we apply the operatoron a point source:

O[point source] = point spread function (1.3)

Or:

How does an operator transform an image?

If the operator is linear, when the point source is a times brighter, the result will be
a times larger:

An image is a collection of point sources (the pixeZs) each with its own brightness
value. We may say that an image is the sum of these point sources. Then the effect of
an operator characterized by point spread function h ( z , ay , P) on an image f ( z ,y)
can be written as:

z=o y=o

where g ( a , P ) is the output “image”, f(z,y) is the input image and the size of the
images is N X N .
Introduction 7

What is the meaning of the point spread function?

The point spread functionh(z,a,y, P) expresses how much the input value at position
(z, y) influences the output value at position (a,P). If the influence expressed by the
point spread function is independent of the actual positions but depends only on the
relative position of the influencing and theinfluenced pixels, we have a shift invariant
point spread function:
h(z,a,y,P) = h ( a - z , P - y ) (1.7)
Then equation (1.6) is a convolution:
N-l N-l

2=0 y=o

If the columns are influenced independently from the rows of the image, then the
point spread function is separable:
h(x7 a7 Y,P) = M z ,a)h,(y,P) (1.9)
where the above expression serves also as the definition of functions h,(z,a) and
h,(y, P). Then equation (1.6) can be written as a cascade of two 1D transformations:

(1.10)
2=0 y=o
If the point spread function is both shift invariant and separable, then equation
(1.6) can be written as a cascade of two 1D convolutions:
N-l
g(a,P) = c
N-l

x=o
hc(a - X) c
y=o
f(.,Y)hT(P - Y) (1.11)

B 1 . l : The formal definition of a point source in the continuous domain

Define an extended source of constant brightness:

Sn(x,y) = n2rect(nx,ny) (1.12)

where n is a positive constant and

rect(nx,ny) = l inside a rectangle 1nx1 5 $ , lny l 5 1

(1.13)
0 elsewhere

The total brightness of this source is given by

8 Image Fundamentals
Processing: The

/"/"
-cc -cc
Sn(x,y)dxdY = n2 111:
\
rect(nz,ny)dzdy= 1
Y
/
(1.14)

area of rectangle

and is independent of n.
As n t 00, we create a sequence, S,, of extended square sources which grad-
ually shrink with their brightness remaining constant. At the limit, S, becomes
Dirac's delta function
#O forz=y=O
{ =0 elsewhere
(1.15)

with the property

LL S(2,y)dzdy = 1. (1.16)

The integral

(1.17)

is the average of image g(z, y) over a square with sides centred at (0,O). At the
limit we have:

SccScc
--CO
--CO
S(z, y)g(n:, y)dzdy = g@, 0) (1.18)

which is the value of the image at the origin. Similarly

(1.19)

is the average value of g over a square k X k centred at n: = a , y = b, since:

Sn(z - a , y - b) = n'rect[n(z - a ) ,n(y - b ) ]

We can see that this is a square source centred at ( a ,b ) by considering that

In(. -.)I +
5 means -+
5 n(n:- U ) 5 i.e. + -& < n: - a <& or a - 1 Ln <-
a+&. T h u s w e h a v e t h a t S , ( n : - a , y - b ) = n 2 i n t h e r e g i o n a - ~ < n : < a + n ,1
<
-

b-2;;<y<b+2;;. 1
At the limit of n + 00, integral (1.19) is the value of the image g at n: = a ,
y = b, i.e.
Introduction 9

S" SW
-W -"
S(%,Y)Sn(z - a,Y - b)dzdy = d a , b) (1.21)

This equation is called the shifting property of the delta function. This equation
also shows that any image g ( z , y) can be expressed as a superposition of point
sources.

How can we express in practice the effect of a linearoperator on an image?

This is done with the help of matrices. We can rewrite equation (1.6) as follows:

g(a,P) =
+
f(O,O)h(O,Q, 0, P) f(l,O)h(l, a , 0, P ) + . . . + f ( N - 1 , O ) W - 1, a , 0, P )
+
+f(O, l)h(O, a , 1, P ) f ( l , l ) h ( l , a , 1, P ) + . . . + f ( N - 1 , l ) W - 1, Q, 1, P)
+
+. . . + f(0, N - l)h(O, a, N - 1,P) f ( 1 , N - l ) h ( l , a, N - 1,P) . . . +
+ f ( N - 1, N - l)h(N - 1, a, N - 1, P ) (1.22)

The right hand side of this expression can be thought of as the dot product of
vector

with vector

This last vector is actually the image f(z,g ) written as a vector by stacking its
columns one under the other. If we imagine writing g ( a , P ) in the same way, then
vectors will arrangethemselves as the rows of a matrix H , where for a = 0, P
will run from 0 to N - 1 to give the first N rows of the matrix, then for a = 1, P will
run again from0 to N - 1 to give the second N rows of the matrix, andso on. Thus,
equation (1.6) can be written in a more compact way as:

(1.25)

This is the Fundamental Equation of linear Image Processing. H here is a square

N 2 X N 2 matrix that is made up of N X N submatrices of size N X N each, arranged
 10 Image Processing: The Fundamentals

in the following way:

X+ X+ X+

X+ I X+ X+ I

In this representation each bracketed expression represents an N X N submatrix

made up from function h(%,a, y, p) for fixed values of y and p and with variables II:
and a taking up all their possible values in the directions indicated by the arrows.
This schematic structure of matrix H is said to correspond to a partition of this matrix
into N 2 square submatrices.

B1.2: What is the stacking operator?

The stacking operator allows us to write an N X N image array as an N 2 X 1

vector, or an N 2 X 1 vector as an N X N square array.
We define some vectors V,, and some matrices N , as:

1 0 ... 0
i n - 1 square N X N ma-
trices on top of each other
with all their elements 0

N, =
0
..

0
.
1 ... 0
..
.
0 ... l i the nth matrix is the unit
matrix

i
N - n square N X N ma-
trices on the top of each
0 otherwith all their ele-
L ments 0
Introduction 11

The dimensions of V, are (N X 1) and of Nn (N2 X N). Then vector f which

corresponds to the (N X N) square matrix f is given by:

(1.27)

It can be shown that if f is an N2 X 1 vector, we can write it as an N X N

matrix f the first column of which is made up from the first N elements off, the
second column from the second N elements of f , and so on, by using the following
expression:
N
f = C N,TfVz (1.28)
n=l

Example 1.1 (B)

You are given a 3 X 3 image f and you are asked to use the stacking
operator to write it in vector form.

Let us say that:

ff1333321

W e define vectors V, and matrices N , for n = 1,2,3:

1 0 0 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0
0 0 0 1 0 0 0 0 0
0 0 0 , N 2 = 0 1 0 , N 3 = 0 0 0
0 0 0 0 0 1 0 0 0
0 0 0 0 0 0 1 0 0
0 0 0 0 0 0 0 1 0
0 0 0 0 0 0 0 0 1
12 Image Processing:
FundamentalsThe

According to equation (1.27):

f = NIf V 1 +NZf V 2 + N3 fV3 (1.29)

W e shall calculate each term separately:

(1 0 0
0 1 0
0 0 1
0 0 0 fll f12
NlfV1 = 0 0 0 f21 f22
0 0 0 f31 f32
0 0 0
0 0 0
\o 0 0
'1 0 0 fll
0 1 0 f21
0 0 1

NlfV1 =
0
0
0
0
0
0
0
0
0
0
0
0
(9;) =
f31
0
0
0
0
0 0 0 0
0 0 0 0
Similarly
0 0
0 0
0 0
f12 0
N2fV2 = f22 7 N3fV3 = 0 (1.30)
f32 0
0 f13
0 f23
0 f33

Then by substituting in (1.29) we get vector f .

14 Image Processing:
FundamentalsThe

I1 Similarly

1 Then by substituting in (1.31) we get matrix f .

What is the implication of the separability assumption on the structure of

matrix H?

According to theseparability assumption,we can replace h ( z ,a , y , p) by h,(z, a)hr(y,

p).
Then inside each partition of H in equation (1.26), h c ( z ,a ) remains constant and we
may write for H :

( hcoo ( hrOO

hrON-l
...
...

...
...

hcol ( '7'
hTOO

hrON-l
...
...

...
...
Introduction 15

How can a separable transform be written in matrix form?

Consider again equation (1.10) which expresses the separable linear transform of an
image:

(1.35)
x=o y=O

Notice that factor Cril f ( z ,y)h,(y,P) actually represents the product of two
N XN matrices, which must be another matrix of the same size. Let us define it as:

(1.36)

Then (1.35) can be written as:

N--l

(1.37)
2=0

Thus in matrix form

m (1.38)

What is the meaning of the separability assumption?

The separability assumption implies that our operator 0 (the point spread function
of which, h(z,a,y,P),is separable)operates on the rows of the image matrix f
independently from the way it operates on itscolumns. These independent operations
are expressed by the two matrices h, and h, respectively. That is whywe chose
subscripts T and c to denote these matrices ( r = rows, c = columns).

B1.3 The formal derivation of the separable matrix equation

We can use equations (1.27) and (1.28) with (1.25) as follows: First express the
output image g using (1.28) in terms of g :
N

9 (1.39)
m=l

Then express g in terms of H and f from (1.25) and replace f in terms of f using
(1.27):

(1.40)
mage 16 Processing:
FundamentalsThe

Substitute (1.40) into (1.39) and group factors with the help of brackets to get:
N N
(1.41)
m=l n=l
H is a ( N 2X N 2 ) matrix. We may think of it as partitioned inN X N submatrices
stacked together. Then it can beshown that NKHN, is the H,, such submatrix.
Under the separability assumption, matrix H is the Kronecker product of matrices
h, and h,:

H = h: @ h: (1.42)

Then partition H,, is essentially hT(m, n)hT.If we substitute this in (1.41) we

obtain:
N N

N N
(1.43)
m=l n=l

The product V n V z is the product between an ( N X 1) matrix with the only

non-zero element at position n , with a (l X N ) matrix, with the only non-zero
element at position m. So, it is an N X N square matrix with the only non-zero
element at position ( n ,m ) .
When multiplied by hT(m,n)it places the ( m , n )element of the h? matrix
in position ( n , m )and sets to zero all other elements. The sum over all m’s and
n’s is h,. So from (1.43) we have:

g = hTfhr ( 1.44)

Example 1.3 (B)

You are given a 9 X 9 matrix H which is partitioned into nine 3 X 3

submatrices. Show that N T H N 3 , where N2 and N3 are matrices of the
stacking operator, is partition H23 of matrix H .

Let us say that:

Introduction 17

H =
18 Image Processing:
FundamentalsThe

What is the “take home” message of this chapter?

Under the assumption that the operator with which we manipulate an image is linear
and separable, this operation can be expressedby an equation of the form:

where f and g are the input and output images respectively and h, and h, are matrices
expressing the point spread function of the operator.

What is the purpose of Image Processing?

The purpose of Image Processing is to solve the following four problems:

Given an image f choose matrices h, and h, so that the output image g is
“better” than f according to some subjective criteria. This is the problem of
Image Enhancement.
0 Given an image f choose matrices h, and h, so that g can be represented by
fewer bits than f without much loss of detail. This is the problem of Image
Compression.
0 Given an image g and an estimate of h ( z ,a , y, p), recover image f. This is the
problem of Image Restoration.
0 Given an image f choose matrices h, and h, so that output image g salien-
ates certain features of f . This is the problem of preparation of an image for
Automatic Vision.
Figures 1.4 and 1.5 show examples of these processes.

What is this book about?

This book is about introducing the Mathematical foundations of Image Processing

in the context of specific applications in the four main themes of Image Process-
ing as identified above. The themes of Image Enhancement, Image Restoration and
preparation for Automatic Vision will be discussed in detail. The theme of Image
Compression is only touched upon as this could be the topic of a whole book on its
own. The book mainly concentrates on linear methods, but several non-linear tech-
niques relevant to Image Enhancement and preparation for Automatic Vision will also
be presented.
Introduction 19

4
!i

Figure 1.4: Top row: An original image and its enhanced version. Bottom row: An
original image and a version of it represented by 40% of the original number of bits.
20 Image Processing:
FundamentalsThe

Figure 1.5: Top row: A blurred original image and its restored version. Bottom
row: An original image and its edge map (indicating locations where the brightness
of the image changes abruptly).
Chapter 2

Image Transformations

What is this chapter about?

This chapter is concerned with the development of some of the most important tools
of linear Image Processing, namely the ways by which we express an image as the
linear superposition of some elementary images.

How can we define an elementary image?

We can define an elementary image as the outer product of two vectors.

What is the outer product of two vectors?

Consider two vectors N x 1:

UT = (uil, ~ i 2.,. . ,U ~ N )
vjT = (vjl, vj2, . . . ,v j ~ )
Their outer product is defined as:

Therefore, the outer product of these two vectors is an N x N matrix which can be
thought of as an image.

How can we expand an image in terms of vector outer products?

We saw in the previous chapter that a general separable linear transformation of an

image matrix f can be written as:
g = h:f hr (2.2)
22 Image Processing:
FundamentalsThe

where g is the output image and h, and h, are the transforming matrices.
We can use the inverse matrices of h: and h, to solve this expression for f in
terms of g as follows: Multiply both sides of the equation with (h:)-' on the left and
h;' on the right:
T T T
(h, ) gh,' = (h, ) h, f hrh,' =f (2.3)
Thus we write:

f = ( hT, 1- l &,l (2.4)

Suppose that we partition matrices (h:)-' and h;l in theircolumnand row

vectors respectively:

Then

We may also write matrix g as a sum of N 2 , N X N matrices, each one having

only one non-zero element:

g=
0
0 ...

0 '.'
0 ...
i) (0 +
0 912

0
0
0
... 0
.'.
...
!) +...+ (0 0 0 ...

0
0 0
" '

...
0

0
gNN
) (2.7)

Then equation (2.6) can be written as:

N N

i=l j=1

This is an expansion of image f in terms of vector outer products. The outer

product uivT may be interpreted as an"image" so that the sumover all combinations
of the outer products, appropriately weighted by the g i j coefficients, represents the
original image f.
 Image 23

Example 2.1

Derive the term i = 2, j = 1 in the right hand side of equation (2.8).

If we substitute g from equation (2.7) into equation (2.6), the right hand side oj
equation (2.6) will consist of N 2 terms of similar form. One such term is:

(U1 U2 ... UN) (g:

0

0
0
0 .'.
0
...

...
i) (':')
0

VNT
...
...
=(U1 U2 ...
...

0)
VNN
U21 ... UN1 ' 0 0 .
U22 ... UN2 921Vll 921v12 . .. Q21UlN
.'
\ o 0 . .. 0

What is a unitary transform?

If matrices h, and h, are chosen to be unitary, equation (2.2) represents a unitary

transform of f , and g is termed the unitary transform domainof image f .

What is a unitary matrix?

A matrix U is called unitary if its inverse is the complex conjugate of its transpose,
i.e.

UUT* = I (2.9)
24 Image Processing:
FundamentalsThe

where I is the unit matrix. We often write superscript “H” instead of “ T P .

If the elements of the matrix arereal numbers, we use the term orthogonal instead
of unitary.

What is the inverse of a unitary transform?

If matrices h, and h, in (2.2) are unitary, then the inverse of it is:

f =

For simplicity, from now on we shall write U instead of h, and V instead of h,, so
that the expansion of an image f in terms of vector outer products can be written as:

f = UgVH (2.10)

How can we construct a unitary matrix?

If we consider equation (2.9) we see that for the matrix U to be unitary the re-
quirement is that the dot product of any two of its columns must be zero while the
magnitude of any of its column vectors must be 1. In other words, U is unitary if its
columns form a set of orthonormal vectors.

How should we choose matrices U and V so that g can be represented by

fewer bits than f?

If we wanted to represent image f with fewer than N Z number of elements, then

we could choose matrices U and V so that the transformed image g was a diagonal
matrix. Then we could represent image f with the help of equation (2.8) using only
the N non-zero elements of g. This can be achieved with a process called matrix
diagonalization, and it is called Singular Value Decomposition (SVD) of the image.

How can we diagonalize a matrix?

It can be shown (see box B2.1) that a matrix g of rank r can be written as:

g = UA4VT (2.11)

where U and V are orthogonal matrices of size N X r and A; is a diagonal r X r

matrix.
Image 25

Example 2.2

If A is a diagonal 2 X 2 matrix and A” is defined by putting all non-zero

elements of A to the power of m, show that:

A-iAA-i = I and A - i A i = I .

Indeed,

This also shows that A-+ A4 = I

Example 2.3 (B)

Assume that H is a 3 X 3 matrix and partition it in a 2 X 3 submatrix

H1 and a 1 X 3 submatrix H2. Show that:

H T H = H , T H l + H,TH2.

Let us say that

Then:
26 Image Processing:
FundamentalsThe

Adding HFHl and HTH2 we obtain the same answer as before.

Example 2.4

You are given an image which is represented by a matrix g. Show that

matrix ggT is symmetric.

A matrix is symmetric when it is equal to its transpose. Consider the transpose

of SST .

(9gTIT = (STITST = SST

Example 2.5 (B)

Show that if we partition an N X N matrix S into an r X N submatrix

S1 and an ( N - r ) X N submatrix S2, the following holds:

S1AST I S1AS,T
I

1 where A is an N
SAST=

X
( -----
S2AST
N matrix.
I
-----
S2AS,T
Image 29

where A and 0 represent the partitions of the diagonal matrix above. Similarly
we can partition matrix S to an T X N matrix S1 and an ( N - T ) X N matrix Sa:

S= ();
Because S is orthogonal, and using the result of Example 2.3, we have:

S ~ =SI +- +s,Ts2= I +-
STSl = I - gs2 +- s T s 1 g = g - s,Ts2g (2.13)

From (2.12) and Examples 2.5 and 2.6 we clearly have:

SlggTST = A (2.14)
S2ggTS? = o* s2g = 0 (2.15)

Using (2.15) in (2.13) we have:

STS1g = g (2.16)

This means that STSl = I , i.e. S1 is an orthogonal matrix. We multiply both

sides of equation (2.14) from left and right by A-$ to get:

A - + S ~ ~ ~ T S T A -=+A - ~ A A - + - I (2.17)

I T
Since A-+ is diagonal, A-+ = (A-?) . So the above equation can be rewritten
as :
T
A - + s ~ ~ ( A - + s ~ ~=) I (2.18)

Therefore, there exists a matrix q = A-iSlg whose inverse is its transpose (i.e.
it is orthogonal). We can express matrix S l g as A i q and substitute in (2.16) to
get :

STRiq=g or g = S T A i q (2.19)

In other words, g is expressed as a diagonal matrix A f made up from the square

roots of the non-zero eigenvalues of g g T , multiplied from left and right by the two
orthogonal matrices S1 and q. This result expresses the diagonalization of image
9.
30 Image Processing:
FundamentalsThe

How can we compute matrices U , V and A + needed for the image

diagonalization?

If we take the transpose of (2.11) we have:

gT = VAiUT (2.20)

Multiply (2.11) by (2.20) to obtain:

ggT = U A ~ V T V A ~ U=
T U +,
A ~ A ~ U T ggT = UAUT (2.21)

This shows that matrix A consists of the T non-zero eigenvalues of matrix ggT while
U is made up from the eigenvectors of the same matrix.
Similarly, ifwe multiply (2.20) by (2.11) we get:

gTg = VAVT (2.22)

This shows that matrix V is made up from the eigenvectors of matrix gTg.

B2.2 What happens if the eigenvalues of matrix ggT are negative?

We shall show that the eigenvalues of ggT are always non-negative numbers. Let
us assume that X is an eigenvalue of matrix ggT and U is the corresponding
eigenvector. We have then:

ggTu = X u

Multiply both sides with uT from the left:

T
U ggT u = uT Xu

X is a scalar and can change position on the right hand side of the equation. Also,
because of the associativity of matrix multiplication, we can write:

(uTg)(g%) = XUTU

Since U is an eigenvector, uTu = 1. Therefore:

(gTu)T(gTu) = A

gTu issome vector y. Then we have: X = yTy 20 since yTy is the square
magnitude of vector y .
Image 31

Example 2.7

If X i are the eigenvalues of ggT and ui the corresponding eigenvectors,

show that gTg has the same eigenvalues with the corresponding eigen-
vectors given by: vi = g T q .

B y definition

ggTui = XZUi

Multiply both sides on the left by gT:

T T T
g ui = g Xiui

X i is ascalarandcanchangeposition. Also, bytheassociativity of matrix

multiplication:

gTg(gTUi)= Xi(gTUi)

This identifiesgTui as an eigenvectorof g T g with X i the corresponding eigenvalue.

Example 2.8

You
are given
an
image: g = . Compute the eigenvectors

U; of ggT and v; of gTg.

Then

SST =
001111

The eigenvalues of ggT will be computed from the characteristic equation:

l-X 2 0
2 6-X 1 = 0 + (1- X)[(6 - X)(1 - X) - l] - 2[2(1- X)] = 0
0 1 l-X
+ (1- X)[(6 - X)(1 - X) - 1- 41 = 0
32 Image Processing:
FundamentalsThe

One eigenvalue is X = 1. The other two are the roots of:

7 f J m -
-
7 f 6.7
6-6X-X+X2-5=0~X2-7X+1=O+X=
2 2
X = 6.85 or X = 0.146

In descending order the eigenvalues are:

X1 = 6.85, = 1, X3 = 0.146

(::)
X2

Let ui = be the eigenvector which corresponds to eigenvalue Xi. Then:

21 + 222 = Xi21

For X i = 6.85
222 - 5.8521 = 0 (2.23)
+
221 - 0.8522 2 3 = 0 (2.24)
2 2 - 5.8523 = 0 (2.25)

Multiply (2.24) by 5.85 and add equation (2.25) to get:

Equation (2.26) is the same as (2.23). So wehavereallyonlytwoindependent

equations for the three unknowns. W e choose the value of x1 to be 1. Then

52 = 2.927 and from (2.244) 5 3 = -2 + 0.85 X 2.925 = -2 + 2.5 = 0.5

Thus the first eigenvector is

and after normalization, i.e. division by d12+ 2.9272 + 0.52 = 3.133, we get:

0.319

For X i = 1, the system of linear equations we have to solve is:

Image 33

X1+ 2x2 = X 1 * X2 = 0
2x1 + X3 = 0 X3 = -221

Choose XI = 1. Then 2 3 = -2. Since 22 = 0, we must divide all components by

d m = fi for the eigenvector to have unit length:

U2 = ( ) -0.894

For X i = 0.146, the system of linear equations we have to solve is:

0.85421 2x2 = 0 +
+
2x1 5.85422 2 3 = 0 +
+ ~2 0.85423 = 0

Choose 2 1 = 1. Then 22 = -?= -0.427 and x3 = -- = 0.5. Therefore,

the third eagenvector is:

and after division b y 41+ 0.4272 + 0.52 = 1.197 we get:

U3 =7):;;;(

The corresponding eigenvectors of gTg are given by gTu;; i.e. the first one as:

2.187

W e normalize it by dividing by 42.1872 + 0.9342 + 1.0942 = 2.618, to get:

0.835

Similarly
v2= ( 01 21 0 )( ) (
Of7
0 1 1-0.894 -0.894
= Of7),
34 Processing:
Image The Fundamentals

I1 while the third eigenvector is

1 20.121
0 0.835

1 which after normalization becomes:

v3 = ( -::;l;)
0.319

What is the singular value decomposition of an image?

The Singular Value Decomposition (SVD) of an image g is its expansion in vector

outer products where the vectors used are the eigenvectors of ggT and gTg and the
coefficients of the expansion are the eigenvalues of thesematrices.In that case,
equation (2.8), applied for image g instead of image f , can be written as:

g = cT

i=l
X3u;vT , (2.27)

since the only non-zero terms are those with i = j .

How can we approximate an image using SVD?

If in equation (2.27) we decide to keep only Ic < T terms, we shall reproduce an

approximated version of the image as:

gk: = c
i=l
k
X$liVT (2.28)

Example 2.9

A 256 X 256 grey image with 256 grey levels is to be transmitted. How
many terms can be kept in its SVD before the transmission of the trans-

1 formed image becomes too inefficient in comparison withthe transmis-

sion of the original image? (Assume that real numbers require 32 bits
each.)
1
Assume that X7 is incorporated into one of the vectors. When we transmit one
term of the SVD expansion of the image we must transmit:
Image 35

2 X 32 X 256 bits.

This is becausewe have to transmit two vectors, and each vector has 256 com-
ponents in the case of a 256 X 256 image, and each component requires 32 bits
since it is a real number. If we want to transmit the full image, we shall have to
transmit 256 X 256 X 8 bits (since each pixel requires 8 bits). Then the maximum
number of terms transmitted before the SVD becomes uneconomical is:
256 X 256 X 8 - 256
Ic= - - = 32
2 X 32 X 256 8

What is the error of the approximation of an image by SVD?

The difference between the original and the approximated image is:

D E g -gk = cT

i=k+l
X3u;vT (2.29)

We can calculate how big this error is by calculating the norm of matrix D , i.e.
the sum of the squares of its elements. From (2.29) it is obvious that the mn element
of D is:
T

i=k+1

i=k+1 i=k+1 j=k+l,j#i

m n

m n i=k+l m n i=k+lj=k+l,j#i
r r r

i=k+l m n i=k+l j=k+l,j#i m n

However, ui, vi are eigenvectors and therefore they form an orthonormal set. So:

m n
36 Image Processing:
FundamentalsThe

n m

since uiuT = 0 and viv; = 0 for i #j. Then

c

(2.30)

Therefore, the errorof the approximate reconstructionof the image using equation
(2.28) is equal to the sum of the omitted eigenvalues.

Example 2.10

For a 3 X 3 matrix D show that its norm, defined as the trace of D T D ,

is equal to the sum of the squares of its elements.

Let us assume that:

Then

+ +
trace[DTD] = (d:l dil dgl) (d:2 di2 + + + dz2)+ (d:3 + d;3 + dz3)
= sum of all termssquared.

How can we minimize the error of the reconstruction?

If we arrange the eigenvalues X i in decreasing order and truncate the expansion at

some integer L < r , we approximate the image g by gk whichis the least square
Image 37

error approximation. This is because the sum of the squares of the elements of the
difference matrix is minimum, since it is equal to the sum of the unused eigenvalues
which have been chosen to be the smallest ones.
Notice that this singular value decomposition of the image is optimal in the least
square error sense but the base images (eigenimages), with respect to which we ex-
panded the image, are determined by the image itself. (They are determined by the
eigenvectors of gTg and ggT.)

Example 2.11

In the singular value decomposition of the image of Example 2.8 only

the first term is keptwhile the others are set to zero.Verify that
the square error of the reconstructed image is equal to the sum of the
omitted eigenvalues.

If we keep only the first eigenvalue, the image is approximated by the first eigen-
image only, weighted by the square root of the corresponding eigenvalue:

91 = Au~vT
= 6%0.934
(1:) (0.835 0.357 0.418)

= (1:)
2.444 (0.835
0.357
0.418) =
(
0.697 0.298
0.349
2.041 0.873 1.022
0.1750.1500.350 1
The error of the reconstruction is given by the difference between g1 and the orig-
inal image:

1
0.303 -0.298 -0.349
g - g1 =
( -0.041 0.127 -0.022
-0.350 -0.150

The sum of the squares of the errors is:

0.825

0.3032 + 0.2982 + 0.34g2 + 0.0412 + 0.1272 + 0.0222 + 0.3502 + 0.1502 + 0.8252

= 1.146

This is exactly equal to the sum of the two omitted eigenvalues X2 and AS.

What are the elementary images in terms of which SVD expands an image?

There is no specific answer to this because these elementary images are intrinsic to
each image; they are its eigenimages.
38 Image Processing:
FundamentalsThe

Example 2.12

Perform the singularvalue decomposition (SVD) of the following

8 n)
image:

g=(!

Thus identify the eigenimages of the above image.

2 0 2

The eigenvalues of ggT are the solutions of:

2-X 0 2
0 l-X 0 =O+(2-X)2(1-X)-4(1-X)=O+
2 0 2-X
(1 - X)(X - 4)X = 0

The eigenvaluesare: X1 = 4, X2 = 1, X3 = 0. Thecorrespondingeigenvectors

are:
2x1 + 2x3 = 4x1 = X3
+ 21
22 =0

W e choose 21 = 23 = 2 so thattheeigenvectorhas
di unit length.Thus
(z h).
UT= 1 0

W e have a n extra constraint as this eigenvector must be orthogonal to the first

one, that is:

u:u2 =0 * Jl Z +J lZ
-21 -23 = 0 j21 = -23

These equations are satisfied if x1 = 2 3 = 0 and 22 is anything. 22 is chosen to

be 1 so that u2 hasalso unit length.Thus: UT = ( 0 1 0).
 40 Image Processing:
FundamentalsThe

1 0 1 X1 + X 3 = 2x1 X1 =X3
1 0 1
* 22 anyvalue

1
Choose x1 = 2 3 = -1
and 2 2 = 0, so u1 = (-, 0, -)1 T .
J2 J2 J2
The second eigenvector must satisfy the same constraints and must be orthog-
onal to u1. Therefore:
T
U2 = (0,1,0)

Calculate the corresponding eigenvectors of gTg using (see Example 2.7):

vi = g T ui

Therefore:

. . = ( l 0 10 0l ) ( $ ) = ( & ; &0)=(+)

0 1 0

Normalize it so that lvll = 1; i.e. set v1 = (0,1,0)T .

vq=
0 1 0
(1 0 1)
0 1 0
(;)=(!)
Normalize it so that Ivz I = 1; i.e. set v2 = (5,0, L). Then the SVD of g is:
f i
T

g = A U l V T + I/%uqv;

= &( ;)++ ; +)
O
O

O
O
L

L
O
O

O
0 0 0

0 1 0 0 0 0

= (:: :)+(: : :)
These two matrices are the eigenimages of g.
Image 41

Example 2.14

Show the different stages of the SVD of the following image:

{ 255 255 255 255 255 255 255 255

255 255 255 100 100 100 255 255
255 255 100 150 150 150 100 255
255 255 100 150 200 150 100 255
g= 255 255 100 150 150 150 100 255
255 255 255 100 100 100 255 255
255 255 255 255 50 255 255 255
\ 50 50 50 50 255 255 255 255

The ggT matrix is:

‘520200 401625 360825 373575 360825 401625 467925 311100
401625 355125 291075 296075 291075 355125 381125 224300
360825 291075 282575 290075 282575 291075 330075 205025
373575 296075 290075 300075 290075 296075 332575 2 17775
360825 291075 282575 290075 282575 291075 330075 205025
401625 355125 291075 296075 291075 355125 381125 224300
467925 381125 330075 332575 330075 381125 457675 258825
\311100 224300 205025 2 17775 205025 224300 258825 270100
Its eigenvalues sorted in decreasing order are:

2593416.500
111621.508
71738.313
34790.875
11882.712
0.009 0.001 0.000

The last three eigenvalues are practically 0, so we compute only the eigenvectors
that correspond to the first five eigenvalues. These eigenvectors are the columns
of the following matrix:

-0.441 0.167 0.080 -0.388 0.764

-0.359 -0.252 0.328 0.446 0.040
-0.321 -0.086 -0.440 0.034 -0.201
-0.329 -0.003 -0.503 0.093 0.107
-0.321 -0.086 -0.440 0.035 -0.202
-0.359 -0.252 0.328 0.446 0.040
-0.407 -0.173 0.341 -0.630 -0.504
-0.261 0.895 0.150 0.209 -0.256

The g T g matrix is:

42 Image Processing:
FundamentalsThe

457675 457675 339100 298300 269025 308550 349350

457675 457675 339100 298300 269025 308550 349350 467925
339100 339100 292600 228550 191525 238800 302850 349350
298300 298300 228550 220050 185525 230300 238800 308550
gTg= 269025 269025 191525 185525 237550 237800 243800 321300
308550 308550 238800 230300 237800 282575 291075 360825
349350 349350 302850 238800 243800 291075 355125 401625
(467925 467925
349350
308550
321300
360825
401625
520200
Its eigenvectors, computed independently, turn out to be the columns of the fol-
lowing matrix:

-0.410 -0.389 0.264 0.106 0.012

-0.410 -0.389 0.264 0.106 0.012
-0.316 -0.308 -0.537 -0.029 -0.408
-0.277 -0.100 0.101 -0.727 -0.158
-0.269 0.555 0.341 0.220 -0.675
-0.311 0.449 -0.014 -0.497 0.323
-0.349 0.241 -0.651 0.200 0.074
-0.443 0.160 0.149 0.336 0.493

In Figure 2.1 the original image and its five eigenimages are shown. Each eigen-
imagehas been scaled so thatitsgreyvaluesvarybetween 0 and 255. These
eigenimages have t o be weighted by the square root of the appropriate eigenvalue
and added t o produce the original image. The five images shown in Figure 2.2
are the reconstructed images when one, two,. . ., five eigenvalues were used for the
reconstruction.
Then we calculate the sum of the squared errors for each reconstructed image
according to the formula

c64

i=l
(reconstructed pixel- original pixeO2

W e obtain:

Error for image (a): + + +

230033.32 ( X 2 X3 X4 X5 = 230033.41)
Error for image (b): 118412.02 (X, + +
X4 X5 = 118411.90)

Error for image (c): +

46673.53 (X4 X5 = 46673.59)
Error for image (d): 11882.65 (X5 = 11882.71)
Error for image (e): 0

W e see that the sum of the omitted eigenvalues agrees very well with the error in
the reconstructed image.
Image 43

Figure 2.1: The original image and its fiveeigenimages,each scaled to have
values from 0 to 255.
44 Image Processing:
FundamentalsThe

r-l
Figure 2.2: Image reconstruction using one, two,. . ., five eigenimages from top
right to bottom left sequentially, with the original image shown in (f).
Image 45

Are there any sets of elementary images in terms of which ANY image
can be expanded?

Yes. They are defined in terms of sets of complete and orthonormal sets of discrete
valued discrete functions.

What is a complete and orthonormal set of functions?

A set of functions Sn(t),where n is an integer,is said to be orthogonal over an interval

[O,T]with weight function w ( t ) if

(2.31)

The set is called orthonormal if Ic = 1. It is called complete ifwe cannot find

any other function which is orthogonal to the set and does not belongto the set. An
example of a complete and orthogonal set is the set of functions ejnt which are used
as the basis functions of the Fourier transform.

Example 2.15

Show that thecolumns of an orthogonal matrixform a set of orthonor-

mal vectors.

Let us say that A is an N X N orthogonal matrix (i.e. AT = A-'), and let us

consider its column vectors u1, u2,. . . ,UN. W e obviously have:

U l TUUl T
U2TU1
l U2
U2TU2
.. .
.. .
UITUN
U2TUN ) 1 0 ... 0
= 10 1 .. 0
.. ..
\
I
. .
0 0 :I
... 1
This proves that the columns of A form an orthonormal set of vectors.
Image 47

where p = 1,2,3,. . . and n = 0,1,. . .,2P - 1.

How are the Walsh functions defined?

They are defined in various ways all of which can be shown to be equivalent. We use
here the definition from the difference equation:

Wzj+q(t) = (-l)[fl+q{Wj(2t) + (-l)j+"j(2t - l)} (2.34)

where [;l means the largestinteger which is smallerorequal to ;, Q = 0 or 1,

j = 0,1,2,. . . and
1 forO<t<l
0 elsewhere
The above equations define these functions in what is called natural order. Other
equivalent definitions order these functions in terms of increasing number of zero
crossings of the functions, and that is called sequency order.

How can we create the image transformation matrices from the Haar and
Walsh functions?

We first scale the independent variable t by the size of the matrix we want to create.
Then we consider only its integer values i . Then HI,( i ) can be written in
a matrix form
for k = 0,1,2,. . . ,N - 1 and i = 0,1,. . .,N - 1 and be used for the transformation
of a discrete 2-dimensional image function. We work similarly for WI,( i ) .
Note that the Haar/Walsh functions defined this way are not orthonormal. Each
has to be normalized by being multiplied with 1in the continuous case, or by 1
fl dN
in the discrete case, if t takes up N equally spaced discrete values.

Example 2.18

Derive the matrix which can be used to calculate the Haar transform
of a 4 X 4 image.

First, by using equation (..33), we shall calculate and plot the Haar functions of
the continuous variablet which are needed for thecalculation of the transformation
matrix.
48 Image Processing:
FundamentalsThe

1 forO<t<+
H(1,t) =
<
-1 for f t < 1 1

-1
0 114 I1 t

In the definition of the Haar functions, when p = 1, n takes the values 0 and 1:

Case p = 1, n = 0:

Jz
-a
forO<t<$
for 1 < t < i
L
0 f o r 4z -< t < l 114 I >
l 12 l t
-L
Case p = 1, n = 1.

0 foro<t< f
+ <t <
H(3,t) =
{ a-Jz for
fori <t < 1

To transform a 4 X 4 image we needa 4 X 4 matrix. Ifwescale the t axis by

multiplying it by 4 and take only the integer values o f t (i.e. t = 0 , 1 , 2 , 3 ) we can
construct the transformation matrix. The plots of the scaled functions looklike
this:
Image 49

t
I I I /

0 l 2 3 4 t

-L
L
i 2
3
-
4 t
-\c2
The entries of the transformation matrix are the values of H ( s ,t ) where S and t
take the values 0 , 1 , 2 , 3 . Obviously then the transformation matrix is:

2 Jz-Jz
1
1
1
1 -1
1

0
-;)
0
0 O J z - J z

The factor f appears so that H H T = I , the unit matrix.

50 Image Processing:
FundamentalsThe

Example 2.19

Calculate the Haar transform of the image:

The Haar transformof the image g is A = H g H T . We shall use matrix H derived

in Example 2.18:

1 1 1

1 -1 0 -4
1 1 1

0 od-d 2 0 -4

-
4 0 0 - 4
0 0 4-4

0 0 1-1

Example 2.20

Reconstruct the image of Example 2.19 using an approximation of its

Haar transform by setting its bottom right element be equal to 0.

The approximate transformation matrix becomes:

Image 51

The reconstructed image is given by 3 = HTAH:

1 J z 2 0 0 0 1 1 1 1
1-1-1
4 1 -1 0 0
1 -1 0 -Jz 0 0 1 0 0 O J z - J z

1 J z 2 2 2

-t(!
- -11
1 -1
-Jz0
0
j
-Jz
)(Jz-Jz
-fi
O fi 0 -fi
0 fi
0 0
0 4 4 0 0 1 1

0 4 0 0 0 1 0

The square error is equal to:

0.52 + 0.5’ + l2 = 1.5

What do the elementary images of the Haar transform look like?

Figure 2.3 shows the basis images for the expansion of an 8 X 8 image in terms of the
Haar functions. Each of these images has been producedby taking the outer product
of a discretized Haar function either with itself or with another one. The numbers
along the left and the bottom indicate the orderof the function used along each row
or column respectively. The discrete values of each image have been scaled in the
range [0,255] for display purposes.
52 Image Processing:
FundamentalsThe

0 1 2 3 4 5 6 1

Figure 2.3: Haar transform basis images. In each image, grey means 0, black means
a negative and white means a positive number. Note that each image has been scaled
separately: black and white indicate different numbers from one image to the next.

Example 2.2 1

Derive the matrix which can be used to calculate the Walsh transform
of a 4 X 4 image.

First, by usingequation (2.34), calculateand plottheWalshfunctions of the

continuous variable t which are needed for the calculation of the transformation
matrix.

1 forOSt<l
W(0,t)=
0 elsewhere

0 1 t
 Image 53

Case j = 0, Q = 1, [i]= 0:

ForO<t<i:

0 5 2t < 1 * W ( 0 , 2 t )= 1
1
--<t--<0*-152(t--)<O*W
2 2- 2
Therefore:
1
W ( 1 , t ) = -1 for 0I t <-
2
For f 5 t < 1:
1 5 2t < 2 * W ( 0 , 2 t )= 0
1 1 1
O<t--<-*O52(t--)<l*W
2 - 2 2
Therefore:
1
W ( 1 , t )= -(-l) = 1 for - <t <1
2 -

-
-1
0 I 112 1 t

Case j = 1, Q = 0, [i]= 0:
W ( 2 ,t ) = W ( 1 , 2 t ) - W

ForO<t<a:
54 Image Processing:
FundamentalsThe

1
0 5 2t < -
2
*
W(1,2t) = -1

2
1
-- < t - -1 < --1
24 - 2 2
* 1
-1 5 2(t - -) < --
2
1
* 1
W(1,2(t - -)) = 0

Therefore:
l
W(2,t) = -1 for 0 5 t <-
4
For$<t<i:

< 1 * W(1,2t) = 1
1
- < 2t
2-
1 1 1 1
--<t--<0*--<2(t--)<O*W
4- 2 2- 2
Therefore:
1 1
W(2,t) = 1 for - t <-
4- 2

3
1 5 2t < - =3 W(1,2t) = 0
2
1 1 1 1
O<t--<-*O<2(t--)<-*W
22 42
1,2(t--)
( 3 =-l

Therefore:
1 3
W(2,t) = 1 for - t <-
2- 4
For 5 t < 1:

32 -< 2t < 2 * W(1,2t) = 0

1
24 -
1 1 1
-<t--<-*-<2(t--)<l*W
2 2-
1
1,2(t--)
2 ( 3 =l

Therefore:
3
W(2,t) = -1 for - < t <1
4-
Image 55

{ ( 91
W ( 3 ,t ) = - W ( 1 , 2 t ) + W 1 , 2 ( t - -

vorQ5t<a:

W ( 1 , 2 t ) = -1, W

Therefore:
1
W ( 3 , t ) = 1 for 0 5 t < -4

For$<t<i:

W ( 1 , 2 t ) = 1, W

Therefore:
1 1
W ( 3 , t )= -1 for -
4-
<t < -2

Fori<t<i:

W ( 1 , 2 t ) = 0, W

Therefore:
1 3
W ( 3 , t ) = 1 for - <t
2 -
< -4

For 5 t < 1:
56 Image Processing:
FundamentalsThe

W ( 1 , 2 t ) = 0, W

Therefore:
3
W(3,t)= -1 for - <t < 1
4-

To create a 4 X 4 matrix, we multiply t by 4 and consider only its integer values

i.e. 0,1,2,3. The first row of the matrix will be formed from W ( 0 ,t ) . The second
from W ( l ,t ) , the third from W ( 2 ,t ) and so on:

W = 2! [ 1:
1 1 1
:)
-11 1 1 -1

1 -1 1 -1

This matrix has been normalized b y multiplying it b y $ so that WTW = I , where

I is the unit matrix.

Example 2.2 2
Calculate the Walsh transform of the image:
Image 57

In the general formula of a separable linear transform A = U g V T , u s e U = V =

W as derived in Example 2.21:

A = 4
-1 1 -1
1

1
-: ; -!)(l::;)(!-; ; -!)
1 01 1 1 0 -1

0 1 1 0 -1
-1 1

-1 1
1 221 001

-11-11
8 0 0 0 2 0 0 0

0 0

Can we define an orthogonal matrix with entries only +l or - l ?

Yes. There are the Hadamard matrices named after the mathematician who studied
them in 1893. The Hadamard matrices are defined only for sizes that are powers of
2, in a recursive way, as follows:

For a general size these matrices have beenshown to exist only for sizesup to 200 X 200.
The rows of such matrices can be shown to be equal to the discretized form of the
Walsh functions of the corresponding order. So the Walsh functions can be calculated
in terms of these matrices for N = 2n. The Walsh functions generated in this way are
said to be in Kronecter or lexicographic ordering, i.e. they are not in the same order
as those produced by using equation (2.34).

What do the basis images of the Hadamard/Walsh transform look like?

Figure 2.4 shows the basis images for the expansion of an 8 X 8 image in terms of
Walsh functions in Kronecker ordering. These images are binary.
58 Image Processing:
FundamentalsThe

0 l 2 3 4 5 6 l

Figure 2.4: Hadamard/Walsh transform basis images.

Example 2.23

Show the different stages of the Haartransformof the image of

Example 2.14:

The eight images shown in Figure 2.5 are the reconstructed images when for the
reconstruction the basis images created by one, two,. . ., eight Haar functions are
used. For example, Figure 2.5b is the reconstructed image when only the coefi-
cients that multiply the four basis images at the bottom left corner of Figure 2.3
are retained. These four basis images are created from the first two Haar func-
tions, H(0,t) and H(1,t). Image 2.59 is reconstructed when all the coeficients
that multiply the basis images along the top row and the right column in Figure
2.3 are set to 0. The basis images used were created from the first seven Haar
functions, i.e. H ( 0 ,t ) ,H(1,t ) ,. . .,H ( 6 ,t ) .
Image 59

cc
r

Figure 2.5: Reconstructed images when the basis images used are those created
from the first one, two, three,. . ., eight Haar functions, from top left to bottom
right respectively.
60 Image Processing:
FundamentalsThe

The sum of the squared errors for each reconstructed image is:

Errorforimage(a): 366394
Errorforimage (b): 356192
Errorforimage (c): 291740
Errorforimage (d): 222550
Errorforimage(e): 192518
Errorforimage (f): 174625
Errorforimage(g): 141100
Error
for
image (h): 0

Example 2.24

Show the different stages of the Walsh/Hadamardtransformof the

image of Example 2.14.

The eight images shown in Figure 2.6 are the reconstructed images when for the
reconstruction the basis images created from the first one, two,. . ., eight Walsh
functions areused.Forexample, Figure 2.6f has been reconstructed from the
inverse Walsh/Hadamard transform, by setting t o 0 all elements of the transfor-
mation matrix that multiply the basisimages in the top two rows and the two
rightmost columns in Figure 2.4. These omitted basis images are those that are
created from functions W(6, t ) and W(7, t ) .
The sum of the squared errors f o r each reconstructed image is:

Error for image (a): 366394

Error for image (b): 356190
Error for image (c): 262206
Error for image (d): 222550
Error for image (e): 148029
Error for image (f): 92078
Error for image (g): 55905
Error for image (h): 0
Image 61

cc

Figure 2.6: Reconstructed images when the basis images used are those created
from the first one, two, three,. . ., eight Walsh functions, from top left to bottom
right respectively.
62 Image Processing:
FundamentalsThe

What are the advantages and disadvantages of the Walsh and the Haar
transforms?

From Figure 2.3 notice that the higher order Haar basis images use the same basic
pattern that scans the whole image, as if every basis image attempts to capturemore
accurately thelocal characteristics of the image focusing every time at one place only.
For example, the 16 basis images in the top right quadrant of Figure 2.3 all use a
window of 2 X 2 pixels to reproduce detail in various parts of the image. If we are
not interested in that level of detail, we can set the corresponding 16 coefficients of
the transform to zero. So, when we truncate a Haar reconstruction we must keep
basis images that are created from the first 2O, 2l, 22,. . . Haar functions instead of
retaining an arbitrary number of them. In this way the reconstruction error will be
uniformly distributed over the whole image. Alternatively, we maywish to retain
those basis images (i.e. retain their corresponding coefficients) that help reconstruct
most accurately those parts of the image that contain most detail. In other words,
the Haar basis functions allow us to reconstruct with different levels of detail different
parts of an image.
In contrast, higher order Walsh basis images try to approximate the image as a
whole, with uniformly distributed detail structure. This is because Walsh functions
cannot take the 0 value. Notice how this difference between the two bases is reflected
in thereconstructedimages:bothimages 2.5g and 2.6g have beenreconstructed
by retaining the same number of basis images. In Figure 2.5g the flower has been
almost fully reconstructed apart from some details on the right and at the bottom,
because the omitted basis images were those that would describe the image in those
locations, and the image happened to have significant detail there. That is why the
reconstructed error in this case is higher for the Haar than the Walsh case. Notice
that the error in the Walsh reconstruction is uniformly distributed over the whole
image.
On the other hand, Walsh transforms have the advantage over Haar transforms
that the Walsh functions take up only two values +l or -1, and thus they are easily
implemented in a computer as their values correspond to binary logic.

What is the Haar wavelet?

The property of the Haar basis functions to concentrate on one part of the image at
a time is a characteristic property of a more general class of functions called wavelets.
The Haar wavelets are all scaled and translated versions of the same function and for
an 8 X 8 image they are shown in the seven top-most and the seven right-most panels
of Figure 2.3. The function represented by the bottom left panel of Figure 2.3, i.e. the
average flat image, is called the scaling function. The basis images represented by the
other panels in the first column and the bottom row of Figure 2.3 are produced from
combinations of the scaling function and the wavelet. All panels together constitute
a complete basis in terms of which any 8 X 8 image can be expanded.
Image 63

What is the discrete version of the Fourier transform?

The l-dimensional discrete Fourier transform of a function f(k) defined at discrete

points k , is defined as:

(2.35)

The 2-dimensional discrete Fourier transform (DFT) for an N X N image is defined

as :
- N-1N-l

I .

b=O 1=0

Unlike the other transforms that were developed directly in the discrete domain,
this transform was initially developed in the continuous domain. To preserve this
“historical” consistency, we shall go back into using function arguments rather than
indices. Further, because we shall have to associate Fourier transforms of different
functions, we shall use the convention of the Fourier transform being denoted by the
same letter as the function, butwith a hat on the top. Different numbers of hats will
be used to distinguish the Fourier transforms that refer to different versions of the
same function. The reason for this will become clear when the case arises. For the
time being, however, we define the Fourier transform of an M X N image as follows:

(2.37)

We must thinkof this formula as a “slot machine” wherewhen we slot in a function,

out pops its DFT:
- M-1N-l
(2.38)

B2.3: What is the inverse discrete Fourier transform?

Multiply both sides of (2.37) by e ” ” J ~ + ~ ] & and sum over all m and n from
0 to M - l, and N - l respectively. We get:
64 Image Processing:
FundamentalsThe

Let us examine the sum:

M-1

c
m=O
e2.rrjtE
(2.40)

where t is an integer. This is a geometric progression with M elements, first term

1 ( m = 0) and ratio q = e 2 T j h . The sum of the first n terms of such a geometric
progression is given by:
n-l
qn - 1
= ~ for q # l (2.41)
k=O q-1
The sum (2.40) is therefore equal to:

If however t = 0, all terms in (2.40) are equal to 1 and we have Czzi 1 = M .

so:
M-l

c
m=O
e2TjtE =
M
0
ift=O
iftf0
(2.43)

Applying this to equation (2.39) once for t =q - lc and once for t =p - 1 we

deduce that the right hand side of (2.39) is:
. M-l N-l

where S(a - b ) is 0 unless a = b. Therefore, the above expression is g ( q , p ) , i.e.

(2.44)

This is the inverse 2-dimensional discrete Fourier transform.

Image 65

How can we write the discrete Fourier transform in matrix form?

We construct matrix U with elements

(2.45)

where 2 takes the values 0,1, . . .,N - 1 along eachcolumn and a takes the same values
along each row. Notice that U constructed this way is symmetric; i.e. UT = U . Then
the 2-dimensional discrete Fourier transform of an image g in matrix form is given
by :
A = UgU (2.46)

Example 2.2 5

Derive the matrix with which the discrete Fourier transform of a 4 X 4

image can be obtained.

Therefore:

-;
U = 2! ( ; 1 -11 -11
1 -1

-1
;)
-j
66 Image Processing:
FundamentalsThe

Example 2.26

Use matrix U of Example 2.25 to compute the discrete Fourier trans-

form of the following image:

0 0 1 0

(1 1 : 1)
9 = 0 0 1 0

Calculate first gU:

0 0 11 01 1 -1 1 -1

0 0 1 0 -1 -j 1 -1 1 -1

Multiplytheresult by U from thelefttogetUgU = A (thediscrete Fouraer

transform):

1 1 1 -1 1 -1

-1 -j 1 -1 1 -1
4 -4 4 -4 1 -1 1 -1

4
0 0 0 0

Is matrix U used for DFT unitary?

We must show that any row of this matrix is orthogonal to any other row. Because
U is a complex matrix, when we test for orthogonality we must use the complex
conjugate of one of the two rows we multiply. The product of rows obtained for
X = X I and X = x2 is given by:

(2.47)

By observing that this sumis the sum of the first N terms of a geometric progres-
2s(.l--.2)a
sion with first element 1 andratio e-j N , we can show that it is zero.
 Image 67

For the matrix to be unitary,we must also show that the magnitude of each row
vector is 1. If we set 2 2 = XI in (2.47), we have for the square magnitude:

(2.48)
O LY=O

This cancels with the normalizing constant that multiplies the matrix.

Example 2.2 7

Derive the matrixU needed for the calculationof DFT of an 8 X 8 imag

By applying formula (2.45) with N = 8 we obtain:

1 1 1 1 1 1
,-j%x2 ,-j%X3
,-j%x4 ,-j%x5 ,-j%xS e-j X'
,-j%X4 ,-j%x6
,-j%xS ,-j%xlO ,-j%X12 ,-j%Xl
,-j%xS ,+%X9
,-j%x12 ,-j%x15 ,-j%xlS ,-j%x2
,-j$xS ,-j 9 X 12 ,-j%xlS ,-j$x20 ,-j%x24 ,-j$x2
,-j%XlO ,-j 9 X 15 ,-j%xZO ,-j$x25 ,-j%x30 ,-j$x3
,-j%Xl2 ,-j 9 X 18 ,--j%x24 ,-j$x30 ,-j%x36 ,-j$x4
,-j%x14 ,+%X21
,-j%x28 ,-j%x35 ,-j%x42 ,-j%X4

the above matrix can be simplified to:

1 1 1 1 1 1 1 1
1 ,-j$xl ,-j$x2 ,-j$x3 ,-j$x4 ,-j$x5 ,-j%xS ,-j%x?
1 ,-j%x2 ,-j%x4 ,-j%xS ,-j%xO ,-j%x2 ,-j%x4 ,-j%xS
1 ,-j%xS ,-j%xS ,-j%xl ,-j%x4 ,-j%x? ,-j%x2 ,-j%x5
1 ,-j%xo ,-j%x4 ,-j%xo ,-j%x4 ,-j%xo ,-j%x4
1 ,-j%x5 ,-j%x2 ,-j%x7 ,-j%x4 ,-j%xl ,-j%xS ,-j%x3
1 ,-j%xS ,-j%x4 ,-j%x2 ,-j%xO ,-j%xS ,-j%x4 ,-j%x2
1 ,-j%x? ,-j%xS ,-j%x5 ,-j%x4 ,-j%xS ,-j%x2 ,-j%xl
68 Image Processing: The Fundamentals

W h i c h are the elementary images in terms of which DFT expands an

image?

As the kernel of DFT is a complex function, these images are complex. They can be
created by taking the outer product between any two rows of matrix U . Figure 2.7
shows the real parts of these elementary images and Figure 2.8 the imaginary parts,
for the U matrix computed in Example 2.27.

0 cT mW
0 1 2 3 4 5 6 7

Figure 2.7: Real part of the Fourier transform basis images.

The values of each image have been linearly scaled to vary between 0 (black) and
255 (white). The numbers along the left and the bottom indicatewhich row of matrix
U was multiplied with which row to produce the corresponding image.
Image 69

0 1 2 3 4 5 6 l

Figure 2.8: Imaginary part of the Fourier transform basis images.

Example 2.28

Compute the real and imaginary

parts of the discrete Fourier transform
of image:

0 0 0 0

(1 : : 1)
9 = 0 1 1 0

W e shall use matrix U of Example 2.25.

70 Image Processing:
FundamentalsThe

A = UgU

g u = -(
2
0 0 0 0
0 1 1 0 )(:
3
j-j
-1
1 1
-; -; =L(;
1
1
0 1
2 2
1 0
0
--1;j
0
1-j 1
0 0
0 j ;- l1)

(S I:p; : ;:;)
0 0 0 0 1 j -1 -j
/ l 1 1 1
~

\ 1 j -1 - j 0 0 0 0
-2 - 2 j 0 2 j - 2

0 0
\ 2j-2 2 0
O -2j
2 ,

1 -1. 0 -1.
2

Re(A) = and Irn(A) =

0 0
1
2
1
2
; 0 0 - ;

Example 2.29

Show the different stages of the Fouriertransform of the image of

Example 2.14.

Theeightimagesshown in Figure 2.9 are thereconstructedimageswhenone,

two,. . ., eight lines of the matrix U were used for the reconstruction. The sum of
the squared errors for each reconstructed image are:

Error for image a: 366394

Error for image b: 285895
Error for image c: 234539
Error for image d: 189508
Error for image e: 141481
Error for image f: 119612
Error for image g: 71908
Error for image h: 0
Image 71

L -1 7
Figure 2.9: Reconstructed image when the basis images used are those created
from the first one, two,. . ., eight lines of matrix U of Example 2.27, from top left
to bottom right respectively.
72 Image Processing:
FundamentalsThe

Why is the discrete Fourier transform more commonlyused than the other
transforms?

The major advantage of the discrete Fourier transform over the Walsh transform is
that it can be defined in such a way that it obeys the convolution theorem. One
can define a corresponding theorem for the Walsh functions but the relationship be-
tween the Walsh transform and the convolution is not as simple and it cannot be
implemented cheaply on a computer. The convolution theorem makes the Fourier
transform by far the most attractive in image processing.
Apart from that, theFourier transform uses very detailed basis functions, so it can
approximate an image with smaller error than the other transformsfor fixed number
of terms retained. This can be judged from the reconstruction errors of Example
2.29, when compared with the reconstruction errors of Examples 2.23 and 2.24. We
must compare the errors for reconstructed images (a), (b) and (d) which correspond
to keeping the first 2’, 2l, and 22 basis functions respectively.
Note, however, that when we say we retained n number of basis images, in the
case of the Fourier transform we require 2n coefficients for the reconstruction, while in
the case of Haar and Walsh transforms we require only n coefficients. This is because
the Fourier coefficients are complex and both their real and imaginary parts have to
be stored or transmitted.

What does the convolution theorem state?

The convolution theorem states that: The Fourier transform of the convolution of
two functions is equal to the product of the Fourier transforms of the two functions.
If the functions are images defined over a finite space, this theorem is true only if we
assume that each image is repeated periodically in all directions.

B2.4: If a function is the convolution of two other functions, what is

the relationship of its DFT with the DFTs of the two functions?

Suppose that we convolvetwo discrete 2-dimensional functions g(n,m) and

w ( n ,m ) to produce another function v ( n , m ) :

(2.49)

Let us say that the discrete Fourier transforms of these three functions are 6 , g
and 8 respectively. To find a relationship between them, we shall try to calculate
the DFT of v(n,m). For this purpose, we multiply both sides of equation (2.49)
with the kernel
Image 73

1
m exp [-j27r (5+ E)] (2.50)

and sum over all m and n. Equation (2.49) then becomes:

- N-1M-I

~ N -M
l -N
l -1M-l

n‘=Om’=O n=O m=O

We recognize the left hand side of this expression as being the discrete Fourier
transform of ‘U;i.e.
~ N -M
l -N
l -lM-l

We would like to split this into the product of two double sums. To achieve this,
we must have independent indices for g and W . We introduce new indices:

n - n‘ n”, m - m‘ m”

Then n = n’ + n”, m = m’ + m” and we must find the limits of m’‘ and n”.
To do that, we map the area overwhich we sum in the ( n , m ) space into the
corresponding area in the (n”,m”) space:

mL m=M-l I dLM-l-m /

The area over which we sum in the ( n ,m) space is enclosed by four lines with
equations given on the left hand side of the list below. Each of these linesis
transformed to a line in the (n”,m”)space, given on the right hand side of the
list below. These transformed lines define the new limits of summation.
74 Image Processing:
FundamentalsThe

Then the last expression for .;(p, q ) becomes:

M-l-m' N-l-n'

m"=m' n,,=n'

Let us concentrate on the last two sums of (2.52). Let us call that factor T . We
can separate the negative from the positive indices and write:

T = [e
m"=-m'
+ c ][ e c]
M-l-m'

m"=O n"=-n'
+
N-l-n'

n" =o
g(n/J,m/J)e-2T./[$+G1 (2.53)

Clearly the two arrays g and W are notdefined for negative indices. We may choose
to extend their definition for indices outside the range [0, N - l], [0, M - l] in
any which way suits us. Let us examine the factor:

We define a new variable m"' = M + m". Then the above expression becomes:
M-]

Now if we choose to define: g(n",m"' - M ) = g(n",m"'),the above sum is:

M-l

m"' is a dummy index and we can call it anything we like.Say we call it m".
Then the above expression becomes:

This term is added to the term

Image 75

and the two together can be written as:

m” =0

We can work in a similar way for the summation over index n”, and assume that
g is periodic also in its first index with period N . Then, under the assumption
we made about the definition of g outside its real area of definition, the double
sum we called T is:

T= c c g(n//,m//)t-%ii./[$+G]
M-l N-l

m“=O n”=O
(2.54)

This does not contain indices m‘, n‘ and therefore it is a factor that multiplies the
double sum over n’ and m‘ in (2.52). Further, it is recognized to be m j ( p ,4 ) .
Similarly in (2.52)we recognize the discrete Fourier transform of W and thus(2.52)
becomes:

(2.55)

under the assumptions that:

g(n,m) = g(n - N , m - M)
W(72,rn) w(n-N,m-M)
9(n,m) dn,m-M )
w(n,m)
g(n,m)
w(n,m)
=
= d.
w ( n , m- M )
-N,m)

w(n - N,m)

i.e. we assume that the image arrays g and w are defined in the whole ( n , m )
space periodically, with period M and N in the two directions respectively. This
corresponds to the time convolution theorem.
The frequency convolution theorem would have exactly the same form. Be-
cause of the symmetry between the discrete Fourier transform and itsinverse, this
implies that the discrete Fourier transforms of these functions are also periodic in
the whole ( 4 , ~space,
) with periods N , M respectively.
Note: The factor m in (2.55) appears because we defined the discrete Fourier
transform so that the direct and the inverse ones are entirely symmetric.
76 Image Processing:
FundamentalsThe

Example 2.30 (B)

g ( n , m ) and w ( n , m ) are two N X M images. Their DFTs are g@, q) and

G ( p ,q ) respectively. We create the image ~ ( n m), by

4 7 2 7 m> = m>X 4%m) (2.56)

m,) , ?(p,q ) , in terms of g(p, q)and G@, 4).
Express the DFT of ~ ( n

Let us take the DFT of both sides of equation (2.56):

Substitute g and W in terms of g and 6 :

r=o s=o

Let us introduce some new variables U and v instead of S and r :

u=r+q+limits:qtoM-l+q
v=p+s+limits:ptoN-l+p

Therefore
M-l N-l M-l N-l M-l+u N-l+D

-2?4y+=] -27rj[+++]
G(w - p , U - q)e ~e

m=o n=O
Image 77

W e know that (see equation (2.43) ):

M-l
M fort =0
0 fort # 0
m=O

Therefore:
- M-lN-lM-l+aN-l+v

Convolution of 6 with 3

Example 2.31

Show that if g(k,Z) is an M X N image defined as a periodic function

with periods N and M in the whole (k,Z)space, its DFT g ( m , n ) is also
periodic in the (m,n)space, with the same periods.

W e m u s t show that ij(m + M , n + N ) = g(m,n ) . W e start from the definition oj

ij(m,n):
. M-lN-l

Then:
78 Image Processing:
FundamentalsThe

Example 2.32 (B)

Show that if v ( n , m ) is defined by:

N-l M-l
v ( n , m) C C g(n - n',m - m ' ) w ( n ' , m') (2.57)
n'=O m'=O

where g(n,m) and w ( n , m ) aretwo periodically defined images with

periods N and M in the two variables respectively, v ( n , m ) is also given
by:

Define some new variables:

Then equation (2.57) becomes:

n-N+l m-M+l
+ w(n,m)= C C g ( k , l ) w ( n - IC,m - Z ) (2.59)
k=n l=m

Consider the sum:

n-Ntl n

k=n k=-N+n+l
-1 -N+n
= C g ( ~l ), w ( n - L, m - Z ) - C g(k,l ) w ( n - L,m - 1)
L=-N L=-N
\ " / \ " /

Change
variable ~ A + N Change
variable i=k+~
n

k=O

k=O k=O
II

+ C g ( ~l ), w ( n - L, m - Z )
k=O
Image 79

g periodic + g(k - N , 1) = g(k, 1)

W periodic w(s + N , t ) = w ( s , t )

Therefore, the last two sums are identicalandcanceleachother,andthesum-

mation over k in (2.59) is from 0 to N - 1. Similarly,wecanshowthatthe
summation over 1 in (2.59) is from 0 to M - 1, and thus prove equation (2.58).

How can we display the discrete Fourier transform of an image?

Suppose that the discrete Fourier transform of an image is g@,q ) . These quantities,
g@,q ) , are thecoefficients of the expansion of the image into discrete Fourier functions
each of which corresponds to a different pair of spatial frequencies inthe 2-dimensional
( k ,1) plane. As p and q increase, the contribution of these high frequencies to the
image becomes less and less important and thus the values of the corresponding
coefficients g@,q ) become smaller. If we want to display thesecoefficients we may find
it difficult because their values will span a great range. So, for displaying purposes,
and only for that, people use instead the logarithmic function:

This function is then scaled into a displayable range of grey values and displayed
instead ofe@, q ) . Notice that when g@,q ) = 0, d@, q ) = 0 too. This function hasthe
property of reducing the ratiobetween the high values of 6 and thesmall ones,so that
small and large values can be displayed in the same scale. For example, if gmaZ = 10
and gmin = 0.1, to draw these numbers on the same graph is rather difficult, as their
ratio is 100. However, log(l1) = 1.041 and log(l.1) = 0.041 and their ratio is 25. So
both numbers can be drawn on the same scale more easily.

What happens to the discrete Fourier transform of an image if the image

is rotated?

We rewrite here the definition of the discrete Fourier transform, i.e. equation (2.37)
for a square image (set M = N ) :

(2.60)
I .
k=O I=O

We can introduce polar coordinateson the planes ( k ,1) and (m,n) as follows: k =
r c o s 8 , 1 = r s i n 8 , m = w c o s + , n = w s i n + . Wenotethatkm+ln=rw(cosOcos++
sin8 sin+)= rw cos(8 - 4). Then equation (2.60) becomes:

(2.61)
..,
k=O I=O
82 Image Processing:
FundamentalsThe

laom+Ion
L' and 1' take values from 0 to N - 1. We alsonotice that factor e-2Kj N is
independent of Ic' and 1' and therefore can come out of the summation. Then we
recognize in (2.66) the DFT of g(k,1 ) appearing on the right hand side. (Note that
Ic', 1' are dummy indices and it makes no difference whether we call them Ic', 1' or Ic,
1.) We have therefore:

(2.67)
kOm+Ion
The DFT of the shifted image = the DFT of the unshifted image xe-2nj N

Similarly, one can show that:

I I
The shifted DFT of an image = the DFT of [ image x e 2 K ~ m o k ~ n o 1 ]

or

What is the relationship between the average value of a function and its
DFT?

The average value of a function is given by:

(2.68)

If we set m = n = 0 in (2.60) we get:

(2.69)
1.
k=O 1=0

Therefore, the mean of an image and the direct component (or de) of its DFT are
related by:

(2.70)

Example 2.34

Confirm the relationship between the average of image

Image 83

0 0 0 0

(1 : : 1)
9 = 0 1 1 0

and its discrete Fourier transform.

Apply thediscreteFouriertransformformula (2.37) for N = M = 4 and for

m=n=O:
3 3
1 1
g(0,O) - X X g ( L , l ) = -(0+0+0+0+0+1+1+0+0+1+1+0
4
k=O z=o
+0+0+0+0+0)=3

The mean of g as:

l l
g -~~;g(L,z)=-(o+O+O+O+O+1+l+O+O+l+l+O
16 k=O I=O 16
4 1
+0+0+0+0+0) =-=-
16 4
Thus N g = 4 X a = 1 and (2.70) as confirmed.

What happens to the DFT of an image if the image is scaled?

When we take the average of a discretized function over an area over which this
function is defined, we implicitly perform the following operation: We divide the area
into small elementary areas of size Ax X Ay say, take the value of the function at
the centre of each of these little tiles and assume that it represents the value of the
function over the whole tile. Thus, we sum and divide by the total number of tiles.
So, really the average of a function is defined as:

(2.71)
2=0 y=o

We simply omit Ax and Ay because X and y are incremented by 1 at a time, so

Ax = 1, Ay = 1. We also notice, from the definition of the discrete Fourier transform,
that really, the discrete Fourier transform is a weighted average, where the value of
g(L, 1) is multiplied by a different weight inside each little tile. Seeing the DFT that
way,we realize that the correct definition of the discrete Fourier transform should
include a factor AL X AZ too, as the area of the little tile over which we assume the
value of the function g to be constant. We omit it because AL = A1 = 1. So, the
84 Image Processing: The Fundamentals

formula for DFT that explicitly states this is:

g(m,n) = - cc
1 N-lN-l
N k=O 1=0
lam+ln
g(k, l)e-2"i(N)AkAl (2.72)

Now suppose that we change the scales in the (k,1)plane and g(L,l) becomes
g ( a k , Pl). We denote the discrete Fourier transform of the scaled g by g(m,n). In
order to calculate it, we must slot function g(&, @l) in place of g ( k , 1) in formula
(2.72). We obtain:

g(m,n)=
+

cc
N-1N-l

k=O 1=0
g(ak,,kll)e-2"j
km+ln
N Aka1 (2.73)

We wish to find a relationship between g(m,n) and g(m,n). Therefore, somehow

we mustmake g(k,1) appear on the right hand side of equation (2.73). For this
purpose, we define new variables of summation k' a k , 1' Pl. Then:

The summation that appears in expression

this spans all pointsover whichfunction
g(k', l') is defined, except that the summation variables k' and 1' are not incremented
by 1 in each step. We recognize again the DFT of g ( k , l ) on the right hand side of
(2.74), calculated not at point (m,n) but at point (E, $). Therefore, we can write:
1 m n
$(m,n) = -g(- -) (2.75)
aP a'P
1
The DFT of the scaled function = lproduct of scaling factorslX the DFT of the
unscaled function calculated at the same point inversely scaled.

B2.5: What is the Fast Fourier Transform?

All the transforms we have dealt with so far are separable. This means that they
can be computed astwo l-dimensional transforms asopposed to one 2-dimensional
transform. The discrete Fourier transform in 2 dimensions can be computed as
two discrete Fourier transforms in l dimension, using special algorithms which
are especially designed for speed and efficiency. We shall describe briefly here the
Fast Fourier Transform algorithm called successive doubling. We shall work in 1
dimension. The discrete Fourier transform is defined by:

(2.76)
Image 85

-2rj
where W N e 7 . Assumenow that N = 2n. Then we can write N as 2 M and
substitute above:

We can separate the odd and even values of the argument of f . Then:

Obviously w ~ =F WE and w2ux+u wFw;~.

Then:
2M -

We can write:
1
f(U) E 5 { f e v e n ( U ) + fodd(U)wgM} (2.80)

where we have defined f e v e n ( uto) be the DFT of the even samples of function
f and fodd to be the DFT of the odd samples of function f . Formula (2.80),
. . ,N ,
however, defines f ( u ) only for U 5 M . We need to define f ( u ) for U = 0,1,.
i.e. for U up to 2 M . For this purpose we apply formula (2.79) for argument of f ,
u+M:

However:

WZM = ,L,,";= WL
,u+M = M
(2.82)
2 M w;Mw2M = -w&4
so:

(2.83)

where
86 Image Processing:
FundamentalsThe

We note that formulae (2.80) and (2.83) with definitions (2.84) fully define f ( u ) .
Thus, an N point transform can be computed as two N/2 point transforms given
by equations (2.84). Then equations (2.80) and (2.83) can used
be to calculate the
full transform. It can be shown that the number of operations required reduces
from being proportional to N 2 to being proportional to N n ; i.e. Nlog,N. This
is another reason why images with dimensions powers of 2 are preferred.

What is the discrete cosine transform?

If the rows of matrices h, and h, are discrete versions of a certain class of Chebyshev
polynomials, we have the ewen symmetrical cosine transform. It is called this because
it is equivalent to assuming that the image is reflected about two adjacent edges to
form an image of size 2N X 2N. Then the Fourier transform of this symmetric image
is taken and this is really a cosine transform. The discrete cosine transform (DCT)
defined in thisway has found wide use in JPEG coding according to which each image
is divided into blocks of size 8 X 8. The cosine transform of each block is computed,
and the coefficients of the transformation are coded and transmitted.

What is the “take home” message of this chapter?

This chapter presented the linear, unitary and separable transforms we apply to im-
ages. These transforms analyse each image intoa linear superposition of elementary
basis images. Usually these elementary images are arranged in increasing order of
structure (detail). This allows us to represent an image with as much detail as we
wish, by using only as many of these basis images as we like, starting from the first
one. The optimal way to do that is to use as basis images those that are defined by
the image itself, the eigenimages of the image. This, however, is not very efficient,
as our basis images change from one image to the next. Alternatively, some bases
of predefined images can be created with the help of orthonormal sets of functions.
These bases try to capture the basic characteristicsof all images. Once the basis used
has been agreed, images can be communicated between different agents by simply
transmitting the weights with which each of the basis images has to be multiplied
before all of them are added to create the original image. The first one of these basis
images is always a uniform image. The form of the rest in each basis depends on the
orthonormal set of functions used to generate them. As these basic images are used
to represent a large number of images, more of them are needed to represent a single
image than if the eigenimages of the image itself were used for its representation.
However, the gains in number of bits used come from the fact that the basis images
are pre-agreed and they do not need to be stored or transmitted with each image
separately.
The bases constructed with the help of orthonormal sets of discrete functions are
more easy to implement in hardware. However, the basis constructed with the help
Image 87

of the orthonormal set of complex exponential functions is by far the most popular.
The representation of an image in terms of it is called a discrete Fourier transform.
Its popularity stems from the fact that manipulation of the weights with which the
basis images are superimposed to form the original image,for the purpose of omitting,
for example, certain details in the image, can be achieved by manipulating the image
itself with the help of a simple convolution.
Chapter 3

Statistical Description of
lmages
What is this chapter about?

This chapter provides the necessary background for the statistical description of
images from the signal processing point of view.

Why do we need the statistical description of images?

In various applications, we often haveto deal with sets of images of a certain type;
for example, X-ray images, traffic scene images, etc. Each image in the set may
be different from all the others, but at the same time all images may share certain
common characteristics. We need the statistical description of images so that we
capture these common characteristics and use them in order to represent an image
with fewer bits and reconstruct it with the minimum error "on average".
The first idea is then to try to minimize the mean square error in the recon-
struction of the image, if the same image or a collection of similar images were to be
transmitted and reconstructed several times, as opposed to minimizing the square
error of each image separately. The second idea is that the data with which we
would like to r present the image must be uncorrelated. Both these ideas lead to
the statistical description of images.

Is there an image transformation that allows its representation in terms of

uncorrelated data that can be used to approximate the image in the least
mean square error sense?

Yes. It is called Karhunen-Loeve or Hotelling transform. It is derived by treating the

image as an instantiation of a random field.
90 Image Processing:
FundamentalsThe

What is a random field?

A random field is a spatial function that assigns a random variable at each spatial
position.

What is a random variable?

A random variable is the value we assign to the outcome of a random experiment.

How do we describe random variables?

Random variables are described in terms of their distribution functions which in turn
are defined in terms of the probability of an event happening. An event is a collection
of outcomes of the random experiment.

What is the probability of an event?

The probability of an event happening is a non-negative number which has the follow-
ing properties:
(A) The probability of the event which includes all possible outcomes of the exper-
iment is 1.
(B) The probability of two events which do not have any common outcomes is the
sum of the probabilities of the two events separately.

What is the distribution function of a random variable?

The distribution function of a random variable f is a function which tells us how

likely it is for f to be less than the argument of the function:

function of f variable
Clearly, Pf(-co) = 0 and Pf(+co)= 1.

Example 3.1

If 21 5 2 2 , show that P f ( z 1 )5 Pf(22).

Suppose that A is the event (i.e. the set of outcomes) which makes f 5 z1 and B
c
is the event which makes f 5 2 2 . Since z1 5 z2, A B + B = ( B - A ) U A; i.e.
the events ( B - A) and A do not have common outcomes (see the figure on the
next page).
 Statistical Description of Images 91

fd Zld z2

f6Z2

Then by property (B) in the definition of the probability of an event:

Pf(z2) =
-
?(B) = P ( B - A ) + P ( A ) +
'?'(B - A )
non-negative
number
+
Pf(z1)+

Example 3.2

Show that:

According to the notation of Example 3.1, z1 5 f 5 z2 when the outcome of the

random experiment belongs t o B - A (the shaded area in the above figure); i.e.
P(z1 5 f I z2) = P f ( B - A ) . Since B = ( B - A ) U A , P f ( B - A )= P f ( B ) - P f ( A )
and the result follows.

What is the probability of a random variable taking a specific value?

If the randomvariable takes values from the setof real numbers, it haszero probability
of taking a specific value. (This can be seen if in the result of example 3.2 we set
f = z1 = 2 2 . ) However, it may have non-zero probability of taking a value within
an infinitesimally small range of values. This is expressed by its probability density
function.
92 Image Processing:
FundamentalsThe

What is the probability density function of a random variable?

The derivative of the distributionfunction of a random variable is called the probability

density functionof the random variable:

The expected or mean value of the random variable f is defined by:

m

(3.4)

and the variance by:

The standard deviation is the positive square root of the variance, i.e. af.

How do we describe many random variables?

If we have n random variables we can define their joint distribution function:

Pflf2 . . ,zn)
...fn (z1,22,. = P{fl I Zl,f2 5 z2,. . .,fn I Zn} (3.6)
We can also define their joint probability density function:

What relationships may n random variables have with each other?

If the distribution of n random variables can be written as:

Pflf2...fn (21, z2,. . ., zn) = Pfl (Z1)Pfz(z2). . . Pfn

(Zn) (3.8)
then these random variables are called independent. They are called uncorrelated if

E{fifj) = E{fi)E{fj>,K j ,i # j (3.9)

Any two random variables are orthogonal to each other if

E{fifj) = 0 (3.10)
The covariance of any two random variables is defined as:

cij = E{(fi - Pfi)(fj - P f j H (3.11)

Description
Statistical of Images 93

Example 3.3

Show that if the covariance cij of two random variables is zero, the two
variables are uncorrelated.

Expanding the right hand side of the definition of the covariance we get:

cij = E1fi.t-j - P f i f j - Pfj fi + Pfi Pfj }

= E U i f j ) - PfiE{fj) - PfjE{fi) + PfiPfj
= E U i f j ) - PfiPfj - PfjPfi + PfiPfj
= E U i f j ) - PfiPfj (3.12)
Notice that the operation of taking the expectation value of a fixed number has no
effect on it; i.e. E { p f i ) = p f i . If cij = 0, we get:

E t f i f j ) = PfiPfj = E { f i ) E { f j > (3.13)

which shows that fi and fj are uncorrelated.

How do we then define a random field?

If we define a random variable at every point in a 2-dimensional space we say that

we have a 2-dimensional random field. The position of the space where a random
variable is defined is like a parameter of the random field:

f (r;W i ) (3.14)
This function for fixed r is a random variable but for fixed wi (outcome) is a 2-
dimensional function in the plane, animage, say. As wi scans all possible outcomes of
the underlying statistical experiment, the random field represents a series of images.
On the other hand, for a given outcome, (fixed w i ) , the random field gives the grey
level values at the various positions in an image.

Example 3.4

Using an unloaded die, we conducted a series of experiments. Each

experiment consisted of throwing the die four times. The outcomes
{ w l , w2, w3, wq) of sixteen experiments aregiven below:
94 Image Processing:
FundamentalsThe

If r is a 2-dimensional vector taking values:

1(1,1>7 (1,217 (1,317 (1,417 (2,117 (2,217(27 31, (2741,

(37 l),(37 2), (373), (374),(47 l),(47 2), (4,3)7 (4,4)1
give the series of images defined by the random field f (r;w i ) .

The first image is formed by placing the first outcome of each experiment in the
corresponding position, the secondby using the second outcomeof each experiment,
and so on. The ensemble of images we obtain is:
1 3 3 1 2 5 4 1 1 2 6 3 6 4 6 2
3 2 1 1 4 6 5 2 4 4 3 6 4 2 6 4
6 3 1 5 5 2 2 1 2 5 4 1 4 6 5 6
(2 3 1 3) ( 5 1 2 2 ) (3 5 1 5) ( l 56 4) (3‘15)

How can we relate two random variables that appear in the same random
field?

For fixedr a random field becomes a random variable with an expectation

value which
depends on r:

(3.16)

Since for different values of r we have different random variables, f ( r l ;w i ) and

f (1-2;w i ) , we can define their correlation, called autocorrelation (we use “auto” because
the two variables come from the same random field) as:

s_,L
+m +m
Rff(r1, r2) = E ( f (r1;w2)f (r2;W i ) } = Zlz2pf(z17 z2; r l , r2)dzldz2 (3.17)

The autocovariance C(rl,r2) is defined by:

cff(rl?r2) =E{[f(rl;wi)
-pf(rl)l[f(r2;Wi)-pf(r2)l} (3.18)
Statistical Description of Images 95

Example 3.5

Show that:

Starting from equation (3.18):

How can we relate tworandomvariables that belong to twodifferent

random fields?

If we have two random fields, i.e. two series of images generated by two different
underlying random experiments, represented by f and g , we can define their cross
correlation:

and their cross covaraance:

Two random fields are called uncorrelated if for any rl and r2:

This is equivalent to:

96 Processing:
Image The Fundamentals

Example 3.6

Show that for two uncorrelated random fields we have:

E{f(rl;Wi)g(r2;wj))
= E{f(rl;wi>>E{g(r2;wj))

It follows trivially from the definition of uncorrelated

random fields
( C f g ( r lr2)
, = 0 ) and the expression:

Cfg(rl,r2)= E{f(rl;wi)g(ra;wj))
- CLfh)CL&2) (3.24)
which can be proven in a similar way as Example 3.5.

Since we always have just one version of an image how do we calculate the
expectation values that appear in all previous definitions?

We make the assumption that the image we have is a homogeneous random field and
ergodic. The theorem of ergodicity which we then invoke allows us to replace the
ensemble statistics with the spatial statistics of an image.

When is a random field homogeneous?

If the expectation value of a random field does not depend on r, and if its autocorre-
lation function is translation invariant, then the field is called homogeneous.
A translation invariant autocorrelation function depends on only one argument,
the relative shifting of the positions at which we calculate the values of the random
field:

Example 3.7

Show thattheautocorrelation function R(rl,r2) of a homogeneous

random field depends only on the difference vector rl - r2.

Theautocorrelation function of a homogeneousrandom field istranslation

invariant. Therefore, for any translation vector ro we can write:

+
R f f ( r l , r 2 )= E { f ( r l ; w i ) f ( r 2 ; W i ) )= E { f ( r 1 + r o ; w i ) f ( r 2 r o ; w i ) )
+
= Rff(r1 ro, r2 ro) + Qro (3.26)
Statistical Description of Images 97

How can we calculate the spatial statistics of a random field?

Given a random field we can define its spatial average as:

Jm
1
, oO 3 S,
f (r;wi)dxdy (3.28)

where ssis the integral over the whole space S with area S and r = ( X , y). The result
p(wi) is clearly a function of the outcome on which f depends; i.e. p(wi) is a random
variable.
The spatial autocorrelation function of the random field is defined as:

(3.29)

This is another random variable.

When is a random field ergodic?

A random field is ergodic when it is ergodic with respectto themean and with respect
to the autocorrelation function.

When is a random field ergodic with respect to the mean?

A random field is said to be ergodic with respect to the mean, if it is homogeneous

and its spatial average, defined by (3.28), is independent of the outcome on which f
depends; i.e. it is a constant and is equal to theensemble average defined by equation
(3.16):

f (r;wi)dxdy = p = a constant (3.30)

When is arandomfield ergodic with respect to the autocorrelation

function?

A random field is said to be ergodic with respect to the autocorrelation function

if it is homogeneous and its spatial autocorrelation function, defined by (3.29), is
independent of the outcome of the experiment on which f depends, and depends
98 Image Processing:
FundamentalsThe

only on the displacement ro, and it is equal to the ensemble autocorrelationfunction

defined by equation (3.25):

E { f ( r ;w i ) f ( r + ro; Q)} = s+ms

'S
lim -
S
f(r;wi)f(r + ro; wi)dxdy = R(r0) (3.31)

Example 3.8

Assuming ergodicity, compute the autocorrelation

matrix
of the
following image:

A 3 X 3 image has the form:

(3.32)

To compute its autocorrelation function we writeit as a column vector by stacking

its columns one under the other:

g = (911
g23
g13
g32
g22
g12
g31
g21 g33 ) T (3.33)
The autocorrelation matrix is given by: C = E{ggT}. Instead of averaging Over
all possible versions of the image, we average over all pairs ofpixels at the Same
relative position in the image since ergodicity is assumed. Thus, the autocorrela-
tion matrix will have the following structure:

g32
g22
g12
g31
g21
911 g13 g33
g23
g l l A B C D E F G H I
g 2 1 B A B J D E K G H
g 3 l C B A L J D M K G
9 1 2 D J L A B C D E F
(3.34)
g 2 2 E D J B A B J D E
g 3 2 F E D C B A L J D
g 1 3 G K M D J L A B C
g 2 3 H G K E D J B A B
9 3 3 I H G F E D C B A
The top row and the left-most column of this matrix show which elements of the
image are associated with which in order to produce the corresponding entry in
the matrix. A is the average square element:
 Statistical Description of Images 99

(3.35)

B is the average value of the product of verticalneighbours. W e have six such

pairs. W e m u s t s u m t h e product of theirvaluesanddivide.Thequestionis
whether we must divide by the actual number of pairs of vertical neighbours we
have, i.e. 6, ordividebythetotalnumber of pixelswehave, i.e. 9. This issue
is relevant to the calculation of all entries of matrix (3.34) apart from entry A .
If we divide by the actual number of pairs, the correlation of the most distant
neighbours (for which very few pairs are available) will be exaggerated. Thus, we
chose t o divide by the total number of pixels in the image knowing that this dilutes
the correlation between distant neighbours, although this might be significant. This
problem arises because of the finite size of the images. Note that formulae (3.29)
and (3.28) really apply for infinite sized images. The problem is more significant
in the case of this example which deals with a very small image for which border
effects are exaggerated.

C is the average product of vertical neighbours once removed. W e have three such
pairs:

I D is the average product of horizontal neighbours. There are six such pairs:

I1 E is the average product of diagonal neighbours. There are four such pairs:
 100 Image Processing:
FundamentalsThe

F : F = =0.44

0.33 G:
Statistical
Images
Description of 101

M : X X M = $ = 0.11

So, the autocorrelation matrix is:

2 1.33 0.67 1.33 0.89 0.44 0.33 0.22 0.11
1.33 2 1.33 0.89 1.33 0.89 0.22 0.33 0.22
0.67 1.33 2 0.44 0.89 1.33 0.11 0.22 0.22
1.33 0.89 0.44 2 1.33 0.67 1.33 0.89 0.44
0.89 1.33 0.89 1.33 2 1.33 0.89 1.33 0.89
0.44 0.89 1.33 0.67 1.33 2 0.44 0.89 1.33
0.33 0.22 0.11 1.33 0.89 0.44 2 1.33 0.67
0.22 0.33 0.22 0.89 1.33 0.89 1.33 2 1.33
0.11 0.22 0.33 0.44 0.89 1.33 0.67 1.33 2

Example 3.9

The following ensemble of images is given:

5 4 6 2 3 5 2 63 4 2 8

6 6 7 1 3 5 4 5 2 2 6 6 4 3 2 2 ’
( 55 34 42 3 ) , ( ; 4 6 6 2: ) , ( S 6 45 4 63 ) , ( 3 5 6 5
3 3 4)
4 3 5 4 4 5 4 5 2 7 6 4 5 3 6 6

Is this ensembleofimagesergodic with respect to the mean?Isit

ergodic with respect to the autocorrelation?

It is ergodic with respect to the mean because the average of each image is 4.125
and the average at each pixel position over all eight images is also 4.125.
102 Image Processing:
FundamentalsThe

It is not ergodic with respect to the autocorrelation function. To prove this let us
calculate one element of the autocorrelation matrix? say element E(g23g34) which
is the average of product values of all pixels at position (2,3) and (3,4) over all
images:
4x1+4x5+4x6+6x2+6x4+2x4+2x7+5x4
E(g23934) =
8
- 4+20+24+12+24+8+14+20 - 126
- - - = 15.75
8 8
This should be equal to the elementof the autocorrelation function which expresses
the spatial average of pairs of pixels which are diagonal neighbours from top left
to bottom right direction. Consider the last image in the ensemble. W e have:

- 5x4+3x5+6x2+4x2+4x3+5x4+4x5+2x4+3x4
-
16
- 20+15+12+8+12+20+20+8+12
- = 7.9375
16
The two numbers are not the same, and therefore the ensemble is not ergodic with
respect to the autocorrelation function.

What is the implication of ergodicity?

If an ensemble of images is ergodic, then we can calculateits mean and autocorrelation

function by simply calculating spatial averages over any image of the ensemble we
happen to have.
For example,suppose that we have a collection of M images of similar type
{ g l ( x ,y), g2(x, y), . . . ,g M ( x , y)}. The mean and autocorrelation function of this col-
lection can be calculated by taking averages over all images in the collection. On the
other hand, ifwe assume ergodicity, we can pick up only one of these images and
calculate the mean and the autocorrelation function from it with the help of spatial
averages. This will be correct if the natural variability of all the different images
is statistically the same as the natural variability exhibited by the contents of each
single image separately.

How can we exploit ergodicity to reduce the number of bits needed for
representing an image?

Suppose that we have an ergodic image g which we would like to transmit over a
communication channel. We would like the various bits of the image we transmit
to be uncorrelated so that we do not duplicate information already transmitted; i.e.
 Statistical
Images
Description of 103

given the number of transmitted bits, we would like to maximize the transmitted
information concerning the image.
The autocorrelation function of a random field that has this property is of a spe-
cial form. After we decide how the image should be transformed so that it consists of
uncorrelated pixel values, we can invoke ergodicity to calculate the necessary trans-
formation from the statistics of a single image, rather than from the statistics of a
whole ensemble of images.

Average across images

at a single position=
Average over all
positions on the same
..../

image

______I
Figure 3.1: Ergodicity in a nutshell
of images
Ensemble

What is the form of the autocorrelation function of a random field with

uncorrelated random variables?

The autocorrelation function Rff(r1, r2) of the two random variables defined at po-
sitions rl and r2 will be equal to E(f(r1; ui)}E{f(rz; u i ) } if these two random vari-
ables are uncorrelated (see Example 3.3). If we assume that we are dealing only
with random variables with zero mean, (i.e. E ( f ( r 1 ; w i ) )= E ( f ( r 2 ; u i ) )= 0), then
the autocorrelation function will be zero for all values of its arguments, except for
rl = r2, in which case it will be equal to E ( f ( r l ; u i ) 2 }i.e.
; equal to the variance of
the random variable defined at position r1.
If an image g is represented by a column vector, then instead of having vectors rl
and r2 to indicate positions of pixels, we have integer indices, i and j , say to indicate
components of each column vector. Then the autocorrelation function R,, becomes
a 2-dimensional matrix. For uncorrelated zero meandata this matrixwill be diagonal
with the non-zero elements along the diagonal equal to the variance at each pixel
position. (In the notation used for the autocorrelation matrix of Example 3.8, A # 0,
but all other entries must be 0.)

Howcanwe transform the image so that its autocorrelation matrix is

diagonal?

Let us say that the original image is g and its transformed version is 3. We shall use
the vector versions of them, g and g respectively; i.e. stack the columns of the two
104 Image Processing:
FundamentalsThe

matrices one on top of the other to create two N 2 X 1 vectors. We assume that the
transformation we are seeking has the form:

g=A(g-m) (3.36)

where the transformation matrix A is N 2 X N 2 and the arbitrary vector m is N 2 X 1.

We assume that the image is ergodic. The mean vector of the transformed image is
given by:

pg = E { g } = E{A(g - m)} = AE{g} - Am = A(pg - m) (3.37)

where we have used the fact that m is a non-random vector, and therefore the ex-
pectation value operator leaves it unaffected. Notice that although we talk about
expectation value and use the same notation as the notationused for ensemble aver-
aging, because of the assumed ergodicity, E { g } means nothing else than finding the
average grey value of image 3 and creating anN 2 X 1 vector all the elements of which
are equalto this average grey value. If ergodicity had not been assumed, E { g } would
have meant that the averaging would have to be done over all the versions of image
I

g.
We can conveniently choose m = pg = E{g} in (3.37). Then pg = 0; i.e. the
transformed image will have zero mean.
The autocorrelation function of g then is the same as its autocovariance function
and is defined by:

Cgg = E{EiZT) = E{& - Pg)[Ak - P g ) l T >

T T
= E{A($ - Pg)k - Pg) A }
= A E{(g-Pg)(g-Pg)T} AT (3.38)
\ " /

Definition of autocovariance of
the untransformed image
Notice that again the expectation operator refers to spatial averaging, and because
matrix A is not a random field, it is not affected by it.
So: Cgg = ACggAT. Then it is obvious that Cgg is the diagonalized version of
the covariance matrix of the untransformedimage. Such a diagonalization is achieved
if the transformation matrix A is the matrix formed by the eigenvectors of the auto-
covariance matrix of the image, used as rows, and the diagonal elements of Cgg are
the eigenvalues of the same matrix. The autocovariance matrix of the image can be
calculated from the image itself since we assumed ergodicity (no large ensemble of
similar images is needed).

Is the assumption of ergodicity realistic?

The assumption of ergodicity is not realistic. It is unrealistic to expect that a single

image will be so large and it will include so much variation in its content that all the
diversity represented by a collection of images will be captured by it. Only images
Statistical Description of Images 105

consisting of pure random noise satisfy this assumption. So, people often divide an
image into small patches,which are expected to be uniform, apart from variation due
to noise, and apply the ergodicity assumption to each patch separately.

B3.1: How can we calculate the spatial autocorrelation matrix of an

image?

To define a general formula for the spatial autocorrelation matrix of an image,

we must first establish a correspondence between the index of an element of the
vector representation of the image and the two indices that identify the position
of a pixel in the image. Since the vector representation of an image is created by
placing its columns one under the other, pixel ( k i ,Zi will be the ith element of
)
the vector, where:

i = (Zi - 1)N + kz + 1 (3.39)

with N being the number of elements in each column of the image. We can solve
the above expression for Z
i and ki in terms of i as follows:

ki = ( ’ - l) modulo N
i-l-ki
li = l+ (3.40)
N
Element Cij of the autocorrelation matrix can be written as:

cij = < gigj > (3.41)

where <> means averaging over all pairs of pixels which are in the same relative
position, i.e. for which li - Zj and ki - Icj are the same. First we calculate li - Zj
and Ici - k j :

k0 E ki - kj = (i - l) module N - ( j - l) module N
i-j-ko
li - 1 . - (3.42)
10
’- N
Therefore:
cij = < QklLJk-k0,l-lo > (3.43)
The values which k , 1, k - ko and 1 - 10 take must be in the range of allowable
values, that is in the range [l,NI for an N X N image:

l<k-ko<N =2 l+ko<k<N+ko
Also 1 5 Ic 5 N
+
+-max(1, I Ico) 5 Ic 5 min(N, N
l+ =2

Ico) (3.44)
Statistical
Images
Description of 111

Whatare the basisimagesin terms ofwhich the Karhunen-Loeve

transform expands an image?

Since g = A(g - pg) and A is an orthogonal matrix, the inverse transformation is

given by g - pg = ATg. We can write this expression explicitly:

UN1 31 1
aN2 321

3N1
321

aN,2N 3 2N

aN,NZ-N+l 3N1
112 Processing:
Image The Fundamentals

(3.47)

This expression makes it obvious that the eigenimages in terms of which the K-L
transform expands an image are formedfrom the eigenvectors of its spatial autocor-
relation matrix, by writing them in matrix form; i.e. by using the first N elements of
an eigenvector to form the first column of the corresponding eigenimage, the next N
elements to form the next column and so on. The coefficients of this expansion are
the elements of the transformed image.

Example 3.11

Consider a 3 X 3 image with column representation g. Write down an

expression for the K-L transform of the image in terms of the elements
of g and the elements aij of the transformation matrix A. Calculate
an approximation to the image g by setting the last six rows of A to
zero. Show that the approximation will be a 9 X 1 vector with the first
three elements equal to those of the full transformation of g and the
remaining six elements zero.

Assume that pg is the average grey value of image g . Then the transformed image
will have the form:

'311
321
331
312
322
832
313
323
333

1 (3.48)
113Statistical
Images
Description of

If we set a41 = a42 = . . . = a49 = a51 = . . . = a 5 9 = . . . = a 9 9 = 0, clearly the

last six rows of the above vector will be 0 and the truncated transformation of the
image will be vector

g' = (311 321 331 0 0 0 0 0 o)T (3.49)

According to formula (3.47) the approximation of the image is then:

g13 g12 g11 all

a17 a14
(921
g33
g32
g31
922
g.3) = (E; E;)E;
PS
PS PS
+ 311 (W; :;; :;;)
a37 a34 a27 a24

a29
a26
a23

Example 3.12 (B)

Show that if A is an N 2 X N 2 matrix the ith row of which is vector UT

and C2 an N 2 X N 2 matrix with all its elements zero except the element
at position (2,2) which is equal to c2, then:
T
ATC2A = C ~ U ~ U ~

Assume that uij indicates the j t h component of vector U;. Then:

... 0 0 ... 0

i
U21
U22 ... 0 c2 ... 0
ATC2A U23 ... 0 0 ... 0
.. ..
. .
U2N2 ... 0 0 ... 0
U12 ...
U22 ...
U32 ...

UN22 ...
114 Image Processing:
FundamentalsThe

Example 3.13 (B)

Assuming a 3 X 3 image, and accepting that we approximate it retaining

only the first three eigenvalues of its autocovariance matrix, show that:

Using the result of Example 3.12 concerning the truncated transform of the image
g‘, we have:
I p11 )
821
831
812
E{ 822 (911 921 8 3I 0 0 0 0 0
832
813
823
l16 Image Processing:
FundamentalsThe

Therefore, the mean square error is:

We can exchange the order of taking the expectation value and taking the trace
because trace means nothing more than the summing of the diagonal elements of
the matrix:

[ {
= trace E (ATg-A’Tg’)(gTA-

= trace[E{ArggTA-ATgg‘TA‘-A‘Tg‘gTA+A’Tg’g’~’}]
(3.57)

Matrices A and A’ are fixed, so the expectation operator does not affect them.
Therefore:

E{llg - g‘ll} = trace[ATE(ggT}A - ATE{gg’T}A’

-A’TE{g‘gT}A + A’TE{g’g’T}A‘] (3.58)

In this expression we recognize E { @ T } and E{g’g’T} as thecorrelation matrices

of the two transforms: Cgg and C‘gg.
The gg’ matrix is the product of a vector and its transpose but with the
last N 2 - K components of the transposereplaced by 0. The expectation operator
will make all the off-diagonal elements of @’ zero anyway (since the transform is
such that its autocorrelation matrix has0 all the off-diagonal elements). The fact
that the last N 2 - K elements of g’T are 0 too will also make the last N 2 - K
diagonal elements 0 (see Example 3.13). So, the result is:
- -I T
E k g 1 = C‘gg (3.59)
Similar reasoning leads to:

(3.60)

so:
Statistical Description of Images 117

E{ Ilg - g’l I} = trace[ATCggA - ATC’ggA’ - AfTC’ggA+ A’TC’ggA’] (3.61)

Consider the sum: -ATC’ggA’ + AJTC’--A’ gg = -(A - A’)TC’ggA’. We can par-

tition A in two sections, a K X N2 submatrix AI and an (N2- K ) X N2 submatrix
A2. A’ consists of AI and an ( N 2- K ) X N 2 submatrix with all its elements zero:

A= (-i2-)A1 A’= (-i-)

A1 +

Then ( A - A’)TC‘ggA’ = ( 0 I A T ) C‘ggA’.

C ’ g g can be partitioned into four submatrices:

ClPL= (3.63)
0 1 0
where C1 is K X K diagonal. Then the product is:

(0 I AT) (5 I
0 1
- -)
0

0
=(O) (3.64)

Using this result in (3.61) we obtain:

E{ I Ig - g‘l I} = trace[ATCggA - A‘TC‘ggA] (3.65)

Consider the term ATCggA. We mayassume that Cgg is the sum of N2

matrices, each one being N2 X N2 and having only one non zero element:

X1 0 ... 0 0 ... 0 0 0 ... 0

. .
0 0 ... 0 0 ... 0 0 ... AN2

(3.66)

A is made up of rows of eigenvectors while AT is made up of columns of eigen-

vectors. Then we can write:
Statistical
Images
Description of 119

Example 3.14

The autocovariance matrix of a 2 X 2 image is given by:

3 0 - 1 0

0 -1

Calculate the transformation matrix A for the image, which when used
for the inverse transform will approximatethe image with mean square
error equal to 2.

We must find the eigenvalues of the matrix b y solving the equation:

13-X 0 -1 0 I

I 0 -1 0 3-XI
(3 - X) [(3- X)3 - (3 - X)] - [(3- X)2 - (-1)2] = 0 =3

(3 - X)2 [(3- X)2 - l] - [(3 - X)2 - l] = 0 *

2
[(3 - - l] = 0 =3

(3 - X - 1)2(3- X + =0 +
(2 - 4 7 4 - = 0 =3
X1 = 4, X2 = 4, X3 = 2, X4 = 2 (3.70)

The corresponding eigenvectors for X = 4 are:

3 0 - 1 0 3x1 - X3 = 4x1
3x2 - X4 = 4x2
-21 +3x3 = 4x3
0 -1 -22 +3x4 = 4x4
X3 = -21, X4 = -22, X1 = -23, X2 = -24 (3.71)

Choose: x1 = 2 3 = 0, x2 = 2 4 = -L
Jz
Or choose: x1 = -,1
x3 = --,
Jz17
x2 = 2 4 = 0.
Jz Jz
eigenvectors,
two
first
The therefore,
(5 0 - 5
are: (0 l
0 ) , which are orthogonal to eachother.For
0 -- a and
X = 2 we
have:
120 Image Processing:
FundamentalsThe

(-:: i
0 -1
0 - 1

-n)
0

(i:)=2(i)+
X4
21 =23, 22 =~
3x1 - X3 = 221
322 - X4 = 222
-xl

4 XI
,
+
3x3 = 2x3
+
-x2 3x4 = 2x4
=23, ~2
+
= 2 4 (3.72)

Choose: XI = 2 3 = 0, 2 2 = x4 = -.1
&i
W e do not need to calculate thefourtheigenvector becauseweare interested
in anapproximatetransformationmatrix. By settingsomeeigenvectorsto
( 0 0 0 0 ) themean squareerrorwecommitwhenreconstructingtheimage
is equal to the sum of the corresponding eigenvalues. In this case, if we consider
as transformation matrix the matrix A:

(3.73)

\ o 0 0 0

the error will be equal to X4 = 2.

Example 3.15

Show the different stages of the Karhunen-Loevetransformof the

image in Example 2.14.

Figure 3.2 showsthe 64 eigenimages of theoriginalimage of Example 2.14

according to the Karhunen-Loeve transform.
Theeightimagesshown in Figure 3.3 are thereconstructedimageswhen
8,16,24,32,40,48,56and 64 terms were used for the reconstruction.
The sums of the mean squared errors for each reconstructed image are:

Errorforimage(a): 5360 (g x i = 5365)

Errorforimage(b): 3846 (2 Xi = 3850)

Errorforimage (c): 2715 (2 Xi = 2718)

Statistical Description of Images 121

Errorforimage (d): 1850 (E Xi = 1852)

Errorforimage(e): 1194 (2 Xi = 1195)

Errorforimage(f): 715 (g Xi = 715)

Errorforimage(g): 321 (2 Xi = 321)

Error
for
image (h): 0

Note that the mean squared errors of the reconstructions agree very well with the
sum of the omitted eigenvalues in each case.

Figure 3.2: The 64 eigenimages, each scaled separately to have values from 0 to
255
122 Image Processing:
FundamentalsThe

I
L l
I r 1
l
l
l 1
(h)
Figure 3.3: Reconstructed image when the first 8,16,24,32,40,48,56and 64
eigenimages shown in Figure 3.2 were used (from top left to bottom right respec-
tively).
Statistical
Images
Description of 123

Example 3.16

The autocovariance matrix of a 2 X 2 image is given by:

4 0-1 0

0 -1

Calculate the transformation matrix A for the image, which,when

used for the inverse transform, will approximatethe image with mean-
square-error equal to 6.

Find the eagenvalues of the matrix first:

4-X 0 -1 0
0 4-X 0 -1
-1 0 4-X 0
=o*
0 -1 0 4-X
4-X 0 -1 0 4-X -1
(4-4 0 4-X 0 - 1 -1 0 0 =o*
-1 0 4-X 0 -1 4-X
(4 - X)[(4 - - (4 - X)] - [(4 - - l] = 0 =3
2
[(4 - X)2 - l] = 1 j(4 - X - 1)2(4- X + =0 j
X1 = 5
X2 = 5
X3 = 3
X4 = 3

Since we allow error of image reconstruction of 6 , we do not need to calculate the

eigenvectors that correspond to X = 3.
Eigenvectors for X = 5:

4 0 -1 4x1 - X3 = 5x1 * X1 = X3
4 x 2 - X4 =5x2 * X2= -X4
-21 - 4 x 3 = 5x3* X1 = -X3
0 -1 -22 -4x4 =5x4 * X2 = -X4

1
Choose x1 = 2 3 = 0, 2 2 = - 4’2 4 = -2 4. For X2 choose an orthogonaleigen-
vector, i.e. x 2 = 2 4 = 0, XI = -, x 3 = - 2 . Then the transformation matrix
d2 d2
whichallowsreconstructionwith mean squareerror 6 (equal to the sum of the
omitted eigenvalues) is:
124 Image Processing:
FundamentalsThe

What is the “take home” message of this chapter?

If we view an image as an instantiationof a whole lot of images which are theresult of

a random process, then we can try to represent it as thelinear superposition of some
eigenimages which are appropriatefor representing the whole ensemble of images. For
an N X N image there may be as many as N 2 such eigenimages, while in the SVD
approach there are only N . The difference is that with these N 2 eigenimages we are
supposed to be able to represent the whole ensemble of images, while in the case of
SVD the N eigenimages are appropriate only for representing the one image. If the
set of eigenimages is arranged in decreasing order of the corresponding eigenvalues,
truncating theexpansion of the image in terms of them, approximates the image with
the minimum mean square error, over the whole ensemble of images. In the SVD
case similar truncation leads to the minimum square error approximation.
The cruxof K-L expansion is the assumptionof ergodicity. This assumption states
that the spatial statisticsof a single image are thesame as theensemble statistics over
the whole set of images. If a restricted type of image is considered, this assumption
is clearly unrealistic: Images are not simply the outcomes of a random process; there
is always a deterministic underlying componentwhich makes the assumption invalid.
So, in such case, the K-L transform effectively puts more emphasis on the random
component of the image; i.e. the noise, rather than the component of interest. How-
ever, if many different images are considered, the average greyvalue over the ensemble
even of the deterministic component may be the same from pixel to pixel, and the
assumption of ergodicity may be nearly valid. On the other hand,if one has available
a collection of images representative of the type of image of interest, the assumption
of ergodicity may not need to be made: The K-L transform can be calculated using
ensemble statistics andused to define a basis of images taylor-madefor the particular
type of image.
Chapter 4

Image Enhancement

What is image enhancement?

Image enhancement is the process by which we try to improve an image so that it

looks subjectively better. We do not really know how the image should look, but we
can tell whether it has been improved or not, by considering, for example, whether
more detail can be seen, or whether unwanted flickering has been removed, or the
contrast is better etc.

How can we enhance an image?

The approach largely depends on what we wish to achieve. In general, there are tw o
major approaches: those which reason about the statistics of the grey values of the
image, and those which reason about the spatial frequency content of the image.

Which methods of the image enhancement reason about the grey level
statistics of an image?

0 Methods that manipulate the histogram of the image for the purpose of increas-
ing its contrast.
0 The method of principal component analysis of a multispectral image for ob-
taining a grey level version of it with the maxirmm possible contrast.
0 Methods based on rank order filtering of the image for the purpose of removing
noise.

What is the histogram of an image?

The histogram of an image is a discrete function that is formed by counting the number
of pixels in the image that have a certain grey value. When this function is normalized
to sum up to 1 for all the grey level values, it can be treated as a probability density
126 Image Processing:
FundamentalsThe

function that expresses how probable is for a certain grey value to be found in the
image. Seen this way, the grey value of a pixel becomes a random variable which
takes values according to the outcome of an underlying random experiment.

When is it necessary to modify the histogram of an image?

Suppose that we cannot see much detail in the image. The reason is most likely that
pixels which represent different objects or parts of objects tend to have grey level
values which are very similar to each other. This is demonstrated with the example
histograms shown in Figure 4.1. The histogram of the “bad” image is very narrow,
while the histogram of the image is morespread.

number number
of pixels of pixels

-
grey level grey level

Figure 4.1: (a) is the histogram of a “bad” image while

(b) is the histogram
of a “good” image.

How can we modify the histogram of an image?

Suppose that the grey levels in the original image are given by the values a variable
r obtains and in the new image by the values a variable s obtains. We would like
to find a transformation s = T ( r ) such that the probability density function p r ( r ) in
Figure 4.la is transformed into a probability density function p,(s) which looks like
that in Figure 4.lb, say.
Since p r ( r ) is the probability density function of random variable T , the number
+
of pixels with grey level values in the range r to r dr is pr(r)dr.The transformation
+
we seek will transform this range to [S,s ds]. The total number of pixels in this
range will remain the same but in the enhanced image this number will be p,(s)ds:
ps(s)ds = pr(r)dr (4.1)
 Image 127

This equation can beused to define the transformation T that must be applied to
variable r to obtain variable S, provided we define function p,(s).

What is histogram equalization?

Histogram equalization is the process by which we make all grey values in an image
equally probable, i.e. we set p s ( s ) = c, where c is a constant. Transformation S = T ( r )
can be calculated from equation (4.1) by substitution of p,(s) and integration. We
integrate from 0 up to an arbitrary value of the corresponding variable, makinguse of
the fact that equation (4.1) is valid for any rangeof values. These limits are equivalent
of saying that we equate the distribution functions of the two random variables S
and r:

Here, in order to avoid confusion we replaced the dummy variable of integration

by z. Figures 4.2a-4.2d show an example of applying this transformation to a low
contrast image. Notice how narrow the histogram 4.2b of the original image 4.2a
is. After histogram equalization, the histogram in 4.2d is much more spread, but
contrary to our expectations, it is not flat, i.e. it does not look “equalized”.

Why do histogram equalization programs usually not produce images

with
flat histograms?

In the above analysis, we tacitly assumed that variables r and S can take continuous
values. Inreality, of course, the grey level values are discrete. In the continuous
domain there is an infinite number of numbers in any interval [r,r + dr]. In digital
images we have only a finite number of pixels in each range. As the range is stretched,
and the number of pixels in it is preserved, there is only this finite number of pixels
with which the stretched range is populated. The histogram that results is spread
over the whole range of grey values, but it is far from flat.

Is it possible to enhance an image to have an absolutely flat histogram?

Yes, ifwe randomly re-distribute the pixels across neighbouring grey values. This
method is called histogram equalization with random additions. We can better follow
it ifwe consider a very simple example. Let us assume that we have N I pixels with
value g1 and NZpixels with value g2. Let us say that we wish to stretch thishistogram
+ +
so that we have ( N I N2)/3pixels with grey value 31, ( N I N2)/3pixels with grey
+
value 3 2 and ( N I N2)/3pixels with grey value 3 3 . Let us also assume that we
have worked out the transformation that leads from gi to &. After we apply this
transformation, we may find that we have fi1 pixels with grey value 31, f i 2 pixels
+
with grey value 3 2 , f i 3 pixels with grey value 3 3 , and that fi1 > ( N I N2)/3,
fi2 <
+ +
( N I N2)/3and f i 3 < ( N I N2)/3. +
We may pick at random ( N I N2)/3- f i 3 pixels
with value 3 2 and give them value 3 3 . Then we may pick at random fi1- ( N I N2)/3+
128 Image Processing: The Fundamentals

I:
4000 r

Gray Levels
1w
200

(a) Original image (b) Original histogram

(c) After histogram (d) Histogram afterhistogram

equalization equalization

L
0 Gray Levels
,W 200

(e) After histogram (f)Histogramafterhistogram

equalization with
ran- equalization with random addi-
dom additions tions

Figure 4.2: Enhancing the image of a bathtub cleaner by histogram equal-

ization
Image

(a) Afterhistogram hy- (b) Histogram afterhistogram

perbolisation hyperboliaation

1w 200
Gray Levels
l

(c) After histogram hy- (d) Histogram afterhistogram

perbolization
with
ran- hyperbolization
with
random
dom additions additions

Figure 4.3: Histogram hyperbolization with a = 0.5 applied to the image

of Figure 4.2a

with value 31 and give them value 3 2 . The result will be a perfectly flat histogram.
An example of applying this method can be seen in Figures 4.2e and 4.3~.

What if we do not wish to have an image with a flat histogram?

We may define p , ( s ) to be any function we wish. Then:

LSP"(")dY = / 0k z ) d z

where in order to avoid confusion we have used z and y as the dummy variables of
integration. This equation defines S directly in terms of T . Since ps(y) is known (the
desired histogram), one can solve the integral on the left hand side to find a function
130 Image Processing:
FundamentalsThe

f~ of S. Similarly, the integral on the right hand side can be performed to yield a
function f2 of r; i.e.
fl(S) = f2(7-) * S = fL1f2(r) (4.3)
In practice, one may define an intermediate transformation by:

W Ei'Pr(x)dx (4.4)

This transformation gives the values W of the equalized histogram of the given image.
Clearly:
I" PdYPY =W (4.5)

This defines another transformation such that W = Tz(s) while we actually need S in
terms of W . So this is a three-step process:
1. Equalize the histogram of the given image.
2. Specify the desired histogram and obtain the transformation W = T ~ ( s ) .
3. Apply the inverse of the above transformation to the equalized histogram.
An example of applying this methodto image 4.2a can be seen in Figures 4.3a and
4.3b. This histogram has been produced by setting p,(s) = ae-"' where Q! is some
positive constant. The effect is to give more emphasis to low grey level values and
less to the high ones. This effect is barely visible in Figure 4.3b because it is masked
by the discretization effect. However, in Figure 4.3d it can be seen clearly because
the method of random additions was used.

Whyshouldonewish to perform something other

than
histogram
equalization?

One may wish to emphasize certain grey values more than others, in order to com-
pensate for a certain effect; for example, to compensate for the way the human eye
responds to the different degrees of brightness. This is a reason for doing histogram
hyperbolization: it produces a more pleasing picture.

Example 4.1

The histogram of an image can be approximated by the probability

density function
p'(.) = Ae-'

where r is the grey-level variable taking values between 0 and b, and A

is a normalizing factor. Calculate the transformation S = T ( r ) ,where S
is the grey level value in the transformed image, such that the trans-
formed image has probability density function
Image 131

p,(s) = Bse-"

where S takes values between 0 and b, and B is some normalizing factor.

The transformation S = T ( r ) can be calculated using equation (4.1).

BsePs2ds = AeC'dr

W e integrate both sides of thisequationtoobtaintherelationship between the

distribution functions of variables S and r. To avoid confusion we use as dummy
variables of integration y on the left hand side and X on the right hand side:

B ye-Y2dy = A
LT e-"dx

The left hand side of (4.6) is:

The right hand side of (4.6) is:

W e substitute from (4.7) and (4.8) into (4.6) to obtain:

1- e-s2 A
2
= -(l
B
- e-') +
e-s2 2A
- 1- -(l -e-') +
B

What if the image has inhomogeneous contrast?

The approach described above is global, i.e. we modify the histogram which refers
to the whole image. However, the image may have variable quality at various parts.
For example, it may have a wide shadow band right in the middle, with its top and
bottom partsbeing adequatelyvisible. In that case we can applythe above techniques
locally: We scan the image with a window inside which we modify the histogram but
132 Image Processing: The Fundamentals

we alter only the value of the grey level of the central pixel. Clearly such a method
is costly and various algorithms have been devised to make it more efficient.
Figure 4.4a shows a classical example of an image that requires local enhancement.
The picture was taken indoors looking towards windows with plenty of ambient light
coming through. All outdoor sections are fine, but in the indoor part the film was
under-exposed. The result of global histogram equalization shown in Figure 4.4b is
not bad, but it makes the outdoor parts over-exposed in order to allow us to see
the details of the interior. The result of the local histogram equalization on the other
hand, shown in Figure 4.4c, is overall a much more balanced picture. The window size
used for this was 40 X 40, with the original image being of size 400 X 400. Notice that
no part of the picture gives the impression of being over-exposed or under-exposed.
There are parts of the image, however, that look damaged: at the bottom of the
picture and a little at the top. They correspond to parts of the original film which
received too little light to record anything. They correspond to flat black patches,
and by trying to enhance them we simply enhance the film grain or the instrument
noise. This effect is more prominent in the picture of the hanging train of Wupertal
shown in Figure 4.5. Local histogram equalization (the result of which is shown in
Figure 4 . 5 ~ )attempts to improve parts of the picture that are totallyblack, in effect
trying to amplify non-existing information. However, those parts of the image with
some information content are enhanced in a pleasing way.
A totally different effect becomes evident in Figure 4 . 6 ~whichshows the local
histogram enhancement of a picture taken at Karlstejn castle in the Czech Republic,
shown in Figure 4.6a. The castle at the back consists of flat grey walls. The process
of local histogram equalization amplifies every small variation of the wall to such a
degree that the wall looks like the rough surface of a rock. Further, on the left of the
picture we observe again the effect of trying to enhance a totally black area. In this
case, the result of global histogram equalization looks much more acceptable, in spite
of the fact that ifwe were to judge from the original image, we would have thought
that local histogram equalization would produce a better result.

Is there an alternative to histogram manipulation?

Yes, one may use the mean and standard deviation of the distribution of pixels inside
a window. Let us say that the mean grey value inside a window centred at (X,y) is
m(z,g ) , the variance of the pixels inside the window is O(X, y), and the value of pixel
(X,y) is f (X,y). We can enhance the variance inside each such window by using a
transformation of the form:

where A is some scalar.

We would like areas which have low variance to have their variance amplified most.
So we choose the amplification factor A inversely proportional to O(X, y):
Image 133

where k is a constant, and M is the average grey value of the image.

Figure 4.4d shows the results of applying this process to image 4.4a with k = 3
and window size 5 X 5. Note that although details in the image have become explicit,
the picture overall is too dark and not particularly pleasing. Figures 4.5d and 4.6d
show the results of applying the same process to theimages 4.5a and 4.6a respectively,
with the additional post-processing of histogram equalization.

U (a) Original image (b) After global histogram equalization

(c) After local histogram equalization (d) After local enhancement

Figure 4.4: Enhancing the image of a young train driver.

134 Image Processing:
FundamentalsThe

. -. .

(a) Original image (b) After global histogram equalization

(c) After local histogram equalization (d) After local enhancement

Figure 4.5: Enhancing the image of the hanging train of Wupertal.

Image 135

(a) Original image (b) After global histogram equalization

(c) After local histogram equalization (d) After local enhancement

Figure 4.6: Enhancing the image at the Karlstejn castle.

How can we improve the contrast of a multispectral image?

A multispectral or multiband or colour image consists of several arrays of the same

scene, one for each spectral component. Each of these bands is a grey level image
giving the intensity of light at the particular spectralcomponent at the position of each
136 Fundamentals
The Processing:
Image

pixel. Suppose for simplicity that we have three spectral bands, Red,Green and Blue.
Then each picture consists of three bands, three grey level images. Alternatively, we
may say that each pixel carries three values, one for each spectral band. We can
plot these triplets in a 3D coordinate space, called RGB because we measure the grey
value of a pixel in each of the three bands along the three axes. The pixels of the
colour image plotted in this space form a cluster.
If we were to use only one of these bands, we would like to choose the one that
shows the most detail; i.e. the one with the maximum contrast, the one in which the
values of the pixels are most spread.
It is possible that the maximum spread of the values of the pixels is not along any
of the axes, but along another line (see Figure 4.7a). To identify this line we must
perform principal component analysis or take the Karhunen-Loeve transformationof
the image.

B 0 0 ./e
B

G
>

R/ /
R

Figure 4.7: The pixels of a colour image form a cluster in

the colour space.
The maximum spread ofthis cluster may be along a line not parallel with
any of the colour axes.

What is principal component analysis?

Principal component analysis (or Karhunen-Loeve transformation) identifies a linear

transformation of the coordinate system such that the three axes of the new coor-
dinate system coincide with the directions of the three largest spreads of the point
distribution. In this new set of axes the data are uncorrelated. This means that if
we form a grey image by using the values of the first co-ordinate of each pixel, it will
contain totally uncorrelated information from the information that will be contained
in the grey image formed by the second coordinate of each pixel and the information
contained in the image formed by the third coordinate of each pixel.
 Image 137

What is the relationship of the Karhunen-Loeve transformation discussed

here and the one discussed in Chapter 3?

They both analyse an ensemble of random outcomes into their uncorrelated compo-
nents. However, in Chapter 3 the whole image was considered as the outcome of a
random experiment, with the other random outcomes in the ensemble not available.
Their lack of availability was compensated by the assumed ergodicity. So, although
the ensemble statistics were computed over the single available image using spatial
statistics, they were assumed to be averages computed over all random outcomes, i.e.
all versions of the image. Here the values of a single pixel are considered to be the
outcomes of a random experiment and we have at our disposal the whole ensembleof
random outcomes made up from all the image pixels.

How can we perform principal component analysis?

To perform principal component analysis we must diagonalize the covariance matrix

of our data. The autocovariance function of the outputs of the assumed random
experiment is:

C(i,j) = E(("i(h1) - "io)("j(k,l) - "jo))

where zi(k,1 ) is the value of pixel ( k ,1 ) at band i , zio is the mean of band i , xj ( k ,1 ) is
the value of the same pixel in band j , xjo is the mean of band j , and the expectation
value is over all outcomes of the random experiment, i.e. over all pixels of the image:

(4.10)
b=l k 1

Since we have three bands, variables i and j take only three values to indicate R,G
and B and the covariance matrix is a 3 X 3 matrix. For data that are uncorrelated,
C is diagonal; i.e. C ( i , j )= 0 for i # j . To achieve this we must transform our data
using the transformation matrix A made up from the eigenvectors of the covariance
matrix of the untransformed data. The process is as follows:
1. Find the mean of the distribution of points in the colour space, say point
(Ro,Go,Bo).
2. Subtract the mean grey levelvalue from each corresponding band.This is
equivalent to translating the RGB coordinate system to be centred at the centre
of the pixel distribution (see axes R'G'B' in Figure 4.7b).
3. Find the autocorrelation matrix C(i,j ) of the initial distribution (where i and
j take the values R, G and B ) .
4. Find the eigenvalues of C(i,j ) and arrange them in decreasing order. Form
the eigenvector matrix A , having the eigenvectors as rows.
138 Image Processing: The Fundamentals

5. Transform the distribution using matrix A. Each triplet X =

K) is trans-

formed into y = (2)

p3
by: y = Ax;i.e. ~k = xi akixi.

This is a linear transformation. The new “colours” are linear combinations of the
intensity values of the initial colours, arranged so that the first principal component
contains most of the information for the image (see Figure 4.7b).

Whatare the advantages of using principal components to express an

image?

The advantages of using principal components are:

1. The information conveyed by each band is maximal for the number of bits used
because the bands are uncorrelated and no information contained in one band
can be predicted by the knowledge of the other bands.
2. If we want to use a monochrome version of the image, we can restrict ourselves
to the first principal component only and be sure that it has the maximum
contrast and contains the maximum possible information conveyed by a single
band of the image.
An example of principal component analysis is shown in Figure 4.8. Although
at first glance not much difference is observed between Figures 4.8a, 4.8b, 4 . 8 ~and
4.8d, at a more careful examination, we can see that the first principal component
combines the best parts of all three bands: For example, the face of the boy has more
contrast in 4.8b and 4 . 8 ~
than in 4.8a, while his right leg has more contrast with his
trousers in 4.8a and 4.8b than in 4.8~. In4.8d we have good contrast in both these
places. Similarly, the contrast between the trousers and the ground is non-existent in
4.8a and 4.8b but it is obvious in 4.8~.Image 4.8d shows it as well.

What are the disadvantages of principal component analysis?

The grey values in the bands created from principal component analysis have no
physical meaning, as they do not correspond to any physical colours. As a result, the
grey value of a pixel cannot beused for the classification of a pixel. This is particularly
relevant to remote sensing applications, where often pixels are classified according to
their grey values. In a principal component band, pixels that represent water, for
example, may appear darker or brighter than other pixels in the image depending on
the image content, while the degree of greyness of water pixels in the various spectral
bands is always consistent, well understood by remote sensing scientists, and often
used to identify them.
Image 139

(a) Red component (b) Green component (c) Blue component

(d)First principal corn- (e) Second principal com- (f) Third principal com-
ponent ponent ponent

Figure 4.8: Example of principal component analysis of a colour image.

Example 4.2

Is it possible for matrix C below to represent the autocovariance matrix

of a three-band image?

-1 0

This matrix cannot represent the autocovariance matrix of an image because from
equation (4.10) it is obvious that C must be symmetric with positive elements
along its diagonal.
Image 141

The transformation matrix A is:

-_

__

Example 4.4

A4 X 4 three-band image is given:

4 3 32 2 33 3 54 4 6
R = (34 5 5 6 ') G = ( ' 4 5 3 ') B = ( ' 4
6 3 3 ')
5

2 2 43 4 5 4 5 5 6
Calculate its three principal components andverify that they are
uncorrelated.

First we calculate the mean of each band:

1
R0 -(3+3+5+6+3+4+4+5+4+5+5+6+4+5+5+6)
=
16
-
73
-
- = 4.5625
16
1
Go = - ( 3 + 2 + 3 + 4 + 1 + 5 + 3 + 6 + 4 + 5 + 3 + 6 + 2 + 4 + 4 + 5 )
16
- 60
-
- = 3.75
16
1
B0 = - ( 4 + 2 + 3 + 4 + 1 + 4 + 2 + 4 + 4 + 3 + 3 + 5 + 2 + 3 + 5 + 5 )
16
50
- -
- = 3.375
16
Next we calculate the elements of the covariance matrix as:
142 Image Processing: The Fundamentals

. 4 4

CGG = 16
l
-
k=l1=1
CC
(G(k,1) - Go)2= 1.9375
- 4 4

I"
k=l1=1
Therefore, the covariance matrix is:
0.996094 0.953125 0.726563
0.953125 1.937500 1.281250
0.726563 1.28125 1.359375

The eigenvalues of this matrix are:

X1 = 3.528765 X2 = 0.435504 X3 = 0.328700

The corresponding eigenvectors are:

0.427670
u1 = (0.708330)
0.561576
uz = ( 0.876742
-0.173808)
-0.448457
u3 = ( 0.220050
-0.684149
0.695355
)
The transformation matrix therefore is:
0.427670 0.708330 0.561576

A= ( 0.220050
0.876742
-0.173808 -0.448457
-0.684149
0.695355

W e can find the principal componentsby using this matrix to transform thevalues
of every pixel. For example, for the first few pixels we find:

0.427670 0.708330
0.876742 -0.173808 -0.448457
0.220050 -0.684149 0.695355
0.561576 ) (i)
Image 143

0.427670 0.708330
0.876742 -0.173808 -0.448457
0.220050 -0.684149
0.427670 0.708330
0.695355
0.561576 )
(f)

0.876742 -0.173808 -0.448457

0.220050 -0.684149 0.695355
0.561576) (,5)
W e use the first element of each transformed triplet to form the first principal
component of the image, the second element for the second principal component,
and the third for the third one. In this way we derive:

(
5.654302 3.822820
2.552915 7.498631
p1= 6.790301 7.364725
4.250490 6.656395
5.948065
4.958820
5.948065
7.779546
7.645640
8.634631
9.623876
8.915546

p2= (
0.314974 1.385694
2.007960 0.844099
1.017905 2.169300
2.262436 2.343106
2.516912
2.088630
2.516912
1.446188
2.771389
1.547035
1.975317
2.149123
1.389125
0.682562
1.133874
1.365131
0.671360
0.240878
0.218468
-0.223219
0.925027
-0.234424
1.133874
0.692187
0.902617
0.449725
1.840433
1.376336

TOconfirm that these new bands contain uncorrelated data we shall calculate their
autocovariance matrix. First we find the mean of each band: P ~ oPm,
, P30. Then
we compute:
4 4
Cplpl =
1
- F,y,(PI(i,j)P10)2= 3.528765
-
l6 i=l j=1
- 4 4
144 Image Processing:
FundamentalsThe

4 4
1
Cp3p3 = - - P ~ o=)
(P3(i,j) ~
0.328700
16 2=13=1
. .

W e see that this covariance matrix is diagonal, so it refers to uncorrelated data.

Example 4.5

For the image in Example 4.4 show that the first principal component
has more contrast than any of the original bands.

The contrast of an image can be characterized by the range of grey values it has.
W e can see that the contrast of the original image was 3 in the red band, 5 in
the green band and 4 in the blue band. The range of values in the first principal
component is 9.623876 - 2.552915 = 7.070961. This is larger than any of the
previous ranges.

Some of the images with enhanced contrast appear very noisy. Can we do
anything about that?

Indeed, this is the case for images 4.4c, 4 . 5 ~and 4.6c, where there are large uniformly
coloured regions, which happen to cover entirely the window inside which local his-
togram equalization takes place. Then the grey values of these pixels are stretched
to the full range of 0-255, and the noise is significantly enhanced. We have to use
then some noise reduction techniques to post-process the image. The technique we
use depends on the type of noise that is present in the image.

What are the types of noise present in an image?

There are various types of noise. However, they fall into two major classes: additive
and multiplicative noise. An example of multiplicative noise is variable illumination.
This is perhaps the most common type of noise in images. Additive noise is often
assumed to be impulse noise or Gaussian noise. Figure 4.9a shows an image corrupted
with impulse noise and Figure 4.9b shows an image corrupted with additive zero-mean
Gaussian noise.
Impulse noise alters at random the value of some pixels. In a binary image this
means that some black pixels become white and some white pixels become black.
This is why this noise is also called salt and pepper noise. Additive zero-mean Gaus-
sian noise means that a value drawn from a zero-mean Gaussian probability density
function is added to the truevalue of every pixel.
Image 145

(a) Image with impulse noise (b) Image with additiveGaussian noise

(c) Median filtering of (d)

(a) Median filtering of (b)

.--!

Q
’
L-
-J..; ’
c,

(e) Smoothing of (a) by averaging (f) Smoothing of (b) by averaging

Figure 4.9: Examples of filtering to remove noise.

146 Fundamentals
The Processing:
Image

We use rankorderfiltering to remove impulse noise and smoothing to reduce

Gaussian noise.

What is a rank order filter?

A rank order filter is a filter the output value of which depends on the ranking of the
pixels according to their grey values inside the filter window. The most common rank
order filter is the median filter.
Figure 4.10 shows the result of trying toremove the noise from output images 4 . 4 ~
and 4 . 5 ~
by median filtering.

I'
t B

Figure 4.10: Improvingimages 4 . 4 ~and 4 . 5 ~by usingmedianfiltering

(median filter size = 3 X 3).

What is median filtering?

The median is the grey level value which divides a distribution in two equally num-
bered populations. For example, ifwe use a 5 X 5 window, we have 25 grey level
values which we order in an increasing sequence. Then the median is the thirteenth
value. This has the effect of forcing points with distinct intensities to be more like
their neighbours, thus eliminating intensity spikes which appear isolated.
Figure 4 . 9 ~shows image 4.9a processed with a median filter and with a window of
size 5 X 5, while Figure 4.9d shows image 4.9b (which contains Gaussian noise) having
been processed in the same way. It is clear that themedian filter removes the impulse
noise almost completely.
 Image 147

What if the noise in an image is not impulse?

The most common type of noise in images is Gaussian. We can remove Gaussian
noise by smoothing the image. For example, we may replace the value of each pixel
by the average value inside a small window around the pixel. Figures 4.9e and 4.9f
show the result of applying this process to images 4.9a and 4.9b respectively. The
size of the window used is the same as for the median filtering of the same images,
i.e. 5 X 5. We note that this type of filtering is much more effective for the Gaussian
noise, but produces bad results in the case of impulse noise. This is a simple form of
lowpass filtering of the image.

Why does lowpass filtering reduce noise?

Usually, the noise which is superimposed on the image is uncorrelated. This means
that it has a flat spectrum. On the other hand, most images have spectra which have
higher values in the low frequencies and graduallyreducing values forhigh frequencies.
After a certain frequency, the spectrum of a noisy signal is dominated by the noise
component (see Figure 4.11). So, ifwe use a lowpass filter, we kill off all the noise-
dominated high-frequency components. At the same time, of course, we kill also the
useful information of the image buried in these high frequencies. The result is a clean,
but blurred image. The process is as follows:
1. Find the Fourier transform of the image.
2. Multiply it with a function which does not alter frequencies below a certain
cutoff frequency but which kills off all higher frequencies. In the 2-dimensional
frequency space, this ideal lowpass filter is schematically depicted in Figure 4.12.
3. Take the inverse Fourier transform of this product.

spectrum
imageClean
Spectral
magnitude

/Noise spectrum

-
r

Figure 4.11: After a certain frequency, the spectrum of the noisy image is
dominated by the noise. Ideally the cutoff frequency of the lowpass filter
should be at TO = F.
Image 148 Processing: The Fundamentals

Multiplication of the two frequency spectra is equivalent to convolution of the actual

functions. So, what we can doinstead of the above procedure, is to find the 2-
dimensional function in real space which has as itsFourier transform the ideal lowpass
filter, and convolve our image with that function. This would be ideal, but it does
not work in practice because the function the Fourier transform of which is the ideal
lowpass filter is infinite in extent.

I H(Wx
Y
1

r
‘0

Figure 4.12: The ideal lowpass filter in 2D in the frequency domain. On the
right a cross-section of this filter with cutoff frequency TO ( r =

What if we are interested in the high frequencies of an image?

It is possible that we may want to enhance the small details of a picture instead of
ironing them out. Such a process is called sharpening and it enhances small fluctua-
tions in the intensity of the image, noise included.
One way to achieve this is to calculate at each pixel the local gradient of the
intensity using numerical difference formulae. If one wants to be more sophisticated,
one can use the filtering approach we discussed in the context of smoothing. Only
now, of course, the filter should be highpass and allow the high-frequency components
to survive while killing the low-frequency components.

What is the ideal highpass filter?

The ideal highpass filter in the frequency domain is schematically depicted in Figure
4.13.
Filtering with such a filter in the frequency domain is equivalent to convolving
in real space with the function that has this filter as its Fourier transform. There is
no finite function which corresponds to the highpass filter and one has to resort to
various approximations.
Image 149

r
‘0

Figure 4.13: The spectrum of the ideal highpass filter is 1 everywhere,

except inside a circle of radius TO in the frequency domain, where it is 0.
On the right, a cross-section of such a filter. Here T E

How can we improve an image which suffers from variable illumination?

This is a problem which can be dealt with ifwe realize that every image function
f(z,y) is the product of two factors: an illumination function i(z,y) and a reflectance
function r ( z ,y) that is intrinsic to the imaged surface:

Illumination is generally of uniform nature and yields low-frequency components

in the Fourier transform of the image. Different materials (objects) on the other
hand, imaged next to each other, cause sharp changes of the reflectance function,
which causes sharp transitions in the intensity of an image. These sharp changes are
associated with high-frequency components. We can try toseparate thesetwo factors
by first taking the logarithm of equation (4.11) so that the two effects are additive
+
rather than multiplicative: In f(z,y) = In i ( z ,y) In ~ ( zy),
Then we filter this logarithmic image by what is called a homomorphic filter.
Such a filter will enhance the high frequencies and suppress the low frequencies so
that the variation in the illumination will be reduced while edges (and details) will
be sharpened. The cross-section of a homomorphic filter looks like the one shown in
Figure 4.14.
Figures 4.15a and 4.16a showtwo images with smoothly varying illumination from
left to right. The results after homomorphic filtering shown in Figures 4.15b and 4.16b
constitute a clear improvement with the effect of variable illumination greatly reduced
and several details particularly in the darker parts of the images made visible. These
results were obtained by applying to thelogarithm of the original image, a filter with
the following transfer function:
Image Enhancement 151

(a) Original image (b) After homomorphic filtering

Figure 4.16: Homomorphic filtering Mrs Petrou at home.

Can any of the objectives of image enhancement be achieved by the linear

methods we learned in Chapter 2?

Yes, often image smoothing and image sharpening is done by convolving the image
with a suitable filter. This is the reason we prefer the filters to be finite in extent:
Finite convolution filters can be implemented as linear operators applied on the image.

Example 4.6

Youhave a 3 X 3 image which can be represented by a 9 X 1 vector.

Derive a matrix which, when it operates on this image, smoothes its
columns by averaging every three successive pixels, givingthem weights
a, i,a.To deal with the border pixels assume that theimage isrepeated
periodically in all directions.

Let us say that the original image is

Image 153

What is the “take home” message of this chapter?

With image enhancement we try to make images look better according to subjective
criteria. Most of the methods used are non-linear, and therefore theycannotbe
described within the framework developed in the previous chapters. Some of the
methods used are not only non-linear, but also inhomogeneous.
Contrast enhancement of a grey image can be achieved by manipulating the grey
values of the pixels so that they become more diverse. This can be done by defining
a transformation that converts the distribution of the grey values to a pre-specified
shape. The choice of this shape may be totally arbitrary. If, however, we are dealing
with a multiband image, we can define a line in the multidimensional space (where
we measure the grey values of a pixel in the different bands along the corresponding
axes) such that when we project the data on this line, their projected values are
maximally spread (see Figure 4.7a). This line defines the first principal axis of the
data (= pixel values) in the multidimensional space. We can create a grey image
by assigning to each pixel its projected value along the first principal axis. This is
called the first principal component of the multiband image. From all grey bands
we can construct from the multiband image, the first principal component has the
maximum contrast and contains the most information. However, the grey values of
the pixels do not have physical meaning as they are linear combinations of the grey
values of the original bands. The basic difference between histogram manipulation
and principal component analysis is that in the former the values the pixels carry are
changed arbitrarily, while in the latter the joint probability density function of the
pixel values remains the same; we simply “read” the data in a more convenient way,
without introducing information that is not already there (compare, for example, the
enhanced images 4 . 6 ~and 4.8d).
We can also enhance an image in a desirable wayby manipulating its Fourier
spectrum: We can preferentially kill frequency bands we do not want. This can be
achieved with the help of convolution filters that have pre-specified spectra. It is
necessary, therefore, to understand how we can develop and use filters appropriate for
each task. This is the topic of the next chapter.
Chapter 5

Two-Dimensional Filters
What is this chapter about?

Manipulation of images often entails omitting or enhancing details of certain spatial

frequencies. This is equivalent to multiplying the Fourier transform of the image with
a certain function that "kills" or modifies certain frequency components. When we
do that, we say that wefilter the image, and the function we use is called a filter.
This chapter explores some of the basic properties of 2D filters and presents some
methods by which the operation we wish to apply to the Fourier transform of the
image can be converted into a simple convolution operation applied to the image
directly, allowing us to avoid using the Fourier transform itself.

How do we define a 2D filter?

A 2D filter is defined in terms of its Fourier transform H(p,u), called the system
function. By taking the inverse Fourier transform of H(p, V) we can calculate the
filter in the real domain. This is called the unit sample response of the filter and is
denoted by h(k, 1).

How are the system function and the unit sample response of the filter
related?

H ( p , u) is defined as a continuous function of b,u). The unit sample response h(k, 1)

is defined as the inverse Fourier transform of H(p, u), but since it has to be used
for the convolution of a digital image, it is defined at discrete points only. Then the
equations relating these two functions are:
156 Image Processing: The Fundamentals

I
Signal

l wewant
to keep
band bands we want

i
to eliminate

Filter
-
Inverse
X
F.T.
2
-

0
11-

r Filter needed

- Inverse
F.T.
;
@

X m

Signal Filtered signal filtered

F.T. of the

Figure 5.1: Top row: a signal and its Fourier transform. Middle row: the
unit sample response function of a filter on the left, and the filter's system
function on the right. Bottom row: On the left the filtered signal that
can be obtained by convolving the signal at the top with the filter in the
middle. On the right the Fourier transform of the filtered signal obtained
by multiplying the Fourier transform of the signal at the top, with the
Fourier transform (system function) of the filter in the middle.
 Two-Dimensional Filters 157

If we are interested in real filters only, these equations can bemodified as follows:

h(Ic,Z) = - (5.3)
2T

H ( P , v) = G cc
+m +m

n , - m m=-m
h(n,m)cos(pn + vm) (5.4)

Why are we interested in the filter function in the real domain?

We can achieve the enhancement of the image as desired, by simply convolving it with
h(Ic,Z) instead of multiplying its Fourier transform with H ( p , v ) . Figure 5.1 shows
this schematically in the 1D case. The 2D case is totally analogous.

Are there any conditions which h(Ic,Z)must fulfil so that it can be used as
a convolution filter?

Yes, h ( k ,1) must be zero for Ic > K and Z > L , for some finite values K and L ; i.e. the
filter with which we want to convolve the image must be a finite array of numbers.
The ideal lowpass, bandpass and highpass filters do not fulfil this condition.

B5.1: What is the unit sample response of the ideal lowpass filter?

The ideal lowpass filterwhich cuts tozero all frequencies abovea certain frequency
R, say, is defined as:

We can use this definition of H ( p , v) t o calculate the corresponding unit sample

response from equation ( 5 . 3 ) :

h(k,Z) =
1
2T
- 1;1; cos(@ + 4 H ( P , v)d@v
We shall introduce polar coordinates (T, 0) in the ( p ,v) frequency space:

Then
158 Image Processing: The Fundamentals

where angle 4 has been defined so that:

k 1
sin 4 E and cos 4
Jm E
Jm
We define a new variable t = B + 4. Then equation (5.6) can be written as:

2 ~ + 4 R

+&L i c o s ( r d m s i n t)rdrdt

In the second term we change the variable t o i = t-27r + t = i+27r + sin t = sin l.
Therefore, we can write:

h(lc,l) = - 2', S,'"1"c o s ( r d w s i n t ) r d r d t

+L
27r /gR cos(rdmsin+-drdi

-
- I
27r
iZT
1" cos(rdwsint)rdrdt

This can be written as:

h(k,l) = -1" 1"

271
c o s ( r d m s i n t)rdrdt

+$S,'" l" c o s ( r d m s i n t)rdrdt

In the second term, we define a new variable of integration: i t - 7r + t =

i + 7r + sint = - s i n i + c o s ( r J m s i n t ) = c o s ( r J m s i n i ) and dt = di.
Then:
Two-Dimensional Filters 159

We know that the Bessel function of the first kind of zero order is defined as:

If we use definition (5.8) in equation (5.7) we obtain:

h(Ic,l) =
I" rJIJ(rdF5)dr

We define a new variable of integration X E r d m + dr = 1m

d x . Then:

xJo(x)dx (5.9)

From the theory of Bessel functions, it is known that:

(5.10)

(5.11)

This function is a function of infinite extent, defined at each point (Ic, l ) of integer
coordinates. It corresponds,therefore, to an array of infinitedimensions. The
implication is that this filter cannot be implemented as a linear convolution filter
of the image.
160 Fundamentals
The Processing:
Image

Example 5.1 (B)

What is the impulse response of the ideal bandpass filter?

The ideal bandpass filter is defined as:

The only difference, therefore, with the ideal lowpass filter derived in box B5.1 is
in the limits of equation (5.9):

This is a function defined for all values (k,1). Therefore the ideal bandpass filter
is an infinite impulse response filter.

Example 5.2 (B)

What is the impulse response of the ideal highpass filter?

The ideal highpass filter is defined as:

The only diference, therefore, with the ideal lowpass filter derived in box B5.1 is
in the limits of equation (5.9):
m
h(k,l) =
1
J
k2 + l2 R-
p x~~(x)dx
Two-Dimensional Filters 161

Bessel function J1(x) tends to 0 f o r X +-CO. However, its asymptotic behaviour

is limx+.mJ1(x) = 2 . This means that J1(x) does not tend to 0 fast enough
fi
to compensate for the factor X whichmultiplies it, i.e. limx+.mx J l ( x ) +- CO.
Therefore, there is no real domain function that has as Fourier transform the
ideal highpass filter. In practice, of course, the highest frequency we may possibly
be interested in is & where N is the number of pixels in the image, so the issue
of CO upper limit in equation (5.12) does not arise and the ideal highpass filter
becomes the same as the ideal bandpass filter.

What is the relationship between the 1D and the 2D ideal lowpass filters?

The 1D ideal lowpass filter is given by:

sin k
h(k) = ~ (5.13)
k
The 2D ideal lowpass filter is given by:

J1 (d m )
h(k,1 ) =
d r n
where J1(x) is the first-order Bessel function of the first kind. Figure 5.2 shows the
plot of h ( k ) versus k and the plot of h ( k , l ) versus k for 1 = 0. It canbe seen
that although the two filters look similar, they differ in significant details: their zero
crossings are at different places, and the amplitudes of their side-lobes are different.
This implies that we cannot take an ideal or optimal (according to some criteria)
1D filter replace its variable by the polar radius (i.e. replace k by d r n in equation
(5.13)) and createthe corresponding “ideal or optimal”filter in 2D. However, although
the 2D filter we shall create thisway will not be the ideal or optimal one according to
the corresponding criteria in 2D, it will be a good suboptimal filter with qualitatively
the same behaviour as the optimal one.

How can we implement a filter of infinite extent?

A filter which is of infinite extent in real space can be implemented in a recursive way,
and that is why it is called a recursive filter. Filters which are of finite extent in real
space are called non-recursive filters. Filters are usually represented and manipulated
with the help of their z-transforms.

How is the z-transform of a digital 1D filter defined?

{ X I , x2,x3,. . . ,xn}.
A filter of finite extent is essentially a finite string of numbers
Sometimes an arrow is used to denote the element of the string that corresponds to
 162 Image Processing: The Fundamentals

Figure 5.2: The cross-section of the 2D ideal lowpass filter is similar to

but different from the cross-section of the 1D ideal lowpass filter.

the zeroth position. The z-transform of such a string is defined as:

X(z)= cm.
...

k=l
xkz-k (5.14)

where 1 and m are defined according to which term of the string of numbers { x k } is
assumed to be the k = 0 term. (For example, if 2 3 is the k = 0 term then 1 = -2 and
m = n - 3 in the above sequence).
If the filter is of infinite extent, the sequence of numbers which represents it is of
infinite extent too and its z-transform is given by an infinite sum of the form:
00 00 0

In such a case we can usually write this sum in closed form as the ratio of two
polynomials in z , as opposed to writing it as a single polynomial in z which is the
case for the z-transform of the finite filter.
Two-Dimensional Filters 163

Why do we use z-transforms?

The reason we use z-transforms is because digital filters can easily be realized in
hardware in terms of their z-transforms. The z-transform of a sequence together
with its region of convergence uniquely defines the sequence. Further, it obeys the
convolution theorem: The z-transform of the convolution of twosequencesis the
product of the z-transforms of the two sequences.

How is the z-transform defined in 2D?

For a finite 2D array of dimensions M X N , the z-transform is a finite polynomial in

two complex variables z1 and z2:
M N
(5.15)
i=o j=o

where cij are the elements of the array. For an infinite array, the double summation
is of infinite extent and it can be written as a ratio of two finite summations:

(5.16)

where M,, N,, Mb and Nb are some integers. Conventionally we choose boo = 1.

B5.2: Why does the extent of a filter determine whether the filter is
recursive or not?

When we convolve an image with a digital filter, we essentially multiply the z-

R(Z1,ZZ)

z-transform of
output image
=
--
transform of the image with the z-transform of the filter:

H(z1,zz)
z-transform
of filter
D(z1,zz)
z-transform of
input image
(5.17)

If we substitute (5.16) in (5.17) and bring the denominator to the left hand side
of the equation, we have:

i=o j = o \i=o j=o

In the sum on the left hand side we separate the k = 0, 1 = 0 term, which by
convention has boo = l:
164 Processing:
Image The Fundamentals

Remember that R(z1,z 2 ) is a sum in z r z ? with coefficients say r,,. It is clear

from the above equation that the value of r,, can be calculated in terms of the
previously calculated values of r,, since the series R(z1,z 2 ) appears on the right
hand side of the equation too. That is why such a filter is called recursive. In the
case of a finite filter all b k l ' s are zero (except boo which is 1) and so the coefficients
r,, of R ( z l , z 2 ) are expressed in terms of aij and the coefficients which appear
in D ( z 1 , z z ) only (i.e. we have no recursion).

Example 5.3 (B)

A 256 X 256 image is to be processed by an infinite impulse response

filter. The z+transform of the filter can be written as the ratio of a
third-degree polynomial ineach of the variables z 1 and z 2 over another
third-degree polynomial in the same variables. Calculate the values of
the output image in terms of the values of the input image and the
filter coefficients.

In equation (5.18) wehave Ma = Na = M(, = NI, = 3. Letussaythatthe

z-transform of the input image is:
255255
k l
D(Zl,.2) = dklZlZ2 (5.19)
k=O l=O

I1 and of the output image is:

Two-Dimensional Filters 165

255255
(5.20)
k=O 1=0

W e wish to derive expressions for rij in terms of aij, bij and dij. If we substitute
(5.19) and (5.20) into equation (5.18) we obtain:
255 255 3 255
3 255

To avoid having to treat separately the terms with 0 indices, we allowed in the
above expression the term i = j = 0, but we must remember that now boo 0.
To be able to relate terms of the same order on the two sides of this equation, we
must define new variables of summation on the right hand side. W e define:

k+i_U
k-iEV * ik=!kkE
- U-W2
2
l+j=t
l-j=s
We must find the limits of summation for the new variables. The figure
* j - t-S
2I
2

below
shows how the area over which we s u m using variables k and i is transformed when
the new variables are used. The transformed area is definedby the transformed
equations of the lines that define the original area, i.e. lines i = 0 , i = 3, k = 0
and k = 255, which become respectively: v = U , v = U-6, v = -U and v = 510-U.

Figure 5.3: The transformation of the area of summation.

Equation (5.21) then becomes:

255 255 3 255 I 2 U 255 U 258 510-U)

k=O 1=0 j=O l=O (u=O W=-U u=3 w=u-6 u=256 w=u-6)

(5.22)
166 Fundamentals
TheProcessing:
Image

ln a similar way we can find the limits of summation of variables s and t t o

obtain:
255255

255 t t
255 255 255 t
510-U 258

t=3 s=t-6 u=O U=-U t=3 s=t-6 u=3 u = u - ~ t=3 s=t-6 ~ = 2 5 6u = u - ~

510-U
258
510-t
258
255
510-t
258

t=256 s=t-6 u=O U=-U t=256 s=t-6 u=3 u = u - ~ t=256 s = t - 6 ~ = 2 5 6 u = u - ~

(5.29

We further define the following quantities:

t 510-U

s=-t u=u-6

t 510-U

s=t-6 u=u-~

s=t-6 U=-U

510-t U

s=t-6 u=u-~

Iut cc
510-t510-U

s=t-6 u=u-~
(UuGv tiSdu:.t+ - b.-,
2
t-sru+v
2 2 2
t-.) (5.24)
 Two-Dimensional Filters 167

Then equation (5.23) can be written as:

255 255
258

u=o t=O
rut.+;
2
= cc 2

t=o u=o
2255
A,t.z,U.zi +
2

t=o u=3
&tz,Uz; + cc
t=O u=256
Cutz,Uz;

255 255 255 2 255

255 258 2 258

(5.25)

Here we changed the dummy summation variables on the left hand side to U and
t as well. Toderiveexpressions for rut in terms of aut, but anddut, we must
equate the coeficients of the terms of the same degree in z1 and z 2 that appear
on the two sides of equation (5.25). W e m u s t remember that boo was redefined in
(5.21) to be 0. To make sure that we deal with all the cases, we draw the diagram
in Figure 5.4, where thetwoaxes correspond to the summing variables on the
right hand side of equation (5.25). Each rectangle in this diagram corresponds to
a combination of ranges of values of the two variables that have to be considered
separately.

01 3 255 258 U

Figure 5.4: Differentrecursiveformulaeapply to differentrangesof

index values.
168 Image Processing: The Fundamentals

Case I: 0 5 U 5 2 and 0 5 t 5 2

W e define new summation variables:

i -
- U-v
with limits i = U (corresponding t o v = - U ) and i = 0 (corresponding t o
v = U).
Thenv=u-2i+?!$?L=y=u-i
j withlimits j = t (correspondingto S = -t) and j = 0 (corresponding t o
S = t).
T h e n s = t - 2 j + y = M =2 t - j
Expression (5.26) can then be written as:
u t
(5.27)
i=o j=o

W e consider first the case U = t = 0 and remember that boo = 0:

Too = aoodoo

Then consider the case U = 0, t = 1 for which j takes the values 0 and 1:

For the case U = 0, t = 2, j takes values 0 , 1 and 2:

Notice that although some r values appear on the right hand side of these equa-
tions, they are always with smaller indices than those of r on the left hand side,
and therefore their values have already been computed.
Case 11: 2 < U 5 255 and 0 5 t 5 2

rut = But = cct

s=-t v=u-6
u
(au2. t2sdu;v t Z s - bu-. t-=ru+.t-.)
2 2 2 2
(5.28)

The only difference with the previous case is that the limits of the new summation
variable i will be f r o m i = 3 (corresponding to v = U - 6 ) t o i = 0 (corresponding
to v = U):
3 t
rut = (aijdu-i,t-j - bijru-i,t-j) (5.29)
i=o j=o
Two-Dimensional Filters 169

Case 111: 255 < U 5 258 and 0 5 t 5 2

In this case there is no corresponding term on the left of the equation since the
maximum value U takes on the left is 255. W e have therefore:

0 = Cut cct 510-U

(auiv t,.dU;. tZs - buzv t,.ru;V t,.) (5.30)

Or 3 t

cc
i=u-255 j=O
(aijdu-i,t-j - bijru-i,t-j) =0 (5.31)

Consider U = 256, t = 0:
c3

i=l
aiod256-i,o - bior256-i,o =0 *
alOd255,O - blOr255,O + a2Od254,O - blOr254,O + a3Od253,O - b30r253,O = 0 (5-32)

Thevalues of r255,0,r254,o and r253,o have already been computed b y equation

(5.29). Equation (5.32), therefore, serves only as a check that these values have
been computed correctly. I n general, this is the role of equation (5.31) and the
equations that will be derived for cases VI, VII, VIII and I X .
Case IV: 0 5 U 5 2 and 2 5 t 5 255

rut = Dut = cc
s=t-6
t

W=-U
u
(au-. t--sdu+vt + =
2 22 2
- bu-,
2
t-=
2
Tu+.
2
t-.)
2
(5.33)

Or
(5.34)

Case V: 2 < U 5 255 and 2 < t 5 255

rut = Eut = cc
s=t-6
t

w=u-6
u
(au-. t--sdu+vt+
2 22 2
- b U - . t--sru+v
2 2
t-.)
2 2
(5.35)

Or 3 3

(5.36)

Case VI: 255 < U 5 258 and 2 < t 255

(5.37)
s=t-6 w=u-6
170 Image Processing: The Fundamentals

Or
U-255 3
(5.38)
i=3 j=o

Case VII: 0 < U 5 2 and 255 < t 5 258

0 = Gut = cc
510-t

s=t-6 U=-U
U

(a.;. t2= d.:. t;, - b.-.

2
t-=
2
r.+.
2 2
t-=) (5.39)

Or
(5.40)

Case VIII: 2 < U < 255 and 255 < t < 258
0 = Hut = cc
510-t

s=t-6 u=u-6
U

( a u - . t-.du+v
2 22 22
t+s -
22 2
bu-. t-.ru+. * - S ) (5.41)

Or 3 t-255

(5.42)

Case VIII: 2 < U 5 255 and 255 < t 5 258

0 = Iut = cc
510-t510-U

s=t-6 u=u-6
(aui. ti8du:v t;, - bug. tZSru:. t l s ) (5.43)

Or 3 t-255

(5.44)

Is there any fundamental difference between 1D and 2D recursive filters?

In one dimension, if we express the z-transform of a function as a ratio of two poly-

nomials, we can factorize these polynomials to obtain the poles and zeroes of the
transform. Once the poles are known, the inverse z-transform can be computed for
each possible region of convergence of the z-transform.
In two dimensions, factorization is much more difficult and even if it is achieved,
the zeroes of the polynomial of the denominator (poles) are not usually isolated dis-
crete points but rather ranges of continuous values of the variables. Further, the ( z 1 ,
2 2 ) space is 4-dimensional, and this makes the visualization of regions of convergence
of the polynomials impossible.
Two-Dimensional Filters 171

How do we know that a filter does not amplify noise?

A filter must be stable in order not to amplify noise. A digital filter is said to be
stable if the output remains bounded for all bounded inputs. A digital filter h(n,m)
is stable if and only if
0 0 0 0

71.=-m m=-m

The above criterion is obviously fulfilled if the filter is of finite extent or non-
recursive. For infinite impulse response (recursive) filters, one has to check the be-
haviour of the filter carefully, with the help of special theorems on the subject.

Is there an alternative to using infinite impulse response filters?

Yes! We can compromise in our requirements on the frequency response of the filter.
In particular, we may adopt the following approach:
1. Decide upon the desired system function (frequency response) of the filter.
2. Choose a finite filter (non-recursive) which approximates as well as possible the
desired system function.
This approach leads to the approximation theory.

Why do we need approximation theory?

Because the required frequency response of a filter cannot be realized exactly by a

finite impulse response filter.

How do we know how good an approximate filter is?

We use as a measure of the quality of the approximation the Chebyshev norm of

the difference between the desired and the approximated response, defined as the
maximum deviation between the two functions.
Suppose F ( p , v ) is the ideal frequency response we require and H ( p , v ) is the
frequency response of the approximated finite filter. Then the closeness of H ( p , v) to
F ( p , v) is measured by:

What is the best approximation to an ideal given system function?

The best approximation to an ideal system function F ( p , v) is that function H ( p , v)

which minimizes the error; i.e.

Best approximation error = mina(.,.){max(,+,) IF(p,v) - H ( p , .)I} (5.46)

172 Image Processing: The Fundamentals

Why do we judge an approximation according to the Chebyshev norm

instead of the square error?

The total square error (S S [&'(p,v) - H ( p , v)12dpdv) is an average measure of how

close &'(p,v) and H ( p , v) are. Two different approximations may have identical square
errors, but one may introduce very undesirable features to the filter at specific fre-
quencies (see Figure 5.5). The Chebyshev norm on the other hand guarantees that
at no frequency will the deviation of the two functions exceed a certain limit.

F(WX A bad approximation,

W W
X X

Figure 5.5: In the frequency domain, a good approximation is one that

never deviates too much away from the desired response.

How can we obtain an approximation to a system function?

There are three techniques of designing 2-dimensional filters. All three are direct
adaptations of corresponding techniques in one dimension. They are:
1. Windowing
2. Frequency sampling
3. Linear programming
Of the above techniques only the last one gives the best approximation in the sense
that it minimizes the maximum deviation between the exact and the approximate
functions.

What is windowing?

The most obvious way to make an infinite impulse response filter finite, is to truncate
it at some desired size. This method is called windowing because it is as ifwe put a
window on the top of the filter and look only at what is shown inside the window.
However, this method creates filters with sharp edges.
Two-Dimensional Filters 173

What is wrong with windowing?

A function which contains discontinuities (like a sharply truncatedfilter) has a Fourier

transform with lots of ripples at high frequencies as lots of higher harmonics are
required to contribute in order to produce those sharp edges. So, the process of
truncation distorts badly the system function of the filter at high frequencies.

How can we improve the result of the windowing process?

Truncating the filter function at certain frequencies is equivalent to multiplying it

with a window function of the form:
1 for - a < x < a a n d -b<y<b
0 elsewhere
where a and b are some parameters. To avoid sharp edges of the truncated filter this
window has to be replaced by a smooth function. Several such smooth windows have
been developed for the case of 1D signals.

Can we make use of the windowing functions that have been developed
for 1D signals, to define a windowing function for images?

Once we have decided upon a good window in the l-dimensional case, the extension
to two dimensions is not hazard-free. One must have in mind that if W(.) is a good
l-dimensional window, its circularly symmetric version w ( r ) W ( d m ) does not
have the same frequency response as w(x). In other words, the Fourier transform of
the circularly symmetric window is not the circular version of the Fourier transform
of the l-dimensional window. This was discussed extensively earlier in this chapter
(see also Figure 5.2).
In general, however, if w(x) is a good l-dimensional window, w 2 ( d m ) is a
good circularly symmetric 2-dimensional window function. In both cases of course,
one should be very careful when one discretizes the window to sample it densely
enough so that aliasing problems are avoided.
The windowing method might yield some reasonably good 2-dimensional filters,
but these filters are not optimal approximations to the desired filter as no effort has
been put into optimizing the approximation.

What is the formal definition ofthe approximation problem we are trying

to solve?

Suppose that F ( p , v) is an ideal, unrealizable filter response. The goal is to find

a set of n coefficients cl, Q , . . . ,cn such that the generalized polynomial P ( p ,v)
cigi(p, v) minimizes the Chebyshev norm of the error function A defined by:
n
(5.47)
174 Image Processing: The Fundamentals

Here g i ( p , v) are known functions of the frequencies p and v, like for example cos(p +
+
v), sin(2p v) etc.
In (5.47) p and v take continuous values. We shall consider, however, a finite set
of samples of A by choosing M points in the ( p ,v) space:
n
Am E F ( p m , vm) - C c i g i ( p m , vm) m = 1 , 2 , .. . , M (5.48)
i=l

Our purpose is to choose the coefficients cl, Q , . . . ,cn so that the maximum value
of the Am’s is as small as possible. This problem can be solved with the help of linear
programming.

What is linear programming?

Linear programming is an optimization method designed to solve the following prob-

lem: Choose nl positive variables x1i and 722 variables x2j in order to

minimize: z = cTxl+ czx2 d +

subjectto: +
Allxl A12x2 2 B1 p1 inequalityconstraints
+
A21x1 A22x2 = B2 p2 equality constraints
Xli L 0 n1 variables X I
x2j free 122 variables xz

where cl, c2, B1, B2 are vectors and All, All, A12, A21, A22 are matrices. Vectors
XI and x2 are made up from variables xli and x2j respectively.
To solve the above problem one can use a commercial package algorithm. It is,
therefore, enough simply to formulate our problem in the above way. Then we can
use one of the commercially available packages to solve it.

How can we formulate the filter design problem as a linear programming

problem?

Suppose that &’(p,v) is the desired frequency response of the filter and suppose that
h(n,m) is the digitized finite filter which we want to use. The frequency response of
this digitized filter is:
N N
(5.49)

where h(n,m) are real numbers. It can be shown (see Box B5.3)that if h ( p , v) =
h(-v, p ) the system function H ( p , v) is real and it can be written as:

H ( p , v) = 2 cc
N

n=-N
N

m=1
h(n,m) cos(pn + vm) + 2 cN

n=l
h(n,0) cos(pn) + h(0,O) (5.50)
 Two-Dimensional Filters 175

This equation expresses the Fourier transform of the filter in terms of the discrete
filter values in real space. We want to specify these values; namely, we want to
specify h(O,O),h(l,O),. . .,h(N,O) , . . . ,h(-N,m),h(-N+l,m), . . . ,h ( N , m )for m =
1,2,. . .,N so that H ( p ,v) approximates optimallythe desired filter response F ( p , v).
This means that we want to choose the above values of h ( i , j )so that the error:

6 = m y , , , ) IH(Pu,v) - Fbu,.>I (5.51)

is minimum.
Variables (p,v) take continuous values. We choose a set of p1 specific points in
the (p,v) space which we shall use as constraint points. For every point (pi,vi) then
the value of vi) - &'(pi,vi)[ will be less or equal to the value of S owing to the
definition of S. So, we can write for every point:

or

(5.52)

The problem is one of trying to minimizing S under the 2pl inequality constraints
(5.52) and the assumption that 6 > 0, while the variables we have to choose h ( i , j )
are free to be positive or negative.
From equation (5.50) it is clear that the number of variables we have to choose is:

(2N+l)N+N+l=2N2+N+N+1=2N2+2N+l

+ +
So, we have 2N2 2N 1 free variables plus one constrained variable (S itself). We
have no equality constraints (p2 = 0) and we have 2pl inequality constraints. In the
language of linear programming:

Minimize: z = 211
under the constraints: +
Allxll A12x2 2 B1 2p1 inequality constraints
(5.53)
X11 2 0 1 non-negative variable
x2j free + +
2N2 2N 1 free variables
176 Image Processing: The Fundamentals

where
211 =6

(2N2+2N+1) X 1 2Pl X 1
X2 = t B1 = t
matrix matrix

All =

+
A12 is a 2pl X ( 2 N 2 2 N +1 ) matrix the elements of which can easily be found
ifwe substitute equation (5.50) into (5.52) or (5.53). It turns out to be:

1 2cos(p1) 2cos(2p1) . . . 2 cos(Np1) 2 cos(-Np1 +V I ) . ..

-1 -2 cos(p1) -2 cos(2p1) . . . -2 c o s ( N p 1 ) -2 cos(-Np1 +VI) ...

1 2 c 0 s ( p p 1 ) 2c0s(2pp1) .. . 2cos(Npp1) 2cos(-Npp1 + vp,) . . .

-1 -2c0s(pp1) -2c0s(2pp1) .. . - 2 c o s ( N p p 1 ) -2cos(-Npp1 + vp,) . . .
Two-Dimensional Filters 177

Once the problem has been formulated in this way any of the commercially avail-
able linear programming packages can be used to solve it. The drawback of this
straightforward application of linear programming is that lots of constraint points are
required for a given number of required coefficients (i.e. for given N ) . The resultant
filter, however, is optimal in the Chebyshev sense.

B5.3: Show that if the impulse response of a filter is symmetric in

both its arguments ( h ( n , m )= h(-n, - m ) ) , the system function is a real
function that c a n b e w r i t t e n in the form ( 5 . 5 0 ) .

From (5.49) we have:

Letus consider the imaginary terms only and split the sum over m into the
positive, the negative and the 0 terms:
N N
C C h(n,m ) sin(pn +vm)=
n=-N m=-N
N r -1 N l

In the negative sum inside the square bracket, change the variable of summation
=
to m -m. Then:
N N N r 1

n=-N L%=N

1
N
+ C h ( n ,m)sin(pn + v m ) + h ( n ,0) sin(pn)
m=l

We can also change the variable for the sum over n for negative values of n:
fi =
-n. Then:
178 Processing:
Image The Fundamentals

N N N

n=l m=l n=l

\ "
N N
-m) sin(-vfi)

+c N

m=l
h ( ~m,) sin(vm) + h ( ~0),sin(p0) = o (5.55)

In the above expression we have also separated the terms with n = 0 or m = 0.

Since:
h(0,m) = h(0,-m)
*
h(n,m) = h(-n, -m)
{ h(n,0) = h(-n, 0)
h(n,-m) = h(-n, m)

and because sin(-2) = - sinx, the terms in (5.55) that are underlined with the
(5.56)

same type of line cancel each other. Therefore, the imaginary part in the equation
(5.54) is zero and the Fourier transform of the filter is real, given by:

H ( p ,v ) = cc
N N
h ( n , m )cos(pn + vm) (5.57)

We can further simplify this expression by changing variables again for the sums
over negative indices (i.e. set fi = -n for negative n and iiz = -m for negative
m ):

5 [e
H ( p , v )=

= h ( n ,-m) cos(pn - v f i )
n=-N e=N

+c m=l
N
h ( n ,m) cos(pn + vm)+ h(n,0) cos(pn)
1
cc 1 1
+cc 1 N
+ vm)
=
fi=N e=N
\
h(-.n,
-"
-m) cos(-p.n - vfi)
fi=N m=l
~
h(-Ln,m) cos(-@

+cc N

n=l e = N
1
h(n,-m) c o s ( p n - v f i ) + cc
N

n=l m=l
N
h(n,m ) cos(pn + vm)
L " /
 Two-Dimensional Filters 179

+c 1

fi=N
h(-.n, 0) cos(-p.n) +c N

n=l
h ( n ,0) cos(pn)

+c +c
1 N
h(0,-m) cos(-rnv) h(0,m) cos(mv) + h(0,O)
77=N m=l

Using (5.56) again and the fact that cos(--5) = cosx, we collect together the
terms that are underlined with the same type of line to obtain:

n=l
H ( p ,v) = 2
m=l
cc
N

n=l
N
h(n,m) cos(pun + vm)+ 2 cc N N
h(-n, m) cos(-pun + vm)

c
+2
N

m=l
h(0,m) cos(mv) + h(0,O) + c2
N
h(n,0) cos(pn)
n=l

Finally, by writing together the first three sums, we get:

H ( p ,v) = 2 cc
N

n=-N m=l
N

h(n,m ) cos(pn + vm)+ 2

n=l
c
N

h(n,0) cos(pn) + h(0,O)

Is there any wayby which we can reduce the computational intensity of

the linear programming solution?

Yes, by breaking the problem into a series of smaller ones within the framework of
an iterative algorithm. At every iteration step of this algorithm the optimal solution
is found within a subset of the set of all the constraint points. The algorithm uses
linear programming at each iteration step and at every step narrows the range over
which the Chebyshev error is allowed to lie. So, it breaks a big linear programming
problem into a series of small, more manageable ones.

What is the philosophy of the iterative approach?

Imagine that we have a metal sheet with which we want to clad a curved surface.
Every time we try to bend the sheet to pass through some points of the surface, it
bulges badly out at some other points owing to its rigidity. We find it difficult to
hold it down everywhere simultaneously. So, we bend and distort it first so that it
fits very well some points (see Figure 5.6). Then we find the point where it deviates
most from the desired shape and deform it again so that it now fits another subset
of points that includes the one with the previously maximum deviation. It is clear
that now it will not fit all that well the original set of points, and since we included
the most deviating one from the remaining points, our new fitting will be a little bit
worse than before. In other words, as our fittingprogresses, we gradually increase the
180 Image Processing: The Fundamentals

fitting error. This is the penalty we pay in order to make our metal sheet fit better
and better overall the shape we want. Such an algorithm is called maximizing because
from one iteration step to thenext it increases the lower limit of the fitting error.

A rigid metal sheet

First bend to match

part of the surface

Figure 5.6: How to fit a surface gradually.

Are there any algorithms that work by decreasing the upper limit of the
fitting error?

Yes, they are called minimizing. There are also algorithms called mini-max that work
by simultaneously increasing the lower limit of the error and decreasing its upper
limit. However, we are not going to discuss them here.

How does the maximizing algorithm work?

It works by making use of the concept of the limiting set of equations and the La
Vallee Poussin theorem.

What is a limiting set of equations?

The set of equations (5.48)is called limiting if all Am’s are # 0 and their absolute
values lAml cannot simultaneously be reduced for any choice of c’s we make.

What does the La Vallee Poussin theorem say?

Suppose that equations (5.48) form a limiting set. We choose some ci’s which give the
best approximation to F according to the Chebyshev sense. Call this approximation
Two-Dimensional Filters 181

P* ctg(p, v). Best approximation in the Chebyshev senseover the set X of

M points

- - . . . , (p,, v,) means that:

(p1, VI),(p2, v2),

muzIP* - F1
over set of points X
5 muzlP - F1
over same set of points X
where P is any other approximation; i.e. any other set of ci’s. The generalized
Vallee Poussin theorem states that the above best Chebyshev error is greater than
the minimum value of the error of the random approximation P:

error of the best Chebyshev

approximation
So, the theorem states that:

i.e. the error of the best Chebyshev approximation is bounded from above and below,
by the minimum and the maximum error of any other approximation.

What is the proof of the La Vallee Poussin theorem?

The right hand side of (5.58) is obvious by the definition of the best Chebyshev ap-
proximation. To prove the left inequality, assume first that it does not hold. Assume:

muzxIP* - F1 < minxlP - F1 + muzxlAkl < minxlA,I

This implies that all A;% are less than all A,% (since the maximum of the A;%
is less than the minimum of the Am’s).This means that when we change the values
of ci’s from ci’s to cf’s, all the absolute values of the A,% reduce simultaneously.
This is in contradiction to the assumption that the set of equations over the points
in X is limiting. So (5.58) holds.

What are the steps of the iterative algorithm?

Suppose that we choose a subset Xk: of the set of points X (these points are thechosen
discrete ( p , v) points over which we want to determine the best approximation). In
this subset of points we choose the best approximation

& ( p , v) = cn

i=l
cfgi(p, v) best in Xk:

and we define its error in Xk: by:

182 Image Processing: The Fundamentals

and its error in the whole set X by:

EI, mazxIPk - F1

Now determine a new set of points whichis a subset of X, such that this Pk
approximation in the Xk+l set has maximum error

and minimum error

lpk - FI 2 'k
and the set of equations

A = F(p7 v) - cn

i=l
w ( p 7 v) V(p7 v) E &+l

is limiting. Then according to the La Vallee Poussin theorem:

lpk - FI 5 ma2Xk+l Ipk+l - 5 maxxk+llpk -

or
dk 5 dk+l 5 Ek (5.59)

where P,+1 is the best approximation in the new set of points Xk+l and:

&+l = maxxk+lI P k + l - FI

From the way we choose every time the extra point to include in the subset X k ,
it is obvious that from one step to the next we narrow the double inequality (5.59)
by increasing its lower limit.

Can we approximate a filter by working fully in the frequency domain?

Yes, by calculating the values of the system function in some frequenciesas functions
of the values of the same function at other frequencies. In other words, express H ( p , v)
in terms of H ( k , 1 ) where k, 1 take up specific integer values and calculate the values
of H ( k , 1 ) for which mazlH(p,v) - F ( p , v)I is minimum. We shall consider certain
discrete values of p, v. These points will be the constraint points.

How can we express the system function of a filter at some frequencies as

a function of its values at other frequencies?

Consider the equations that express the relationship between the impulse response of
the filter and its system function:

(5.60)
Two-Dimensional Filters 183

Note that here ( p ,v) take integer values and they no longer signify the angular
frequencies like they did in all other expressions in this chapter so far. The inverse
discrete Fourier transform of this system function is:
N N
(5.61)

+
Call 2N 1 p . Notice that ifwe assume that H ( k , 1) = H ( - k , -1) the filter will
be real (as we want it) because the imaginary terms will cancel out; i.e.
N N

p L=-N l=-N 1 (5.62)

From (5.62) it is obvious that h(m,n)= h(-m, -n) and using that in (5.60) we
have:

(5.63)

Since we want to express H ( p , v) in terms of other values of H , we must substitute

from (5.62) into (5.63):

H(p,v)=$ c c cc
N N N

n=-Nm=-Nk=-Nl=-N
N
H(k,Z)cos
1 (5.64)

It can be shown (see Box B5.4)that this expression can be written as:

Z)+c 1
N N
1
1)@(0, c H ( k ,Z)@(k,1) (5.65)
k=ll=-N

where @ ( k ,1) is some known function of the trigonometric functions.

Example 5.4

Wewant to construct a 3 X 3 real and symmetric (when both its ar-

gumentschange sign) filter, with the followingfrequencyresponse:
-
m
0 1 0
1 0 1
184 Image Processing: The Fundamentals

i.e.

Formulate the problem so that it can be solved bylinearprogram-

ming. State clearly the dimensions and elements of each arrayinvolved.
(Suggestion: Use at most nine constraint points in Fourier space.)

Because of symmetry
h(-l, -1) = h(1,l)
h(-1,O) = h(1,O)
h(0,-1) = h(0,l)
h(-1,l) = h(1,-l)

For a 3 X 3 filter, clearly N = 1 in equation (5.65). Then:

+
H ( p , V ) = 2h(-1,1) COS(Y - p) 2h(0,1) COS V + 2h(l, 1)COS(,U + V )
+ +
2h(l, 0) cosp h(0,O)

h(1, l),h(l,O),h(O, l),h(l, -1) and h(0,O) are the unknowns we must specify.
Since they can be positive or negative, these must be our free unknowns. Define
the error as 6 m a ~ ~ , ~ I H v)(-p F, ( p ,.)I. Then:

OIH-FI+{ S > H - F j 6 - H 6> >

- F- ( H - F ) 6 + H > F

W e use two inequalitieslike the above for every constraint pointin frequency space,
i.e. a total of 18 inequalities.
W e want to minimize 6 which must always be non-negative, so it must be our
non-negative unknown. So: Minimize Z = X1 under the constraints

AllXl + A12X2 > B1 18 inequalities

X1 > 0 1 unknown
X2 free 5 unknowns

where X1 = 6

W e consider the points in the Fourier space in the following order:

Two-Dimensional Filters 185

Then
0
0
-1
1
0
0
-1
1
0
(Each element i s the value of- F ( p , v) or F ( p , v))
0
-1
1
0
0
-1
1
0
0
-1 -2cosl -2 -2cosl -2cos2 ’
1 2cosl 2 2cosl 2 cos 2
-1 -2cosl -2cosl -2 -2 cos 1
1 2cosl 2cosl 2 2 cos 1
-1 -2cosl -2cos2 -2cosl -2
1 2cosl 2cos2 2cosl 2
-1 -2 -2cosl -2cosl -2 cos 1
1 2 2cosl 2cosl 2 cos 1
-1 -2 -2 -2 -2
A12 = 1 2 2 2 2
-1 -2 -2cosl -2cosl -2 cos 1
1 2 2cosl 2cosl 2 cos 1
-1 -2cosl -2cos2 -2cosl -2
1 2cosl 2cos2 2cosl 2
-1 -2cosl -2cosl -2 -2 cos 1
1 2cosl 2cosl 2 2 cos 1
-1 -2cosl -2 -2cosl -2 cos 2
1 2cosl 2 2cosl 2cos2 ,
 186 Image Processing: The Fundamentals

B5.4: Prove expression (5.65).

Ifwe use the identity cos a cos p = $ [cos(a + p) + cos(a - p)] and define P
=A
equation (5.64) can be written as:
- N N N N

+ cos[nA(k - p) + mA(Z - V)]}

We define:
A(p+ k ) = IC

A(Z+v) y =
A(k - p ) E U
A(1- V ) = V
Then:

& c cH ( k , Z ) c c
, N
N N
N
H(P,v)= { c o s ( n ~ ~ + m y ) + c o s ( n u + m(5.66)
~)}

Consider the first of the terms inside the curly brackets:

N N
C C cos(nz + my) = C C (cos(nz)cos(my) + sin(rny)sin(nz))
n=-N m=-N n=-N
N N
= C cos(n2) C cos(my)
n=-N m=-N
N N

m=-N n=-N

As sine is an odd function, sin(my) summed over a symmetric interval of values

of m gives zero. So the second double sum in the above expression vanishes. On
the other hand, cos(nz) being an even function allows us to write:
N

n=-N m=-N
N
l +2 cN
cos(n2)
N

We use the identity:

n
C cos(kz) = cos )
X
?
(
nIC 1
sin -~
2 sin I
k=l
Two-Dimensional Filters 187

We apply it here to obtain:

cc
N N
cos(nz + m y ) =
2 cos (?X) sin F 2 cos (yy) sin F
n=-Nm=-N
sin 5

We also use the identity: 2 cos a s i n p = sin(a + p) - sin(a - p). So:

cc
N N
sin (YX)
-sin I sin (9
y) -sin %
n=-Nm=-N
cos(nz+my) =
( (9%))(
l+

sin
sin I
sin ( 2N+1
T y )
l+ sin!

-
- (5.67)
sin sin 5
We define:

4(u,P) = ssin[r(u
.
+ P)] (5.68)
m [ F ( a +P)]

From the definition ofp, X and A we have: y x = z”+lA ( p + k ) = ;%(p+v) =

+
r(p V). Similarly for v y . Then equation (5.67) can be written as:
N N
(5.69)

Working in a similar way we obtain:

Then (5.66) can be written as:

. N N

We define:

From the definition of 4 (equation (5.68)) it is clear that:

@(/c,1) = @ ( - k , -1)

We also have H ( k ,1 ) = H ( - k , - 1 ) . Then equation (5.70) can be further simplified:

188 Image Processing: The Fundamentals

N l

In a similar manner we can split the summationover k into the0 term, thepositive
and the negative terms and define a new variable i= -k to sum over negative
values of k:
.l I N N N N
H ( p , v ) = 3112 I H(-& -Z) + y y(
-1)@ -i,
H(-i, l)@(-& 1)

+ c
k=l
H ( - k 0 p - k 0) + 2 2 H ( k , -Z)@(k, -1)
k=l2=1
~

N N

+ c
N

k=l
H ( k , O ) @ ( k 0)
, + c
N

1=1
H ( 0 , -1)@(O, -1)

+ c
N

1=1
H@,W ( 0 ,1) + H@,O)@(O, 0)

Owing to the symmetry properties of functions @ ( k ,1) and H ( k , Z ) , the terms

1
that are underlined by the same type of line are equal, and therefore they can be
combined t o yield:
N N

+2 c
k=l
N
H ( k , 0 ) @ ( k 0)
, +2 c
1=1
N
H(0,1)@(0,1 ) + H(O,O)@(O,0)
1 (5.72)

The first three sums on the right hand side of the above expression can be com-
bined into one as they include summationover all positive values of 1, all negative
values of 1 and the term for 1 = 0:

cc c 1
N
H ( p ,v) = ~ 2 H ( k , Z)@(k,1 ) +2 H ( 0 ,1)@(0,1 ) + H(O,O)@(O,0)
2p2 k = 1 l=-N 1=1
 Two-Dimensional Filters 189

Finally, upon substituting the value of p we obtain:

N N l
(5.73)

What exactly are we trying to do whenwe design the filter inthe frequency
domain only?

We try to find optimal valuesfor the coefficients H ( k ,1 ) for Ic = 0 , 1 , . . ., N ,

Z = 0 , 1 , . . . ,N so that the error in approximating F ( p , v) by H ( p , v) at a fixed
number of constraint points (pi, vi) is minimum.
In practice, what people usually do is to fix some values of the H(k,Z) and try to
optimize the choice of the remaining values. For example, in the case of a lowpass
filter, we can put H(Ic,Z) = 1 for the frequencies we want to pass and H ( k ,1 ) =
0 for the undesired frequencies and then leave the values of H(Ic,Z) which refer to
transitional frequencies to be determined (see Figure 5.7).

.f plane frequency
0 0 0

stop band: putall H(k,l) there

values
e-, 0 / equal to 0

transition band: calculate optimal

0 ‘\ 0 1, 0
values of H(k,l) there
I l

I t‘ e
I
I
, -
I
l
l 1

pass band:
t ’ -grid
mk
values
continuous values
put all H(k,l) there
equal to 1

Figure 5.7: Fixing some of the filter values in the frequency domain and
letting others be free.
190 Image Processing:
FundamentalsThe

How can we solve for the unknown values H ( k , l ) ?

Equation (5.65) is similar to equation (5.50) and can besolvedby using linear
programming.

Does the frequency sampling method yield optimal solutions according to

the Chebyshev criterion?

No, because the optimization is not done for all H ( k , 1) since some of them are fixed.

Example 5.5

Formulate the problem of designing a filter inthe frequency domain so

that it can be solved by a linear programming package.

If compare
we equations (5.65) and (5.50), we see
that
the
prob-
lemisverysimilartotheproblem of specifying the
values of the fil-
ter in the real domain using
linear
programming: In that case
we
wereseeking t o specify h(O,O),h(l,O),. . .,h(N,O),. . . ,h(-N,rn), h(-N +
l , m ) , . . .,h ( N , m ) f o r m = 1,2,. . . , N . Nowwewishtospecify
H(O,O),H(O,l),...,H ( O , N ) , H ( k , - N ) , H ( k , - N + l ) , . .., H ( k , N ) f o r 5 =
1 , 2,...,N .
In that case, we had function cos(pn+vm) appearing in equation (5.50). Now
we have function @ ( p ,v) instead. @ ( p ,v), however, is also an even function with
respect t o both its arguments, and therefore it behaves in a similar way to the
cosine function as far as the solution of the problem is concerned. The problem
therefore can be formulated as follows:
Minimize:

under the constraints:

where:

and
Two-Dimensional Filters 191

X2 =
- +
(2N2 2J ? + 1) X 1
B1 = + 2Pl X 1
matrix matrix

A12 is a 2pl X (2N2+ 2N + 1) matrix the elements of which can easily be found
if we substitute equation (5.65) into the inequality constraint. It turns out to be:

. .
.. ..
I . . .

.. .. .. ..
192 Image Processing:
FundamentalsThe

What is the “take home” message of this chapter?

If we specify exactly the desired frequency response of a filter to be zero outside

a finite range, we obtain a filter of infinite extent in the real domain. A digital
filter of infinite extent can be realized as a recursive filter with the help of its z-
transform. z-transforms in two dimensions are muchmoredifficult to be inverted
than l-dimensional z-transforms. Also, recursive filters have to fulfil certain criteria
to be stable. The theory of 2-dimensional digital filters, their stability criteria and
their design problems are beyond the scope of this book.
To avoid using infinite impulse response filters we resort to approximations. We
specify exactly the frequency response of the filter we require and then try tofind the
finite impulse response filter which approximates the desired frequency behaviour as
well as possible.
Sometimes it is important to have exactly the desired frequency response to some
frequencies, while we do not care much about some other frequencies. Then we fix
the frequency response of the filter at the specific frequenciesand try tofind the value
of the filter at the remaining frequencies in an approximate way.
In filter design the criterion of approximation we use isthe Chebyshev norm instead
of the least square error. This is because we are interested in the worse artifact we
create at a single frequency,rather than in an overall good approximation which may
have a very unevenly distributed error among the various frequencies.
Chapter 6

Image Rest orat ion

What is image restoration?

Image restoration is the improvement of an image using objective criteria and prior
knowledge as to what the image should look like.

What is the difference between image enhancement and image restoration?

In image enhancement we try to improve the image using subjective criteria, while
in image restoration we are trying to reverse a specific damage suffered by the image,
using objective criteria.

Why may an image require restoration?

An image may be degraded because the grey values of individual pixels may be altered,
or it may be distorted because the position of individual pixels may be shifted away
from their correct position. The second case is the subject of geometric lestoration
Geometric restoration is also called image registration because it helps in finding
corresponding points between two images of the same region taken from different
viewing angles. Image registration is very important in remote sensing when aerial
photographs have to be registered against the map, or two aerial photographs of the
same region have to be registered with each other.

How may geometric distortion arise?

Geometric distortion may arise because of the lens or because of the irregular move-
ment of the sensor during image capture. In the former case, the distortion looks
regular like those shown in Figure 6.1. The latter case arises, for example, when an
aeroplane photographs the surface of the Earth with a line scan camera. As the aero-
plane wobbles, the captured image may be inhomogeneously distorted, with pixels
displaced by as much as 4-5 interpixel distances away from their true positions.
194 Processing:
Image The Fundamentals

(a) Original (b) Pincushion distortion (c) Barrel distortion

Figure 6.1: Examples of geometric distortions caused by the lens.

-X A
X

Corrected image Distorted image

Figure 6.2: In this figure the pixels correspond to the nodes of the g1rids.
Pixel A of the corrected grid corresponds to inter-pixel position A’ of the
original image.

How can a geometrically distorted image be restored?

We start by creating an empty array of numbers the same size as the distorted image.
This array will become the corrected image. Our purpose is to assign grey values to
the elements of this array. This can be achieved by performing a two-stage operation:
spatial transformation followed by grey level interpolation.

How do we perform the spatial transformation?

Suppose that the true position of a pixel is (z, y) and the distorted position is ( g , $ )
(see Figure 6.2 ). In general there will be a transformation which leads from one set
of coordinates to the other, say:
Image Restoration 195

First we must find to which coordinate position in the distorted image each pixel
position of the corrected image corresponds. Here we usually make someassumptions.
For example, we may say that the above transformation has the following form:

where cl, c2, . . . ,c8 are some parameters. Alternatively, we may assume a more general
form, where squares of the coordinates X and y appear on the right hand sides of the
above equations. The values of parameters c l , . . . ,C8 can be determined from the
transformation of known points called tie points. For example, in aerial photographs
of the surface of the Earth, there are certain landmarkswith exactly known positions.
There are several such points scattered all over the surface of the Earth. We can use,
for example, four such points to find the values of the above eight parameters and
assume that these transformation equations with the derived parameter values hold
inside the whole quadrilateral region defined by these four tie points.
Then, we apply the transformation to find the position A' of point A of the
corrected image, in the distorted image.

Why is grey level interpolation needed?

It is likely that point A' will not have integer coordinates even though the coordinates
of point A in the (z, y) space are integer. This means that we do not actually know the
grey level valueat position A'. That is whenthe grey level interpolation process comes
into play. The grey level value at position A' can be estimated from the values at its
four nearest neighbouring pixels in the (?,G) space, by some method, for example by
bilinear interpolation. We assume that inside each little square the grey level value is
a simple function of the positional coordinates:

g(D,jj) = a0 + pjj + y2i.g + 6

where a , . . . , 6 are some parameters. We apply this formula to the four corner pixels
to derive values of a , p, y and 6 and then use these values to calculate g($, jj) at the
position of A' point.
Figure 6.3 below shows in magnification the neighbourhood of point A' in the
distorted image with the four nearest pixels at the neighbouring positions with integer
coordinates.
Simpler as well as more sophisticated methods of interpolation may be employed.
For example, the simplest method that can be used is the nearest neighbour method
where A' gets the grey level value of the pixel which is nearest to it. A more sophisti-
cated method is to fit a higher order surface through a larger patch of pixels around
A' and find the value at A' from the equation of that surface.
196 Processing:
Image The Fundamentals

Figure 6.3: The non-integer position of point A' is surrounded by four

pixels at integer positions, with known grey values ( [ X ] means integer part
of X ) .

Example 6.1

In the figure below the grid on the right is a geometrically distorted

image and has to be registered with the reference image on the left
using points A, B , C and D as tie points. The entries in the image
on the left indicate coordinate positions. Assuming that the distortion
within the rectangle ABCD can be modelled by bilinear interpolation
and the grey level value at an interpixel position can be modelled by
bilinear interpolation too, find the grey level value at pixel position
(2,2) in the reference image.

Suppose that the position (i, 6) of a pixel in the distorted image is given in terms
of its position (X,y ) in the reference image by:

W e have the following set of the corresponding coordinates between the two grids
using the four tie points:
Image Restoration 197

Distorted ( 2 , y ) coords.Reference (X,y) coords.

Pixel A (0,O) (090)
Pixel B (3,1> (390)
Pixel C (1,3) ((43)
Pixel
4) D (4, (3,3)
W e can use these to calculate the values of the parameters cl, . . . ,CS:

Pixel A:

Pixel B:

Pixel C 1=3~2
3 = 3cs
}+{ 1

4=3+3x$+9c3
Pixel D
4=3x++3+9c7
The distorted coordinates, therefore, of any pixel within the square ABDC are
given by:

? = a : +y-=Y, - + y
X
3 3
For X = y = 2 we have 2 = 2 + 2 , y = $ + 2. So, the coordinates of pixel (2,2)
in the distorted image are (2$, 2 i ) . This position is located between pixels in the
distorted image and actually between pixels with the following grey level values:

- 213
W e define a local coordinate system ( 2 ,y”), so that the pixel at the top left corner
hascoordinateposition (0,O) thepixelatthetoprightcornerhascoordinates
(1,0), the one at the bottom left ( 0 , l ) and the one at the bottom right ( 1 , l ) .
Assuming that the grey level value between four pixels can be computed from the
grey level values in the four corner pixels with bilinear interpolation, we have:

Applying this for the four neighbouring pixels we have:

Image Restoration 199

We recognize now that equation (6.7) is the convolution between the undegraded
image f (z,y) and the point spread function, and therefore we can write it in terms
of their Fourier transforms:

G(u,W) = F(u,v)H(u,W) (6.8)

where G, F and H are the Fourier transforms of functions f , g and h respectively.

What form does equation (6.5) take for the case of discrete images?

N N

k=l k 1

We have shown that equation (6.9) can be written in matrix form (see equation
(1.25)):
g=Hf (6.10)

What is the problem of image restoration?

The problem of image restoration is: given the degraded image g, recover the original
undegraded image f .

How can the problem of image restoration be solved?

The problem of image restoration can besolved if we have prior knowledge of the point
spread function or its Fourier transform (the transfer function) of the degradation
process.

How can we obtain information on the transfer function ~ ( u , wof) the

degradation process?

1. From the knowledge of the physical process that caused degradation. For exam-
ple, if the degradation is due to diffraction, H ( u ,W ) can be calculated. Similarly,
if the degradation is due to atmospheric turbulence or due to motion, it can be
modelled and H(u,W) calculated.
2. We may try toextract information on k ( u ,W ) or h(a - z, ,B -9) from the image
itself; i.e. from the effect the process has on the images of some known objects,
ignoring the actual nature of the underlying physical process that takes place.
200 Processing:
Image The Fundamentals

I Example 6.2

When a certain static scene was being recorded, the camera underwent
planar motion parallel to the image plane (X,y ) . This motion appeared
as if the scene moved in the X,y directions by distances zo(t) and yo(t)
which are functions of time t. The shutterof the camera remained open
from t = 0 to t = T where T is a constant. Write down the equation
that expresses the intensity recorded at pixel position (X,y ) in terms of
the scene intensity function f (X,y ) .

The total exposure at any point of the recording medium (say the film) will be T
and we shall have for the blurred image:
T
(6.11)

Example 6.3

In Example 6.2, derive the transfer function with which you can model
the degradation suffered by the image due to the camera motion, as-
suming that the degradationis linear with a shift invariant point spread
function.

Consider the Fourier transform of g ( z , y )defined in Example 6.2.

+a +a
i*(u,v) = S_ S_,
, g(z, y)e-2"j(uz+uY)d XdY (6.12)

If we substitute (6.11) into (6.12) we have:

W e can exchange the order of integrals:

f (X - x o ( t ) y, - yo(t))e-2"j(ur+ur)drdy} dt (6.14)

This is the Fourier transfoA of the shifted function by

20,yo in directions X,y respectively
Image Restoration 201

W e have shown (see equation (2.67)) that the Fourier transform of a shifted func-
tion and the Fourier transform of the unshifted function are related by:

( F . T. of shifted function) = ( F . T. of unshifted function)e-2aj(uzo+vyo)

Therefore:

&(U, v) =
I' fi(u,y)e-2?rj(uzo+vy'J)dt

where F(u,v) is the Fourier transformof the scene intensity functionf (X,y ) , i.e.
the unblurred image. F ( u , v ) is independent of time, so it can come out of the
integral sign:

.)
=
I'
.) e-2?rj(uzo(t)+vYo(t))dt

Comparing this equation with (6.8) we conclude that:

(6.15)

Example 6.4

Suppose that the motion in Example 6.2 was in the X direction only
and with constant speed F, so that yo(t) = 0, zo(t) = F. Calculate the
transfer function of the motion blurring caused.

I n the result of Example 6.3, equation (6.15), substitute yo(t) and xO(t) to obtain:
 202 Processing:
Image The Fundamentals

Example 6.5 (B)

It was established that during the time interval T when the shutter
was open, the camera moved in such a way that it appeared as if the
objects in the scene moved along the positive y axis, with constant
acceleration 2 a and initial velocity SO, starting from zero displacement.
Derive the transfer function of the degradation process for this case.

In this case x:o(t)= 0 and

dt
%
dt2
dY0 = 2 a t + b + y o ( t ) = at2 + bt + C
= 2a + -

where a is half the constant acceleration and b and c some integration constants.
W e have the following initial conditions:

t=O zero
shifting i.e. c = 0
t = 0 velocity of shifting = SO + b = SO
Therefore:
yo(t) = at2 + sot

W e substitute xo(t) and yo(t) in equation 6.15 f o r H ( u , v ) :

H(u,w) =
I’ e-~?rj71(Cd2+Sot)dt

= LT cos [2.rrvat2 + 2 m s o t ] d t - j
I’ sin[27rvat2 +2~vsot] dt
We may use the following formulae:

S cos(ax2+bx+c)dx =

S sin(ax2+bx+c)dx =

where S ( x ) and C ( x ) are

S ( x )E clx sin t2dt

C ( x )E 6lx cos t2dt

and they are called Fresnel integrals.

204 Image Processing:
FundamentalsThe

1
lim S(z) = -
X+CC 2
1
lim C(%)= -
X+oO 2
lim S(z) = 0
x+o
lim C(%)= 0
x+o

Therefore, for SO N 0 and T + m, we have:

c ( E ( a T + s o ) )+ ; c (Eso)
+ 0

S (E,,,,,))
(Eso) +; S +o
2 7 4 27Fvs;
cos -+ 1 sin -+ O
a! a
Therefore equation (6.17) becomes:

H ( u , v ) c! ~

2
1
['+a]
6 2
= ~

4
1-j
6

Example 6.7

How can we infer the point spread function of the degradation process
from an astronomical image?

W e k n o w t h a tby definition the point spread function is the output of the imaging
system when the input isa point source. In a n astronomical image, a very distant
star can be considered as a point source. B y measuring then the brightness profile
of a star we immediately have the point spread functionof the degradation process
this image has been subjected to.
Image Restoration
205

Example 6.8

Suppose that we have an ideal bright straight line in the scene parallel
to the image axis X. Use this information to derive the point spread
function of the process that degrades the captured image.

Mathematically the undegraded image of a bright line can be represented by:

f (2,Y ) = S(Y)
where we assume that the line actually coincides with the X axis. Then the image
of this line will be:

h1 (X,Y ) = /'" /'"

-m -m
h ( -~X ' , y - y')S(y')dy'dx' =
L
+"
h ( -~X ' , y ) d d

W e changevariable 3 X - X' + dx' = -d3. The limits of 3 are f r o m +m t o

-ca. Then:
+"
hZ(X,Y) = -S,, -"
h(?,Y)d? = S_, h(?,Y)d? (6.18)

The right hand side of this equation does not depend on X, and therefore the left
hand side should not depend either; i.e. the image of the line will be parallel t o
the X axis (or rather coincident with it) and the same all along it:

hl(X,Y) = hl(Y) =
- +"
S_, hl(%Y)d3

3 is a d u m m y variable,
independent of X
(6.19)

(6.20)

The point spread function has as Fourier transform the transfer function given
by:

(6.21)
J -m

If we set U = 0 in this expression, we obtain

206 Image Processing:
FundamentalsThe

1
k(0,W) = 1: L[:
- h(x,y)dx]

~I(Y)f r o m

B y comparing equation (6.20) with (6.22) we get:

e-2?rjwydy

(6.19)
(6.22)

H ( 0 , W) = H l ( W ) (6.23)

That is, the imageof the ideal line givesus the profile of the transfer function along
a single direction; i.e. the direction orthogonal to the line. This is understandable,
as the cross-section of a line orthogonal to its length is no different from the cross-
section of a point. By definition, the cross-section of a point is the point spread
function of theblurringprocess. If now we have lots of ideallines in various
directions in the image, we are going to have information as to how the transfer
function looks along the directions orthogonal to the lines in the frequency plane.
B y interpolation then we can calculateH(u,W) at any pointin the frequency plane.

Example 6.9

It is known that a certain scene contains a sharp edge. How can the
image of the edge be used to infer some information concerning the
point spread function of the imaging device?

Let us assume that the ideal edge can be represented by a step function along the
X axis, defined by:

1 for y >0
0 for y 50
The image of this function will be:

/
00

he(%,Y ) = h(x - X', y - y')u(y')dx'dy'

W e m a y definenewvariables 53 X - X', J y - y'.Obviouslydx' = -d3and

dy' = -dJ. The limits of both 3 and J are f r o m +m t o -m. Then:
Image Restoration 207

Let us take the partial derivative of both sides of this equation with respect to y:

It is known that the derivative of a step function with respect to its argument is a
delta function:

(6.24)

If we compare (6.24) with equation (6.18) we see that the derivative of the image
of the edge is the image of a line parallel to the edge. Therefore, we can derive
information concerning the pointspread function of the imaging process by obtain-
ing images of ideal step edges at various orientations. Each such image should
be differentiated alonga directionorthogonal to the direction of the edge. Each
resultant derivative image should be treated as the image of a n ideal line and used
to yield the profile of the point spread function along the direction orthogonal to
the line, as described in Example 6.8.

Example 6.10

Use the methodology of Example6.9 to derive the point spreadfunction

of an imaging device.

Using a ruler and black ink we create the chart shown in Figure 6.4.

Figure 6.4: A test chart for the derivation of the point spread function
of an imaging device.
208 Image Processing:
FundamentalsThe

This chart can be used to measure the point spread function of our imaging sys-
tem at orientations 0", 45", 90" and 135". First the test chart is imaged using
our imaging apparatus. Then the partial derivative of the image is computed by
convolution at orientations O", 45", 90" and 135" using the Robinson operators.
These operators are shown in Figure 6.5.

0 -2-1

-1 -2 -1 0 -1 -2 2 1

( 4 MO (b) M1 (c) M2 ( 4 M3

Figure 6.5: Filters used to compute the derivative in 0,45,90 and 135
degrees.

190.0

so 0

-10 0 -1000 I ' ' ' ' '

0.0 50.0 1w.o 55.0 60.0 65.0

(a) Four profiles of the PSF (b) Zooming into (a)

-1000 I ' ' ' ' '

55 0 60.0 65.0

(c) PSF profile for orientations (d) PSF profile for orientations
0' and 90' 45' and 135'

Figure 6.6: The point spread function (PSF) of an imaging system, in

two different scales.
Image Restoration 209

The profiles of the resultant images along several lines orthogonal to the original
edges are computed and averaged t o produce the four profiles for0", 45", 90" and
135" plotted in Figure 6.6a. These are the profiles of the point spread function. In
Figure 6.6b we zoom into the central part of the plot of Figure 6.6a. Two of the
four profiles of the point spread functions plotted there are clearly narrower than
the other two. This is because they correspond to orientations 45" and 135" and
the distance of the pixels along these orientations is fi longer than the distance
of pixels along 0" and 90". Thus, the value of the point spread function that is
plotted as being 1 pixel away from the peak, in reality is approximately 1.4 pixels
away. Indeed, if we take the ratio of the widths of the two pairs of the profiles, we
find thevalue of 1.4.
In Figures 6 . 6 ~and 6.6d we plot separately the two pairs
of profiles and see that
the system has the same behaviour along the 45", 135" and 0", 90" orientations.
Takingintoaccountthe fi correction for the 45" and 135", weconcludethat
the point spread function of this imaging system is to a high degree circularly
symmetric.
In a practical application these four profiles can be averaged t o produce a
single cross-section of a circularly symmetric point spread function. The Fourier
transform of this 2D function is the system transfer function of the imaging device.

Ifweknow the transfer function of the degradation process, isn't the

solution to the problem of image restoration trivial?

If we know the transfer function of the degradation andcalculate the Fourier transform
of the degraded image, it appears thatfrom equation (6.8)we can obtain the Fourier
transform of the undegraded image:

(6.25)

Then, by taking the inverse Fourier transform of $ ( U , W), we should be able to

recover f ( z ,y ) , which is what we want. However, this straightforward approach pro-
duces unacceptably poor results.

What happens at points (U, W ) where B(u,W ) = O?

H(u,W ) probably becomes 0 at some points in the ( U , W) plane and this means that
G(u,v) will also be zero at the same points as seen from equation (6.8). The ratio
G(u,o)/I?(u, W) as appears in (6.25)will be O/O; i.e. undetermined. All this means is
that for the particular frequencies ( U , W ) the frequency content of the original image
cannot be recovered. One can overcome this problem by simply omitting the corre-
sponding points in the frequency plane, provided of course that they are countable.
210 Image Processing: The Fundamentals

Will the zeroes of B(u,v) and G(u7v) always coincide?

No, if there is the slightest amount of noise in equation (6.8), the zeroes of k ( u ,v)
will not coincide with the zeroes of G(u7v).

How can we take noise into consideration when writing the linear degra-
dation equation?

For additive noise, the complete form of equation (6.8) is:

G(u7v) = $(U7 v ) B ( u ,v) + fi(u,v) (6.26)

where fi(u,
v) is the Fourier transform of the noise field. @(U, v) is then given by:

A G(,, v) fi(u,v)
F(u,v)= ~ - ~ (6.27)
k ( u ,v) k ( u ,v)

In places where k ( u ,v) is zero or even just very small, the noise term may be
enormously amplified.

How can we avoid the amplification of noise?

In many cases, Ik(u,v)l drops rapidly away from the origin while Ifi(u, v)l remains
more or less constant. To avoid the amplification of noise then when using equation
(6.27), we do not use as filter the factor l / k ( u , v ) , but a windowed version of it,
cutting it off at a frequency before Ik(u,v)[becomes too small or before its first zero.
In other words we use:

@(U, v) = &!(U, v)G(u,v) - & ( U , v)fi(u,

v) (6.28)

where
(6.29)

where WO is chosen so that all zeroes of H ( u ,v) are excluded. Of course, one may use
other windowing functions instead of the above window with rectangular profile, to
make & ( U , v) go smoothly to zero at W O .

Example 6.11

Demonstrate the application of inverse filtering in practice by restoring

a motion blurred image.
Image Restoration 211

Let us consider the image of Figure 6 . 7 ~ .To imitate the way this imagewould loot
if it were blurred by motion, we take every 10 consecutive pixels along the X axis,
find their average value, and assign it to the tenth pixel. This is what would have
happened if, when the image was being recorded, the camera had moved 10 pixels
to the left: the brightness of a line segment in the scene with length equivalent to
10 pixels would have been recorded by asingle pixel. The result would loot lite
Figure 6.76. The blurred image g ( i , j ) in terms of theoriginalimage f ( i , j ) is
given by the discrete version of equation (6.11):

g(i,j)= 7
2T
C f(i-t,j)
1 ZT-l
i = O , l , ...,N - l (6.30)
k=O

where i~ is the total number of pixels with their brightness recorded by the same
cell of the camera, and N is the total number of pixels in a row of the image. I n
this example i~ = 10 and N = 128.
The transfer function of the degradation is given by the discrete version of
the equation derived in Example 6.4. W e shall derive it now here. The discrete
Fourier transform of g ( i , j ) is given by:

If we substitute g(1, t ) from equation (6.30) we have:

W e rearrange the order of summations to obtain:

D F T of shifted f (1, t )

By applying the property of the Fourier transforms concerning shifted functions,

we have:

where F ( m , n )is the Fourier transform of the original image.

As F ( m , n )does not depend o n t , it can be taken out of the summation:
212 Image Processing: The Fundamentals

W e identify then the Fourier transform of the degradation process as

(6.32)

The sum on the right hand side of this equation is a geometric progression with
ratio between successive terms
.arm
q Ee - 3 7
W e apply the formula

Cqk=-
n-l
qn - 1
where q # 1
k=O q-1
t o obtain:

Therefore:

(6.33)

Notice that for m = 0 we have q = 1 and we cannot apply the formula of the
geometric progression. Instead we have a s u m of 1 'S in (6.32) which is equal t o
i~ and so

fi(o,n)= 1 for o5n5N -1

It is interesting to compare equation (6.33) with its continuous counterpart, equa-

tion (6.16). W e can see that there is a fundamental diference between the two
equations: in the denominator equation (6.16) has the frequency U along the blur-
ring axis appearing on its own, while in the denominator of equation (6.33) we
havethesine of thisfrequencyappearing.Thisis because discreteimages are
treated by the discrete Fourier transform as periodic signals, repeated ad injini-
tum in all directions.
W e can analyse the Fourier transform of the blurred image in its real and
imaginary parts:

G(m,n) E G1 (m,n) + jG2 (m,n)

Image Restoration 213

W e can then write it in magnitude-phase form:

G(m,n) = dG:(m, n) + G;(m,n)ej@(m>n) (6.34)

where

(6.35)

To obtaintheFouriertransform of theoriginalimage we divide G(m,n) by

H(m,n):

Therefore, the real and the imaginary parts n), Fl(m,n) and F2(m,n)
of @(m,
respectively, are given by:

If we use the formulae +

cos(a b) = cosacos b - sinasin b and sin(a b) = +
+
cos a sin b sin a cos b and substitute for cos 4(m,n) and sin $(m,n) from equa-
tions (6.35) we obtain:

W e also remember to set:

F l ( 0 , n ) = G l ( 0 , n ) for 0 5 n 5 N - 1
F2(0,n) = G a ( 0 , n ) for 0 5 n 5 N - 1

If we use F1 (m,n) and F2 (m,n) as the real and the imaginary partsof the Fourier
transform of the undegraded image and Fourier transform back, we obtain image
6.7d. This is totally wrong because in equations (6.36) we divide by 0 for several
values of m.
214 Image Processing: The Fundamentals

Indeed, the denominator sin 9becomes 0 every time 9is a multiple of X:

iTXm kN
N
= k~ +m = -
2T
where k = 1 , 2 , . . .

Our image is 128 X 128, i.e. N = 128, and i~ = 10. Therefore, we divide b y 0
when m = 12.8,25.6,38.4, etc. As m takes only integer values, the denominator
becomes very small form = 13,26,38, etc. It is actually exactly0 only form = 64.
Let us omit this value for m, i.e. let us use:

F1 (64, n)
= G1 (64, n) f o r 0 5 n 5 127
F2(64,n) = G2(64,n) f o r 0 5 n 5 127
The rest of the values of F1 (m,n) and FZ(m,n) are as defined by equations (6.36).
If we Fourier transform back, we obtain the image in Figure 6.7e. The image
looks now almost acceptable, apart from some horizontal interfering frequency. In
practice, instead of trying to identify the values of m or n for which the denom-
inator of equations (6.36) becomes exactly 0, we find the first of those 0’s and
apply the formula only up to that pair of values. In our case, the first zero is for
c
k = 1, i.e. f o r m = ZT = 12.8.
W e use formulae (6.36), therefore, only for 0 5 m 5 12 and 0 5 n 5 127.
Otherwise we use:
F1 (m,n)= G1 (m,n) 13 5 m 5 127
FZ(m,n) = G2 (m,n) } for 0 5 n 5 127
If we Fourier transform back we obtain the image shown in Figure 6.7f. This
image looks more blurred than the previous with the vertical lines (the horizontal
interfering frequency) still there, but less prominent. The blurring is understand-
able: we have effectively done nothing to improve the frequencies above m = 12,
so the high frequencies of the image responsible for any sharp edges will remain
degraded. As for the vertical lines, we observe that we have almost 13 of them
in an image of width 128, i.e. they repeat every 10 pixels. They are due to the
boundaryeffect:TheFouriertransformassumesthattheimageis repeated ad
infinitum in all directions. So it assumes that the pixels on the left of the blurred
image carry the true values of the pixels on the right of the image. In reality of
course this is not the case, as the blurred pixels on the left carry the true values
of some points further left that do not appear in the image. To show that this
explanation is correct, we blurred the original image assuming cylindrical bound-
aryconditions, i.e. assumingthattheimageis repeated ontheleft.Theresult
is the blurred image of Figure 6.712. The results of restoringthisimage b y the
three versions of inverse filtering are shown at the bottom row of Figure 6.7. The
vertical lines have disappeared entirely and we have a remarkably good restoration
in 6.7h, obtained by simply omitting the frequency for which the transfer function
is exactly 0.
Image Restoration 215

(a) Original image (b) b a l i s t i c blurring (c) Blurring with

cylindrical boundary
condition

l
(d) Inverse filtering (e) Inverse filtering (f) Inverse filtering
of (b) of (b) omitting divi- of (b) omitting divi-
sion by 0 sion with terms be-
yond the first 0

(g) Inverse filtering (h) Inverse filtering (i) Inverse filtering of

of (c) of (c) omitting divi- (c) omitting division
sion by 0 withterms beyond
the first 0

Figure 6.7: Image restoration with inverse filtering.

216 Image Processing: The Fundamentals

. .. .. . .~

..
' '. :l
(a) b a l i s t i c blurring (b)
Blurring using (c) Realistic blurring
with added Gaussian cylindrical boundary with added Gaussian
noise ( U = 10) with added Gaussian noise ( U = 20)
noise ( U = 10)

(d) Inverse filtering (e) Inverse filtering (f) Inverse filtering

of (a), omitting divi- of (b), omitting divi- of (c), omitting divi-
sion by 0 sion by 0 sion by 0

.. . -
-..

(g) Inverse filtering (h) Inverse filtering (i) Inverse filtering of

of (a), but omitting of (b), but omitting (c), but omitting di-
division with terms division withterms vision with termsbe-
beyond the first 0 beyond the first 0 yond the first 0

Figure 6.8: Image restoration with inverse filtering in the presence of

noise.
 Image Restoration 217

Unfortunately, in real situations, the blurring is going to be like that of Figure

6.7b and the restoration results are expected to be more like those in Figures 6. ?e'
and 6.7f than those in 6.7h and 6.7i.
To compare how inverse filtering copes with noise, weproduced the blurred
and noisy images shown in Figure 6.8 by adding white Gaussian noise. The noisy
images were subsequently restored using inverse filtering and avoiding the division
by Q.The results, shown in Figures 6.8d-6.8f are really very bad: High frequencies
dominated by noise are amplified by the filter to the extent that they dominate the
restored image. When the filter is truncated beyond its first Q, the results, shown
in Figures 6.89-6.8i are quite reasonable.

How can we express the problem of image restoration in a formal way?

If f(r) is an estimate of the original undegraded image f(r), we wish to calculate

f(r) so that the norm of the residual image f(r) - f(r) is minimal over all possible
versions of image f(r). From Chapter 3 we know that this is equivalent to saying
that we wish to identify f(r) which minimizes

e2 = E{[f(r) - fWl2} (6.37)

Equation (6.7) in this section has been assumed to have the form:
+m
+m
h(r - r')f(r')dr' + v(.)
g ( r ) =S_, S_, (6.38)

where g ( r ) , f(r) and v(.) are considered to be random fields, with v(.) being the
noise field.

What is the solution of equation (6.37)?

If no conditions are imposed on the solution, the least squares estimateof f (r) which
minimizes (6.37) turns out tobe the conditional expectation of f(r) given g ( r ) which
in general is a non-linear function of g ( r ) and requires the calculation of the joint
probability density function of the random fields f(r) and g ( r ) . This can becalculated
with the help of non-linear methods like simulated annealing. However, such methods
are beyond the scope of this book.

Can we find a linear solution to equation (6.37)?

Yes, by imposing the constraint that the solution f(r) is a linear function of g ( r ) .
Clearly, the solution found this way will not give the absolute minimum of e but it
will make e minimum within the limitations of the constraints imposed. We
218 Image Processing: The Fundamentals

decide that we want the estimated image f(r) to be expressed as a linear function
of the grey levels of the degraded image, i.e.:

(6.39)

where m(r,r’) is the function we want to determine and which gives the weight by
which the grey level value of the degraded image g at position r’ affects the value of the
estimated image f at position r. If the random fields involved are homogeneous, the
weighting function m(r, r’) will depend only on the difference of r and r‘ as opposed
to depending on them separately. In that case (6.39) can be written as:

(6.40)

This equation means that we wish to identify a filter m(r) with which to convolve
the degraded image g(r‘) in order to obtain an estimatef(r) of the undegraded image
f
What is the linear least mean square errorsolution of the image restoration
problem?

If k ( u , v ) is the Fourier transform of the filter rn(r), it can be shown that the linear
solution of equation (6.37) can be obtained if

(6.41)

where Sfg(u, v) is the cross-spectral density of the undegraded and the degraded
image and Sgg(u,v) is the spectral density of the degraded image. G ( u ,W) is the
Fourier transform of the Wzener filter for image restoration.

Since the original image f(r) is unknown, how can we use equation (6.41)
which relies on itscross-spectral density with thedegraded image, to derive
the filter we need?

In order to proceed we need to make some extra assumption: the noise and the true
image are uncorrelated and at least one of the two has zero mean. This assumption
is a plausible one: we expect the process that gives rise to the image to be entirely
different from the process that gives rise to the noise. Further, if the noise has a
biasing, i.e. it does not have zero mean, we can always identify and subtract this
biasing to make it have zero mean.
Since f(r) and v(.) are uncorrelated and since E { v ( r ) )= 0, we may write:
Image 219
Restoration

To create the cross-spectral density between the original and the degraded image,
we multiply both sides of equation (6.38) with f ( r - S ) and take the expectation

- --
value:

E{g(r)f(r - s ) ) = /+" /+" h(r - r') E{f(r')f(r - S)) dr' + E{f(r - s)v(r))
-m -m
R,f (S) Rff (r'-r+s) 0 from (6.42)
Therefore:
+m
+m
~ , f ( s )= S_, h(r - r')Rff(r'
S _ , -r + s)dr'
In terms of Fourier transforms this equation can be written (see Box B6.4 and Ex-
ample 6.12) as:
(6.43)

From equation (6.38) we can also show that (see Box B6.3):

(6.44)
If we substitute equations (6.43) and (6.44) into (6.41), we obtain:

(6.45)

or

(6.46)

If we multiply numerator and denominator by fi(u,

W ) we obtain:

(6.47)

This equation gives the Fourier transform of the Wiener filter for image restoration.

Howcanwe possibly use equation (6.47)if weknow nothing about the

statistical properties of the unknown image f(r)?

If we do not know anything about the statistical properties of the image we want to
restore, i.e. we do not know S f f ( W ~ ),, we may replace theterm in equation

m
(6.47) by a constant l? and experiment with various values of l?.
This is clearly rather an oversimplification, as the ratio
( U , W ) and not a constant.
is a function of
220 Image Processing: The Fundamentals

What is the relationship of the Wiener filter (6.47) and the inverse filter
of equation (6.25)?

In the absence of noise, Svv(u,v)= 0 and the Wiener filter becomes the inverse
transfer function filter of equation (6.25). So the linear least square error approach
simply determines a correction factor with which the inverse transfer function of the
degradation process has to bemultiplied before it is used as a filter, so that the effect
of noise is taken care of.

Assuming that we know the statistical properties of the unknown image

f(r), how can we determine the statistical properties of the noise expressed
by Svv(r)?
We usually make the assumption that the noise is white; i.e. that

Svv(u,
W ) = constant = Svv(O,
0) = S_,+or, S_,Rvv(X,Y)dXdY
+a
(6.48)

If the noise is assumed to be ergodic, we can obtain Rvv(x,y) from a single pure
noise image (the recorded image g(z, y) when there is no original image, i.e. when
f(2,Y) = 0).

B6.1: Show that if m(r - r’) satisfies

m(r - r’)g(r’)dr’ (6.49)

then it minimizes the error defined by equation (6.37).

Ifwe substitute equation (6.40) into equation (6.37) we have:

e2 =E { - S_:1’) rn(r - r’)g(rt)drt (6.50)

Consider now another function rn’(r) which does not satisfy (6.49). We shall show
that rn’(r) when used for the restoration of the image, will produce an estimate
f’(r) with error d 2 , greater than theerror of the estimate obtained by m(r) which
satisfies (6.49):
Image Restoration 221

Inside the integrand we add to and subtract from m ( r - r’) function m’(r - r’).
We split the integral into two parts and then expand the square:

e’’= E { [f(r) -
+a +a
J’, J’,[m’(r - r’) + m(r - r’) - m(r - r’)lg(r’)dr’

=E { [(m -
+a
JI, La+a
m ( r - r’Mr’)dr’ 1
+ (/:L: [m(r - r’) - m’(r

+a +a
- r’)]g(r’)dr’
11 2 }

(6.51)

rename r’+~

The expectation value of the first term is e2 and clearly the expectation value of
the second term is a non-negative number. In the last term, in the second factor,
change the dummy variable of integration from r‘ t o S . The last term on the right
hand side of (6.51) can then be written as:

The first factor in the above expression does not depend on S and thus it can be
put inside the sign:

The difference [m(r - S ) - m’(r - S)] is not a random field but the difference
of two specific functions. If we change the order of integrating and taking the
expectation value, the expectation is not going t o affect this factor so this term
will become:
222 Image Processing:
FundamentalsThe

[m(r-s)-m'(r-S)]&

However, the expectationvalue in the above term is 0 according t o (6.49), so from

(6.51) we get that:

e r 2 = e2 + a non-negative term

We conclude that the error of the restoration created with the m'(r) function is
greater than or equal to the error of the restoration created with m(r). So m(r)
that satisfies equation (6.49) minimizes the error defined by equation (6.37).

Example 6.12 (B)

If $'(u,v), k ( u , v ) and G(u,w)are the Fourier transforms of f(r), h(r) and

g(r) respectively, and

g(r> = S0Os_m_
-00
h(t - r)f (t)dt (6.52)

show that
G(u,W) = H * @ , v)P(u,W) (6.53)

where I?*(u,W) is the complex conjugate of k ( u ,W).

Assumethat r = (X,y) and t = (53, g ) . TheFouriertransforms of thethree

functions are:

(6.54)
J-00 J-00

(6.55)
J-m J-m

(6.56)
J-m J-m

The complex conjugate of H(u,W ) is

H*(u,W) = (6.57)
Image Restoration 223

Let us substitute g(x,y ) f r o m (6.52) into the right hand side of (6.54):

G(u,v) =
-m
-m
-m
1: h(2 - X , J - y )f ( 2 ,J)d2dJe-j(u”+”Y)dxdy

W e definenewvariablesforintegration s1 2 - X and s2 y” - y t o replace

integration over X and y . Since d x = -dsl and d y = -ds2, dxdy = d s l d s 2 . Also,
as the limits of both s1 and s2 are f r o m +COt o -CO, we can change their order
without worrying about a change of sign:

The two double integrals are separable:

G(u,v) = 1- 0 0 0 0

h(s1,~ 2 ) e j ( ~ ~SldS2 1, 0 0 0 0

~ + ~ ~ ~ ) df ( -X, Y- k -j(uz+wij)&dJ

O n t h e right hand side of this equation we recognize the product of $’(u,v) and
H * ( U , v) fromequations (6.55) and (6.57) respectively. Therefore equation (6.53)
is proven.

B6.2: Show that the Fourier transform of the spatial autocorrelation

function of a random field f ( z , y) is equal to thespectral density Ifi(u,. ) l 2
of the field (Wiener-Khinchine theorem).

The spatial autocorrelation function of f ( z , y) is defined as

R f f ( 2 ,G) = f(. + 2 , Y + G ) f ( z ,Y)dzdY (6.58)

--CO
--CO

We multiply both sides of equation (6.58) with the kernel of the Fourier transform
and integrate to obtain theFourier transform of Rff(5, G), kff(u, v):
224 Image Processing: The Fundamentals

= +
We define new variables of integration s1 X 5 and s2 y ?j to replace the = +
integral over 5 and ?j. We have 5 = s1 - X , ?j = s2 - y, d5dQ = dslds2 and no
change in the limits of integration:

R f f ( U , v) = S-*; S-*; S_*,

-C-C
O-C
OO
S-*; f ( s 1 , s a ) f ( z , y)e-j((sl-")"+(S2-y)~)dXdydsldsz

The two double integrals on the right hand side are separable, so we may write:

We recognize the first of the double integrals onthe right hand side of this equation
to be the Fourier transform P(u,v) of f ( s 1 , s2) and the second double integral its
complex conjugate $ * ( U , U). Therefore:

ir'ff(U,U) = P(U,V)P*(U,V) = IP(U,U)12

B6.3: Derive an expression for the Fourier transform of function m(r)

that minimizes the error defined by equation (6.37).

Equation (6.49) which is satisfied by m(.) that minimizes (6.37) can be written
as :

where g ( s ) has gone inside the integral sign because it does not depend on r'.
The expectation operator applied to the second term operates really only on the

- -
random functions g(r') and g ( s ) . Therefore, we can write:

E {f(r)g(s)>
= /+-CO
-CO
S'" -00
m(r - 4 E{g(r')g(s)l dr' (6.60)
Rf9 ( . > S ) R,, (r' > S )

In this expression we recognize the definitions of the autocorrelation and cross-

correlation functions of the random fields, so we can write:
+CO+CO
J', J', m(r - r')Rgg(r',s)dr' = Rfg(r,S ) (6.61)
Image Restoration 225

We have seen that for homogeneous random fields, the correlation function can
be written as a function of the difference of its two arguments (see Example 3.7).
so:

m(r - r’)Rgg(r’- s)dr’ = Rf,(r - S) (6.62)

We introduce some new variables: r’ - S = t and r - S = r . Therefore dr‘ = dt

and r - r‘ = r - t . Then:
+m +m

J’, J’, m ( r - t)R,,(t)dt = R f g ( r ) (6.63)

This is a convolution between the autocorrelation functionof the degraded image

and thesought filter. According to theconvolution theorem theeffect is equivalent
to the multiplication of the Fourier transforms of the two functions:

where S,, and Sf, are the spectral density of the degraded image and the cross-
spectraldensity of the degraded andundegradedimagesrespectively;i.e.the
Fourier transforms of the autocorrelation function of g and cross-correlation of f
and g functions respectively. Therefore:

(6.65)

The Fourier transform of the optimal restorationfilter which minimizes the mean
square error between the real image and the reconstructed one, is equal to the
ratio of the cross-spectral density of the degraded image and the true image, over
the spectral density of the degraded image.

B6.4: If S,,(U, v), S f f ( u ,v) and SVV(u, v) are the spectral densities of
the homogeneous random fields g(z, y), f ( z , y ) and v(z,y) respectively,
f i ( u , v ) is the Fourier transform of h ( z ,y) and

h(. - 2,y - j j ) f ( % , jj)d5djj + V(., y) (6.66)

226 Image Processing: The Fundamentals

with the additional assumption that f ( z ,y) and v ( z ,y) are uncorrelated
and at least one of the two has zero mean.

If we multiply both sides of equation (6.66) with g ( z s1,y sa) and take the + +
ensemble average over all versions of random field g(%,y), we have:

Since g(z, y) is a homogeneous random field, we recognize on the left hand side
the autocorrelation function of g with shifting arguments s1, sa, R,,(sl, sa). The
noise random field v(z,y) is also homogeneous, so the last term on the right hand
side is the cross-correlation R g v ( s l s2)
, between random fields g and v. Further,
+ +
g(z S I , y sa) does not depend on the variables of integration Ir: and g, so it
may go inside the integral in the first term of the right hand side:

l
R , , ( s l , s a ) = E { [ ~ - ~ ( z - i ; Y - , ) f ( ~ , , ) g ( z + s l , ~ + s a ) d ~ d+ R, g v ( S l , S a )

Taking the expectation value and integrating are two linear operations that can
be interchanged. The expectation operator operates only on randomfields f and
g, while it leaves unaffected function h. We can write therefore:

R,, (s1,sz)= [ ~ - ~ ( z - . i . , y - y ) E { f ( f , i ) g ( 5 f s 1 , y + s - i ) } d ~ d i + R g r ( s 1 , s ? )

We recognize inside the integral the cross correlation R,f between fields f and g
calculated for shifting values z + s1 - 2 and y + s2 - 5:

~,,(~~,s~)=Sii_lxh(z--S,y-(i)~,~(z--S+s~,y-I/+s~)dZd~+~~,(~~,s~)(6.
- -Q3

We may define new variables of integration: z - 5 a , y - jj =

p. Then =
d 5 d i = d a d p , and the change of sign of the two sets of limits of integration
cancel each other out:
c o r n

R,, (SI,~ 2 =) J’, J’,

h(a,P)Rgf( a + SI, P + s2)dadP + R,, ( S I , 5-21

We can change variables of integration again, t o W a + sl, z p + s2. Then= =

Q: = W - S I , p = z - s2, dadp = dwdz and the limits of integration are not
affected:

R,, ( ~ 1s2)
, = 1,0 0 0 0

h(w - SI,z - s2)R,f ( W , z)dwdz + Rgv(s1,S Z ) (6.69)

Image Restoration 227

If we take the Fourier transform of both sides of this expression, and makeuse of
the result of Example 6.12, we can write:

&(U,U) = k * ( U , v ) i l g f ( U , U ) +&&,v)

where the quantities with the hat signify the Fourier transforms of the corre-
( A )

sponding quantities that appear in equation (6.69).

If the fields were ergodic, the ensemble auto- and cross-correlation functions
we calculated here would have been the same as the spatial auto- andcross- corre-
lation functions. Then the Fourier transforms of these functions would have been
the spectral or cross-spectral densities of the corresponding random fields (see Box
B6.2). In general,however, the fields are not ergodic. Then, it is postulated
that the Fourier transforms of the auto- and cross-correlation functions are the
spectral and cross-spectral densities respectively, of the corresponding random
fields.
Thus, it must be noted that in the development of the Wienerfilter, the
ergodicity assumption is tacitly made. With this in mind, we can write:

S,, ( U , v) = I?* ( U , v1Sg.f ( U , U) + Sgv(U,U) (6.70)

We notice that we need to calculate the cross-spectral density between random
fields f and g. We start again from equation (6.66), but now we multiply both
sides with f ( z - s l , y-sz) and take the expectationvalue. The reason we multiply
+ +
with f ( z - s1, y - s2) and not with f ( z s1, y s2) is because we formed the
shifting arguments of R,f in (6.68) by subtracting the arguments of f from the
arguments of g. We must follow the same convention again. Proceeding as before,
we have:
c o r n

Rgf(S1,sZ) = S_,h(z - 2 , Y -
S _, % I f ( . - 3 1 , Y - sz)} d?d%
U(?.,
+E If(. - s1, Y - sz)v(z,Y)} (6.71)

We remember that random fields f and v are uncorrelated and at least one of
them has 0 mean. Then:

Inside the integral on the right hand side of equation (6.71) we recognize the
autocorrelation function of random field f computed for shifting argument ( 2 -
z + s1, g - y+ sa). The reason we subtract the arguments of f(x - s1, y - s2)
from the argumentsof f ( 2 ,g), and not the otherway round, is because on theleft
hand side we subtracted the argumentsof the “new” function from the arguments
of the existing one (i.e. the arguments of f ( z - s1, y - sz) from the arguments of
g(., y)) t o from R,f. So we have:
228 Image Processing: The Fundamentals

0 0 0 0

Rgf(S1,S2) = h(x - 5,y - i j ) R f f ( 5- X + SI, g - y + s2)diXj

We define new variables of integration a X - 5, p = y - g:
0 0 0 0

R g f ( S 1 , S21 = J’, J’, P)Rff(Sl - Q,S2 - PPQdP

The aboveequation is a straightforwardconvolution,andinterms of Fourier

transforms it can be written as:

&f ( U , v) = f i b , v)& ( U , v)

Invoking again the postulate of the equivalence between Fourier transforms of

correlation functions and spectral densities, we can write:

Sgf (U, v) = fib, 4 S f f ( U , v) (6.73)

In equation (6.68) we also need the cross-spectral density between random fields
g and v. We start again from equation (6.66), multiply both sides with v(11:-
SI,y - s2) and take the expectation value:

kgv(s1,s2)=/m
--DC1
/--(z-%v -W{f(~,1J)v(x-sl,Y - SZ)}d5dg+Rvv(Sl,sz)

The integral term vanishes because of equation (6.72). In terms of Fourier trans-
forms we can therefore write:

Sgv(u,v) = Svv(u,v) (6.74)

If we substitute from equations (6.73) and (6.74) into equation (6.70) we obtain
the equation we set out t o prove, i.e. equation (6.67).

Example 6.13

Demonstrate how to apply Wiener filtering in practice by restoring a

motion blurred image.

Let us consider again the blurred image of Figure 6.7a. From equation (6.33) we
have:
Image Restoration 229

The Wiener filter as given by equation (6.46), with the ratio of the spectral den-
sities in the denominator replaced by a constant l? is:

Or

W e m u s t be careful for the case m = 0, when we have

h 1
M(0,n) = - for O l n l N - l
l+r
If we multiply with this function the Fourier transform of the blurred image as
defined by equation (6.3441, we obtain:

For the case m = 0, we have:

The real and the imaginary parts of @(m7n) are given by:
iT sin y sin 9 ( i -~ 1 ) r m
1 (m, n) =
F
sin2 *+ rig sin2 y N

If we use formulaecos(a + b) = cos a cos b - sin a sin b and sin(a + b) = sin a cos b+
cosasin b and substitute cos +(m, n) and sin+(m,n) from equations (6.35) we
obtain:
230 Image Processing: The Fundamentals

For m = 0 we must remember to use:

If we Fourier transform back using functions F l ( m ,n) an Fz(m,n ) we obtain the

restored image shown in Figure 6.9a. This image should be compared with images
6.7e and 6.7f, which are obtained by inverse filtering.
The restoration of the noisy images of Figures 6.8a and 6 . 8 ~b y Wiener fil-
tering is shown in Figures 6.9b and 6 . 9 ~ . These images should be compared with
Figures 6.89 and 6.8i respectively. In all cases Wiener filtering produces superior
results.

(a) Wiener filtering with (b) Wiener filtering with (c) Wiener filtering with
l? = 0.01 of image 6.7b l? = 0.5 of image 6.8a (re- l? = 1 of image 6 . 8 ~(re-
(realistic blurring) alistic blurring with addi- alistic blurring withaddi-
tive Gaussian noise (a = tive Gaussian noise (a =
10)) 20) 1

Figure 6.9: Image restoration with Wiener filtering.

If the degradation process is assumed linear, why don’t we solve a system

of linear equations to reverse its effect instead of invoking the convolution
theorem?

Indeed, the system of linear equations we must invert is given in matrix form by
equation (6.10), g = Hf. However, we saw that it is more realistic to include in this
equation an extra term representing noise (see equation (6.26)):

g=Hf+u (6.75)
Image Restoration 231

where v is the noise field written in vector form.

Since we assumed that we have some knowledge about the point spread function
of the degradation process, matrix H is assumed to be known. Then:

where H is an N 2 X N 2 matrix and f , g and v are N 2 X 1 vectors.

Equation (6.76) seems pretty straightforward, why bother with any other
approach?

There are two major problems with equation (6.76):

1. It is extremely sensitive to noise. It has been shown that one needs impossibly
low levels of noise for the method to work.
2. The solution of equation (6.76) requires the inversion of an N 2 X N 2 square
matrix with N typically being 500, which is a formidable task even for modern
computers.

Example 6.14

Demonstrate the sensitivity to noise of the inverse matrix restoration.

Let us consider the signal given by:

27Tx
f ( X ) = 25 sin - f o r x = O , l , ... , 2 9
30
Suppose that this signal is blurred by a function that averages every three samples
after multiplying them with some weights. W e can express this by saying that the
discrete signal is multiplied with matrixH given below, t o produce a blurred signal
dx).
0.4 0.3 0 0 . . . 0 0 0 0.3
0.3 0.4 0.3 0 . . . 0 0 0
.. . 0 0 0
.. .. .. ..
. . . .
0.3 0 0 0 . . . 0 0 0.4 0.3
To introduce some noise we round the elements of g(x) to the nearest integer. To
recover the original signal we multiply the blurred signal g(x) with the inverse of
matrix H. The original and the restored signal are shown in Figure 6.10.
232 Image Processing: The Fundamentals

f(x) 100
T-

-60 I
-100
I
11 I I l,
0 10 20 30
X

Figure 6.10: An original and a restored signal by direct matrix inver-

sion. The noise in the signal was only rounding error.

Is there any way by which matrix H can be inverted?

Yes, matrix H can easily be inverted because it is a block circulant matrix.

When is a matrix block circulant?

A matrix H is block circulant if it has the following structure:

H0 H M - ~ H M - ~ . . . H1

H = [ H1
. H0
H2 HH0
M . . H2 H3
H1- ~ . .. (6.77)

HM-l HM-2 HM-3

where H o , H .I ., . ,H M - ~are partitionsof matrix H , and they arethemselves circulant
matrices.
Image Restoration 233

When is a matrix circulant?

A matrix D is circulant if it has thefollowing structure:

D= [ d(0)
41)
d(2)
d ( M - 1) d ( M - 2 ) ... d(1)
d(0)
d(:)

d ( M - 1) d ( M - 2 )
d ( M - 1) ... 4 2 )
d(0)

d(M-3)
.. . 4 3 )

... d(0)
(6.78)

In such a matrix, each column can be obtained from the previous one by shifting all
elements one place down and putting the last element at the top.

Why can block circulant matrices be inverted easily?

Circulant and block circulant matrices can easily be inverted because we can find
easily their eigenvalues and the eigenvectors.

Which are the eigenvalues and the eigenvectors of a circulant matrix?

We define the following set of scalars:

X ( k ) E d(O) + d ( M - 1)exp

1 (6.79)

and the following set of vectors:

(6.80)

where k can take up thevalues k = 0,1,2,. . . ,M - 1. It can be shown then by direct

substitution that:

Dw(k)= X(k)w(k) (6.81)

i.e X ( k ) are the eigenvalues of matrix D (defined by equation (6.78)) andw ( k ) are its
corresponding eigenvectors.
234 Image Processing: The Fundamentals

How does theknowledge of the eigenvalues and the eigenvectors of a matrix

help in inverting the matrix?

If we form matrix W which has the eigenvectors of matrix D as columns, we know

that we can write
D = WAW-' (6.82)
where W-' has elements

(6.83)

and A is a diagonal matrix with the eigenvalues along its diagonal. Then, theinversion
of matrix D is trivial:
D-1 = (WAW-l)-' = (W-1)-lA-1W-1 = WA-lW-1 (6.84)

Example 6.15

Consider matrix W the columns of which w(0), W (l),. . . ,w(M - 1) are

given by equation (6.80). Show that matrix 2 with elements:

M
is the inverse of matrix W .

We have :
1 1 ... 1
Z r j 2-i 2 eq(M-l)
e r ...
eM ... es2(M-l)

(1 eM
2"i(M-l) es2(M-1) ... e 2"i(&1)2
M

z = -1
M
Image Restoration 235

M-1 =k
All the off-diagonal elements of this matrix are of the form: Ck=oe M where
t is some positive or negative integer. This sum is a geometric progression with
first term 1 andratio q e y . W e apply the formula
n-l
"-1
C q k = k
k=O

and obtain that:

M-l
M for t = 0
0 for t # 0
CeVk=(
k=O

Therefore ZW = I , the identity matrix, i.e. Z = W - l .

Example 6.16

For M = 3 show that X ( k ) defined by equation (6.79) and w(k) defined

by (6.80) are the eigenvalues and eigenvectors respectively, of matrix
(6.78), for k = 0,1,2.

Let us redefine matrix D for M = 3 as:

W e also have:

x(~c)= do + d 2 e y k + d l e T 2k
2mj
for /c = 0,1,2

W e m u s t show that
Dw(k) = X(k)w(k) (6.85)

W e compute first the left hand side of this expression:

Image Restoration 237

Example 6.18 (B)

The elements of a matrix WN are given by:

where k and n take values 0,1,2...,N - 1. The elements of the inverse

matrix W', (see Example 6.10) are given by:

a (
l
W, ( k , n ) = -exp --
2:kn)
We define matrix W as the Kronecker product of WN with itself. Show
that the inverse of matrix W is formed by the Kronecker product of
matrix W', with itself.

Let us consider an element W(m, 1) of matrix W. W e write integers m and 1 in

terms of their quotients and remainders when divided by N :
238 Image Processing: The Fundamentals

m m l N +m2
1 11N +l2

Since W is WN@ WN we have:

w ( m , l ) = -e(
1 ymlzl)e(~m2z2) (6.88)
N
(ml,11) identifies in which partition of matrix W element W ( m ,1 ) belongs. m2
and l2 vary inside each partition, taking all their possible values.
m
1,
I

In a similar way we can write an element of matrix z W;' 8 W;':

2a j
Z ( t , n )= -e-
1 qtlnle-ytznz (6.89)
N
An element of the product matrix A W Z is given by:
NZ-1

1.
t=O
If we write again t t l N + t 2 , we can break the sum over t to two sums, one over
tl and one over t 2 :

From Example 6.10 we know that:

M-l

0 ifsf0
m=O
240 Image Processing: The Fundamentals

Notice that here we assume that h(w,u) is periodic with period N in each of its
+
arguments, and so h(1- N,u)= h((1 - N ) N,u)= h(1,u) etc.
The full matrix H can be written in the form:

(6.94)

where again owing to the periodicity of h(w,u),H-1 = H M - ~ , H - M=

+ H1
~ etc.

How can we diagonalize a block circulant matrix?

Define a matrix with elements

1
W ~ ( k , n=) -exp (6.95)
fl
and matrix

W=WN@WN (6.96)
where @ is the Kronecker product of the two matrices. The inverse of W N ( n~) ,is a
matrix with elements:
1
w,'(k,n) = exp (6.97)
fl
~

The inverse of W is given by (see Example 6.18):

W,'(k, n) = W,' @ W,' (6.98)

We also define a diagonal matrix A as follows:

(6.99)

where H is the discrete Fourier transform of the point spread function h:

H(u,w) =
1
cc
N-lN-l

2=0 y=o
-2?rj(y+T)
h(x,Y>e
(6.100)

It can be shown then by direct matrix multiplication that:

H = WAW-l (6.101)

Thus H can be inverted easily since it has been written as the product of matrices
the inversion of which is trivial.
Image Restoration 241

B6.5: What is the proof of equation (6.101)?

First we have t o find how an element H ( f ,g) of matrix H is related to the point
spread function h(%,y). Let us write indices f , g as multiples of the dimension N
of one of the partitions, plus a remainder:

f = f1N + f2

g = g1N + Q2

As f and g scan all possible values from 0 t o N2 - 1 each, we can visualize the
N X N partitions of matrix H, indexed by subscript U , as follows:

We observe that each partition is characterized by index fl - g1 and inside each

partition the elements computed from h ( z ,y) are computed for various values of
f 2 - 92. We conclude that:

Let us consider next an element of matrix WAW-l:

N ~ - ~ N ~ - I

z=o t=O

Since matrix A is diagonal, the sumover t collapses only t o values of t = 1. Then:

1=0

All is a scalar and therefore it may change position inside the summand:

A(m,n)=
NZ-l
c
1=0
WmlW&bl (6.103)

In Box B6.2 we showed how the elements of matrices Wml and Wlil can be
written if we write their indices in terms of their quotients and remainders when
divided by N:
242 Image Processing: The Fundamentals

m = Nm1-t m2
l E Nl1+ l2
n E Nnl + n2

Using these expressions, and the definition of All as N k ( l 2 , 1 1 ) from equation

(6.99), we obtain:

On rearranging, we have:

We recognize this expression the

as inverse
Fourier transform of
h(m2 - n2,ml - n1). Therefore:

A ( m , n )= h(m2 - n2,ml - n1) (6.105)

By comparing equations (6.102) and

(6.105) we can see that theelements of matrix
H and WAW-l have been shown to be identical.

B6.6: What is the transpose of matrix H?

We shall show that H T = WA*W-l, where A* means the complex conjugate of

matrix A.
According to equation (6.102) of Box B6.5, an element of the transpose of
matrix H will be given by:

m f ,9 ) = h(92 - f 2 , 9 1 - fl) (6.106)

An element A(m,n)of matrix WA*W-l will be givenby an equation similar

t o (6.104) of Box B6.5, but instead of having factor k(Z2,Zl) it will have factor
k ( - Z 2 , -Z1), coming from the element of h,*,
being defined in termsof the complex
conjugate of the Fourier transform k ( u ,U) given by equation (6.100):
Image 243
Restoration

We change the dummy variables of summation to:

= -12
X X

11 -11 and 12

Then:

Since we are dealing with periodic functions summed over a period, the range
over which we sum does not really matter, as long as N consecutive values are
considered. Then we can write:

We recognize on the right hand side of the above expression the inverse Fourier
transform of l ? ( . f 2 , . f l ) computed at (n2 - m2, n1 - m l ) :

A(m,n)= h(n2 - mz,n1 - m ~ ) (6.107)

By directcomparisonwithequation (6.106), we prove that matrices H T and

WA*W-I are equal, element by element.

Example 6.19

Show that the Laplacian, i.e. the sum of the second derivatives, of a
discrete image at a pixel position ( i , j ) can be estimated by:

A2f(i,j) = f(i - 1 , j ) + f ( i , j - +
1) f(i + 1,j)+ f ( i , j + 1) - 4f(i,j)

At inter-paxel position (i + i , j ) , the first deravate of the image function along the
i axas is approximated by the first diference:

&f(i + 21 , j ) = f(i + 1,j) - f ( i , j )

Similarly, the first diference at (i - i , j ) along the i axasas:
244 Processing:
Image The Fundamentals

1 Similarly, the second derivative at ( i , j ) along the j axis can be approximated by:
1 1
A ; f ( i , j ) = A j f ( i , j + -) - Aj f ( i , j - 2)
2
= f ( i ,j + +
1) - 2f ( i , j ) f ( i ,j - 1) (6.109)

Adding equations (6.108) and (6.109) part by part we obtain the result.

Example 6.20

Consider a 3 X 3 image representedby a column vector f. Identify a 9 X 9

matrix L such that if we multiply vector f by it, the output will be a
vector with the estimateof the value of the Laplacian at each position.
Assume that image f is periodic in each direction with period 3. What
type of matrix is L?

From Example 6.19 we know that the matrix operator that returns the estimate of
the Laplacian at each position is:

0 1 0
1 -4 1
0 1 0

To avoid boundary effects, we first extend the image in all directions periodically:

f31 f 3 32
f11f 1 3 f 1 2 fll

t:
f33 (f31
f11
f21
f23)
f22
1 f33 f32
f 2f31
f12 f13
Image Restoration 245

By observing which values will contribute to the value of the Laplacian at a pixel,
and with what weight, we construct the 9 X 9 matrix below with which we must
multiply the column vector f:

-4 1 1 1 0 0 1 0 0 ) ff")
1 - 4 1 0 1 0 0 1 0 f21
1 1 - 4 0 0 1 0 0 1 f31
1 0 0 - 4 1 1 1 0 0 f12
0 1 0 1 - 4 1 0 1 0 f22
0 0 1 1 1 - 4 0 0 1 f32
1 0 0 1 0 0 - 4 1 1 f13
0 1 0 0 1 0 1 - 4 1 f23
\ 0 0 1 0 0 1 1 1 - 4 ) \f33)

This matrix is a block circulant matrix with easily identifiable partitions of size
3 X 3.

Example 6.21 (B)

Using the matrix defined in Example 6.20, estimate the Laplacian for
the following image:

(i S i)
Then re-estimate the Laplacian of the above image using the formula
of Example 6.19.

f-4 1 1 1 0 0 1 0 0 ) 3) f -7 )
1 - 4 1 0 1 0 0 1 0 2 -4
1 1 - 4 0 0 1 0 0 1 0 6
1 0 0 - 4 1 1 1 0 0 2 -4
0 1 0 1 - 4 1 0 1 0 o = 5
0 0 1 1 1 - 4 0 0 1 0 3
1 0 0 1 0 0 - 4 1 1 1 3
0 1 0 0 1 0 1 - 4 1 1 0
0 0 1 0 0 1 1 1 - 4 \l -2
Image 246 Processing:
FundamentalsThe

If we use the formula, we need to augment first the imageby writing explicitly the
boundary pixels:
0 0 1

3 2 1
The Laplacian is:
1 + 2 + 2 - 4 x 33 + 1 - 4 x 21 + 2 + 1 + 3 - 4 x l
3+1-4~2+2+1+1+2-4x1
1 + 1 - 4 x21+ 1 2 + 1 + 3
)=(-; :) -7 -4
5
3 -2
Note that we obtain the same answer, whether we use the local formula or the
matrax multaplacation.

Example 6.22 (B)

Find the eigenvectors and eigenvaluesof the matrix derived in Example

6.20.

Matrax L derived an Example 6.20 is:

f - 4 1 1 1 0 0 1 0 0
1 - 4 1 0 1 0 0 1 0
1 1 - 4 0 0 1 0 0 1
1 0 0 - 4 1 1 1 0 0
0 1 0 1 - 4 1 0 1 0
0 0 1 1 1 - 4 0 0 1
1 0 0 1 0 0 - 4 1 1
0 1 0 0 1 0 1 - 4 1
0 0 1 0 0 1 1 1 - 4

W e can clearly see that this matrix is a block circulant matrix with easily iden-
tifiable 3 X 3 partitions. To find its eigenvectors we first use equation (6.80) f o r
M = 3 t o define vectors W:
Image Restoration 247

These vectors are used as columns to create a matrix. W e normalize this matrix by
dividing by = fi, andthusconstructthematrix defined by equation(6.95):

W e take the Kronecker product of this matrix with itself to create matrix W as
defined by equation (6.96):
1 1 1 1 1 1 1 1
ez
2aj

e 7
4aj
e 8aj7
e 7
1
1
ez
ez
2aj
2aj

ez 4aj
e 8aj7
ez ez
e Zaj7
1
1
4aj
Zaj 4aj
e 823’
7

3
1 1
%
e3 7 e% 7 e 7
9
e% 7 e 7
9
e3 7
e 4aj
3 e 3 e 2a
3j e 6a
3j e 3 e 4a
3j e 3 e12a
3j
e s
1
2-3’
8aj
e 7
1
4aj
ez ez ez ez
e 7
4-3’
e 7
6-3’
lOaj

e 8 a7j e 7
8aj
8a j
e 8 a7j
e lOaj
7 e 7
12-3’
e 7
4-3’
e 8 a7j e 7
4-3’
e 7
8aj
e lZaj
7 e 7
8aj
e lZaj3 e 3
16aj
e 7 e 7 e 7 e 7 e 3 e 7 e 3 e 3
The columns of this matrix are theeigenvectors of matrix L. Note that these
eigenvectors are the same for all block circulant matrices with the same structure,
independent of what the exact values of the elements are. The inverse of matrix
W can be constructed using equation (6.97), i.e. b y taking the complex conjugate
of matrix W . (Note that for a general unitary matrix we must take the complex
conjugate of its transpose in order to construct its inverse. This is not necessary
here as W is a symmetric matrix and therefore it is equal to its transpose.)

The eigenvalues of matrix L can be computed from its Fourier transform, using
equation (6.99). First, however, we need to identify the kernel 1(x,y ) of the opera-
tor represented by matrix L and take its Fourier transform L(u,v) using equation
(6.100).
From Example 6.20 we know that the kernel function is:
248 Image Processing: The Fundamentals

0 1 0
1 -4 1
0 1 0

W e can identify then the following values for the discrete function1(x,y ) :

l(0,O) = -4, l(-l, -1) = 0, l(-1,O) = 1, l(-l, 1) = 0

l(0, -1) = 1, 1(0,1) = 1, l(1, -1) = 0, l(1,O) = 1, 1(1,1) = 0

Note that these values cannot be directly used in equation (6.100), which assumes
a function h(x,y ) defined with positive values of its arguments only. W e therefore
need ashifted version of our kernel, one that puts the value -4 at the top left
corner of the matrix representation of the kernel. W e can obtain such a version
b y reading the first column of matrix L and wrapping it around to form a 3 X 3
matrix:
-4 1
1 0 0
1 0 0
Then we have:

l(0,O) = -4, 1(0,1) = 1, 1(0,2) = 1, l(1,O) = 1

l(2,O) = 1, 1(1,1) = 0, 1(1,2) = 0, 1(2,1) = 0, 1(2,2) = 0

W e can use these values in equation (6.100) to derive:

1 2rr j 2rrj 2rr j
i ( u ,v) = - [-4 + e - 7 " + e-72V + e - 7 " + e-?2~] (6.110)
3
Formula (6.99) says that the eigenvalues of matrix L, which appear along the di-
agonal of matrix A ( k , i ) , are the values of the Fourier transformi ( u ,v), computed
f o r U = km& 3 and v = [$] where k = 0,1,. . . ,8. These values can be computed
using formula (6.110):

i(0,O) = 0
1 1
22j
i ( 0 , l ) = - [-4+ e - 7
3
+ e - 7 + 1+ l] = -[-2
423

3
- 2cos60°] = -1

i(1,O) = i ( 0 , l ) = -1
Image Restoration 249

L(2,O) = L(0,2) = -1
Jt(2,l) = Jq1,2) = -2
1
i(2,2) = -
3
[-4 + 2 e - Y + 2e-?] = -2

where we have made use of the following:

e- 9 = -cos60° + j s i n 6 0 ° = --1 + 3 .- d 3
2 2

Notethatthefirsteigenvalue of matrix L is 0. Thismeansthatmatrix L is

singular, and even though we can diagonalize it using equation (6.101), we cannot
invert it by taking the inverse of this equation. This should not be surprising as
matrix L expresses the Laplacian operator for an image, and we know that from
the knowledge of the Laplacian alone we can never recover the original image.
Applying equation (6.99) we define matrix A for L to be:

0 0 0 0 0 0 0 0 0
0 - 3 0 0 0 0 0 0 0
0 0 - 3 0 0 0 0 0 0
0 0 0 - 3 0 0 0 0 0
0 0 0 0 - 6 0 0 0 0
0 0 0 0 0 - 6 0 0 0
0 0 0 0 0 0 - 3 0 0
0 0 0 0 0 0 0 - 6 0
0 0 0 0 0 0 0 0 - 6

Having defined matrices W , W-' and A we can write:

L = WAW-l

This equation can be confirmed by direct substitution. First we compute matrix

AW-l:
250 Image Processing:
FundamentalsThe

0 0 0 0 0 0 0 0

OK, now we know how to overcome the problem of inverting H ; however,

how can we overcome the extreme sensitivity of equation (6.76) to noise?

We can do it by imposing a smoothness constraint to the solution, so that it does not

fluctuate too much. Let us say that we would like the second derivative of the recon-
structed image to be small overall. At each pixel, the sum of the second derivatives
of the image along each axis can be approximated by:
A2f (i, k) = f (i - 1,k) + f (i, k - 1) + f (i + 1,k) + f ( i ,5 + 1) - 4 f ( i , 5) (6.111)
This is the value of the Laplacian at position (i,k). The constraint we choose to impose
then is for the sum of the squares of the Laplacian values at each pixel position to be
minimal:
- .-N. N
[A2f ( i , k)I2 = minimal (6.112)
k=1 i=l

The value of the Laplacian at each pixel position can be computed by the Laplacian
operator which has the form of an N 2 X N 2 matrix acting on the column vector f (of
size N 2 X l),Lf. Lf is a vector. The sum of the squares of its elements are given by
(Lf)TLf. The constraint then is:

(LflTLf = minimal (6.113)

Image Restoration 251

How can we incorporate the constraint in the inversion of the matrix?

Let us write again in matrix form the equation we want to solve for f:
g=Hf+v (6.114)
We assume that the noise vector U is not known but some of its statistical properties
are known; say we know that:
uTu = E (6.115)
This quantity E is related to the variance of noise and could be estimated from the
image itself using areas of uniform brightness only. If we substitute U from (6.114)
into (6.115) we have:

(g - Hf)T(g- Hf) = E (6.116)

The problem then is to minimize (6.113) under the constraint (6.116). The solution
of this problem is a filter with Fourier transform:

(6.117)

By multiplying numerator and denominator by H(u,Y),we can extract the transfer

function of the restoration filter as being:

(6.118)

where y is a constant and ~ ( u , wis)the Fourier transform of an N X N matrix L,

with the following property: If we use it to multiply the image from the left, the
output will be an array the same size as the image, with an estimate of the value of
the Laplacian at each pixel position. The role of parameter y is to strike the balance
between smoothing the output and paying attention to the data.

B6.7: Find the solution of the problem: minimize (Lf)TLf with the
constraint

[ g - H f ]T [ g - Hf] = E (6.119)

According to the method of Lagrunge multipliers (see Box B6.9) the solution
must satisfy:

a
-[f T
L T Lf + X(g - Hf)] = 0 (6.120)
af
252 Image Processing: The Fundamentals

where X is a constant. This differentiation is with respect t o a vector and it will

yield a system of N2 equations (one for each component of vector f ) which with
+ +
equation (6.116) form a system of N2 1 equations, for the N 2 1 unknowns:
N 2 the components of f plus X.
If a is a vector and b another one, then it can be shown (see Example 6.23)
that :
afTa
-=a (6.121)
af
dbTf
-=b (6.122)
af
Also, if A is an N2 X N2 square matrix, then (see Example 6.24):

df Af
= ( A + AT)f (6.123)
af
We apply equations (6.121), (6.122) and (6.123) to (6.120) to perform the differ-
entiation:

-
fT(LTL)f +X(gTg- gTHf
+
eqn(6.122)
- fTHTg

eqn(6.121)
+

-l
fTHTHf ) =O
eqn(6.123) with
with b = HTg with a = HTg A = HTH

+ (2LTL)f + X(-HTg HTg + 2HTHf) = 0

-

+ (HTH+yLTL)f = HTg
(6.124)

where y i. Equation (6.124) can easily be solved interms of blockcirculant

matrices.Then:

f = [ H T H + "iLTL]-lHTg (6.125)

y can be specified by substitution in equation (6.119).

Image Restoration 253

B6.8 Solve equation (6.124).

Since H and L are block circulant matrices (see Examples 6.20, 6.22 and Box
B6.6), they can be written as:

H = WAhW-' H T = WAEW-l
L = WA1W-l LT = WAt W-l (6.126)

Then:
H ~ +Hy ~ +
T W~A E W - ~ W A ~ W -y~ ~ ~ ; ~ -
= l ~ ~ l ~ - l
= W A ~ A ~ WY +
- ~~ h ; ~ l ~ - l
+
= W(AZAh yAiAl)W-' (6.127)

We substitute from (6.126) and (6.127) into (6.124) to obtain:

W ( h ; l A h+ 7AZ;Al)W-lf = Wh;/W-lg (6.128)

First we multiply both sides of the equation from the left with W - ' , to get:

(AEAh + 7ArRl)W-'f = AEW-lg (6.129)

Notice that as A;, A;lhh and h;hl are diagonal matrices, this equation expresses
a relationship between the corresponding elements of vectors W-lf and W-lg
one by one.
Applying the result of Example 6.26, we can write:
2 2
A;lhh = N21I?(u,v)I and A;Al = N21L(u,u)I
2
where If,(u,u)Iis the Fourier transform of matrix L . Also, by applying the results
of Example 6.25, we can write:

W-'f = @ ( u , v ) and W-lg = G ( u , v )

Finally, we replace A;L by its definition, equation (6.99), so that (6.129) becomes:

1 I;T*(u,U ) G ( U , U)
F(u,w) = - (6.130)
II;T(u,U ) 1 2 + yli(u, U)12
254 Image Processing: The Fundamentals

What is the relationship between the Wiener filter and the constrained
matrix inversion filter?

Both filters look similar (see equations (6.47) and (6.118)), but they differ in many
ways:
1. The Wiener filter is designed to optimize the restoration in an average statistical
sense over a large ensemble of similar images. The constrained matrix inversion
deals with one image only and imposes constraints on the solution sought.
2. The Wiener filter is based on the assumption that therandom fields involvedare
homogeneous with known spectral densities. In the constrained matrix inversion
it is assumed that we know only some statistical property of the noise.
In the constraint matrix restoration approach, various filters may be constructed
using the same formulation by simply changing the smoothing criterion. For example,
one may try tominimize the sum of the squares of the first derivatives at all positions
as opposed to the second derivatives. The only difference from the formula (6.117)
will be in matrix L.

Example 6.23 (B)

Differentiation by a vector is defined as differentiation by each of the

elements of the vector. For vectors a, b and f show that:

dfTa dfTb
=a and =b
~

af ~

af
Assume that vectors a, b and f are N X 1. Then we have:

Use this in:

Similarly:
Image Restoration 255

Example 6.24 (B)

If f is an N X 1 vector and A is an N X N matrix, show that

df Af
= (A + AT)f
df

Using the results of Example 6.23 we can easily see that:

dfTAf - dfT(Af)
d(fTA)f
-
df df df
+

a(ATf)Tf
= Af +
df
= Af + ATf = (A AT)f +
( w e m a k e u s e of the fact that Af and ATf are vectors.)

Example 6.25 (B)

If g is the column vector that corresponds to a 3 X 3 image G and matrix

W-' is defined as in Example 6.18 for N = 3, show that vector W - l g is
the discrete Fourier transform G of G.

Assume that:
256 Image Processing: The Fundamentals

simplifies somehow. So we get:

Image Restoration

This shows that W-'g yields the Fourier transform of G, as a column vector.

Example 6.26 (B)

Show that if matrix A is defined by equation (6.99) then A*A is a diag-

2
onal matrix with its kth element along the diagonal being N'Ifi(k2, k1)l
where k2 E kmod N and kl E [X].
From the definition of A, equation (6.99), we can write:
NH(0,O) 0 0 ... 0
0 NH(1,O) 0 ... 0
A = 0 0 NH(2,O) . . . 0

0 0 0 . . . N H ( N - 1,N
Then:
NH*(O,0) 0 0 ... 0
0 N H * ( l ,0) 0 ... 0
A* = 0 0 N H * ( 2 , 0 ) .. . 0

0 0 0 . . . N H * ( N - 1,N - 1)
258 Image Processing: The Fundamentals

Obviously

‘NIH(O70)I2 0 0 ... 0
0 Nlfi(k 0 ... 0
h* h= 0 0 NIH(2, 0)l2 . . . 0

\ o 0 0 ... N I f i ( N - l , N - 1)12

B6.9 Why does the method of Lagrange multipliers work?

Suppose that we wish to satisfy two equations simultaneously:

f(z,y) = 0
g(z,y) = 0 (6.131)

Let
us
assume that in the ( ~ , y ) plane
the first of these
equations
is satisfied at point A and the
otherat point B , so thatit is im-
possible to satisfy bothequationsexactly for the same value of (X,y).

contours of constant
values of g(x,y)

We wish to find a point C on the plane where we make the least compromise in
violating these two equations. The location of this point will depend on how fast
the values of f ( z , y) and g(z, y) change from 0 as we move away from points A
and B respectively.
Image Restoration 259

Let us consider the isocontours of f and g around each of the points A and
B respectively. As the contours grow away from point A, function I f ( z ,y) I takes
larger and larger values, while as contours grow away from point B, the values
function l g ( X , y)l takes become larger aswell. Point C where the values of If ( X ,y)I
and g1(X,y) I are as small aspossible (minimum violation of the constraints which
demand that If ( ~ , y ) l= Ig(X,y)l = 0), must be the point where an isocontour
around A just touches an isocontouraround B, without crossingeach other.
Whentwocurvesjusttouch each other,theirtangents becomeparallel. The
tangent vector to a curve along which f =constant is O f , and the tangent vector
t o a curve along which g =constant is Vg. The two tangent vectors do not need
t o have the same magnitude for the minimum violation of the constraints. It is
enough for them to have the same orientation. Therefore, we say that point C
is determined by the solution of equation Of = pOg where p is some constant
thattakescare of the (possibly)differentmagnitudes of the twovectors. In
other words, the solution to the problem of simultaneous satisfaction of the two
incompatible equations (6.131) is the solution of the differential set of equations:

Of+XOg=O

where X is the Lagrange multiplier, an arbitrary constant.

Example 6.27

Demonstrate how to apply the constraint matrix inversion in practice,

by restoring a motion blurred image.

To apply the filter given by equation (6.117) to the blurred image of Figure 6.7a,
we must first define matrix ~ ( I Lv), which expresses the constraint.
Following the steps of Example 6.19, we can see that matrix L ( i , j ) with which
we have to multiply an N X N image in order to obtain the value of the Laplacian
at each position is given by an N 2 X N 2 matrix of the following structure:
260 Image Processing: The Fundamentals

m
N- 1 unit matrices NxN

I
matrix L

..
* * * r~
matrices
0
NXN

Matrixhasthefollowing form:
~ - - 3zeroes

f -4 1 0 0 ... 0 1
1 -4 1 0 ... 0 0
0 1 -4 1 ... 0 0
X
0 0 1 -4 ... 0 0
L= 0 0 0 1 ... 0 0
. ...
0 0 0 0 ... -4 1
( 1 0 0 0 ... 1 -4
To form the kernel we require, we must take the first columnof matrix L and wrap
it to form an N X N matrix. The first column of matrix L consists of the first
column of matrix (N elements) plus the first columns of N - 1 unit matrices of
size N X N . These N 2 elements have to be written as N columns of size N next
to each other, to form an N X N matrix L', say
' -4 1 1 ... 1
1 0 0 ... 0
0 0 0 ... 0
.. .. ..
. . . .. . .
0 0 0 ... 0
, 1 0 0 ... 0

It is the Fourier transform of this matrix that appears in the constraint matrix
inversion filter.
Image Restoration 261

W e calculate the Fourier transform using formula (6.100) with N = 128. Let
us say that the real and the imaginary parts of this transform are L1 (m, n)and
L2(m,n) respectively. Then

IQm, .)l2 +
= L?(m,n> L;(m,n>

The transfer function of the filter we must use then is given by substituting the
transfer function (6.33) into equation (6.117):

Note that for m = 0 we must use:

Working as for the case of the Wiener filtering we can find that the real and
imaginary parts of the Fourier transform of the original image are given by:

(iT-l)?rm (iT-l)?rm
1 (m,n) cos - G2 ( m ,n)sin
Fl(m,n)=-
N sin2 iTTm + y (Lf(m,n) + L;(m,n))i$ sin2 N E

1 iTsin
F2(m,n)= -
sin p 1(m,n)sin
(iT-l)?rm
+ G2(m,n)cos (iT-l)?rm]
N sin2 7
iT?rm +
y (Lf(m,n) + L;(m,n))i$ sin2 N E

These formulae are valid for 0 < m 5 N - 1 and 0 5 n 5 N - 1. For m = 0 we

must use the formulae:

If we take the inverse Fourier transform using functions Fl(m, n)and F2(m,n)
as the real and the imaginary parts, we obtain the restored image. The results of
restoring images 6.7b, 6.8a and 6 . 8 ~are shown in Figure 6.11. Note that different
values of 7, i.e. different levels of smoothing, have to be used for different levels
of noise in the image.
Image 262 Processing: The Fundamentals

(a) Constraint matrix in- (b) Constraint matrix in- (c) Constraint matrix in-
version with y = 0.001 version with y = 0.05 for version with y = 0.1 for
for image 6.7b (realistic image 6.8a (realistic blur- image 6 . 8 ~(realistic blur-
blurring) ring and Gaussian noise ring and Gaussian noise
(a = 10)) (a = 20))

Figure 6.11: Restoration with constraint matrix inversion.

What is the “take home” message of this chapter?

This chapter explored some techniques used to correct (i.e. restore) the damaged
values of an image. The problem of restoration requires some prior knowledge con-
cerning the original uncorrupted signal or the imaged scene, and in that way differs
from the image enhancement problem. Geometric restoration of an image requires
knowledge of the correct location of some reference points.
Grey level restoration of an image requires knowledge of some statistical proper-
ties of the corrupting noise, the blurring process and the original image itself. Often,
we bypass the requirement for knowing the statistical properties of the original im-
age by imposing some spatial smoothness constraints on the solution, based on the
heuristic that “the world is largely smooth”. Having chosen the correct model for the
degradation process and the uncorrupted image, we have then to solve the problem
of recovering the original image values.
The full problem of image restoration is a very difficult one as it is non-linear.
It can be solved with the help of local or global optimization approaches. However,
simpler solutions can be found, in the form of convolution filters, ifwe make the
assumption that thedegradation process is shift invariant and if we restrict the domain
of the sought solution to that of linear solutions only. Figure 6.12 summarizes the
results obtained by the various restoration methods discussed, for an image blurred
by motion along the horizontal axis.
Image Restoration 263

7- - - - - -

. ..

Figure 6.12: (a) Original images. (b) Restoration by inverse filtering with
omission of all terms beyond the first zero of the filter transfer function.
(c) Restoration by Wiener filtering. (d) Restoration by constraint matrix
inversion.
Chapter 7

Image Segmentation and

Edge Detection
What is this chapter about?

This chapter is about those Image Processing techniques that are used in order to
prepare an image as an input to an automatic vision system. These techniques perform
image segmentation and edge detection, and their purpose is to extract information
from an image in such a way that the output image contains much less information
than the original one, but the little information it contains is much more relevant to
the other modules of an automatic vision system than the discarded information.

What exactly is the purpose of image segmentation and edge detection?

The purpose of image segmentation and edge detection is to extract the outlines of
different regions in the image; i.e. to divide the image in to regions which are made
up of pixels which have something in common. For example, they may have similar
brightness, or colour, which may indicate that they belong to the same object or facet
of an object.

How can we divide an image into uniform regions?

One of the simplest methods is that of histogramming and thresholding If we plot the
number of pixels which have a specific grey level value, versus that value, we create
the histogram of the image. Properly normalized, the histogram is essentially the
probability density function for a certain grey level value to occur. Suppose that we
have images consisting of bright objects on a dark background and suppose that we
want to extract the objects. For such an image, the histogram will havetwo peaks
and a valley between them.
We can choose as the threshold then the grey level value whih corresponds to the
valley of the histogram, indicated by to in Figure 7.1, and label all pixels with grey
266 Image Processing: The Fundamentals

I
lOW high grey level value
threshold
threshold

Figure 7.1: The histogram of an image with a bright object on a dark

background.

level values greater than t o as object pixels and pixels with grey level values smaller
than t o as background pixels.

What do we mean by “labelling” an image?

When we say we “extract” an object in an image, we mean that we identify the pixels
that make it up. To express this information, we create an array of the same size
as the original image and we give to each pixel a label. All pixels that make up the
object are given the same label and all pixels that make up the background are given
a different label. The label is usually a number, but it could be anything: a letter or
a colour. It is essentially a name and it hassymbolic meaning only. Labels, therefore,
cannot be treated asnumbers. Label images cannot be processed in the same way as
grey level images. Often label images are also referred to as classified images as they
indicate the class to which each pixel belongs.

What can we do if the valley in the histogram is not very sharply defined?

If there is no clear valley in the histogram of an image, it means that there areseveral
pixels in the background which have the same grey level value as pixels in the object
and vice versa. Such pixels are particularly encountered near the boundaries of the
objects which may be fuzzy and not sharply defined. One can use then what is called
hysteresis thresholding: instead of one, two thresholdvalues (see Figure 7.1) are chosen
on either side of the valley.
Image Segmentation
Edge and Detection 267

The highest of the two thresholds is used to define the “hard core” of the object.
The lowest is used in conjunction with spatial proximity of the pixels: a pixel with
intensity value greater than the smaller threshold but less than the larger threshold
is labelled as object pixel only if it is adjacent to a pixel which is a core object pixel.
Figure 7.2 shows an image depicting a dark object on a bright background and
its histogram. In 7 . 2 ~the image is segmented with a single threshold, marked with a
t in the histogram, while in 7.2d it has been segmented using two thresholds marked
tl and t 2 in the histogram.

Grey Levels

(a) Original image (b) Histogram of (a)

(c) Thresholded with t=91 (d) Thresholded with tl = 68 and

t2= 100

Figure 7.2: Simple thresholding versus hysteresis thresholding.

Alternatively, we may try to choose the global threshold value in an optimal way.
Since we know we are bound to misclassify some pixels, we may try to minimize the
number of misclassified pixels.

How can we minimize the number of misclassified pixels?

We can minimize the number of misclassified pixels if we have some prior knowledge
about the distributionsof the grey values that make up theobject and the
background.
268 Image Processing: The Fundamentals

For example, if we know that theobjects occupy a certain fraction 8 of the areaof the
picture then this 8 is the prior probability for a pixel to be an object pixel. Clearly
the background pixels occupy 1 - 8 of the area anda pixel has 1- 8 prior probability
to be a background pixel. We may choose the threshold then so that the pixels we
classify as object pixels are a 8 fraction of the total number of pixels. This method
is called p-tile method. Further, ifwe also happen to know the probability density
functions of the grey values of the object pixels and the background pixels, then we
may choose the threshold that exactly minimizes the error.

B7.1 Differentiation of an integral with respect t o a parameter.

Suppose that the definite integral I ( X ) depends on a parameter X as follows:

I(4 = sf:‘ f(x;4 d x

Its derivative with respect to X is given by the following formula, known as the
Leibnitz rule:

How can we choose the minimum error threshold?

Let us assume that the pixels which make up the object are distributed according to
the probability density function p o ( x ) and the pixels which make up the background
are distributed according to function Pb(x).
Suppose that we choose a threshold value t (see Figure 7.3). Then the error
committed by misclassifying object pixels as background pixels will be given by:
t
S_,
PO(X)dX

and the error committed by misclassifying background pixels as object pixels is:

4’” Pb ( x ) d x

In other words, the error that we commit arises from misclassifying the two tails of
the two probability density functions on either side of threshold t. Let us also assume
that the fraction of the pixels that make up the object is 8, and by inference, the
Edge
Segmentation
and
Image Detection 269

1.5

1.o

0.5

0.0 4
5
X

Figure 7.3: The probability density functions of the grey values ofthe pix-
els that make up the object @,(X)) and the background (pb(x)). Their sum,
normalized to integrate to 1, is what we obtain if we take the histogram
of an image and normalize it.

fraction of the pixels that make up the background is 1 - 13. Then the total error is:

We would like to choose t so that E ( t ) is minimum. We take the first derivative

of E ( t ) with respect to t (see Box B7.1)and set it to zero:

The solution of this equation gives the minimum error threshold, for any type of
distributions the two pixel populations have.
270 Processing:
Image The Fundamentals

Example 7.1 (B)

Derive equation (7.3) from (7.2).

W e apply the Leibnitz rule given by equation (7.1) to perform the differentiation
of E ( t ) given by equation (7.2). W e have the following correspondences:
Parameter X corresponds to t.
For the first integral:

.(X) + -m (aconstant,withzeroderivative)
b(X) + t
(independent from the parameter with respect
f(X; + differentiate)
which
weto
For the second integral:

.(X) + t
b(X) + -cm (aconstant,withzeroderivative)
f(z;X) + pb(x)(independentfrom t)
Equation (7.5')then follows.

Example 7.2

The grey level values of the object and the background pixels are dis-
tributed according to the probability density function:

3 [a2 - (X - b)2] for b - a 5 X 5 b +a

otherwise

with a = 1, b = 5 for the background and a = 2, b = 7 for the ob-

ject. Sketch the two distributions and determine the range of possible
thresholds.
Image Segmentation
Edge and Detection 271

0.8

0.6

0.4

0.2

0.0
0 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3
X
Range of possible thresholds: 5 4 ~ 6

Example 7.3

If the object pixels are eight-ninths (p)

of the total number of pixels,
determine the threshold that minimizes the fraction of misclassified
pixels for the problem of Example 7.2.

W e substitute into equation (7.5’)the following:

3 3
pb(t) = -(-t2 - 24 + lot) po(t) = -(-t2 - 45 + 14t)
4 32
Then:
8 3
1 !(-t2 - 24 + lot) = - -(-t2 - 45 + 14t) +
9 4 9 32
21
-24 + lot = -45 + 14t + 4t = 21 + t = - = 5.25 (7.4;
4
272 Processing:
Image The Fundamentals

Example 7.4

The grey level values of the object and the background pixels are dis-
tributed according to the probability density function:

with 20 = 1, a = 1 for the objects, and 20 = 3, a = 2 for the back-

ground. Sketch the two probability density functions. If one-third of
the total number of pixels are object pixels, determine the fraction of
misclassified object pixels by optimal thresholding.

X
Range of possible tiesholds
1<t<2

Apply formula (7.3) with:

1 2
,q=-+l-o=-
3 3
n- ( X - 1)n- n- (X - 3)n-
p&) = - cos p&) = - cos
4 2 8 4
Equation (7.3) becomes:
1 7r (t - 1)7r - 2 n- (t - 3).
- -cos
3 4
~

2
- - -cos
3 8
~

4
+
Edge
Segmentation
and
Image Detection 273

(t - 1)T (t - 3)T
* cos - 2
= cos -*
4
*- 2 --~(t -43).
(t - l)T +

Consider first = +r + 2t - 2 = t - 3 + t = -1.

This value is outside the acceptable range, so it is a meaningless solution.
Then:
(t - 1)T - --(t - 3)T
--
5
*2t-2=-t+3*3t=5*t=-
2 4 3
This is thethreshold for minimum error. The fraction of misclassified object pixels
will be given by all those object pixels that have grey value greater than g.
We
define a new variable of integration y X - 1, to obtain:

sin
2

-
2 - 1.7 - 0.3
~-
- - = 0.075 = 7.5%
4 4

What is the minimum error threshold when object and backgroundpixels

are normally distributed?

Let us assume that the pixels that make up the object are normally distributed with
mean p, and standard deviation uo and the pixels that make up the background are
normally distributed with mean pb and the standard deviation a b :

Upon substitution into equation (7.3) we obtain:

274 Fundamentals
The Processing:
Image

This is a quadratic equation in t . It has two solutions in general, except when the
two populations have the same standard deviation. If 0, = O b , the above expression
takes the form:

This is the minimum error threshold.

What is the meaning of the two solutions of (7.6)?

When g,, # c b , the quadratic term in (7.6) does not vanish and we have two thresholds,
tl and t 2 . These turn out to be one on either side of the sharpest distribution. Let
us assume that the sharpest distribution is that of the object pixels (see Figure 7.3).
Then the correct thresholding will be to label as object pixels only those pixels with
grey value X such that tl < X < t 2 .
The meaning of the second threshold is that the flatter distribution has such a
long tail that the pixels with grey values X 2 t 2 are more likely to belong to the long
tail of the flat distribution, than to thesharper distribution.
Image Segmentation
Edge and Detection 275

Example 7.5

Derive the optimal threshold for image 7.4a and use it to threshold it.

Figure 7.4d shows the image of Figure 7.2a thresholded with the optimal threshold
method. First the two main peaks in its histogram were identified. Then a Gaus-
sian was fitted to the peak on the left and its standard deviation was chosen by
trial and error so that the best fitting was obtained. The reason we fit first the peak
on the left is because it is flatter, so it is expected to have the longest tails which
contribute to the valueof the peak of the other distribution. Once the firstpeak has
been fitted, the values of the fitting Gaussian are subtracted from the histogram. If
the result of this subtraction is negative, it is simply set to zero. Figure 7.4a shows
the full histogram. Figure 7.4b shows the histogram with the Gaussian with which
the first peak has been fitted superimposed. The mean and the standard deviation
of this Gaussian are p. = 50 and IT, = 7.5. Figure 7 . 4 ~shows the histogram that
is left after we subtract this Gaussian, with the negative numbers set to zero, and
asecond fitting Gaussiansuperimposed.The second Gaussianhas = 117 and
f f b = 7. The amplitude of the first Gaussian was A, = 20477, and of the second
Ab = 56597. W e can estimate 8, i.e. the fraction of object pixels, b y integrating
the two fitting functions:

Therefore
Aoffo
8=
Aoffo +Abffb
In our case we estimate 8 = 0.272. Substituting these values into equation (7.6)
we obtain two solutions tl = -1213 and t 2 = 74. The original image 7.2a thresh-
olded with t 2 is shown in Figure 7.4d. After thresholding, we may wish to check
how the distributions of the pixels of each class agree with the assumed distribu-
tions. Figures ".de and 7.4f show the histograms of the pixels of the object and
the background respectively, with the assumed Gaussians superimposed. One can
envisageaniterativescheme according to whichthesetwohistograms are used
to estimate new improved parameters for each class which are then used to de-
fine a new threshold, and so on. However, it is not certain that such a scheme
will converge. What is more, this result is worse than that obtained by hysteresis
thresholding with two heuristically chosen thresholds. This demonstrates how pow-
erful the combination of using criteria of spatial proximity and attribute similarity
is.
276 Image Processing:
FundamentalsThe

(a) Histogram with the opti- (b) Histogram with theGaus-

mal threshold marked sian model for the object pix-
els superimposed

(c) Histogram after subtrac- (d) Image thresholdedwith

tion of the Gaussian used to the optimal threshold
model the object pixels with
the Gaussian model for the
background pixels superim-
posed.

(e) Gaussian model used for (f) Gaussian model used for
the object pixels, andtheir the background pixels, and
real histogram their real histogram

Figure 7.4: Optimal thresholding ( t = 74).

Image Segmentation and Edge Detection 277

Example 7.6

The grey level values of the object pixels are distributed according to
the probability density function:

while the grey level values of the background pixels are distributed
according to the probability density function:

1
Pb = -exp
2ab
(-7)
1% - f i b I
If p, = 60, p(, = 40 and C, = 10 and = 5 , find the thresholds that
minimize the fraction of misclassified pixels when weknow that the
object occupies two-thirds of the area of the image.

We substitute in equation (7.3):

278 Image Processing:
FundamentalsThe

So, there are two thresholds, tl = 20 and t2 = 47.

t
l.o

0.8

0.6

0.4

0.2

0.0
X

Onlypixelswiththegrey level values between tl and t2 should be classified as

background pixels in order to minimize the error.

What are the drawbacks of the minimum error threshold method?

The method has various drawbacks. For a start, we must know the prior probabilities
for the pixels to belong to the objector the background; i.e. we must know 8.
Next, we must know the distributions of the two populations. Often it is possible to
approximate these distributions by normal distributions, but even in that case one
would have to estimate the parametersIT and p of each distribution.

Is there any method that does not depend on the availability of models
for the distributions of the object and the background pixels?

A method which does not depend on modelling the probability density functions has
been developed by Otsu. Unlike the previous analysis,this method hasbeen developed
directly in the discrete domain.
Consider that we have an image with L grey levels in total and its normalized
histogram, so that for each grey level value X , p , represents the frequency with which
the particular value arises. Then suppose that we set the threshold at t. Let us
assume that we are dealing with the case of a bright object on a dark background.
Edge
Segmentation
and
Image Detection 279

The fraction of pixels that will be classified as background ones will be:

The mean grey level value of the background pixels and the objectpixels respectively
will be:

(7.10)

(7.11)

(7.12)

Similarly, we may define the variance of each of the two populations created by the
choice of a threshold t as:

Let us consider next the total variance of the distribution of the pixels in the
image:
L

x=1

We may split this sum into two:

c t
c
L

4= x=1 ( X - d 2 P X + x=t+1 (X - p)2px

 280 Image Processing:
FundamentalsThe

As we would like eventually to involve the statistics defined for the two populations,
we add and subtract inside each sum the corresponding mean:
t
c$ = c(Z-jlb+pb -pl2Px
x=1
+ c
L
(X - P O + P O - p )
2
PZ
t t t

x=1 x=1
L L L

x=t+l x=t+l z=t+l

Next we substitute thetwo sums onthe left of each line in termsof c; and c,”using
equations (7.13). We also notice that the two sums in the middle of each line can be
expressed in terms of equations (7.8) and (7.9), since p, and p. are constants and
do not depend on thesumming variable X:

a% = e(t)c,”+ (pb - p)’e(t) + 2(pb - p ) c(X

t

x=1
-pb)Px

+ (PO- p)’e(t)
+ ( l - e(t))c,” + 2(po - p) cL

x=t+l
(X - P b ) P z

The two terms with the sums arezero, since, for example:
t t t

x=l
x=1 x=1

Then by rearranging the remaining terms:

c$ = + (1- W O , ” +(Pb - p)”(t) + (p0 - PI2(1- O(t))
“ “
terms depending on the variance terms depending on the variance
within each class between the two classes
a&@) + a $ ( t ) (7.14)

where &(t) is defined to be thewithin-class variance and u$(t) is defined to be the

between-class variance. Clearly c$ is a constant. We want to specify t so that &(t)
is as small as possible, i.e. the classes that arecreated are ascompact as possible, and
&(t) is as large as possible. Suppose that we choose to work with a$@),i.e. choose
t so that it maximizes a$(t). We substitute in the definition of &(t) the expression
for f i b and p. as given by equations (7.10) and (7.11) respectively:

d(t) = b b - p>2ee(t> + (PO- - @(t))

Image Segmentation
Edge and Detection 281

(7.15)

This function expresses the interclass variance g;(t) in terms of the mean grey value
of the image p, and quantities that can be computed once we know the values of the
image histogram up to the chosen threshold t.
The idea is then to start from the beginning of the histogram and test each grey
level value for the possibility of being the threshold that maximizes a i ( t ) ,by cal-
t
culating the values of p(t) = Cz=l xpz and @(t)= C zt = l p z and substituting into
equation (7.15). We stop testing once the value of IS; starts decreasing. This way
we identify t for which &(t) becomes maximal. This method tacitly assumes that
function a;(t) is well-behaved; i.e. that it has only one maximum.

Example 7.7

Calculate Otsu's threshold for the image of Figure 7.2a and use it to
threshold the image.

Figure 7.5a shows how a;(t) varies as t scans all possible grey values. The first
maximum of this function is at t = 84 and we choose this threshold to produce
theresultshown in Figure 7.5b. W e can see thattheresultisnotnoticeably
different from the result obtained with the empirical threshold (Figure 7.512)and
a little worse than the optimal threshold result (Figure 7.4d). It is much worse
than the result obtained by hysteresis thresholding, reinforcing again the conclu-
sion that spatial and grey level characteristics used in thresholding is a powerful
combination.
282 Image Processing: The Fundamentals

-1w 1
100 200
t

(a) c r ~ ( tversus
) t. (b) Imagethresholded
with
Otsu’s
threshold.

I Figure 7.5: Otsu’s thresholding (t = 84).

Are there any drawbacks to Otsu’s method?

Yes, a few:
1. Although the method does not make any assumption about the probability
density functions po(x) and pb(x), it describes them by using only their means
and variances. Thus it tacitly assumes that these two statistics are sufficient in
representing them. This may not be true.
2. The method breaks down when the two populations are very unequal. When
the two populations become very different in size from each other, a g ( t ) may
have two maxima and actually the correct maximum is not necessarily the global
maximum. That is why in practice the correct maximum is selected from among
all maxima of a i ( t )by checking that the value of the histogram at the selected
threshold, p i , is actually a valley (i.e. pt < p p 0 ,pt < p p b )and only if this is true
should t be accepted as the best threshold.
3. The method,as presentedabove,assumes that the histogram of the image
is bimodal; i.e. the imagecontainstwo classes. For more than two classes
present in the image, the method has to bemodified so that multiple thresholds
are defined which maximize the interclass variance and minimize the intraclass
variance.
4. The method will divide the image into two classes even if this division does not
make sense. A case when the method should not be directly applied is that of
variable illumination.
 Image Segmentation and Edge Detection 283

How can we threshold images obtained under variable illumination?

In the chapter on image enhancement, we saw that animage is essentially the product
of a reflectance function r ( z ,y) which is intrinsic to the viewed surfaces, and an
illumination function i(z,y):

f(z,9) = r ( z ,y>i(zCY)
,
Thus any spatial variation of the illumination results in a multiplicative interfer-
ence to the reflectance function that is recorded during the imaging process. We can
convert the multiplicative interference into additive, ifwe take the logarithm of
the image:

In f(z,y) = lnr(z, y) + In i ( z ,y) (7.16)

Then instead of forming the histogram of f(z,y), we can form the histogram of
1n f (2, Y).

If we threshold the image according to the histogram of l n f ( z , y), are

we thresholding it according to the reflectance properties of the imaged
surfaces?

The question really is, what the histogram of In f(z,y) is in terms of the histograms
of lnr(z, y) and lni(z, y). For example, if l n f ( z , y) is the sum of lnr(z, y) and
In i ( z ,y) which may be reasonably separated functions apart from some overlap, then
by thresholding In f(z,y) we may be able to identify the lnr(z,y) component; i.e. the
component of interest.
Let us define some new variables:

4 2 ,Y) = In f(2,Y)
F(z,y) lnr(z, y)
i ( z ,y) Ini(z, y)

Therefore, equation (7.16) can be written as:

If f(z,y), r ( z , y) and i(z,y) are thought of as random variables, then z ( z , y), F(%, y)
and i(z,y) are also random variables. So, the question can be rephrased into: What
is the histogram of the sum of two random variables in terms of the his-
tograms of the two variables? A histogram can be thought of as a probability
density function. Rephrasing the question again, we have: What is the probability
density function of the sum of two random variables interms of the proba-
bility density functions of the two variables? We have seen that theprobability
density function of a random variable is the derivative of the distribution function of
the variable. So, we can rephrase the question again: What is the distribution
284 Image Processing: The Fundamentals

function of the sum of two random variables in terms of the probability

density functions or the distribution functions of the two variables? In the
(5, r") space, equation (7.17) represents a line for a given value of z. By definition, we
know that:

Distribution function of z = PZ(u)= Probability of z 5U P(z 5 U)

Figure 7.6: z is less than U in the shadowed half plane.

The line r" + = divides the (i

U F),planeinto twohalf planes, one in which
z > U and one where < U . The probability of z < U is equal to the integral of
z
the probability density function of pairs (5, r") over the area of the half plane in which
z < U (see Figure 7.6):

Pz(U) =
+m

2=-m l=-m U-i

p& i)dr"dl

where p,;(r", l)is the joint probability density function of the two random variables r"
and 5.
To find the probability density function of z , we differentiate PZ(u)with respect
to U , using Leibnitz's rule (see Box B7.1),applied twice, once with

f (X;X) + S"'
r=-m
p,;(r", i)dr"

b(X) + +m
.(X) + -m

and once more when we need to differentiate f (z; X) which itself is an integral that
 Image Segmentation
Edge and Detection 285

depends on parameter U with respect to which we differentiate:

X
I X

= /m P,;(u - i, i)di (7.18)

2=-00

The two random variables r" and 5, one associated with the imaged surface and
one with the source of illumination, are independent, and therefore their joint proba-
bility density function can be written as the product of their two probability density
functions:

p,;(r",S) = P,(r")&(i)

Upon substitution in (7.18) we obtain:

(7.19)
J -00

which shows that the histogram (= probability density function) of z is equal to the
convolution of the two histograms of the two random variables r" and 5.
If the illumination is uniform, then:

+
I I

i(z,y) = constant i = lni(z, y) = i, = constant

Then &(l) = S(; - i,), and after substitution in (7.19) and integration we obtain
P2 ( U ) = P?(U).
That is, under uniform illumination, the histogram of the reflectance function
(intrinsic to the object) is essentially unaffected. If, however, the illumination is not
uniform then, even if we had a perfectly distinguishable object, thehistogram is badly
distorted and the various thresholding methods break down.

Sincestraightforward thresholding methods break downundervariable

illumination, how can we cope with it?

There aretwo ways in which we can circumvent the problem of variable illumination:

1. Divide the image into more or less uniformly illuminated patches and histogram
and threshold each patch as if it were a separate image. Some adjustment may
be needed when the patches are put together as the threshold essentially will
jump from one value in one patch to another value in a neighbouring patch.
2. Obtain an image of just the illumination field, using the image of a surface
with uniform reflectance and divide the image f ( z ,y) by i(z,y); i.e. essentially
subtract the illumination component ;(X, y) from z ( z ,y). Then multiply
with a reference value, say i ( O , O ) , to bring the whole image under the same
illumination and proceed using the corrected image.
286 Image Processing: The Fundamentals

Example 7.8

Threshold the image of Figure 7.7a.

This image exhibitsan illumination variationfrom left to right. Figure 7.7b shows
the histogram of theimage.UsingOtsu’smethod, we identify threshold t = 75.
The result of thresholding the image with this threshold is shown in Figure 7 . 7 ~ .
The result of dividing the imageinto four subimages from left to rightand applying
Otsu’s method to each subimage separately is shown in Figure 7.7d.

l
3000

1I (a) Original image (b) Histogram

(c) Global thresholding

(d) Local thresholding

Figure 7.7: Global versus local thresholding for an image with variabh
illumination.
Edge
Segmentation
and
Image Detection 287

Are there any shortcomings of the thresholding methods?

With the exception of hysteresis thresholding whichis of limited use, the spatial
proximity of the pixels in the image is not considered at all in the segmentation
process. Instead, only the grey level values of the pixels are used.
For example, consider the two images in Figure 7.8.

Figure 7.8: Two very different images with identical histograms.

Clearly, the first image is the image of a uniform region, while the second image
contains two quite distinct regions. Even so, both images have identical histograms,
shown in Figure 7.9. Their histograms are bimodal and we can easily choose a thresh-
old. However, ifwe use it to segment the first image, we shall get nonsense.

40000 L
40000 T
30000 -

-
f 20000

5
10000 -

L
0-5.0 50 150 250

(a) Histogram of image in Figure 7.8a (b) Histogram of image in Figure 7.8b

Figure 7.9: The two identical histograms

of the very different images shown
in Figure 7.8.
288 Fundamentals
The Processing:
Image

Howcanwe cope with images that contain regions that are not uniform
but they are perceived as uniform?

Regions that are not uniform in terms of the grey values of their pixels but are
perceived as uniform, are called texturedregions. For segmentation purposes then,
each pixel cannot only be characterized by its grey level value but also by another
number or numbers which quantify the variation of the grey values in a small patch
around that pixel. The point is that the problem posed by the segmentation of
textured images can be solved by using more than one attribute to segment the
image. We can envisage that each pixel is characterized not by one number but by
a vector of numbers, each component of the vector measuring something at the pixel
position. Then each pixel is represented by a pointinamultidimensionalspace,
where we measure one such number, a feature, along each axis. Pixels belonging to
the same region will have similar or identical values in their attributes and thus will
cluster together. The problem then becomes one of identifying clusters of pixels in a
multidimensional space. Essentially it is similar to histogramming only now we deal
with multidimensional histograms. There are several clustering methods that may be
used but they are in the realm of Pattern Recognition and thus beyond the scope of
this book.

Are there any segmentation methods that takeintoconsideration the

spatial proximity of pixels?

Yes, they are called regiongrowing methods. In general, one starts from some seed
pixels and attaches neighbouring pixels to them provided the attributes of the pixels
in the region created in this way vary within a predefined range. So, each seed grows
gradually by accumulating more and more neighbouring pixels until all pixels in the
image have been assigned to a region.

How can one choose the seed pixels?

There is no clear answer to this question, and this is the most important drawback
of this type of method. In some applications the choice of seeds is easy. For example,
in target tracking in infrared images, the target will appear bright, and one can use
as seeds the few brightest pixels. A method which does not need a predetermined
number of regions or seeds is that of split and merge.

How does the split and merge method work?

Initially the whole image is considered as one region. If the range of attributes within
this region is greater than a predetermined value, then the region is split into four
quadrants and each quadrant is tested in the same way until every square region
created in thisway contains pixels with range of attributes within the given value. At
the end all adjacent regions with attributes within the same range may be merged.
Edge
Segmentation
and
Image Detection 289

An example is shown in Figure 7.10 where for simplicity a binary 8 X 8 image

is considered. The tree structure shows the successive splitting of the image into
quadrants. Such a tree is called quadtree.

A F R

D O U H L C

Figure 7.10: Image segmentation by splitting.

We end up having the following regions:

(AFIE)(FBGI)(GKNJ)(JNMI)(MNLH)(KCLN)
(RSXV)(SMTX)(THUX)(XUOV)(QIMR)(EQRP)(PROD)
i.e all the children of the quadtree. Any two adjacent regions then are checked for
merging and eventually only the two main regions of irregular shape emerge. The
above quadtree structure is clearly favoured when the image is square with N = 2n
pixels in each side.
Split and merge algorithms often start at some intermediate level of the quadtree
(i.e some blocks of size 2' X 2' where 1 < n) and check each block for further splitting
into four square sub-blocks and any two adjacent blocksfor merging. At the end
again we check for merging any two adjacent regions.

Is it possible to segment an image by considering the dissimilarities be-

tween regions, as opposed to considering the similarities between pixels?

Yes, in such an approach we examine the differences between neighbouring pixels and
say that pixels with different attribute values belong to different regions and therefore
we postulate a boundary separating them. Such a boundary is called an edge and the
process is called edge detection.

How do we measure the dissimilarity between neighbouring pixels?

We may slide a window across the image and at each position calculate the statistical
properties of the pixels within each half of the window and compare the two results.
290 Fundamentals
The Processing:
Image

The places where these statistical properties differ most are where the boundaries of
the regions are.
x x x x x x x x x

X I X X XIXI X X X I X
x x x x x x x x x
x x x x x x x x x
x x x x x x x x x
x x x x x x x x x

Figure 7.11: Measuring the dissimilarity between two image regions using
a sliding widow.

For example, consider the 8 X 8 image in Figure 7.11. Each X represents a pixel.
The rectangle drawn is a 3 X 7 window which could be placed so that its centre 0
coincides with every pixel in the image, apart from those too close to the edge of the
image. We can calculate the statistical properties of the nine pixels on the left of the
window (part A) and those of the nine pixels on the right of the window (part B)
and assign their difference to pixel 0. For example, we may calculate the standard
deviation of the grey values of the pixels within each half of the window, say ITA
and I T B ,calculate the standard deviation of the pixels inside the whole window, say
IT, and assign the value E E IT - ITA - ITB to the central pixel. We can slide this
window horizontally to scan the whole image. Local maxima of the assigned values
are candidate positions for vertical boundaries. Local maxima where the value of E is
greater than a certain threshold are accepted as vertical boundaries between adjacent
regions. We can repeat the processby rotating the windowby 90” and sliding it
vertically to scan the whole image again. Clearly the size of the window here plays a
crucial role as we need a large enough window to calculate the statistics properly and
a small enough window to include within each half only part of a single region and
avoid contamination from neighbouring regions.

What is the smallest possible window we can choose?

The smallest possible windowwe can choose consists of two adjacent pixels. The only
“statistic” we can calculate from such a window is the difference of the grey values of
the two pixels. When this difference is high we say we have an edge passing between
the two pixels. Of course, the difference of the grey values of the two pixels is not a
statistic but is rather an estimateof the first derivative of the intensity function with
respect to the spatial variable along the direction of which we take the difference.
Image Segmentation
Edge and Detection 291

This is because first derivatives are approximated by first differences in the discrete
case:
Af,c = f ( i + 1 , j ) - f ( i , j )
Afy = f ( i , j+ 1) - f ( i , j )
Calculating A fz at each ixel osition is equivalent to convolving the image with a
mask (filter) of the form I i n the X direction, and calculating A f y is equivalent

to convolving the image with the filter B in the y direction.

The first and the simplest edge detection scheme then is to convolve the image
with these two masks and produce two outputs. Note that these small masks have
even lengths so their centres are not associated with any particular pixel in the image
as theyslide across the image. So, the outputof each calculation should be assigned to
the position in between the two adjacent pixels. These positions are said to constitute
the dual grid of the image grid. In practice, we seldom invoke the dualgrid. We usually
adopt a convention and try to be consistent. For example, we may always assign the
output value to the first pixel of the mask. If necessary, we later may remember that
this value actually measures the difference between the two adjacent pixels at the
position half an interpixel distance to the left or the bottom of the pixel to which it
is assigned. So, with this understanding, and for simplicity from now on, we shall be
talking about edge pixels.
In the first output, produced by convolution with mask , -1 any pixel that
has an absolute value larger than values of its left and right neighbours is a candidate

mask B
pixel to be a vertical edge pixel. In the second output, produced by convolution with
, any pixel that has an absolute value larger than thevalues of its top and
bottom neighbours is a candidate pixel to be a horizontal edge pixel. The process of
identifying the local maxima as candidate edge pixels (=edgels) is called non-maxima
suppression.
In the case of zero noise this scheme will clearly pick up the discontinuities in
intensity.

What happens when the image has noise?

In the presence of noise every small and irrelevant fluctuation in the intensity value
will be amplified by differentiating the image. It iscommon sense then that one

Figure 7.12 shows an original image and the output obtained if the m
should smooth the image first with a lowpass filter and then find the local differences.
and

B convolution filters are applied along the horizontal and vertical directions
respectively. The outputs of the two convolutions are squared, added and square
rooted to produce the gradient magnitude associated with each pixel. Figure 7.13
shows the results obtained using these minimal convolution filters and some more
sophisticated filters that take consideration the noise present in the image.
292 Fundamentals
The Processing:
Image

Figure 7.12: (a) Original image. (b) The image of the gradient magnitudes
computed by simpledifferencingwithoutapplyingany smoothing. For
displaying purposes the gradient image has been subjected to histogram
equalization.

Letus consider for example a l-dimensional signal. Suppose that one uses as
lowpass filter a simple averaging procedure. We smooth the signal by replacing each
intensity value with the average of three successive intensity values:

(7.20)

Then we estimate thederivative at position i by averaging the two differences between

the value at position i and its left and right neighbours:
F. = (&+l - Ai)
a -
+ (Ai - Ai-l) -
-
Ai+l - Ai-l
(7.21)
2 2
If we substitute from (7.20) into (7.21), we obtain:
1
Fi = :[Ii+2 + Ii+l - Ii-1 - Ii-21 (7.22)

It is obvious from this example that one can combine the two linear operations of
smoothing and finding differences into one operation if one uses large enough masks.
In this case, the first difference at each position could be estimated by using a mask
I 1
like - B I - B
0 I f l - ;I . It is clear that the larger the mask used, the better
Image Segmentation and Edge Detection 293

Figure 7.13: (a) Result obtained by simply thresholding the gradient values
obtained without any smoothing. (b) The same as in (a) but using a higher
threshold, so some noise is removed. However, useful image information
has also been removed. (c) Result obtained by smoothing first along one
direction and differentiating afterwards along the orthogonal direction,
using a Sobel mask. (d) Result obtained using the optimal filter of size
7 X 7. In all cases the same value of threshold was used.
294 Image Processing:
FundamentalsThe

is the smoothing. But it is also clear that the more blurred the edge becomes so the
more inaccurately its position will be specified (see Figure 7.14).
For an image whichis a 2D signal one should use 2-dimensional masks. The
smallest mask that one should use to combine minimum smoothing with differencing
is a 3 X 3 mask. In this case we also have the option to smooth in one direction
and take the difference along the other. This implies that the 2D mask may be the
result of applying in a cascaded way first a 3 X 1 smoothing mask and then a 1 X 3
differencing masks, or vice versa. In general, however, a 2D 3 X 3 mask will have the
form:

How can we choose the weights of a 3 X 3 mask for edge detection?

We are going to use one such mask to calculate Afz and another to calculate Af,.
Such masks must obey the following conditions:
1. The mask which calculates A f z must be produced from the mask that calculates
Af, by rotation by 90”. Letus consider from now on the mask whichwill
produce A f, only. The calculated value will be assigned to the central pixel.
2. We do not want to give any extra weight to the left or the right neighbours of
the central pixel, so we must have identical weights in the left and right columns.
The 3 X 3 mask therefore must have the form:

3. Let us say that we want to subtract the signal “in front” of the central pixel
from the signal “behind” it, in order to find local differences, and we want these
two subtracted signals to have equal weights. The 3 X 3 mask therefore must
have the form:

4. If the image is absolutely smooth we want to have zero response. So the sum of
all the weights must be zero. Therefore, a22 = - 2 ~ 2 1 :

I -all I -a12 I -all I

Image Segmentation and Edge Detection 295

(a) A noisy signal (b) First derivative of (a) by

differencing

(c) Signal smoothed by aver- (d) First derivative of (c) by

aging over every 3 samples differencing

(e) Signal smoothed by aver- (f) First derivative of (e) by

aging over every 10 samples differencing

Figure 7.14: The left column shows a noisy signal and two smoothed ver-
sions of it. On the right is the result of estimating the first derivative of
the signal by simple differencing. The edge manifests itself with a sharp
peak. However, the more the noise is left inthe signal, the more secondary
peaks will be present, and the more the smoothing, the more blunt the
main peak becomes.
296 Image Processing: The Fundamentals

5 . In the case of a smooth signal,and aswe differentiate in the direction of columns,

we expect each column t o produce 0 output. Therefore, a21 = 0:

We can divide these weights throughout by all so that finally, this mask depends
only on one parameter:

(7.23)

What is the best value of parameter K?

It can be shown that the orientations of edges which are almost aligned with the
image axes are not affected by the differentiation, if we choose K = 2. We have then
the Sobel masks for differentiating an image along two directions:

Note that we have changed convention for the second mask and subtract thevalues
of the pixels “behind” the central pixel from the values of the pixels “in front”. This
is intentional so that the orientation of the calculated edge, computed by using the
components of the gradient vector derived using these masks, is measured from the
horizontal axis anticlockwise (see Box B7.2).

B7.2: Derivation of the S o b e l m a s k s

Edges are characterized by their strength and orientation defined by:

Strength = E ( i , j ) = 4 [ A f Z ( i , j ) +
l 2[Af,(i,j)12

The idea is to try tospecify parameter K of mask (7.23) so that the outputof the
operator is as faithful as possible to the truevalues of E and a which correspond
to the non-discretized image:
Image Segmentation
Edge and Detection 297

Consider a straight step edge in the scene, passing through the middle of a pixel.
Each pixel is assumed to be a tile of size l X 1. Suppose that the edgehas
orientation 8 and suppose that 8 is small enough so that the edge cuts lines AB
and CD as opposed to cutting lines AC and BD (0 5 8 5 tanp1(+)) (see Figure
7.15).
i- A F G B

I
L

M N

C D

Figure 7.15: Zooming into a 3 X 3 patch of an image around pixel ( i , j ) .

Also, assume that on the left of the edge we have a dark region with grey
value G1 and on the right a bright region with grey value GZ. Then clearly the
pixel values inside the mask are:

f ( i - 1 , j- 1) = f ( i - 1,j)= f ( i - 1 , j+ 1) = G1
+
f ( i 1 , j- l) f ( i + 1,j) f(i + 1 , j+ l) = G2
The pixels in the central column have mixed values. If we assume that each pixel
is like a tile with dimensions l X l and denote the area of a polygon by the name
of the polygon inside brackets, then pixel A B I L will have value:
298 Processing:
Image The Fundamentals

1 1
f ( i , j - l ) = G l ( A F J L ) + G Z ( F B I J )= G1[T-(FGHJ)]+G2[T+(FGHJ)](7.24)

We must find the area of trapezium F G H J . From the triangles OJH and OFG
we have:
1 3
JH = - t a n 0 , FG = - t a n 0
2 2
Therefore, ( F G J H ) = i ( J H + FG) = tan@, and by substitution into equation
(7.24) we obtain:

By symmetry:
1 1
f ( i , j + 1) = G2(-2 - t a n @ )+ G1(-2 + tano)

Clearly

Let us see now what mask (7.23) will calculate in this case:

AfZ = f ( i + 1 , j+ l) + f ( i + 1 , j- l) + K f ( i + 1,j)
- [f(i - 1,j + 1)+ f ( i - 1 , j- 1)+ K f ( i - l , j ) ]
= (G2 + G2 + KG2) (G1 + G1 + KG1) = (G2 G1)(2 + K )
- -

Af, = - [ f ( i 1 , j+ 1)+ f ( i + 1 , j+ 1)+ K f ( i , j + l)]

-

+ f ( i - 1 , j- l) + f ( i + 1 , j- l) + K f ( i ,j - l)

= -G1 - G 2 - K G 2

+KG1 (i - tang)
G+ (k1 (a 1
--tan@

KG2
- KG1

+tang)
- + t a n @ + G 1+ G 2

1
= K(G2 - G1) (-2 +tan0 +
= 2K(G2 - G1)t a n 0

The magnitude of the edge will then be

E = d ( G 2 - G1)'(2 + K ) 2+ (2K)'(G2 - G1)2tan2@

 Image Segmentation and Edge Detection 299

and the orientation of the edge:

tana: = 4f,
~
-
-
2K tan d
Afz 2+K
Note that if we choose K = 2:

tan a: = tan 8

i.e. the calculated orientation of the edge will be equal to the true orientation.
One can perform a similar calculation for the case tan-' 5 8 5 45'. In that
case we have distortion in the orientation of the edge.

Example 7.9

Write down the formula which expresses the output O ( i ,j ) at position

( i , j ) of the convolution of the image with the Sobel mask that differ-
entiates along the i axis, as a function of the input values I ( i ,j ) .
(Note: Ignore boundary effects, i.e. assume that ( i ,j ) is sufficiently far
from the image border.)

O ( i ,j ) = - I ( i - 1 , j- 1) - 2 I ( i - 1,j)- I ( i - 1 , j+ 1)
+ I ( i + 1,j - 1) + 2I(i + 1,j)+ I ( i + 1,j+ 1)

Example 7.10

You have a 3 X 3 image which can be represented by a 9 X 1 vector.

Construct a 9 X 9 matrix which, when it operates on the image vector,
will produce another vector each element of which is the estimate of
the gradient component of the image along the i axis. (To deal with
the boundary pixels assume that the image is repeated periodically in
all directions.)

1 Consider a 3 X 3 image:
300 Image Processing: The Fundamentals

f31 f32 f33

Periodic repetition of this image implies that we have:

f33 f31 f32 f33 f31

f12 fll

flz
f33
f13
f21
f31
(fll
fll
f22
f32
f12
f23
f33
f13)
f13
f21
f31
fll

Using the result of Example 7.9, we notice that the first derivative at position
(1,l) is given by:

f32 + 2f12 + f22 - f33 - 2f13 - f23

The vector representation of the image is:

fll
f2l
f31
f12
f22
f32
f13
f23
f33

If we operate with a 9 X 9 matrix on this image, we notice that in order to get the
desired output, the first row of the matrix should be:

0 0 0 2 1 1 - 2 - 1 - 1
The derivative at position (1,2), i.e. at pixel with value f21, is given by:

f12 + 2f22 + f32 - f13 - 2f23 - f33

Therefore, the second row of the 9 X 9 matrix should be:

0 0 0 1 2 1 - 1 - 2 - 1
The derivative at position (1,3)is given by:

f22 + 2f32 + f12 - f23 - 2f33 - f13

Edge
Segmentation
and
Image Detection 301

Therefore, the third row of the 9 X 9 matrix should be:

0 0 0 1 1 2 -1
-1 -2

Reasoning this way, we conclude that the matrix we require must be:
( 0 0 0 2 1 1 -2 -1 -1
0 0 0 1 2 1 - 1 - 2 - 1
0 0 0 1 1 2 -1-1-2
-2-1-1 0 0 0 2 1 1
-1-2-1 0 0 0 1 2 1
-1-1-2 0 0 0 1 1 2
2 1 1 - 2 - 1 - 1 0 0 0
1 2 1 - 1 - 2 - 1 0 0 0
\ l 1 2 -1-1 -2 0 0 0

Example 7.11

Using the matrix derived in Example7.10, calculate the first derivative

along the i axis of the following image:

(; ; :)
3 1 0

What is the component of the gradient along the i axis at the centre
of the image?

/o 0 0 2 1 1 -2 -1 -1 13 ( 4
0 0 0 1 2 1 - 1 - 2 - 1 3 4
0 0 0 1 1 2 -1-1-2 3 4
-2-1-1 0 0 0 2 1 1 1 - 12
-1-2-1 0 0 0 1 2 1 1 - 12
-1-1-2 0 0 0 1 1 2 1 - 12
2 1 1 -2-1-1 0 0 0 0 8
1 2 1 - 1 - 2 - 1 0 0 0 0 8
\ l 1 2 -1-1 -2 0 0 0 \O \ 8
At the centre of the image the component of the gradient along the i axis is -12.
302 Image Processing: The Fundamentals

Figure 7.16: (a) Original image. (b) The gradient magnitude “image”: the
brighter the pixel, the higher the value of the gradient of the image at that
point. (c) After non-maxima suppression: only pixelswith locally maximal
values of the gradient magnitude retain their values. The gradient values
of all other pixels are set to zero. (d) The edge map: the gradient values in
(c) are thresholded: all pixels with values above the threshold are labelled
0 (black) and all pixels with values below the threshold are labelled 255
(white).
Image Segmentation
Edge and Detection 303

In the general case, how do we decide whether a pixel is an edge pixel or

not?

Edges are positions in the image where the image function changes. As the image
is a 2D function, to find these positions we have to calculate the gradient of the
function Vf(z,y). The gradient of a 2D function is a 2D vector with components
the partial derivatives of the function along two orthogonal directions. In thediscrete
case these partialderivatives are the partial differences computed along two orthogonal
directions by using masks like, for example, the Sobel masks. If we convolve an image
with these masks, we have a gradient vector associated with each pixel. Edges are
the places where the magnitude of the gradient vector is a local maximum along the
direction of the gradient vector (i.e. the orientation of the gradient at that pixel
position). For this purpose, the local value of the gradient magnitude will have to be
compared with the values of the gradient estimated along this orientation and at unit
distance on either side away from the pixel. In general, these gradient values will not
be known because they will happen to be at positions in between the pixels. Then,
either a local surface is fitted to thegradient values and used for the estimation of the
gradient magnitude at any interpixel position required, or the value of the gradient
magnitude is calculated by interpolating the values of the gradient magnitudes at the
known integer positions.
After this process of non-maxzma suppression takes place, the values of the gradi-
ent vectors that remain are thresholded and only pixels with gradient values above the
threshold are considered as edge pixels identified in the edge map (see Figure 7.16).

Example 7.12

The following image is given

Use the followingmasks to estimate the magnitude and orientation

of the localgradient at all pixel positions of the image except the
boundary pixels:
304 Image Processing: The Fundamentals

l I
1-11
I
01 11I

Then indicate which pixels represent horizontal edge pixels and which
represent vertical.

If weconvolve the image with the first mask, weshall have an estimate of the
gradient component along the X (horizontal) axis:

If we convolve the image with the second mask, we shall have an estimate of the
gradient component along the y (vertical) axis:

The gradient magnitude is given by: IGI = d ( A I z ) 2+ ( A I , ) z

(7.25)

The gradient orientation is given by:

Image Segmentation
Edge and Detection 305

W e k n o w t h a t f o r a nangle 8 to be in the range 0" t o 45" (0" 5 181 5 45") its

tangent must satisfy 0 5 I tan81 5 1. Also, a n angle 8 is in the range 45" t o 90"
(45" 181 5 90") if 1 I tan81 5 m.
As we want to quantize the gradient orientations to vertical and horizontal
ones only, we shall make all orientations 0" 5 181 5 45" horizontal, i.e. set them
t o 0, and all orientations with 45" < 181 5 90" vertical, i.e. we shall set them to
90".
B y inspectingtheorientationarray above weinferthefollowinggradient
orientations:

A pixel is a horizontal edge pixel if the magnitude of its gradient is a local

maximum when comparedwithitsverticalneighboursanditsorientationis 0.
A pixel is a vertical edge if its orientation is 90" and its magnitude is a local
m a x i m u m in the horizontal direction.
B y inspecting the gradient magnitudesin (7.25)and the quantized orientations
we derived, we identify the following horizontal (-) edges only:
306 Processing:
Image The Fundamentals

Example 7.13

Arethe masks used in Example 7.12 separable? How can we take

advantage of the separability of a 2D mask to reduce the computational
cost of convolution?

The masks used in Example 7.12 areseparable because they can be implemented

m
as a sequence of two 1D filters applied in a cascaded way one after the other:

-1
H followed
by

m j followed by
1
Any 2D N X N separable mask can be implemented as a cascade of two 1D masks
of size N . Thisimplementationreplaces N 2 multiplicationsandadditionsper
pixel by 2N such operations per pixel.

Are Sobel masks appropriate for all images?

Sobel masks are appropriate for images with low levels of noise. They are inadequate
for noisyimages. See forexample Figure 7.17 which showsthe results of edge detection
using Sobel masks for a very noisy image.

How can we choose the weights of the mask if we need a larger mask owing
to the presence of significant noise in the image?

We shall consider the problem of detecting abrupt changes in the value of a 1-

dimensional signal, like the one shown in Figure 7.18.
Let us assume that the feature we want to detect is U(.) and it is immersed in
additive white Gaussian noise n(x). The mask we want to usefor edge detection
should have certain desirable characteristics, called Canny’s criteria:
Image Segmentation and Edge Detection 307

Figure 7.17: Trying to detect edges in a blurred and noisy image using
the Sobel edge detector may be very tricky. One can play with different
threshold values but the result is not satisfatory. The three results shown
have been obtained by using different thresholds within the same edge
detection framework, and they were the best among many others obtained
for different threshold values.
308 Image Processing: The Fundamentals

l signal

0 X

Figure 7.18: A noisy 1D edge.

1. Good signal to noise ratio.

2. Good locality; i.e. the edge should be detected where it actually is.
3. Small number of false alarms; i.e. the maxima of the filter response should be
mainly due to the presence of the true edges in the image rather than tonoise.
Canny showed that a filter function f(x) has maximal signal to noise ratio if it is
chosen in such a way that it maximizes the quantity:

(7.26)

where no is the standarddeviation of the additive Gaussiannoise. As this is a constant

for a particular image, when we try to decide what function f(x) should be, we omit
it from the computation. So, we try to maximize the quantity:

Canny also showed that thefilter function f(x) detects an edge with the minimum
deviation from its true location if it is chosen in such a way that its first and second
derivatives maximize the quantity:

(7.27)

Finally, he showed that the output of the convolution of the signal with the filter
f(x) will contain minimal number of false responses if function f(x) is chosen in such
Edge
Segmentation
and
Image Detection 309

a way that its first and second derivatives maximize the quantity:

(7.28)

One, therefore, can design an optimal edge enhancing filter by trying to maximize
the above three quantities:

Canny combined the first two into one performance measure P , which he maxi-
mized under the constraint that C = const. He derived a convolution filter that way,
which, however, he did not use because he noticed that it could be approximated by
the derivative of a Gaussian. So he adopted the derivative of a Gaussian as a good
enough edge enhancing filter.
Alternatively, one may combine all three quantities S , L, and C into a single
performance measure P:

and try tochoose f(.) so that P is maximum. The free parameters which will appear
in the functional expression of f(z)can be calculated from the boundary conditions
imposed on f(z)and by substitution into the expression for P and selection of their
numerical values so that P is maximum.
It must be stressed that the filters defined in this way are appropriate for the
detection of antisymmetric features; i.e step or ramp edges when the noise in
the signal is additive, white and Gaussian.

Can we use the optimal filters for edges to detect lines in an image in an
optimal way?

No. Edge detection filters respond badly to lines. The theory for optimal edge detector
filters has been developed under the assumption that there is a single isolated step
edge. We can see that from the limits used in the integrals in equation (7.26), for
example: the limits are from --CO to +-CO,assuming that in the whole infinite length
of the signal there is only a single step edge. If we have a line, its profile looks like
the one shown in Figure 7.19.
It looks like two step edges back to back. The responses of the filter to the two
step edges interfere and the result is not satisfactory: the two steps may or may not
be detected. If they are detected, they tend to be detected as two edges noticeably
misplaced from their true position and shifted away from one another.
Apart from this, there is another more fundamental difference between step edges
and lines.
Image 310 Processing:
FundamentalsThe

1 grey value

I X

Figure 7.19: The profile of a line.

What is the fundamental difference between step edges and lines?

A step edge is scale invariant: it looks the same whether we stretch it or shrink it.
A line has an intrinsic length-scale: it is its width. From the moment the feature we
wish to detect has a characteristic “size”, the size of the filter we must use becomes
important. Similar considerations apply also if one wishes to develop appropriate
filters for ramp edges as opposed to step edges: the length over which the ramp rises
(or drops) is characteristic of the feature and it cannot be ignored when designing
the optimal filter. Petrou and Kittler used the criterion expressed by equation (7.29),
appropriately modified, to develop optimal filters for the detection of ramp edges of
various slopes. Figure 7.20 shows attempts to detect edges in the image of Figure
7.17a, using the wrong filter; the edges are blurred, so they actually have ramp-like
profiles and the filter used is the optimal filter for step edges. Figure 7.21 shows the
results of using the optimal filter for ramps on the same image, and the effect the size
of the filter has on the results. Figure 7.22 shows an example of a relatively “clean”
and “unblurred” image for which one does not need to worry too much about which
filter one uses for edge detection.
Petrou also modified Canny’s criteria for good edge filters for the case of lines and
used them to derive optimal filters for line detection that depend on the width and
sharpness of the line.

Example 7.14(B)

Assume that U(.) is defined for positive and negative values of II: and
that wewant to enhanceafeatureof U(.) at position X = 0. Use
equation (7.26) to show that if the feature we want to enhance makes
U ( % ) an even function, we must choose an even filter, and if it makes it
an odd function, we must choose an odd filter.
Any function f (X) can be written as the s u m of an odd and an even part:
Image Segmentation
Edge and Detection 311

In general, therefore, f (X) can be written as:

f(g) = f e ( x )+ f o ( z )
Suppose that u(x)is even. Then the integral in the numerator of S is:

L
00

The integral in the denominator in the expression for S is:

S_,
00

f2(x)dx = S_
00

f,"(x)dx
, + -00
f,"(x)dx +
-
2
00

odd intergand
f efo(x)dx

integrated
over a symmetric interval:
it vanishes
- S_
00

, +
f,"(z)dx S_,f,"(x&
00

So, we see that the odd part of the filter does not contribute at all in the signal
response, while it contributes to the noise response; i.e. it reduces the signal to
noise ratio. Thus, to enhance an even feature we must use even filters. Similarly,
to enhance an odd feature, we must use odd filters.

Example 7.15

What type of feature are the digital filter masks below appropriatefor
enhancing?

The first enhances vertical lines and the second horizontal lines, in bothcases
brighter than the background.
312 Processing:
Image The Fundamentals

(a) Assuming a step edge and using a (b) The same parameters as in (a)with
filter of size 17 X 17
the size of the filter increased to 21 X 21.
Some detail is lost and no improvement
has been achieved

(c) Assuming a step edge and a filter of (d) As in (c) with thethreshold increased
size 17 X 17 and increasing the threshold even further

Figure 7.20: Trying to deal with the high level of noise by using an optimal
filter of large size and playing with the thresholds may not help, if the
edges are blurred and resemble ramps, and the "optimal" filter has been
developed to be optimal for step edges.
Image Segmentation
Edge and Detection 313

(a) Assuming a shallow ramp and using (b) Exactly the same as in(a) but setting
a filter of size 7 X 7 the filter size to 13 X 13

a,
(c) Assuming a ramp edge and filter of
size 17 X 17
(d) Assuming a ramp edge and using a
filter of size 21 X 21

Figure 7.21: Having chosen the right model for the features we wish to
detect, using the right filter size for the level of noise may prove crucial.
316 Image
The Processing: Fundamentals

1 These are the feature maps of the given image.

B7.3: Convolving a random noise signal with a filter.

Suppose that our noise signal is .(X) and we convolve it with a filter f(x). The
result of the convolution will be:

(7.30)

As the input noise is a random process, this quantity is a random process too. If
we take its expectation value, it will be 0 as long as the expectation value of the
input noise is 0 too. We may also try tocharacterize it by calculating itsvariance,
its mean square value. This is given by:

E{[9(z0)l2)= E{g(zo)S(~o)I (7.31)

This is the definition of the autocorrelation function of the output calculated at
argument 0. So we must calculate first the autocorrelation function of g , R,,(O).
+
We multiply both sides of equation (7.30) with g(x0 7) and take expectation
values:

* Rgg(7T) = 1, 00

f ( z ) E { g ( z o+ .r)n(xo - %)Id%

L
00

* RSS(T) = f(z)Rg,(zo + 7 - 50 + x)dx

Q3

* Rgg(7) = J’,f(z)R,n(.r+ x)dx (7.32)

where R g n ( a )is the cross-correlation function between the input noise signal and
the output noise signal at relative shift a. We must calculate Rgn. We multiply
both sides of (7.30) with n(z0-T) and then take expectationvalues. (We multiply
+
with n(z0 - T ) instead of n(z0 T ) because in (7.32) we defined the argument of
R,, as the difference of the argument of g minus the argument of n.)
Image Segmentation
Edge and Detection 317

where &,(a) is the autocorrelation function of the input noise signal at relative
shift a. However, n ( z ) is white Gaussian noise, so its autocorrelation function is
a delta function given by Rnn(7)= n;S(r) where ni is the variance of the noise.
Therefore:

(7.34)

Therefore, R,, with argument T + z as it appears in (7.32) is:

+
Rgn(r II:)= ngf(7 II:) + (7.35)

If we substitute in (7.32) we have:

Rgg(r) = 4 S_
cc
, + z)dz * R g g ( 0 ) = ni
f(.)f(. l, f'(z)dz (7.36)
cc

If we substitute in equation (7.31) we obtain:

(7.37)

Themeanfilterresponse t o noise is givenby thesquareroot of the above

expression.

B7.4: Calculation of the signal to noise ratio after convolution of a

noisy edge signal with a filter.

Suppose that f ( x ) is the filter we want to develop, so that it enhances an edge in

a noisy signal. The signal consists of two components: the deterministic signal of
interest U(.) and the random noise component n ( z ) :

I ( z ) = U(.) + .(X)
318 Image Processing: The Fundamentals

The response of the filter due to thedeterministic component of the noisy signal,
when the latter is convolved with the filter, is:

(7.38)

Let us assume that the edge we wish to detect is actually at position zo = 0.

Then:

(7.39)

The mean response of the filter due to the noise component (see Box B7.3) is
nod-. The signal t o noise ratio, therefore, is:

(7.40)

B7.5: Derivation of the good locality measure.

The position of the edge is marked at the point where the output is maximum.
The total output of the convolution is the output due to the deterministic signal
U(.) and the output due to the noise component .(X). Let us say that the noisy
signal we have is:

I(X) = U(.) +.(X)

The result of the convolution with the filter f ( z ) will be:

00

O(z0) = JI
f (,
z)I(zo - z)dz = f(z0 - z)I(z)dz

L L
00
-

-
+
f ( z 0 - z)u(z)dz f(z0 - z)n(z)dz= s(z0) + g(z0)
where s(z0) is the output due to the deterministic signal of interest and q(z0) is
the output due t o noise.
Image Segmentation
Edge and Detection 319

The edge is detected at the local maximum of this output; i.e. at the place
where its first derivative with respect to zo becomes 0:

We expand the derivative of the filter, about point zo = 0, the true position of
the edge, and keep only the first two terms of the expansion:

f ’ ( z o- X) E f’(-X) + zof”(-z) + . . .
Upon
substitution
into we obtain:

(7.42)

The feature we want to detect is an antisymmetric feature; i.e. an edge has a

shape like 1.
According to the result proven in Example 7.14, f ( z ) should also be an an-
tisymmetric function. This means that its first derivative will be symmetric and
when it is multiplied with the antisymmetric function U(.) it will produce an
antisymmetric integrand which upon integration over a symmetric interval will
make the first term in equation (7.42) vanish. Then:

set ZE-X

The derivative of the filter response to noise may be written ina more convenient
way:

The position of the edge will be marked at the value of z that makes the sum of
the right hand sides of equations (7.43) and (7.44) zero:

=o*

(7.45)
320 Image Processing: The Fundamentals

The right hand side of this expression is a random variable, indicating that
the location of the edge will be marked at various randomly distributed positions
around the true position, which is at zo = 0. We can calculate the mean shifting
away from the true position as the expectation value of 20. This, however, is
expected to be 0. So, we calculateinsteadthe variance of the zo values. We
square both sides of (7.45) and take the expectationvalue of

Note that the expectationvalue operator applies only to the randomcomponents.

In Box B7.3 we saw that if a noise signal with variance ni is convolved by a
filter f ( z ) ,the mean square value of the output signal is given by:

ni 1-
cc
f"4dX

(see equation (7.37)). The right hand side of equation (7.45) here indicates the
convolution of the noise component by filter f'(z). Its mean squarevalue therefore
is ni S-", [ f ' ( ~ ) ] ~Then:
dz.

(7.46)

The smaller this expectation value is, the better the localization of the edge. We
can define, therefore, the good locality measure as the inverse of the square root
of the above quantity. We may also ignore factor no as it is the standard deviation
of the noise during the imaging process, over which we do not have control. So a
filter is optimal with respect t o good locality, if it maximizes the quantity:

(7.47)

I I
Example 7.17 (B)

Show that the differentiation of the output of a convolution of a signal

U(%)with a filter f ( x ) can be achieved by convolving the signal with the
derivative of the filter.
Image Segmentation
Edge and Detection 321

The output of the convolution is:

or
(7.48)

Applying Leibnitz's rule for differentiating an integral with respect to a parameter

(Box B7.1) we obtain from equation (7.48):

which upon changing variables can be written as:

B7.6: Derivation of the count of false maxima.

It has been shown by Rice that the average density of zero crossings of the con-
volution of a function h with Gaussian noise is given by:
1

"' [ -&h@)
R"hh (0) ] (7.49)

where Rhh(7) is the spatial autocorrelation function of function h ( z ) ,i.e:

(7.50)

Therefore:

Using Leibnitz's rule (see Box B7.1) we can differentiate (7.50) with respect, t o
r to obtain:
322 Image Processing:
FundamentalsThe

00

Rthh(7) = J'
-cc
h(X)h'(X + 7)dX
We define a new variable of integration 5 = X +r + X = 5 - r and dx = d5 to
obtain:
00

R$&) = J'
-cc
h(5 - r)ht(5)d5 (7.51)

We differentiate (7.51) once more:

00 00
R'thh(7)= - [ ht(5 - T)ht(5)d5 + R'lhh(0) = [ - (h'(x))2dx
J-, J -cc

Therefore, the average distance of zero crossings of the output signal when a noise
signal is filtered by function h(x) is given by:

(7.52)

From the analysis in Box B7.5,we can see that the false maxima in our casewill
come from equation S-", f'(x)n(zo- z)dz = 0 which will give the false alarms
in the absence of any signal (U(.) = 0). This is equivalent to saying that the
false maxima coincide with the zeroes in the output signal when the noise signal
is filtered with function f'(x). So, ifwe want to reduce the number of false local
maxima, we should make the average distance between thezero crossings as large
as possible, for the filter function f'(z). Therefore, we define the good measure
of scarcity of false alarms as:

What is the "take home" message of this chapter?

This chapter dealt with the reduction of the information content of an image so that
it can be processed more easily by a computer vision system. It surveyed the two
basic approaches for this purpose: region segmentation and edge detection. Region
segmentation triesto identify spatially coherent sets of pixels that appear toconstitute
uniform patches in the image. These patches may represent surfaces or parts of
surfaces of objects depicted in the image. Edge detection seeks to identify boundaries
between such uniform patches. The most common approach for this is based on the
estimate of the first derivative of the image. For images with low levels of noise the
Image Segmentation
Edge and Detection 323

Sobel masks are used to enhance the edges. For noisy images the Canny filters should
be used. Canny filters can be approximated by the derivative of a Gaussian; i.e. they
2
have the form x e - 9 . Although parameter (T in this expression has to have a specific
value for the filters to be optimal according to Canny’s criteria, often people treat (T
as a free parameter and experiment with various values of it. Care must be taken
in this case when discretizing the filter and truncating the Gaussian, not to create a
filter with sharp ends. Whether the Sobel masks are used or the derivatives of the
Gaussian, the result is the enhancement of edges in the image. The output produced
has to be further processed by non-maxima suppression (i.e. the identification of the
local maxima in the output array) and thresholding (i.e. the retention of only the
significant local maxima).
Edge detectors consist of all three stages described above and produce fragmented
edges. Often a further step is involved of linking the fragments together to create
closed contours that identify uniform regions. Alternatively, people may bypass this
process by performing region segmentation directly and, if needed, extract thebound-
aries of the regions afterwards. Region-based methods are much more powerful when
both attribute similarity and spatial proximity are taken into consideration when
deciding which pixels form which region.
Bibliography

The material of this book is based largely on classical textbooks. The method de-
scribed in example 6.10 was taken fromreference 3. Chapter 7 is based mainly on
research papers. From the references given below, n u d e r s 13, 15 and 23 refer to
thresholding methods. References 5, 6, 13, 18-22 and 28 refer to linear edge de-
tectors. Paper 17 refers to non-linear edge detectors with an application of them
presented in 10. Reference 21 is an extensive review of the Canny-related approaches
to edge detection.

1. M Abramowitz and I A Stegun (eds), 1970. Handbook of Mathematical Func-

tions, Dover Publications, ISBN 0-486-61272-4.
2. K G Beauchamp, 1975. Walsh Functions and their Applications, Academic
Press, ISBN 0-12-084050-2.
3. C R Boukouvalas, 1996. Colour Shade Grading and its Applications to Visual
Inspection PhD thesis, University of Surrey, UK.
4. R N Bracewell, 1978. The Fourier Transform and its Applications, McGraw-
Hill, ISBN 0-07-007013-X.
5. J Canny, 1983. Finding Edges and Lines in Images. MIT A1 Lab Technical
Report 720.
6. J Canny, 1986. "A computational approach to edge detection". IEEE Transac-
tions on Pattern Analysis and Machine Intelligence, Vol P AMI-8, pp 679-698.
7. R A Gabel and R A Roberts, 1987. Signals and Linear Systems, John Wiley &
Sons, ISBN 0-471-83821-7.
8. R C Gonzalez and R E Woods, 1992. Digital Image Processing Addison-Wesley ,
ISBN 0-201-50803-6.
9. I S Gradshteyn and I M Ryzhik, 1980. Table of Integrals, Series and Products,
Academic Press, ISBN 0-12-294760-6.
10. J Graham and C J Taylor, 1988. "Boundary cue operators for model-based
image processing". Proceedings of the Fourth Alvey Vision Conference, AVC88,
University of Manchester, 31 August-2 September 1988, pp 59-64.
326 Image Processing: The Fundamentals

11. H P Hsu, 1970. Fourier Analysis, Simon & Schuster, New York.
12. T S Huang (ed), 1979. “Picture processing and digital filtering”. In Topics in
Applied Physics, Vol 6, Springer-Verlag, ISBN 0-387-09339-7.
13. J Kittler, 1983. “On the accuracy of the Sobel edge detector”. Imageand
Vision Computing, Vol 1, pp 37-42.
14. J Kittlerand J Illingworth, 1985. “Onthreshold selection using clustering
criteria”. IEEE Transactions on Systems, Man, and Cybernetics, Vol SMC-15,
pp 652-655.
15. N Otsu, 1979. “A threshold selection method from gray level histograms”. IEEE
Transactions on Systems, Man, and Cybernetics, Vol SMC-9, pp 62-66.
16.A Papoulis, 1965. Probability,RandomVariablesandStochasticProcesses,
McGraw-Hill Kogakusha Ltd, Library of Congress Catalog Card number 64-
22956.
17. I Pitas and A N Venetsanopoulos, 1986. “Non-linear order statistic filters for
image filtering and edge detection”. Signal Processing, Vol 10, pp 395-413.
18. M Petrou and J Kittler, 1991. “Optimal edge detectors for ramp edges”. IEEE
Transactions on Pattern Analysis and Machine Intelligence, Vol PAMI-13, pp
483-491.
19. M Petrou and A Kolomvas, 1992. “The recursive implementation of the optimal
filter for the detection of roof edges and thin lines”. In Signal Processing VI?
Theory and Applications, pp 1489-1492.
20. M Petrou, 1993. “Optimal convolution filters and an algorithm for the detection
of wide linear features.” IEE Proceedings: Vision, Signal and Image Processing,
V01 140, pp 331-339.
21. M Petrou, 1994. “The differentiating filter approach to edge detection”. Ad-
vances in Electronics and Electron Physics, Vol 88, pp 297-345.
22. M Petrou, 1995. “Separable 2D filters for the detection of ramp edges”. IEE
Proceedings: Vision, Image and Signal Processing, Vol 142, pp 228-231.
23. M Petrou and A Matrucceli, 1998. “On the stability of thresholding SAR im-
ages”. Pattern Recognition, Vol 31, pp 1791-1796.
24. W K Pratt, 1978. Digital Image Processing, John Wiley & Sons, ISBN 0-471-
01888-0.
25. R M Pringle and A A Rayner, 1971. Generalized Inverse Matrices with Ap-
plications to Statistics, No. 28 of Griffin’s Statistical monographs and courses,
edited by A Stuart, ISBN 0-85264-181-8.
Bibliography 327

26. S 0 Rice, 1945. “Mathematical analysis of random noise”. Bell Systems Tech.
J., V01 24, pp 46-156.
27. A Rosenfeld and A C Kak, 1982. Digital Picture Processing, Academic Press,
ISBN 0-12-597301-2.
28. L A Spacek, 1986. “Edge detection and motion detection”. Image and Vision
Computing, Vol4, pp 43-53.
29. P H Winston, 1992. Artificial Intelligence, Addison-Wesley, ISBN 0-20-153377-
4.
30. T Y Young and K S Fu (eds), 1986. Handbook of Pattern Recognition and Image
Processing, Academic Press, ISBN 0-12-774560-2.
Index
A C
additive noise 144 Canny’s criteria for an optimal edge filter
antisymmetric image features 309 306-309,318-322
approximation theory for filter design characteristic equation of a matrix 31
171-191 Chebyshev norm 171, 172
approximation of an image using its K-L checkerboard effect 2
transform 110, 115 circulant matrix 233
approximation of an image by SVD 34,35 clustering 288
autocorrelation function of a random field complete set of functions 45
9.4, 103 conditional expectation 217
autocorrelation matrix of an image (how constraint matrix inversion 251
to compute) 98-102, 105-106 contrast enhancement of a multispectral
autocovariance of a random field 94 image 135
automatic vision 18 contrast manipulation 132
convolution 7, 72-77, 157
convolution theorem (assumptions of) 75
B covariance of two random variables 92
cross correlation of two random fields 95
bandpass filter 160 cross covariance of two random fields 95
bands of an image 1, 138 cross spectral density of two random fields
barrel distortion 194 218
basis images forthe Fourier transform 68,
69
basis images for the Haar transform 52 D
basis images for the Hadamard transform
58 delta function 8
basis images from the K-L transform 111 DFT (see discrete Fourier transform)
basis images for the Walsh transform 58 direct component of an image 82
bilinear interpolation 195 discrete cosine transform 86
bits needed for an image 2 discrete Fourier transform 63-86
block circulant matrix 232, 239-240 discrete Fourier transform (display of) 79
brightness of an image pixel 2 discrete Fourier transform and image
restoration 210-215, 228-230
discrete Fourier transform of a rotated
image 79
discrete Fourier transform of a scaled
330 Image Processing: The Fundamentals

image 83 expected value of a random variable 92

discrete Fourier transform of a shifted
image 81
distribution function of a random variable F
90
distribution function of many random false contouring 2
variables 92 fast Fourier transform 84
dual grid 291 feature map 315-316
features 288
FFT (see fast Fourier transform)
E filter 155
Fourier transform 45, 63, 72
edge detection 265, 289-322 frequency convolution theorem 75
edge map 303 frequency sampling 172, 182-191
edge pixels 291 Fresnel integrals 202-204
edgels 291
eigenimages from the K-L transform 121
eigenimages from SVD 37 G
eigenvalues of a circulant matrix 233
eigenvalues of a matrix 30, 140, 234 Gaussian noise 144
eigenvectors of a circulant matrix 233 geometric progression 64
eigenvectors of a matrix 30, 31, 140, 234 geometric restoration 193-198
ergodic random field 96, 97 grey level 1
ergodic random field with respect to the grey level interpolation 195
mean 97
ergodic random field with respect to the H
autocorrelation function 97
ergodicity lU2, 103, 104, 227 Haar functions 46
error in approximating an image by its Haar transforms 47, 62
K-L transform 115 Haar wavelet 62
error in approximating an image by its Hadamard matrices 57
SVD 35 Hadamard transforms 57
even symmetrical cosine transform 86 highpass filter 148, 160
expansion of an image in terms of Fourier histogram equalization 127
basis functions 68 histogram hyperbolization 129
expansion of an image in terms of histogram modification with random
eigenimages 34 additions 127, 130
expansion of an image in terms of Haar histogram of an image 125
basis functions 51-52, 58-60 histogram of an image under variable
expansion of an image in terms of Walsh/ illumination 283-285
Hadamard matrices 57-58, 60-61 homogeneous random field 96
expansion of an image using its K-L homomorphic filter 149-150
transform 110 Hotelling transform 89
expansion of an image in terms of vector hysteresis thresholding 266
outer products 21-22
Index 331

I K
ideal highpass filter 148, 160 Karhunen-Loeve expansion of an image
ideal lowpass filter 148, 157-161 110
image 1 Karhunen-Loeve transform 89
image as a linear superposition of point Karhunen-Loeve transform of a
sources 9 multispectral image 136
image as a random field 89-121 K-L (see Karhunen-Loeve)
image classification 266 Kronecker ordering of Walsh functions 57
image compression 18 Kronecker product of matrices 14, 237
image compression using K-L 102
image compression using SVD 24
image enhancement 18, 125-153 L
image geometric restoration 193-198
image labelling 266 La Vallee Poussin theorem 180, 181
image registration 193 Lagrange multipliers 251-258
image resolution 2 Laplacian 243
image restoration 18, 193-263 least square error solution for image
image restoration by inverse filtering restoration 218
209-217 least square error approximation of an
image restoration by matrix inversion image 37
Leibnitz rule for differentiating an
230-262
image restoration by Wiener filtering integral with respect to a parameter
218-230 268
image segmentation 265 lexicographic ordering of Walsh functions
image sharpening 148 57
image smoothing 147 limiting set of equations 180
image thresholding 255-286 line detection 309-310
image thresholding under variable linear operator 6
illumination 285 linear programming 172, 174
impulse noise 144 local contrast enhancement 131
independent random variables 92 low pass filter 148, 157, 161
inhomogeneous contrast 131 lowpass filtering 147

J M
joint distribution function of many matrix diagonilization 24-35
random variables 92 maximizing algorithms for filter design 180
joint probability density function of many mean square error for K-L transform 118
random variables 92 mean value of a random variable 92
median filter 146
minimizing algorithms for filter design 180
mini-max algorithms for filter design 180
minimum error threshold 268-278
minimum mean square error
332 Processing:
Image The Fundamentals

approximation of an image 118, 124 principal component analysis of a

minimum square error approximation of multispectral image 136-144
an image 36 probability of an event 90
motion blurring 200-204, 210-217, probability density function of a random
228-230,259-262 variable 91, 92
multiband image 135 probability density function of many
multiplicative noise 144, 149, 283 random variables 92
multispectral image 135 properties of a discrete Fourier transform
79-84

N
natural order of Walsh functions 47
Q
nearest neighbour interpolation 195 quadtree 289
noise 144
noise convolved with a filter 316-317
noise in image restoration 210 R
non-maxima suppression 303
non-recursive filter 161 ramp edges 309
norm of a matrix 35 random field 90, 93
random variable 90
rank order filtering 146
0 rectangle function 7
recursive filter 161
operator 6 region growing 288
optimal threshold 268-278 resolution 2
orthogonal matrix 24 restoration by matrix inversion 230-262
orthogonal random variables 92 restoration by Wiener filtering 218-230
orthogonal set of functions 45 restoration of motion blurring 210-217,
orthonormal set of functions 45 228-230, 259-262
orthonormal vectors 24 Rice’s formula for filtered noise 321
Otsu’s thresholding method 278-282 Robinson operators 208
outer product of vectors 21

S
P
salt and pepper noise 144
partition of a matrix 10 scaling function 62
pattern recognition 288 seed pixels for region growing 288
pel l separability assumption 14-15
pincushion distortion 194 separable masks 306
pixel 1 separable point spread function 7
point source 7-9 separable transform 15
point spread function of a linear sequency order of Walsh functions 47
degradation process 198-209 sharpening 148
point spread function of an operator 6-7 shift invariant point spread function 7
Index 333

shifting property of the delta function 9 uncorrelated random variables 92

simulated annealing 217 unit sample response of a filter 155
singular value decomposition of an image unitary matrix 23-24
34 unitary transform 23
singular value decomposition of a matrix
24
smoothing 147 V
Sobel masks 296-299,306
variable illumination 149, 283-286
spatial statistics of a random field 97
spatial autocorrelation matrixof an image variance of a random variable 92
vector outer product 21
105
spectral bands 1, 138 vector outer product (expansion of an
spectral density of a random field 223 image in terms of) 21-22
split and merge algorithms 288
stability condition in filter design 171 W
stacking operator 10
standard deviation of a random variable Walsh functions 46, 47
92 Walsh transforms 47, 62
successive doubling algorithm 84 wavelets 62
SVD (see singular value decomposition) white noise 220
system function 155 Wiener filter 218-230, 254
Wiener-Khinchine theorem 223
windowing 172, 173, 210
T
textured regions 288 Z
thresholding 255
tie points 195 z-transform 161-171
time convolution theorem 75
trace of a matrix 36
transfer function from an astronomical
image 204
transfer function from a bright edge
206-209
transfer function from a bright line
205-206
transfer function of a degradation process
199-209
transfer function of motion blurring
200-204,210-217

U
uncorrelated data 144
uncorrelated random fields 95