Image Restoration

Image Restoration

• Image restoration vs. image enhancement

 Enhancement:
 largely a subjective process
 Priori knowledge about the degradation is not a must
(sometimes no degradation is involved)
 Procedures are heuristic and take advantage of the
psychophysical aspects of human visual system
 Restoration:
 more an objective process
 Images are degraded
 Tries to recover the images by using the knowledge
about the degradation
 An Image Degradation Model
• Two types of degradation
 Additive noise
 Spatial domain restoration (denoising) techniques are preferred
 Image blur
 Frequency domain methods are preferred
• We model the degradation process by a degradation function
h(x,y), an additive noise term, (x,y), as g(x,y)=h(x,y)*f(x,y)+
(x,y)
 f(x,y) is the (input) image free from any degradation
 g(x,y) is the degraded image
* is the convolution operator


 The goal is to obtain an estimate of f(x,y) according to the

knowledge about the degradation function h and the additive noise

 In frequency domain: G(u,v)=H(u,v)F(u,v)+N(u,v)
• Three cases are considered in this Chapter
 g(x,y)=f(x,y)+ (x,y)
 g(x,y)=h(x,y)*f(x,y)
 g(x,y)=h(x,y)*f(x,y)+ (x,y) (5-7~5-9)
A Model of the Image
Degradation/Restoration Process
Noise Model

• We first consider the degradation due to noise

only
 h is an impulse for now ( H is a constant)
• White noise
 Autocorrelation N  1 M function
1 is an impulse function
a
a ( x, y )by
multiplied  ( s, t )  ( s  x, t  y )  N 0 ( x, y )
constant
t 0 s 0


 It means there is no correlation between any two pixels

in the noise image
 There is no way to predict the next noise value
 The spectrum of the autocorrelation function is a
constant (white)
Gaussian Noise

• Noise (image) can be classified according the

distribution of the values of pixels (of the noise
image) or its (normalized) histogram
• Gaussian noise is characterized by two
parameters,  (mean) and σ2 (variance), by

1  ( z   ) 2 / 2 2
p( z )  e
2 

• 70% values of z fall in the range [(-σ),(+σ)]

• 95% values of z fall in the range [(-2σ),(+2σ)]
Gaussian Noise
Other Noise Models
• Rayleigh noise
2 2
 ( z  a )e  ( z  a ) / b for z a
p ( z )  b
0 for z  a
 The mean and variance of this
density are given by
b( 4   )
 a  b / 4 and  2 
 a and b can be obtained4through
mean and variance
Other Noise Models
• Erlang (Gamma) noise
 a b z b  1  az
 e for z 0
p ( z )  (b  1)!
0 for z  0
 The mean and variance of this
density are given by
b
 b / a and  2  2
 a and b can be obtained a
through mean and variance
Other Noise Models

• Exponential noise
ae  az for z 0
 p ( z ) 
0 for z  0
 The mean and variance of
this density are given by

 Special case 1
pf Erlang
 1 / a and 2  2
PDF with b=1 a
Other Noise Models
• Uniform noise
 1
 if a  z b
p ( z )  b  a
  0 otherwise
 The mean and
variance of this
density are given by
2
(b  a )
 (a  b) / 2 and  2 
12
Other Noise Models
• Impulse (salt-and-
pepper) noise
 Pa for z a
 
p ( z )  Pb for z b
0 otherwise

 If either Pa or Pb is
zero, the impulse noise
is called unipolar
 a and b usually are
extreme values
because impulse
corruption is usually
large compared with
the strength of the
image signal
 It is the only type of
noise that can be
Periodic Noise
• Arises typically from
electrical or
electromechanical
interference during image
acquisition
• It can be observed by
visual inspection both in
the spatial domain and
frequency domain
• The only spatially
dependent noise will be
considered
Estimation of Noise Parameters

• Periodic noise
 Parameters can be estimated by inspection of
the spectrum
• Noise PDFs
 From sensor specifications
 If imaging sensors are available, capture a set of
images of plain environments
 If only noisy images are available, parameters of
the PDF involved can be estimated from small
patches of constant regions of the noisy images
Estimation of Noise Parameters

• In most cases, only mean and variance are

to be estimated
 Others can be obtained from the estimated
mean and variance
• Assume a sub-image with plain scene is
available and is denoted by S

1
̂   z ( xi , yi )
N S ( xi , yi )S

 2 1 2 1
0   ( z ( xi , yi )   )
N S ( xi , yi )S
2
1 
NS
 i i
( z ( x ,
( xi , yi )S
y )  
ˆ ) 2

 2 1
2   i i
N S  1 ( xi , yi )S
( z ( x , y )  ˆ ) 2
Restoration in the Presence of Noise
Only (De-Noising)
• Mean filters
1
 Arithmetic mean filter fˆ ( x, y )   g ( s, t )
 g(x,y) is the corrupted image mn ( s ,t )S x , y
 S is the mask 1
x,y
  mn
 ˆ
Geometric mean filtersf ( x, y )   g ( s, t )
 Tends to preserve more details  ( s ,t )S x , y 
 Harmonic mean filter fˆ ( x, y ) 
mn
1

( s ,t )S x , y g ( s, t )
 Works well for salt noise but fails for pepper noise
 Contraharmonic mean filter
 Q: order of the filter
 Positive Q works for pepper noise
 g ( s, t ) Q 1
ˆ ( x, y )  ( s ,t )S x , y
f
 Negative Q works for salt noise
 Q=0arithmetic mean filter
 g ( s, t ) Q
( s ,t )S x , y
 Q=-1harmonic mean filter
Filters Based on Order Statistics
(De-Noising)
fˆ ( x, y ) median{g ( s, t )}
• Median filter (s,t)S x,y

 Median represents the 50th percentile of a

ranked set of numbers
• Max and min filter
 Max filter uses the 100th percentile of a ranked
set of numbers
 Good for removing pepper noise
 Min filter uses the 1 percentile of a ranked set
of numbers
 Good for removing salt noise
• Midpoint filter ˆf ( x, y )  1  max {g ( s, t )}  min {g ( s, t )}
2  ( s ,t )S xy ( s ,t )S xy 
 Works best for noise with symmetric PDF like
Gaussian or uniform noise
Alpha-Trimmed Mean Filter
(De-Noising)

• Alpha-trimmed mean filter takes the mean

value of the pixels enclosed by an m×n mask
after deleting the pixels with the d/2 lowest
and the d/2 highest gray-level
1 values
fˆ ( x, y )   g ( s, t ) r
mn  d ( s ,t )S xy
 gr(s,t) represent the remaining mn-d pixels
 It is useful in situations involving multiple
types of noise like a combination of salt-
and-pepper and Gaussian
Adaptive Filters
(De-Noising)
• Adaptive Local Noise Reduction Filter
 Assume the variance of the noise  2 is either
known or can be estimated satisfactorily
 Filtering operation changes at different regions
of an image according to localvariance L
2

calculated within an M×N region

If  L    , the filtering operation
2 2
 is defined as
ˆf ( x, y )  g ( x, y )    [ g ( x, y )  m ]
2

L
 2
L
If     , the output takes the mean value
 2 2
L

 That is:
 2
is set to be 1
L 2

  2
At edges, it is assumes thatL   2

Adaptive Median Filter
(De-Noising)

• Median filter is effective for removing salt-

and-pepper noise
 The density of the impulse noise can not be too
large
• Adaptive median filter
 Notation
 Zmin: minimum gray value in Sxy
 Zmax: maximum gray value in Sxy
 Zmed: median of gray levels in Sxy
 Zxy: gray value of the image at (x,y)
 Smax: maximum allowed size of Sxy
Adaptive Median Filter
(De-Noising)

• Two levels of operations

 Level A: Used to test whether Zmed
 A1= Z
med –Zmin is part of s-and-p noise. If
 A2= Z yes, window size is
med –Zmax
 If A1 > 0 AND A2 < 0, Go to level B increased
else increase the window size by 2
 If window size <= S
max repeat level A
else output Zxy
 Level B: Used to test whether Zxy
 B1= Z –Z is part of s-and-p noise. If
xy min
 B2= Z –Z yes, apply regular
xy max median filtering
 If B1 > 0 AND B2 < 0, output Z
xy
else output Zmed
Periodic Noise Reduction by
Frequency Domain Filtering

• Lowpass and highpass filters for image

enhancement have been studied
• Bandreject, bandpass, and notch filters
as tools for periodic noise reduction or
removal are to be studied in this section.
Bandreject Filters

• Bandreject filters remove or attenuate a

band of frequencies about the origin of the
Fourier transform.
• Similar to those LPFs and HPFs studied, we
can construct ideal, Butterworth, and
Gaussian bandreject filters
Bandreject Filters
• Ideal bandreject filter
 W
 1 if D (u , v )  D0 
2
 W W
H (u , v) 0 if D0  D(u , v) D0 
 2 2
1 if D(u , v)  D0  W
 2
Bandreject Filters
• Butterworth bandreject
filter
1
H (u , v)  2n
 D(u , v)W 
1  2 2
 D (u , v )  D0 

n =1
Bandreject Filters
• Gaussian bandreject filter
1  D 2 ( u ,v )  D02 
  
2  D ( u ,v )W 
H (u , v) 1  e
Bandbass Filters
Bandpass filter performs the opposite of a
bandpass filter

H bp (u , v) 1  H br (u , v)
Notch Filters

• Notch filter ejects frequencies in predefined

neighborhoods about a center frequency.
• It appears in symmetric pairs about the origin
because the Fourier transform of a real valued
image is symmetric.
Notch Filters

• Ideal notch filter

0 if D1 (u , v) D0 or D2 (u , v) D0
H (u , v) 
1 otherwise

 
D1 (u , v)  u  M / 2  u0   v  N / 2  v0 
2 2 1/ 2

D (u , v) u  M / 2  u   v  N / 2  v  
2 2 1/ 2
2 0 0
 Notch Filters

• Butterworth notch filter

1
H (u , v)  2n
 D 2

1  0

D
 1 (u , v ) D2 (u , v ) 
n=2
Notch Filters

• Gaussian notch filter

1  D1 ( u ,v ) D2 ( u ,v ) 
  
2  D02 
H (u , v) 1  e
Notch Filters

Notch filters that pass, rather than suppress:

H np (u , v) 1  H nr (u , v)

• NR filters become highpass filters

u0 v
if 0 0

• NP filters become lowpass filtersu0ifv0 0

Optimum Notch Filtering
• In the ideal case, the original image can be
restored if the noise can be estimated
completely.f ( x, y )  g ( x, y )   ( x, y )
 That is:

• However, the noise can be only partially

fˆ ( x, y )the
estimated. This means  g restored
( x, y )  ˆ (image
x, y ) is
not exact.
ˆ ( x, y ) IFT
 Which means
H (u, v)G (u, v)
Optimum Notch Filtering
• In this section, we try to improve the restored
image by introducing a modulation function

fˆ ( x, y )  g ( x, y )  w( x, y )ˆ ( x, y )
 Here the modulation function is a constant within a
neighborhood of size (2a+1) by (2b+1) about a point
(x,y)
 We optimize its performance by minimizing the local
variance of the1 restoreda image
b at the position (x,y) 2
 fˆ ( x  s, y  t )  fˆ ( x, y )
 2 ( x, y ) 
(2a  1)(2b  1)
  
s  at  b

a b
1
fˆ ( x, y )    fˆ ( x  s, y  t )
(2a  1)(2b  1) s  at  b
Optimum Notch Filtering
Points on or near Edge of the image can be treated by
considering partial neighborhoods
a b
1
 ( x, y ) 
2
 
(2a  1)(2b  1) s  at  b
{[ g ( x  s, y  t )

 w( x  s, y  t )ˆ ( x  s, y  t )]
 [ g ( x, y )  w( x, y )ˆ ( x, y ]}2

Assumption:w( x  s, y  t ) w( x, y ) for  a s a and  b t b

 w( x, y )ˆ ( x, y ) w( x, y )ˆ ( x, y )

Optimum Notch Filtering
b
1
 s  a 
a
 ( x, y ) 
2
{[ g ( x  s, y  t )
(2a  1)(2b  1) t  b

 w( x  s, y  t )ˆ ( x  s, y  t )]
 [ g ( x, y )  w( x, y )ˆ ( x, y )}2

To minimize 2 ( x, y )
 2 ( x, y )
0
w( x, y )

g ( x, y )ˆ ( x, y )  g ( x, y )ˆ ( x, y )
 w( x, y ) 
ˆ 2 ( x, y )  ˆ 2 ( x, y )
Linear, Position-Invariant
Degradation
• Degradation Model
g ( x, y ) H [ f ( x, y )]  ( x, y )
In the absence of additive noise:
For scalar values of a and b, H is
linear
H [if:
a f ( x, y )  b f ( x, y )] a H [ f ( x, y )]  b H [ f 2 ( x, y )]
1 2 1

H is Position-Invariant if:
g ( x, y ) H [ f ( x, y )]  H [ f ( x   , y   )]  g ( x   , y   )
Linear, Position-Invariant
Degradation
In the presence of additive noise:

g ( x, y )  f ( ,  )h( x   , y   )d d  ( x, y )
  

g ( x, y ) h( x, y ) * f ( x, y )  ( x, y )
G (u , v) H (u , v) F (u , v)  N (u , v)

• Many types of degradation can be

approximated by linear, position-invariant
processes
• Extensive tools of linear system theory are
available
Estimating the Degradation Function

• Principal way to estimate the

degradation function for use in image
restoration:
 Observation
 Experimentation
 Mathematical modeling
Estimating by Image Observation
• We look for a small section of the image
that has strong signalgcontent
s ( x, y ) ( )
and then construct an un-degradation of
fˆs ( x, ygray
this section by using sample ) levels (
).
Gs (u , v)
H s (u , v) 
Fˆs (u , v)

H (u , v)
Now, we construct a function on a large
scale, but having the same shape.
Estimating by Experimentation
• We try to obtain impulse response of the
degradation by imaging an impulse (small
dot of light) using the system. Therefore
G (u , v)
H (u , v) 
A
 Estimating by Modeling
 k ( u 2 v 2 ) 5 / 6
H (u , v) e
Atmospheric turbulence model:

High
turbulence
k=0.0025

Negligible
turbulence

Low
Mid turbulence
turbulence k=0.00025
k=0.001
H.R. Pourreza
Estimating by Modeling
Blurring by linear motion:
T
g ( x, y ) f [ x  x0 (t ), y  y0 (t )]dt
0
T
G (u , v) F (u , v) e  j 2 [ux0 ( t ) vy0 (t )]dt
0
T
 H (u , v) e  j 2 [ux0 (t ) vy0 (t )]dt
0 T

if x 0 ( t )  at / T and y 0 ( t )  0  H (u, v) e  2uat / T dt

0

T
 sin( ua )e  j ua
 ua
Estimating by Modeling
if x 0 ( t )  at / T and y 0 ( t )  bt / T 
T
H (u, v)  sin[ (ua  vb)]e  j ( ua vb )
 (ua  vb)

H.R. Pourreza
Inverse Filtering
The simplest approach to restoration is direct
inverse filtering:
ˆ G (u , v) Even if we know the
F (u , v)  degradation function,
H (u , v)
we cannot recover
N (u , v) the un-degraded
Fˆ (u , v) F (u, v)  image
H (u, v)

If the degradation has zero or very small values, then

the ratio N/H could easily dominate our estimation of F .

One approach to get around the zero or small-value

problem is to limit the filter frequencies to value near
the origin.
Minimum Mean Square Error Filtering
(Wiener Filtering)
This approach incorporate both the degradation
function and statistical characteristic of noise
into the restoration process.
Image and
noise are
random process

e 2 E[( f  fˆ ) 2 ]

The objective is to find an estimation for f such that

minimized e2
 Minimum Mean Square Error Filtering
(Wiener Filtering)
 H *
(u , v ) S (u , v ) 
Fˆ (u, v) 
f
2
 G (u , v)
 S f (u, v) H (u, v)  S (u, v) 
If the noise is zero,
 *
H (u, v)  then the Wiener
 2
 G (u, v) Filter reduces to the
 H (u , v)  S (u , v) / S f (u , v)  inverse filter.
 1 H (u, v)
2

 2
 G (u, v)
 H (u, v) H (u, v)  S (u, v) / S f (u , v) 

2
S ( u , v )  N ( u , v )  power spectrum of the noise
2
S f (u , v )  F ( u , v )  power spectrum of the undegraded im age
Minimum Mean Square Error Filtering
(Wiener Filtering)
 1 H (u , v )
2

Fˆ (u, v)  2
 G (u, v)
 H (u, v) H (u, v)  S (u, v) / S f (u , v) 

Constan Unknow
t n

 1 H ( u , v )
2

Fˆ (u , v)  2
 G (u, v)
 H (u, v) H (u , v)  K 
Geometric Transformations
• Unlike the techniques discussed so far,
geometric transformations modify the
spatial relationships between pixels in an
image.

Geometric transformation: RUBBER-SHEET

TRANSFORMATION
Basic Operations:
1. Spatial
Transformation
2. Gray-level
Interpolation
Spatial Transformations

x r ( x, y )
y  s ( x , y )

x r ( x, y ) c1 x  c2 y  c3 xy  c4
y s ( x, y ) c5 x  c6 y  c7 xy  c8
Gray-level Interpolation

v( x, y ) ax  by  cxy  d