Image Processing and Analysis

Image denoising using deep learning techniques

Oge Marques, PhD

Professor
College of Engineering and Computer Science
Outline
1. Noise and noise models
2. Classical image denoising techniques (i.e., before the deep learning
era)
3. Image denoising using MATLAB
4. Image denoising using deep learning in MATLAB
5. Hands-on assignment
What will we learn?
• What is noise (in the context of image processing) and how can it be
modeled?
• What are the main types of noise that may affect an image?
• Which noise removal techniques are typically used in image
processing?
• Which denoising techniques are available in MATLAB?
• Which deep learning architectures can be used for image denoising?
• How to implement image denoising using deep learning techniques in
MATLAB?
Noise and Noise Models
 What is image noise?
• Image noise is random variation of brightness or color information in
images.
• It is usually an aspect of electronic noise, which can be produced by the
image sensor and circuitry of a scanner or digital camera.
• It is an undesirable by-product of image capture that obscures the desired
information.
• In the context of signal (and image) processing, “noise” can be
interpreted as “unwanted signal”.

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise

 Types of noise
• Gaussian noise
• Salt-and-pepper noise
• Shot noise
• Quantization noise (uniform noise)
• Film grain
• Periodic noise

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise

 Gaussian noise
• Gaussian noise, named after Carl Friedrich Gauss, is statistical noise
having a probability density function (PDF) equal to that of the
normal distribution, which is also known as the Gaussian distribution.
• Image sensors have inherent noise due to the level of illumination
and their own temperature, and the electronic circuits connected to
the sensor inject their own share of electronic circuit noise.
• A typical model of image / sensor noise is Gaussian, additive,
independent at each pixel, and independent of the signal intensity,
caused primarily by Johnson–Nyquist noise (thermal noise).

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise and https://en.wikipedia.org/wiki/Gaussian_noise

 Salt-and-pepper noise
• Sometimes referred to as "impulsive”, speckle, or spike noise, it
appears in the form of dark pixels in bright regions and bright pixels in
dark regions of an image.

• This type of noise can be caused by analog-to-digital converter errors,

bit errors in transmission, among other reasons.

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise and https://en.wikipedia.org/wiki/Salt_and_pepper_noise

 Shot noise
• Photon shot noise is the dominant noise in the brighter parts of an
image from an image sensor is typically that caused by statistical
quantum fluctuations, that is, variation in the number of photons
sensed at a given exposure level.
• Shot noise has a root-mean-square value proportional to the square
root of the image intensity, and the noises at different pixels are
independent of one another.
• Shot noise follows a Poisson distribution, which except at very high
intensity levels approximates a Gaussian distribution.

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise

 Quantization noise
• The noise caused by quantizing the pixels of a sensed image to a
number of discrete levels is known as quantization noise.

• It has an approximately uniform distribution.

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise

 Film grain
• The grain of photographic film is a signal-dependent noise, with
similar statistical distribution to shot noise.
• If film grains are uniformly distributed (equal number per area), and if
each grain has an equal and independent probability of developing to
a dark silver grain after absorbing photons, then the number of such
dark grains in an area will be random with a binomial distribution.
• In areas where the probability is low, this distribution will be close to
the classic Poisson distribution of shot noise.
• A simple Gaussian distribution is often used as an adequately
accurate model.

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise

 Periodic noise
• Periodic noise
• a type of noise that usually arises as a result of electrical or electromechanical
interference during the image
NOISEacquisition process.
REDUCTION USING FREQUENCY-DOMAIN TECHNIQUES 255

(a) (b) (c)

Source: Wikipedia https://en.wikipedia.org/wiki/Image_noise

 Selected noise models

• Gaussian
• Impulse
(salt-and-pepper)
• Uniform
• Rayleigh
• Gamma (Erlang)
• Exponential

Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Effects of noise
on images and
histograms

• Gaussian

• Exponential

• Impulse
(salt-and-pepper)

Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Effects of noise
on images and
histograms

• Rayleigh

• Gamma (Erlang)

• Uniform

Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Classical Image
Denoising Techniques
(i.e., before the Deep Learning Era)
 Modeling of the image degradation
and restoration problem

Spatial domain Frequency domain

Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Noise reduction – spatial domain
• Mean filters
• Arithmetic
• Geometric
• Harmonic
• Contra-harmonic
• Order-statistic (rank) filters
• Median
• Min
• Max
• Mid-point
• Alpha-trimmed mean filter
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 In this section we revisit the arithmetic mean filter introduced in Section 10.3.1 – from the
perspective of its noise reduction capabilities – and expand the discussion to include other
related filters and discuss their performance.
Noise reduction – Mean filters
Arithmetic mean filter The arithmetic mean filter, also known as averaging filter,
operates on an m ⇥ n sliding window by calculating the average of all pixel values within
the window, and replacing the center pixel value in the destination image with the result.
• Arithmetic mean
Its mathematical filter (averaging
formulation filter)
is given as follows.
1 X
ˆ
f (x, y) = g(r, c) (12.19)
mn
(r,c)2W

where g is the noisy image, fˆ is the restored image, and r and c are the row and column
Causes a certain
•coordinates, amount
respectively, of blurring
within a window (proportional
W of size mto⇥the window
n where the size) to thetakes
operation image,
thereby reducing the effects of noise.
place.
The arithmetic mean filter causes a certain amount of blurring (proportional to the
•window
Can besize)
usedtotothereduce
image,noise
thereby
of reducing
differentthe effects
types, butofworks
noise. best
It can
forbeGaussian,
used to reduce
uniform,
or Erlang
noise noise.types, but works best for Gaussian, uniform, or Erlang noise.
of different

Geometric mean filter The geometric mean filter is a variation of the arithmetic mean
filter, and is primarily used on images with Gaussian noise. This filter is known to retain
image detail better than the arithmetic mean filter. Its mathematical formulation is as
follows.
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 window size) to the image, thereby reducing the effects of noise. It can be used to reduce
noise of different types, but works best for Gaussian, uniform, or Erlang noise.

Noise reduction
Geometric mean filter The– Mean
geometric meanfilters
filter is a variation of the arithmetic mean
filter, and is primarily used on images with Gaussian noise. This filter is known to retain
image detail better than the arithmetic mean filter. Its mathematical formulation is as
Geometric mean filter
• follows.
2 3 mn
1

Y
fˆ(x, y) = 4 g(r, c)5 (12.20)
(r,c)2W

Harmonic mean filter The harmonic mean filter is yet another variation of the arith-
metic
• a mean filter,
variation ofand
theisarithmetic
useful formean
imagesfilter
with Gaussian or salt noise. Black pixels (pepper
noise) are not filtered. The filter’s mathematical formulation is as follows.
• primarily used on images with Gaussian noise mn
ˆ
f (x, y) = P (12.21)
1
• retains image detail better than the arithmetic(r,c)2W mean
g(r,c) filter

Contra-harmonic mean filter The contra-harmonic mean filter is another variation

of the arithmetic mean filter, and is primarily used for filtering salt or pepper noise (but
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Y
fˆ(x, y) = 4 g(r, c)5 (12.20)

Noise reduction – Mean filters

(r,c)2W

Harmonic mean filter The harmonic mean filter is yet another variation of the arith-
metic mean filter, and is useful for images with Gaussian or salt noise. Black pixels (pepper
Harmonic
noise)•are mean
not filtered. Thefilter
filter’s mathematical formulation is as follows.
mn
fˆ(x, y) = P 1 (12.21)
(r,c)2W g(r,c)

Contra-harmonic mean filter The contra-harmonic mean filter is another variation

of the arithmetic
• anothermean filter,
variation andarithmetic
of the is primarily
meanused
filter for filtering salt or pepper noise (but
not both). Images with salt noise can be filtered using negative values of R, where pepper
noise can •beuseful for images
filtered with Gaussian
using positive or salt
values of R.noise
The filter’s mathematical formulation is:
P R+1
• black pixels (pepper noise) are not filtered g(r, c)
ˆ (r,c)2W
f (x, y) = P R
(12.22)
(r,c)2W g(r, c)
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Noise reduction – Order-statistic filters
The median filter The most popular and useful of the rank filters is the median filter. It
works by selecting the middle pixel value from the ordered set of values within the m ⇥ n
neighborhood (W ) around the reference pixel. If mn is an even number (which is not
common), the arithmetic average of the two values closest to the middle of the ordered set
• Median
is used instead. filter
Mathematically:
fˆ(x, y) = median {g(r, c)|(r, c) 2 W } (12.23)
There have been many variants, extensions, and optimized implementations of the
• Most
median filter popular
proposed andliterature.
in the useful ofPlease
the rank filters.
refer to the “Learn more about it” section at
the end of the chapter for useful pointers.
• It works by selecting the middle pixel value from the ordered set of values within the
The min and m ×max filters The(W)
n neighborhood minaround
and max thefilters also work
reference pixel.on a ranked set of pixel
values. Contrary to the
• If mn is anmedian filter(not
even number – which replaces
common), the reference
the arithmetic averagepixel
of thewith the median
two values closest to the
middle
of the ordered set – the of the
minordered
filter, set
alsois used
knowninstead.
as the 0th percentile filter, replaces it with
the lowest value, instead.
• Many variants, extensions, and optimized implementations in the literature.
Mathematically:
fˆ(x, y) = min {g(r, c)|(r, c) 2 W } (12.24)
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
Similarly, the the max filter, also known as the 100th percentile filter, replaces the
 There
There have
have been
been many
many variants, extensions, and
variants, extensions, and optimized
optimizedimplementations
implementationsofoftheth
median
medianfilter
filterproposed
proposed inin the literature. Pleaserefer
literature. Please refertotothe
the“Learn
“Learnmore
moreabout
aboutit”it” section
section at a
Noise
the
theend reduction
endof
ofthe
thechapter –forOrder-statistic
chapter for useful pointers.
pointers. filters
The
Themin minand
and max
max filters
filters The min min and
and max
maxfilters
filtersalso
alsowork
workonona aranked
rankedsetsetofof pixe
pixel
values.
values. Contrary
• MinContrary
and Max to
to the
the median filter –– which
filters whichreplaces
replacesthe thereference
referencepixel
pixelwith
withthethe median
median
ofofthe
theordered
ordered set
set –– the
the min filter, also
also known
known as asthe
the0th0thpercentile
percentilefilter,
filter,replaces
replacesit it with
with
thelowest
the lowest value,
value,
• Min: instead.
instead.
replaces the reference pixel with the lowest value of the ordered set
Mathematically:
Mathematically:
fˆ(x, y) =
= min
min{g(r, c)|(r,c)c)22WW}}
{g(r,c)|(r, (12.24)
(12.24
Similarly, the
Similarly, the the
the max
max filter, also
also known
known as as the
the 100th
100thpercentile
percentilefilter,
filter,replaces
replacestheth
referencepixel
reference pixelwithin
within the
the window with the highest value, i.e:
• Max: replaces the reference pixel with the highest value ofi.e:
with the highest value, the ordered set

ffˆˆ(x,
(x, y) = max
y) = max{g(r, c)|(r,c)c)22WW}}
{g(r,c)|(r, (12.25)
(12.25
Themin
The minfilter
filterisis useful
useful for
for reduction
reduction of
of salt
saltnoise
noisewhereas
whereasthe
themax
maxfilter
filtercan
canhelp remove
help remov
peppernoise.
pepper noise.
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 The min filter is useful for reduction of salt noise whereas the max filter can help remove
pepper noise.
Noise reduction – Order-statistic filters
The midpoint filter The midpoint filter calculates the average of the highest and lowest
pixel values within a window, thereby combining order statistics and averaging into one
filter. It is used to reduce Gaussian and uniform noise in images.
—Midpoint filter
Mathematically:

ˆ 1
f (x, y) = [max {g(r, c)|(r, c) 2 W } + min {g(r, c)|(r, c) 2 W }] (12.26)
2

The alpha-trimmed mean filter The alpha-trimmed filter uses another combination
of order — Calculates
statistics and the average
averaging, of the inhighest andan
this case lowest pixelof
average values withinvalues
the pixel a window,
closest to the
thereby combining order statistics and averaging into one filter.
median, after the D lowest and the D highest values in an ordered set have been excluded.
The rationale behind this filter is to allow its user to control its behavior by specifying
— Used to reduce Gaussian and uniform noise.
the parameter D: for D = 0 the filter behaves like a regular arithmetic mean filter; for
D = (mn 1)/2, it is equivalent to the median filter. It is used in cases where the image
has been corrupted by more than one type of noise, e.g., salt-and-pepper (where the median
Source: filter performs
Marques (2011) well)
“Practical Image and Videoand Gaussian
Processing (where the arithmetic mean filter shows satisfactory
Using MATLAB”, Wiley.
 The rationale behind this filter is to allow its user to control its behavior by specifying
the parameter D: for D = 0 the filter behaves like a regular arithmetic mean filter; for
Noise reduction – Alpha-trimmed mean filter
D = (mn 1)/2, it is equivalent to the median filter. It is used in cases where the image
has been corrupted by more than one type of noise, e.g., salt-and-pepper (where the median
filter performs well) and Gaussian (where the arithmetic mean filter shows satisfactory
performance).
• Alpha-trimmed mean
The mathematical filterof the alpha-trimmed filter is as follows:
description
1 X
fˆ(x, y) = g(r, c) (12.27)
mn 2D
(r,c)2W

• Uses another
where combination
D is the of order
number of pixel statistics
values excludedand averaging,
at each in this
end of the caseset,
ordered an which
average of
therange
can pixelfrom
values
0 toclosest
(mn to the median, after the D lowest and the D highest values in
1)/2.
an ordered set have been excluded.

• Rationale: to allow its user to control its behavior by specifying the parameter D:
• for D = 0: the filter behaves like a regular arithmetic mean filter;
• for D =(mn-1)/2: it is equivalent to the median filter.

• Used in cases where the image has been corrupted by more than one type of noise,
e.g., salt-and-pepper and Gaussian.
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Figure 12.9 shows an example of an image corrupted by periodic noise, the noise
Noise reduction – frequency domain
component, and its Fourier spectrum. The bright dots in the spectrum indicate four
pairs of impulse functions, each pair corresponding to a sinusoidal noise source.

• Bandreject filter
12.4.2 Bandreject filter
• attenuates frequency components within a certain range (the stopband of the
filter)filter,
A bandreject whileasleaving all other
its name frequency
suggests, components
attenuates untouched
frequency (or amplifying
components within a certain
them by a certain gain).
range (the stopband of the filter) while leaving all other frequency components untouched
(or amplifying them by a certain gain). The mathematical formulation for an ideal ban-
Ideal bandreject filter
dreject •filter is:
8
< 1 if D(u, v) < D0 W 2
i
Hbr (u, v) = 0 if D0 W
2  D(u, v)  D 0 + W
2
(12.28)
:
1 if D(u, v) > D0 + W 2
where D(u, v) is the distance from the origin of the frequency spectrum, W is the width
of the band, and D0 is the radius of the circle-shaped band.
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 component; (c) the Fourier spectrum of the noise component (bright dots were enl
component; (c) the Fourier spectrum of the noise component (bright dots were
purposes).
Noise reduction – frequency domain
purposes).

A Butterworth
• Bandreject filter bandreject filter of order n is described mathematically
A Butterworth bandreject filter of order n is described mathematical
b 1
• Butterworth bandreject filter 1 Hbr (u, v)
i2n = h
b
Hbr (u, v) = 1 + h DD(u,v)W
2 (u,v) D 2 i2n
D(u,v)W0
1+
whereas the transfer function for a GaussianDbandreject
2 (u,v) D 2
filter is given b 0

whereas the transfer function for a Gaussian bandreject

h 2 2
filter
i2 is given
1 D (u,v) D0
g
H (u, v) = 1 e
2
h D(u,v)W
2
i
2 2
br 1 D (u,v) D
0
• Gaussian bandreject filter g 2 D(u,v)W
Hbr (u, v) =1 e
EXAMPLE 12.7

EXAMPLE 12.7 shows an example of an ideal bandreject filter (of radius

Figure 12.10
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
en successfully removed, the resulting image shows an undesirable ringing effect
a result of the sharp transition between passband and stopband in the bandreject
er. Noise reduction – frequency domain
• Bandpassfilter
3 Bandpass filter

• Allows
ndpass filter allows certain frequencies
certain (within its
frequencies passband)
(within its topassband)
be preserved
towhile
be preserved wh
attenuating all others.
ating all others. It is, in effect, the opposite of a bandreject filter (Section 12.4.
s mathematical
• It is, informulation can beofsimply
effect, the opposite stated
a bandreject as:
filter:

Hbp (u, v) = 1 Hbr (u, v) (12.3

ere Hbr (u, v) is the transfer function of a bandreject filter.
plying Eq.( 12.31) to Equations ( 12.28) through ( 12.30) leads to the mathemati
lations for the different types of bandpassfilter, as follows:
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 and its mathematical formulation can be simply stated as:

Hbp (u, v) = 1 Hbr (u, v) (12.31)

Noise
where H reduction
(c)
– frequency
(u, v) is the transfer function
br of a bandreject filter. domain
(c) 12.31) to Equations ( 12.28) through ( 12.30)
Applying Eq.( (d) leads to the mathematical
(d)

formulations
Example offorusing
re 12.10 —Bandpass the different
a types offilter
bandreject bandpassfilter,
to reduce as follows:noise: (a) noisy image; (b)
periodic
filter (BPF)
y Figure
image spectrum (the eight
Ideal Example
12.10 BPF spotsa corresponding
of using to reduce
bandreject filter to the noise havenoise:
periodic been made brighter
(a) noisy image;and(b) bigger
isualization
noisy image purposes);
spectrum (the(c) thespots
eight Fourier8spectrumto
corresponding ofthe
thev)image
noise haveafter
W applying
been the bandreject
made brighter and bigger filter;
< 0 if D(u, <D
esulting image. — Ideal
for visualization purposes);H(c)
i the Fourier spectrum of the image
(u, v) = 1 if D0 W
0
after 2
 D(u, v)  D0the
applying + bandreject
W filter;
(12.32)
bp 2 2
(d) resulting image. :
0 if D(u, v) > D0 + W
2

terworth BPF
Butterworth BPF h i
— Butterworth h D(u,v)W i2n
D(u,v)W
2n

bb (u,v) D2D02
D22(u,v)
HH (u,
bpbp (u,v)
v) =
=
D
hh
0
i2ni2n (12.33)
(12.33)
D(u,v)W
D(u,v)W
1+
+ 2 (u,v)D 2D 2
2 (u,v)
DD 0 0

Gaussian
ussian BPFBPF hh i i
— Gaussian 2
1 DD(u,v)
2 2
2 (u,v)D0D 2 2
g 21 D(u,v)W 0
H
g bp (u, v) = e 2 D(u,v)W (12.34)
Hbp (u, v) = e (12.34)
12.4.4 Notch filter
4.4 Notch filter
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Noise reduction – frequency domain
• Notch filter
• Special kind of frequency domain filter that attenuates (or
allows) frequencies within a neighborhood around a center
frequency.
• Due to the symmetry property of the Fourier transform, the spectrum
of notch filters shows symmetric pairs around the origin (except for a
notch filter located at the origin, of course).

Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
 Noise reduction – frequency domain
IMAGE DEBLURRING TECHNIQUES 257
• Ideal notch filter
⇢
i 0 if D1 (u, v) < D0 or D2 (u, v) < D0
Hnr (u, v) = (12.35)
1 otherwise
where:
⇥ 2 2
⇤1/2
D1 (u, v) = (u M/2 u0 ) + (v N/2 v0 ) ) (12.36)
and
⇥ 2 2
⇤1/2
D2 (u, v) = (u M/2 + u0 ) + (v N/2 + v0 ) ) (12.37)
It has been assumed that the spectrum has been shifted by (M/2, N/2) 3 and that the
values of (u0 , v0 ) are relative to the shifted center.
Source: Marques (2011) “Practical A Butterworth
Image notch
and Video Processing filterWiley.
Using MATLAB”, of order n can be mathematically described as:
 D1 (u, v) = (u M/2 u0 ) + (v N/2 v0 ) ) (12.36)
and

Noise reduction
D (u, v) = (u– M/2
frequency
2 domain
⇥
+ u ) + (v N/2 +v ) ) 0
2
0
2
⇤1/2
(12.37)
It has been assumed that the spectrum has been shifted by (M/2, N/2) 3 and that the
• values of (u0 , vnotch
Butterworth 0 ) are relative
filter to the shifted center. IMAGE DEBLURRING TECHNIQUES 257
A Butterworth notch filter of order n can be mathematically described as:

b ⇢ 1
Hnr (u, v) = h in (12.38)
i 0 if D (u, v) D<
1 + 1 D1 (u,v)D
2
D 0 or D 2 (u, v) < D 0
Hnr (u, v) = 0
(12.35)
1 otherwise 2 (u,v)

where D 1 (u, v) and D2 (u, v) are given by equations ( 12.36) and ( 12.37), respectively.
where:
A Gaussian notch filter is mathematically described by the equation:
⇥ ⇤
2 1/2
h 2
D1 (u, v) = (u M/2 u0 ) + (v N/2 v0 ) ) i (12.36)
1 D1 (u,v)D2 (u,v)
g 2 D2
and Hnr (u, v) = 1 e 0 (12.39)
To convert a notch filter that rejects ⇥ certain frequencies 2 into one that allows⇤ those same
2 1/2
frequencies to pass one D 2 (u,
has to v) = (u
simply M/2 + u0 ) + (v N/2 + v0 ) )
compute: (12.37)
It has been assumed that the spectrum has been shifted by (M/2, N/2) 3 and that the
values of (u0 , v0 ) are relative Hnp (u, to v) the=shifted
1 Hcenter.
nr (u, v) (12.40)
A Butterworth
Source: Marques (2011) “Practical Image
where H and Video Processingnotch
(u, v) is theUsing filterfunction
MATLAB”,
transfer of orderofn the
Wiley. cannotch
be mathematically
pass filter thatdescribed as: to the
corresponds
 b 1
Hnr (u, v) = h in (12.38)
D02
Noise reduction – frequency domain 1+ D1 (u,v)D2 (u,v)

where D1 (u, v) and D2 (u, v) are given by equations ( 12.36) and ( 12.37), respectively.
A •Gaussian
Gaussiannotch filter
notch is mathematically described byIMAGE
filter the equation:
DEBLURRING TECHNIQUES 257
h i
1 D1 (u,v)D2 (u,v)
D2
(u,⇢v)0= 1 if De1 (u, v) < D0 or D2 (u, v) < D0
g 2

i
Hnr 0 (12.39)
Hnr (u, v) = (12.35)
1 otherwise
To convert a notch filter that rejects certain frequencies into one that allows those same
where:
frequencies to pass one has to simply compute:
⇥ 2 2
⇤1/2
D1 (u, v) = (u M/2 u0 ) + (v N/2 v0 ) ) (12.36)
Hnp (u, v) = 1 Hnr (u, v) (12.40)
and
where Hnp (u, v) is the transfer function
⇥ of the notch
2
pass filter that
⇤ corresponds to the
2 1/2
D2 (u, v) = (u M/2 + u0 ) + (v N/2 + v0 ) ) (12.37)
notch reject filter whose transfer function is Hnr (u, v).
It has been assumed that the spectrum has been shifted by (M/2, N/2) 3 and that the
values of (u0 , v0 ) are relative to the shifted center.
A Butterworth
Source: Marques (2011) “Practical Image and Video Processingnotch filterWiley.
Using MATLAB”, of order n can be mathematically described as:
 1+ 0
D1 (u,v)D2 (u,v)

here D1 (u, v) and D2 (u, v) are given by equations ( 12.36) and ( 12.37), respectively.
Noise reduction – frequency domain
Gaussian notch filter is mathematically described by the equation:
h i
1 D1 (u,v)D2 (u,v)

To convert a nr
notch
(u, v)filter
= 1 that
e rejects certain frequencies into one D2
g 2
H (12.39) 0

o convertthat allows
a notch filter those same
that rejects frequencies
certain tointo
frequencies passoneone
thathas to those
allows simplysame
encies tocompute:
pass one has to simply compute:

Hnp (u, v) = 1 Hnr (u, v) (12.40)

here Hnp (u, v) is the transfer function of the notch pass filter that corresponds to the
h reject filter whose transfer function is Hnr (u, v).
where Hnp(u, v) is the transfer function of the notch pass filter that
corresponds to the notch reject filter whose transfer function is Hnr
IMAGE DEBLURRING TECHNIQUES
(u, v).
goal of image deblurring techniques is to process an image that has been subject to
ing caused, for example, by camera motion during image capture or poor focusing of
Source: Marques (2011) “Practical Image and Video Processing Using MATLAB”, Wiley.
Image denoising using
MATLAB
 Image denoising using MATLAB: useful functions

• imnoise
• https://www.mathworks.com/help/images/ref/imnoise.html
• Used to add noise to an image
• Noise types: Gaussian, Poisson, salt and pepper, multiplicative (speckle)

• imfilter
• https://www.mathworks.com/help/images/ref/imfilter.html
• N-D filtering of multidimensional images with user-defined filter mask (kernel), which
can be defined manually or with the help of the fspecial function

• fspecial
• https://www.mathworks.com/help/images/ref/fspecial.html
• Predefined 2D filter kernels, e.g.: average, disk, gaussian

Source: https://www.mathworks.com/help/images/noise-removal.html
 Image denoising using MATLAB: useful functions

• medfilt2
• https://www.mathworks.com/help/images/ref/medfilt2.html
• Performs median filtering on an image

• ordfilt2
• https://www.mathworks.com/help/images/ref/ordfilt2.html
• Performs 2D order-statistic filtering (e.g., max, min) on an image

• wiener2
• https://www.mathworks.com/help/images/ref/wiener2.html
• Applies a pixel-wise adaptive low-pass Wiener filter to an image

Source: https://www.mathworks.com/help/images/noise-removal.html
 Image denoising using MATLAB: useful functions

• imdiffusefilt
• https://www.mathworks.com/help/images/ref/imdiffusefilt.html
• Applies anisotropic diffusion filtering to an image

• imnlmfilt
• hhttps://www.mathworks.com/help/images/ref/imnlmfilt.html
• Applies a non-local means-based filter to an image

• locallapfilt
• https://www.mathworks.com/help/images/ref/locallapfilt.html
• Filters an image with an edge-aware, fast local Laplacian filter.

Source: https://www.mathworks.com/help/images/noise-removal.html
 Image denoising using MATLAB: useful functions

• imgaussfilt
• https://www.mathworks.com/help/images/ref/imgaussfilt.html
• Filters image with a 2-D Gaussian smoothing kernel

• imbilatfilt
• https://www.mathworks.com/help/images/ref/imbilatfilt.html
• Applies an edge-preserving Gaussian bilateral filter to an image

Source: https://www.mathworks.com/help/images/noise-removal.html
Image denoising using
deep learning in
MATLAB
 Image denoising using deep learning in MATLAB

3 main options:

1. Use the built-in pretrained denoising neural network, (DnCNN)

2. Train your own denoising CNN using built-in layers
3. Use an autoencoder.

Reference: https://www.mathworks.com/help/images/train-and-apply-denoising-neural-networks.html
 Image denoising using deep learning in MATLAB

• OPTION 1 (Simplest and fastest solution): use the built-in pretrained

denoising neural network, DnCNN.

• Limitations:
1. Noise removal works only with 2D monochrome (grayscale) images.
• For images with multiple color channels, or 3D images, noise removal must be done by
treating each channel or plane separately.

2. The network recognizes only Gaussian noise, with a limited range of

standard deviation.

Reference: https://www.mathworks.com/help/images/train-and-apply-denoising-neural-networks.html
 Image denoising using pretrained DnCNN in MATLAB

• Steps:
1. To load the pretrained DnCNN network, use the denoisingNetwork function.
2. Pass the DnCNN network and a noisy 2D single-channel image
to denoiseImage.

Reference: https://www.mathworks.com/help/images/train-and-apply-denoising-neural-networks.html
 Image denoising using pretrained DnCNN in MATLAB

• Example code
(for grayscale
images):

Reference: https://www.mathworks.com/help/images/train-and-apply-denoising-neural-networks.html
 Color image denoising using pretrained DnCNN in MATLAB

In practice, image color channels frequently have correlated noise.

To remove correlated image noise, follow these steps:

1. Load color image into the workspace and convert the data to double.
2. Add zero-mean Gaussian white noise with a variance of 0.01 to the image.
3. Convert the RGB image to a color space with a luminance channel, such as the L*a*b* color
space
4. Load the pretrained DnCNN network.
5. Use the DnCNN network to remove noise from the luminance channel only.
6. Convert the denoised image back to the RGB color space.
7. Display images (original, noisy, denoised).
8. Compute figures of merit (PSNR, SSIM, etc.)

Reference: https://www.mathworks.com/help/images/remove-noise-from-color-image-using-pretrained-neural-network.html
 Image denoising using deep learning in MATLAB

• OPTION 2: Build and

train your own
denoising neural
network

Reference: https://www.mathworks.com/help/images/train-and-apply-denoising-neural-networks.html
 Image denoising using deep learning in MATLAB

Building and (re-)training your own network

Steps:
1. Create an ImageDatastore object that stores pristine images.
2. Create a denoisingImageDatastore object that generates noisy training
data from the pristine images.
3. Get the predefined denoising layers using the dnCNNLayers function.
4. Define training options using the trainingOptions function.
5. Train the network, specifying the denoising image datastore as the data
source for trainNetwork.
6. After you have trained the network, pass the network and a noisy
grayscale image to denoiseImage.

Reference: https://www.mathworks.com/help/images/train-and-apply-denoising-neural-networks.html
 Image denoising using deep learning in MATLAB

OPTION 3: Use a convolutional autoencoder.

• Convolutional autoencoders consist of two stages: an encoder and a
decoder.

• The encoder compresses the original input image into a latent representation
that is smaller in width and height, but deeper in the sense that there are many
feature maps per spatial location than the original input image.
• The compressed latent representation loses some amount of spatial resolution in its ability
to recover high frequency features in the original image, but it also learns to not include
noisy artifacts in the encoding of the original image.

• The decoder repeatedly upsamples the encoded signal to move it back to its
original width, height, and number of channels.

Reference: https://www.mathworks.com/help/deeplearning/ug/image-to-image-regression-using-deep-learning.html
 Convolutional autoencoders and image-to-image regression

• Image-to-image regression: the process of estimating new values for

each pixel of an image (provided as input to a deep neural network)
• Architectures include:
• Convolutional autoencoders
• U-nets
• etc.
• Applications include:
• Semantic image
segmentation
• Denoising
• Colorization
• etc.
Reference: https://blogs.mathworks.com/deep-learning/2019/02/21/image-to-image-regression/
 Image denoising using deep learning in MATLAB

• Convolutional autoencoders are convenient for denoising operations!

• Since the encoder removes noise (as part of its job in building a more compact
representation of the input image), the decoded final image has fewer noise
artifacts.

Reference: https://www.mathworks.com/help/deeplearning/ug/image-to-image-regression-using-deep-learning.html | Figure from: https://blog.keras.io/building-autoencoders-in-keras.html

Hands on!
MATLAB Assignment:
Image denoising using deep learning (denoising_dl.mlx)
• Part 1:
• Quick review of classical of image noise types and measures of image quality
• Part 2:
• Simplest workflow for removing image noise from an individual image using a
pre-trained denoising CNN
MATLAB Assignment:
Image denoising using deep learning (denoising_dl.mlx)
• Part 3:
• Complete workflow for
training a denoising CNN
using a datastore of
pristine images and
eventually deploying the
resulting model
MATLAB Assignment:
Image denoising using deep learning (denoising_dl.mlx)
• Part 4 (OPTIONAL):
• Workflow for training a convolutional autoencoder for denoising purposes.