3.

2 Frequency Analysis

EECS 4422/5323 Computer Vision !1 J. Elder

 Outline
❖ Linear Shift-Invariant Systems

❖ The Fourier Transform

❖ The Wiener Filter

EECS 4422/5323 Computer Vision !2 J. Elder

 Outline
❖ Linear Shift-Invariant Systems

❖ The Fourier Transform

❖ The Wiener Filter

EECS 4422/5323 Computer Vision !3 J. Elder

 1D Signal Coding
❖ A 1D signal (e.g., a slice of a luminance image f(x) over horizontal location x) can be
coded as a sequence of values

❖ This can also be viewed as a superposition of shifted and weighted impulses

#104
6

4
f

1
0 20 40 60
x

EECS 4422/5323 Computer Vision !4 J. Elder

 Impulse (Delta) Functions

1
Δ

0 Δ 0 Paul Dirac
Discrete pulse lim δ Δ (x) Dirac delta function 1902 - 1984
Δ→0
⎧ 1 ⎧ ∞ if x = 0
⎪ if 0 ≤ x ≤ Δ δ (x) = ⎨
δ Δ (x) = ⎨ Δ ⎩ 0 otherwise
⎪ 0 otherwise
⎩

EECS 4422/5323 Computer Vision J. Elder

 Representing a Signal with Impulses

= + + + +
... ...

∞ Figure 1: Staircase approximation to a continuous-time signal.

f (x) ! ∑ f ( kΔ )δ ( x − kΔ ) Δ
Δ
k=−∞
Representing signals with impulses. Any signal can be expressed as a sum of scaled and
∞
shifted unit impulses. We begin with the pulse or “staircase” approximation to a continuous
f (x) = lim ∑ f ( kΔ )δ Δ ( x − kΔ ) Δ
signal Δ→0 ,k=−∞
as illustrated in Fig. 1. Conceptually, this is trivial: for each discrete sample of the
original ∞signal, we make a pulse signal. Then we add up all these pulse signals to make up the
= ∫ f ( usignal.
approximate )δ ( x − uEach
) du of these pulse signals can in turn be represented as a standard pulse
scaled by
−∞ the appropriate value and shifted to the appropriate place. In mathematical notation:

= f (x)∗ δ (x)

As we let approach zero, the approximation becomes better and better, and the in the limit
equals . Therefore,

EECS 4422/5323 Computer Vision J. Elder

!6

Also, as , the summation approaches an integral, and the pulse approaches the unit impulse:
 Representing a Filter with Impulses
❖ Of course we can also code a filter h(x) using impulses.

❖ This is why we refer to h(x) as the impulse response function of the filter

h(x) = h(x)∗ δ (x)

= + + + +
... ...

Figure 1: Staircase approximation to a continuous-time signal.

Representing signals with impulses. Any signal can be expressed as a sum of scaled and
shifted unit impulses. We begin with the pulse or “staircase” approximation to a continuous
signal , as illustrated in Fig. 1. Conceptually, this is trivial: for each discrete sample of the
original signal, we make a pulse signal. Then we add up all these pulse signals to make up the
approximate signal. Each of these pulse signals can in turn be represented as a standard pulse
scaled by the appropriate value and shifted to the appropriate place. In mathematical notation:

EECS 4422/5323 Computer Vision J. Elder

!7
 Alternative Linear Codes
❖ The impulse code is not the only way to code a signal or a filter!

❖ In particular, there are many alternative linear codes, including

๏ Fourier transforms

๏ Discrete coding transforms (DCTs)

๏ Wavelet transforms

❖ These linear codes are simply linear transformations of the impulse code.

❖ We begin with the Fourier code, which arises naturally from linear systems theory.

EECS 4422/5323 Computer Vision !8 J. Elder

 What is a linear system?
❖ A system h is linear if it satisfies the principle of superposition:

Additivity

h (α f1 + β f2 ) = α h ( f1 ) + β h ( f2 )

Homogeneity

EECS 4422/5323 Computer Vision J. Elder

!9
 Shift Invariance
❖ A system h is shift-invariant if a shift in the input produces an identical shift in the
output:

g(x) = h ( f (x)) → g(x − u) = h ( f (x − u))

EECS 4422/5323 Computer Vision J. Elder

!10
 The Impulse Response Function
❖ The output of a linear shift-invariant system at x is a weighted sum of the input, where
the weights are fixed relative to x.

❖ These filter weights are simply the reversed impulse response function.

h(x) = h(x)∗ δ (x)

past present future

δ (s) 0 0 0 1 0 0 0 0 0 input (impulse)

h(x − s) 1/8 1/4 1/2 weights

h(x) 0 0 0 1/2 1/4 1/8 0 0 0 output (impulse response)

0 0 0 1 1 1 1 1 1 input (step)

1/8 1/4 1/2 weights

0 0 0 1/2 3/4 7/8 7/8 7/8 7/8 output (step response)

EECS 4422/5323 Computer Vision J. Elder
!11
 which the major and minor axis orientation and lengths can be computed xusing φ
nvalue analysis.7 Sinusoids
h(x)
stics can then
❖ beWhen
used for
wefurther
inputprocessing,
an impulsee.g.,tofor
a sorting
linear the regions
s(x) by the area
shift-invariant system, we get a complicatedo(x)
nsider the largest regions
output first)
(the or for preliminary
impulse response)matching of regions in different
Figure 3.24 The Fourier Transform as the response of a filter h(x) to an inpu
s(x) = ej!x yielding an output sinusoid o(x) = h(x) ⇤ s(x) = Aej!x+ .
urier transforms
be the input sinusoid whose frequency is f , angular frequency is ! = 2⇡f , and p
❖ However,
3.4 Fourier
3.2, we mentioned when
transforms
that Fourier we input a
analysis could sinusoid,
be used to we get another
analyze the sinusoid of the same frequency,
frequency 133
Note that in this section, we use the variables x and y to denote the spatial coordin
but scaled and shifted in phase.
tics of various filters. In this section, we explain both how Fourier analysis lets us
image, rather than i and j as in the previous sections. This is both because the lett
hese characteristics (orsequivalently, the frequency content
are used for theof imaginary
an image) number
and ohow(the usage depends on whether you are readin
ast Fourier Transform (FFT) lets us perform large-kernel
variables orconvolutions Aliterature) and because it is clearer how to disti
in time that
electrical engineering
dent of the kernel’s size. More comprehensivehorizontal
introductions
(x) to
andFourier
verticaltransforms
(y) components
φ in frequency space. In this section, w
x x
d by Bracewell (1986); Glassner (1995); Oppenheim and Schafer (1996); Oppen-
letter j for the imaginary number, since that is the form more commonly found in
fer, and Buck (1999). processing literature (Bracewell 1986; Oppenheim and Schafer 1996; Oppenheim
n we analyze what a given filter does to high,andmedium, and
h(x)
low frequencies? The
s(x) Buck 1999). o(x)
o simply pass a sinusoid of known frequency through the filter and to observe by
If we convolve the sinusoidal signal s(x) with a filter whose impulse respon
it is attenuated. Let we get another sinusoid of the same frequency but different magnitude A and phas
Figure 3.24 The Fourier Transform as the response of a filter h(x) to an input sinusoid
s(x)==ej!x
s(x) sin(2⇡f x + an
yielding i ) output
= sin(!x + i ) o(x) = h(x) ⇤ s(x)
sinusoid = h(x)
o(x)(3.47)
= Aej!x+ . = A sin(!x + o ),
⇤ s(x)
s can also be computed using Green’s theorem applied to
astheshown
boundary
in pixels
Figure (Yang and To
3.24. Albregtsen
see that
this is the case, remember that a convolut
❖ beThis makes sinusoids a natural code
expressed f ,for
is as linearfrequency
aangular
weighted shift invariant
summation of = systems.
shifted ,input signalsis(3.14)
the input sinusoid whose frequency is ! 2⇡f and phase i . and that the sum
Note that in this section, we usea the
bunch of shifted
variables sinusoids
x and of the same
y to denote frequency
the spatial is just a of
coordinates single
an sinusoid at that f
image, rather than i and j as in The new magnitude
the previous A isThis
sections. called
is the
bothgain or magnitude
because of the
the letters i andfilter,
j while the phase
are used
EECS 4422/5323 Computer imaginary number =
for theVision (theo usage i is called the
depends
!12 onshift or phase.
whether you are reading complex J. Elder
 weighted
igure 3.24.summation
To see that of this
shifted input
is the signals
case, (3.14) and
remember that that the summation
a convolution of
can be
ted sinusoids
weighted of the same
summation frequency
of shifted inputissignals
just a single
(3.14)sinusoid
and thatatthe
that frequency.of8
summation
itude A is called
ted sinusoids thesame
of the gain or magnitude
frequency Complex Sinusoids
of the
is just filter,sinusoid
a single while theatphase difference8
that frequency.
is called
itude A isthe shiftthe
called or gain
phase.or magnitude of the filter, while the phase difference
ore compact
is called
❖ It is often
the notation
convenient
is to
shift or phase.
to work withsinusoid
use the complex-valued complex sinusoids:
more compact notation isj!x to use the complex-valued sinusoid
s(x) = e = cos !x + j sin !x. (3.49)
s(x) = ej!x = cos !x + j sin !x. (3.49)
e can simply write,
e can simply write, j
o(x) = h(x) ⇤ s(x) = Aej!x+ . (3.50)
o(x) =additional
ral (non-linear) transform, s(x) = frequencies
h(x) ⇤ harmonic Aej!x+ .are introduced. This was traditionally
(3.50)
hiles,
ral who insisted
(non-linear) on equipment
transform, with
additional no harmonic
harmonic distortion.
frequencies Now that digital
are introduced. audio
This was has intro-
traditionally
on-free sound,
hiles, who someonaudiophiles
insisted equipmentare buying
with retro tube
no harmonic amplifiersNow
distortion. or digital signalaudio
that digital processors that
has intro-
ortions
on-freebecause
sound, of their
some “warmer sound”.
audiophiles are buying retro tube amplifiers or digital signal processors that
ortions because of their “warmer sound”.

EECS 4422/5323 Computer Vision !13 J. Elder

 Outline
❖ Linear Shift-Invariant Systems

❖ The Fourier Transform

❖ The Wiener Filter

EECS 4422/5323 Computer Vision !14 J. Elder

 Fourier Series
❖ We have already seen that any signal f(x) or filter h(x) can be expressed exactly
as an infinite sum of impulses.

❖ It turns out that any signal can alternatively be expressed exactly as an infinite
sum of sinusoids.

❖ This is known as a Fourier series.

Joseph Fourier
1768 - 1830
❖ For a finite signal f(x) defined on [0, X], we have:
∞
f (x) = ∑ An sin ( 2π nx / X + φn )
n=1

EECS 4422/5323 Computer Vision J. Elder

!15
 Computer
puter Vision:Vision: Algorithms
Algorithms and Applications The Fourier Transform
and Applications (September
(September 3, 2010 draft)
3, 2010 draft)

ransform
rm is simply
❖
is simply a In thea tabulation
limit
tabulation of the magnitude
as X→∞,
of the theand
magnitude andresponse
Fourier
phase phasebecomes
series response
at each at
theeach
Fourier transform.
Computer Vision: Algorithms and Applications (September 3, 2010 draft)
H(!) =H(!)
F = F {h(x)}
{h(x)} = Ae j = Aej ,
, (3.51) (3.51)
The aFourier
ransform is❖simply transform
tabulation of a signal
of the magnitude f(x) or
and phase filter h(x)
response is the response to a complex
at each
oonse
a complex sinusoid
to a complex
sinusoid of at
sinusoid each
frequencyfrequency
of frequency
! passed passed through
! through the filterthe filter h(x).
h(x).
form
pair pair is
is also also
often often
= written
written
H(!) Fas{h(x)}as = Aej , (3.51)

onse to a complex F
sinusoid F
of frequency ! passed through the(3.52)filter h(x).
h(x)
h(x) $ H(!). $ H(!). (3.52)
form pair is also often written as
y,) does
(3.51) does
not H
givenot ω
an(give
) isancalled
actual actualthe
formula fortransfer
formula function
for computing
computing ofthe
the Fourierthe filter h(x).
Fourier
transform. transform.
F
ae,recipe,
i.e., convolve the filter
i.e., convolve thewith
h(x) $ a sinusoid,
filter observe observe
with a sinusoid,
H(!). the magnitude and(3.52)and
the magnitude
Joseph Fourier
unately,
at. closed
Fortunately, form
closedequations
❖ Continuous for the Fourier
form domain:
equations transform
for the Fourier exist both
transform in both in
exist 1768 - 1830
y, (3.51) does not give an actual formula for computing the Fourier transform.
omain, Z 1 Z 1
recipe, i.e., convolve the filterj!x
with a sinusoid, observe the magnitude and
H(!) = = dx, j!x dx, (3.53)
t. Fortunately, closed1form equations for the Fourier transform exist both(3.53)
H(!) h(x)e h(x)e
in
1
omain, Z 1
ain,
e domain,
H(!) = h(x)e j!x dx, (3.53)
N 1
1 X11 N
X1 j 2⇡kx
H(k) =H(k) = h(x)e h(x)e N , j 2⇡kx
N , (3.54) (3.54)
e domain, N x=0 N
x=0
N
X1
fngth
the signal 1analysis. These
of the or region
signal
H(k) =or of
region of analysis.
h(x)e
formulas
j 2⇡kx
These
N
apply both to filters, filters,
, formulas apply both to (3.54)
dnals or images,
to signals such asN
or images, s(x)
such
x=0or s(x)
as g(x).or g(x). space
f the Fourier
form of the transform
Fourier (3.54) is(3.54)
transform known is as the Discrete
known as the Fourier Trans-
Discrete Fourier Trans-
ngth of the signal or region of analysis. These formulas apply both to filters,
whileEECS
etothat (3.54) can
4422/5323
while beComputer
(3.54) evaluated
can be for any for
Vision
evaluated valueany ofvalue
k, it only
of k,makes
it !1sense
only 6makes sense J. Elder
signals or images, such as s(x) or g(x).
 Fourier Transforms

Signal Amplitude of Fourier Transform

x(t) 1
|X(f)|

cos 8π x 0 0
1

-1
0 1 0
0 4 8

x(t) 1 |X(f)|
1
cos16π x 0

-1 0
0 1 0 4 8
Time (sec) Frequency (Hz)

EECS 4422/5323 Computer Vision J. Elder

!17
 Amplitude & Phase

F [ cos 6π x ] F [ sin 6π x ]
cosine sine

real part
-8 0 8 -8 0 8

imaginary part
-8 0 8 -8 0 8

amplitude
-8 0 8 -8 0 8
Frequency (Hz) Frequency (Hz)

phase 0 π/2

Figure 10: Fourier transforms of sine and cosine signals. The amplitudes are the same, but the
phases are different.

Do not be put off by the negative frequencies in the plots. The equations for theJ.Fourie
EECS 4422/5323 Computer Vision Elder
!18
 End of Lecture
Oct 3, 2018

EECS 4422/5323 Computer Vision !19 J. Elder

 H(!) = h(x)e j!x
dx, (3.53)
1

omain, The Discrete Fourier Transform (DFT)

N 1
1 X j 2⇡kx
H(k) = h(x)e N , (3.54)
N x=0

h of the signal or region of analysis. These formulas apply both to filters,

where N is the number of samples in the signal.
signals or images, such as s(x) or g(x).
m of the Fourier transform (3.54) is known as the Discrete Fourier Trans-
❖ Interpreting frequencies:
hat while (3.54) can be evaluated for any value of k, it only makes sense
nge k 2 [ N2 , ๏ 2 ]. If
N
This
N isisodd:
because larger values of k alias with lower
ce provide no additional information, as explained in the discussion on
k = 0 ⇔ DC value (mean)
.3.1.
2 k = 1 ⇔ 1 cycle per image, 1/ N cycles per pixel
e DFT takes O(N ) operations (multiply-adds) to evaluate. Fortunately,
k = 2 ⇔ 2 cycles per image, 2 / N cycles per pixel
algorithm called the Fast Fourier Transform (FFT), which requires only
!
ions (Bracewell 1986; Oppenheim, Schafer, and Buck 1999). We do not
1⎛ 1⎞
of the algorithm here, k =except
( N − 1) to
/2⇔ say( Nthat
− 1)it/ 2involves image, of ⎜log
a series
cycles per 1− 2 N⎟ cycles per pixel (Nyquist limit)
2⎝ N⎠
tage performs small 2 ⇥ 2 transforms (matrix multiplications with known
ed by some semi-global 1⎛ 1⎞
k = (permutations.
N + 1) / 2 = N − ((You
N − 1)will/2⇔often
−( Nsee
− 1)the term but-
/ 2 cycles per image, − ⎜ 1− ⎟ cycles per pixel (Nyquist limit)
2⎝ N⎠
se stages because of the pictorial shape of the signal processing graphs
! be found in most numerical and signal processing
ntations for the FFT can
k = N − 2 ⇔ −2 cycles per image, − 2 / N cycles per pixel
= N − 1 ⇔ −1
ve defined the Fourierk transform, cycles
what areper
some of its− 1/
image, N cyclesand
properties per how
pixel
able 3.1 lists a number of useful properties, which we describe in a little

EECS 4422/5323 Computer Vision !20 J. Elder

 H(!) = h(x)e j!x
dx, (3.53)
1

omain, The Discrete Fourier Transform (DFT)

N 1
1 X j 2⇡kx
H(k) = h(x)e N , (3.54)
N x=0

h of the signal or region of analysis. These formulas apply both to filters,

where N is the number of samples in the signal.
signals or images, such as s(x) or g(x).
m of the Fourier transform (3.54) is known as the Discrete Fourier Trans-
❖ Interpreting frequencies:
hat while (3.54) can be evaluated for any value of k, it only makes sense
nge k 2 [ N2 , ๏ 2 ]. If
N
This
N isiseven:
because larger values of k alias with lower
ce provide no additional information, as explained in the discussion on
k = 0 ⇔ DC value (mean)
.3.1.
k = 1 ⇔ 1 cycle per image, 1/ N cycles per pixel
e DFT takes O(N )koperations
2
(multiply-adds) to evaluate. Fortunately,
= 2 ⇔ 2 cycles per image, 2 / N cycles per pixel
algorithm called the Fast Fourier Transform (FFT), which requires only
!
ions (Bracewell 1986; Oppenheim, Schafer, and Buck 1999). We do not
1 1
k = N / 2 − 1 ⇔ N / 2 − 1 cycles per
of the algorithm here, except to say that it involves a seriesimage, −of log
cycles
N per pixel
2 N 2
tage performs small 2 ⇥ 2 transforms (matrix multiplications 1 with known
k = N / 2 ⇔ N / 2 cycles per image, cycles per pixel (Nyquist limit)
ed by some semi-global permutations. (You will often2see the term but-
se stages because of the pictorial shape of the signal processing graphs ⎛1 1⎞
k = N / 2 + 1 = N − ( N / 2 − 1) ⇔ −( N / 2 − 1) cycles per image, − ⎜ − ⎟ cycles per pixel
ntations for the FFT can be found in most numerical and signal processing ⎝2 N⎠
!
ve defined the Fourierk transform,
= N − 2 ⇔ −2what are per
cycles some of its−properties
image, and
2 / N cycles perhow
pixel
able 3.1 lists a number
k = of
N− useful
1 ⇔ −1properties,
cycles per which
image, we
− 1/describe
N cyclesinper
a little
pixel

EECS 4422/5323 Computer Vision !21 J. Elder

 H(!) = h(x)e j!x
dx, (3.53)
1

omain, The Discrete Fourier Transform (DFT)

N 1
1 X j 2⇡kx
H(k) = h(x)e N , (3.54)
N x=0

h of the signal or region of analysis. These formulas apply both to filters,

where N is the number of samples in the signal.
signals or images, such as s(x) or g(x).
m of the Fourier transform (3.54) is known as the Discrete Fourier Trans-
❖ What is the computational complexity for computing the DFT?
hat while (3.54) can be evaluated for any value of k, it only makes sense
nge k 2 [ N2 , N2 ]. This is because larger values of k alias with lower
๏ Naïve: O( )
N 2
ce provide no additional information, as explained in the discussion on
.3.1.
e DFT takes O(N ๏ )Fastoperations
Fourier(multiply-adds) O ( N logFortunately,
to evaluate. N)
2
Transform (FFT):
algorithm called the Fast Fourier Transform (FFT), which requires only
ions (Bracewell 1986; Oppenheim, Schafer, and Buck 1999). We do not
of the algorithm here, except to say that it involves a series of log2 N
tage performs small 2 ⇥ 2 transforms (matrix multiplications with known
ed by some semi-global permutations. (You will often see the term but-
se stages because of the pictorial shape of the signal processing graphs
ntations for the FFT can be found in most numerical and signal processing

ve defined the Fourier transform, what are some of its properties and how
able 3.1 lists a number of useful properties, which we describe in a little

EECS 4422/5323 Computer Vision !22 J. Elder

 The Inverse Fourier Transform
❖ Continuous domain:
∞
1
h(x) =
2π ∫
−∞
H (ω )e jω x dω

❖ Discrete domain:
N /2 2 π kx
1
∑
j
h(x) = H (k)e N
N k==N /2

Joseph Fourier
1768 - 1830

space

EECS 4422/5323 Computer Vision !23 J. Elder

urier transforms Properties of the Fourier Transform

Property Signal Transform

superposition f1 (x) + f2 (x) F1 (!) + F2 (!)
shift f (x x0 ) F (!)e j!x0
reversal f ( x) F ⇤ (!)
convolution f (x) ⇤ h(x) F (!)H(!)
correlation f (x) ⌦ h(x) F (!)H ⇤ (!)
multiplication f (x)h(x) F (!) ⇤ H(!)
differentiation f 0 (x) j!F (!)
domain scaling f (ax) 1/aF (!/a)
real images f (x) = f ⇤ (x) , F (!) = F ( !)
P P
Parseval’s Theorem x [f (x)]2
= ! [F (!)]2

3.1 Some useful properties of Fourier transforms. The original transform pa

= F{f (x)}.
EECS 4422/5323 Computer Vision !24 J. Elder
 3.4 Fourier transforms 137

Fourier Pairs
Name Signal Transform
1.0 1.0

impulse (x)
0.5 0.5

-1.0000 -0.5000
0.0
0.0000 0.5000 1.0000
, 1 -0.5000
0.0
0.0000 0.5000

-0.5 -0.5

1.0 1.0

shifted
(x
0.5 0.5

u) , e j!u
impulse -1.0000 -0.5000
0.0
0.0000 0.5000 1.0000 -0.5000
0.0
0.0000 0.5000
(phase plot)
-0.5 -0.5

1.0 1.0

box(x/a)
0.5 0.5

box filter 0.0

, asinc(a!) 0.0
-1.0000 -0.5000 0.0000 0.5000 1.0000 -0.5000 0.0000 0.5000

-0.5 -0.5

1.0 1.0

tent(x/a) asinc2 (a!)

0.5 0.5

tent -1.0000 -0.5000

0.0
0.0000 0.5000 1.0000
, -0.5000
0.0
0.0000 0.5000

-0.5 -0.5

1.0 1.0

p
G(x; ) 2⇡
G(!; 1
)
0.5 0.5

Gaussian -1.0000 -0.5000

0.0
0.0000 0.5000 1.0000
, -0.5000
0.0
0.0000 0.5000

-0.5 -0.5

1.0 1.0

p
Laplacian 0.5
2
( x4 1
)G(x; ) 2⇡
! 2 G(!; 1
)
0.5

of Gaussian -1.0000 -0.5000

0.0
0.0000 0.5000 1.0000
2 , -0.5000
0.0
0.0000 0.5000

-0.5 -0.5

1.0 1.0

p
cos(!0 x)G(x; ) 2⇡
G(! ± !0 ; 1
)
0.5 0.5

Gabor -1.0000 -0.5000

0.0
0.0000 0.5000 1.0000
, -0.5000
0.0
0.0000 0.5000

-0.5 -0.5

(1 + )
1.5 1.5

unsharp 1.0

(1 + ) (x) p
1.0

mask
0.5

0.0 G(x; ) , 2⇡
G(!; 1
)
0.5

0.0
-1.0000 -0.5000 0.0000 0.5000 1.0000 -0.5000 0.0000 0.5000

-0.5 -0.5

1.0 1.0

windowed rcos(x/(aW ))
(see Figure 3.29)
0.5 0.5

sinc -1.0000 -0.5000

0.0
0.0000 0.5000 1.0000
sinc(x/a) , -0.5000
0.0
0.0000 0.5000

-0.5 -0.5

Table 3.2 Some useful (continuous) Fourier transform pairs: The dashed line in the Fourier
EECS 4422/5323 Computertransform
Vision of the shifted impulse indicates its
!25(linear) phase. All other transforms have zero J. Elder
 3.4 Fourier transforms 139

Fourier transforms of simple filters

Name Kernel Transform Plot

1.0

0.8

0.6

1
1 1 1 3 (1 + 2 cos !)
0.4
1
box-3 3 0.2

0.0
0 0.1 0.2 0.3 0.4 0.5
-0.2

-0.4

1.0

0.8

0.6

1
1 1 1 1 1
0.4
1
box-5 5 5
(1 + 2 cos ! + 2 cos 2!) 0.2

0.0
0 0.1 0.2 0.3 0.4 0.5
-0.2

-0.4

1.0

0.8

0.6

1
1 2 1 2 (1 + cos !)
0.4
1
linear 4 0.2

0.0
0 0.1 0.2 0.3 0.4 0.5
-0.2

-0.4

1.0

0.8

0.6

1
1 4 6 4 1 4 (1 + cos !)2
0.4
1
binomial 16 0.2

0.0
0 0.1 0.2 0.3 0.4 0.5
-0.2

-0.4

1.0

0.8

0.6

1
1 0 1
0.4

Sobel 2 sin ! 0.2

0.0
0 0.1 0.2 0.3 0.4 0.5
-0.2

-0.4

1.0

0.8

0.6

1
1 2 1 2 (1 cos !)
0.4
1
corner 2 0.2

0.0
0 0.1 0.2 0.3 0.4 0.5
-0.2

-0.4

Table 3.3 Fourier transforms of the separable kernels shown in Figure 3.14.
EECS 4422/5323 Computer Vision !26 J. Elder
 Computer Vision: Algorithms and Applications (September 3, 2010 draft)
Computer
Computer Vision: Vision:and
Algorithms Algorithms and (September
Applications Applications3,(September
2010 draft) 3, 2010 draft)
2 Two-dimensional Fourier transforms The 2D Fourier Transform
2 Two-dimensional
mensional Fourier transforms
Fourier transforms
The extension
❖ insights
formulas and we have to 2D images
developed and filters is straightforward.
for one-dimensional signals and their trans-
formulas we andhave
insights we have fordeveloped for one-dimensional signals and their trans-
msinsights
translate directly developed
to two-dimensional one-dimensional
images. Here, signals
insteadandoftheir
just trans-
specifying a hor-
ms translate
rectly directly to two-dimensional images. Here, instead of just specifying a hor-
tal or to two-dimensional
vertical frequency !images. or ! , Here,
we instead
can create of anjust specifying
oriented a
sinusoidhor-of frequency
tal or ❖
vertical The 2D
frequency Fourier
!
x
or transform
!
y
, we can tabulates
create an the amplitude
oriented sinusoid andof phase of sinusoidal gratings for
frequency
l!frequency
y ), !x or !y , we can create an oriented sinusoid of frequency
x y
!y ), all combinations of horizontal and vertical frequency:
s(x, y) = sin(!x x + !y y). (3.62)
s(x, y) = sin(! s(x,xy) + =
! sin(!
y). x x + !y y). (3.62) (3.62)
corresponding two-dimensionalxFouriery transforms are then
gcorresponding
two-dimensional two-dimensional Z 1Fourier
Fourier transforms Z 1aretransforms
then are then
H(!
Z 1x ,Z!y1 ) = Z 1 Z 1h(x, y)e j(!x x+!y y) dx dy, (3.63)
H(!x , !y ) = H(!x , !y ) h(x, = 1 j(!x x+!y y)
y)e
1 xh(x,
j(! x+!yy)e y)
dx dy, dx dy, (3.63) (3.63)
1 1
in the discrete domain, 1 1
indomain,
the discrete domain,
M 1N 1
1 X X kx x+ky y
H(kx , M ky )1=N 1 1 M X 1N X 1h(x, y)e j2⇡
kx x+ky,y
MN (3.64)
H(k X
1 x , ky ) = X M N j2⇡
(3.64)
j2⇡ h(x, y)e , (3.64)
k x x+k y y MN
H(kx , ky ) = h(x, x=0 y=0
y)e ,
M N x=0 y=0M N x=0 y=0
M N

re M and N are the width and height of the image.

re
re M
All of and
the the
width are the
N and
Fourier width
transform
height of and height offrom
properties
the image. the image.
Table 3.1 carry over to two dimensions if
All
eplace
urier oftransform
the
the Fourier transform
scalarproperties
variables x, !,
from properties
x0 and3.1
Table afrom
with
carryTable
their 3.1
over2D carry
tovector over to twoifdimensions
counterparts
two dimensions x = (x, y), if
eplace
alar x , !the
(!variablesy ), scalar
x0x,=!,variables
(xx0 , yand
0 ), aand
x,with =0 and
!,axtheir ,aawith
(ax2D ), and
yvector their
use2D vector
vector
counterparts xcounterparts
inner =products x = (x,ofy),
(x, y), instead
0
=iplications.
(!x , !y ), x0 = (x0 , y0 ), and a = (ax , ay ), and use vector inner products instead of
0 = (x0 , y0 ), and a = (ax , ay ), and use vector inner products instead of
iplications.
3 Wiener filtering
3 Wiener filtering
filtering
le the Fourier transform is a useful tool for analyzing the frequency characteristics of a
kernel or image,
letransform
the Fourier it can also abeuseful
used tool
to analyze the frequency spectrumcharacteristics
of a whole class
is atransform
useful
EECS 4422/5323 toolis for
Computer analyzing
Vision
for
the analyzing
frequencythe frequency
characteristics
!27 of a of a
J. Elder
 Outline
❖ Linear Shift-Invariant Systems

❖ The Fourier Transform

❖ The Wiener Filter

EECS 4422/5323 Computer Vision !28 J. Elder

 Noise
❖ Images formed by a camera or the eye are corrupted by noise.

❖ This noise can often be approximated as a zero-mean, additive and stationary random
process.
f ( x, y ) = g ( x, y ) + n ( x, y )

Optical image

Noisy sensed image Added noise

= +

EECS 4422/5323 Computer Vision !29 J. Elder

 Noise Filtering
f ( x, y ) = g ( x, y ) + n ( x, y )

❖ Denoising is a core problem in image processing.

❖ The linear systems solution to this problem is well understood.

❖ The problem is to find the optimal filter h(x,y) that will maximize the accuracy of the
filtered image in the least squares sense.

❖ By the convolution theorem, this is equivalent to identifying the optimal transfer

(
function H ω x , ω y )
( ) (
h(x, y)∗ f (x, y) ⇔ H ω x , ω y F ω x , ω y )

EECS 4422/5323 Computer Vision !30 J. Elder

 Probabilistic Model
f ( x, y ) = g ( x, y ) + n ( x, y )

❖ To solve this problem, we assume that the optical image g(x,y) and the noise n(x,y)
are both independent, stationary, random processes whose power spectral densities
are known

๏ Power spectral densities:

( ) ( ) (
= E ⎡ F ω x ,ω y ) ⎤
2 2
Pf ω x , ω y = F ω x , ω y
⎢⎣ ⎥⎦

( ) ( ) (
= E ⎡ G ω x ,ω y ) ⎤
2 2
Pg ω x , ω y = G ω x , ω y
⎢⎣ ⎥⎦

( ) ( ) (
= E ⎡ N ω x ,ω y ) ⎤
2 2
Pn ω x , ω y = N ω x , ω y
⎢⎣ ⎥⎦

EECS 4422/5323 Computer Vision !31 J. Elder

 End of Lecture
Oct 15, 2018

EECS 4422/5323 Computer Vision !32 J. Elder

 Power Spectral Density
❖ Natural images tend to be lowpass - most of the energy is in the low spatial
frequencies.

Image Log Fourier Energy

ωy

ωx

g ( x, y ) ( )
2
log G ω x , ω y
EECS 4422/5323 Computer Vision !33 J. Elder
 Noise Spectral Density
❖ In contrast, the expected energy in image noise tends to be more flat (white) across
spatial frequency

Image
Noise Log Fourier
Log Energy
Fourier Energy

ωy

ωx

n ( x, y ) ( )
2
log N ω x , ω y
EECS 4422/5323 Computer Vision !34 J. Elder
 The Wiener Filter
❖ When the frequency distribution of the image energy and the noise energy differ, we
can improve the signal-to-noise ratio (SNR) by boosting the Fourier amplitudes where
the image is strong relative to the noise and attenuating the Fourier amplitudes where
it is relatively weak.

❖ Typically this means a lowpass filter.

❖ The Wiener filter is given by

H (ω , ω ) =
( )
Pg ω x , ω y
=
P (ω , ω )
g x y
, where
P (ω , ω ) P (ω , ω ) + P (ω , ω )
x y
f x y g x y n x y

P (ω , ω ) is the power spectral density of the noisy sensed image

f x y

P (ω , ω ) is the power spectral density of the optical image before noise was added
g x y

P (ω , ω ) is the power spectral density of the noise

n x y

Norbert Wiener 1894 - 1964

EECS 4422/5323 Computer Vision !35 J. Elder
 The Wiener Filter
H (ω , ω ) =
( )
Pg ω x , ω y
=
P (ω , ω )
g x y
, where
P (ω , ω ) P (ω , ω ) + P (ω , ω )
x y
f x y g x y n x y

P (ω , ω ) is the power spectral density of the noisy sensed image

f x y

P (ω , ω ) is the power spectral density of the optical image before noise was added
g x y

P (ω , ω ) is the power spectral density of the noise

n x y

❖ The Wiener filter minimizes the expected mean square error (MSE) of the estimated
image relative to the original image before noise was added.

❖ It is the optimal linear shift-invariant solution to this problem

❖ Note that this optimality is general - it does not depend upon either the noise or the
image being Gaussian. (Be careful with the textbook here.)

EECS 4422/5323 Computer Vision !36 J. Elder

 Estimating the Wiener Filter
H (ω , ω ) =
( )
Pg ω x , ω y
=
P (ω , ω )
g x y
, where
P (ω , ω ) P (ω , ω ) + P (ω , ω )
x y
f x y g x y n x y

P (ω , ω ) is the power spectral density of the noisy sensed image

f x y

P (ω , ω ) is the power spectral density of the optical image before noise was added
g x y

P (ω , ω ) is the power spectral density of the noise

n x y

❖ To calculate the Wiener filter we need to know the power spectral density of the
optical image and of the noise.

❖ Typically, we employ simple approximations.

EECS 4422/5323 Computer Vision !37 J. Elder

 Wiener Filter Example
H (ω , ω ) =
( )
Pg ω x , ω y
=
P (ω , ω ) g x
, where
y

P (ω , ω ) P (ω , ω ) + P (ω , ω )
x y
f x y g x y n x y

P (ω , ω ) is the power spectral density of the noisy sensed image

f x y

P (ω , ω ) is the power spectral density of the optical image before noise was added
g x y

P (ω , ω ) is the power spectral density of the noise

n x y

❖ Assume isotropic spectral densities for both image and noise

๏ Image spectral density is lowpass

α2
(
Pg ω x , ω y ) = 2 , where ω 2 = ω x2 + ω y2
ω
๏ Noise spectral density is white
(
Pn ω x , ω y ) = σ n2 Field, 1987

๏ Then

H (ω x , ω y ) =
(α /ω )
2
1
= 2 , where β = α / σ n is the SNR.
(α / ω )2 + σ n2 1+ (ω / β )

EECS 4422/5323 Computer Vision !38 J. Elder

 Wiener Filter Example
(
H ω x ,ω y ) =
1
1+ (ω / β )
2 , where β = α / σ n is the SNR.

❖ Observe that: -=0.63

-=1.00
⎛β⎞
2

β →∞
(
lim H ω x , ω y = 1 ) (
lim H ω x , ω y
β →0
) =⎜ ⎟
⎝ω ⎠
-=1.58
1
0.8
0.6

h
0.4
0.2
0
-2 0 2
!x (cycles per pixel)

EECS 4422/5323 Computer Vision !39 J. Elder

 Wiener Filter Example
(
H ω x ,ω y ) =
1
1+ (ω / β )
2 , where β = α / σ n is the SNR.

❖ Note that:

๏ h(r) is the inverse Hankel transform of H(ω), not the Fourier transform.

๏ h(r) has no analytic form, but the discrete form of h(x, y) can be determined by taking
the inverse Fourier transform of H(ωx,ωy).
⎛ ⎞
⎜ The Hankel transform of β 2
1 ⎟
e− β r is actually
⎜ 2π
( ) ⎟
3/2

⎝ 1+ (ω / β ) 2
⎠

-=0.63 -=0.63 -=0.63 -=0.63

-=1.00 -=1.00 -=1.00 -=1.00
-=1.58 -=1.58 -=1.58 -=1.58
1 1 0.4 0.4
0.8 0.8 0.3 0.3
0.6 0.6 0.2 0.2
H

h
0.4 0.4 0.1 0.1

0.2 0.2 0 0

0 0 -0.1 -0.1
-2 0 2-2 0 -5 2 0 -5 5 0 5
!x (cycles per ! (cycles per pixel) x (pixels)
pixel)
x
x (pixels)
EECS 4422/5323 Computer Vision !40 J. Elder
 State of the Art
❖ Deep convolutional neural networks

๏ Zhang, K., Zuo, W., Chen, Y., Meng, D., and Zhang, L. (2017). Beyond a Gaussian
denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on
Image Processing, 26(7):3142–3155.

๏ Wang, R. and Tao, D. (2016). Non-local auto-encoder with collaborative stabilization for
image restoration. IEEE Transactions on Image Processing, 25(5):2117–2129.

❖ Nonlinear filtering with learned parameters

๏ Chen, Y. and Pock, T. (2017). Trainable nonlinear reaction diffusion: A flexible

framework for fast and effective image restoration. IEEE Transactions on Pattern Analysis
and Machine Intelligence, 39(6):1256–1272.

EECS 4422/5323 Computer Vision !41 J. Elder

 Outline
❖ Linear Shift-Invariant Systems

❖ The Fourier Transform

❖ The Wiener Filter

EECS 4422/5323 Computer Vision !42 J. Elder