B14 Image Analysis

4 Lectures Michaelmas Term 2014

1 Examples Sheet Prof. A. Zisserman

• Lecture 1: Image enhancement: image basics and terminology: point

operations, spatial filters, matched filters.

• Lecture 2: 2D Fourier transforms and applications: spatial

frequencies, convolution theorem, aliasing.

• Lecture 3: Image restoration: inverse and Wiener filters,

applications to defocus and motion deblurring, MAP estimation

• Lecture 4: Non-linear filters & Image compression: bilateral filter, non-

local means, median filter, JPEG, inpainting
A Pot-Pourri of 2D Signals

aerial image synthetic aperture radar (SAR)

X-ray inspection image Magnetic Resonance Image (MRI)

More 2D Signals

scanning electron microscope

image of a mosquito’s head
Preview 1

noise reduction

3x3
median
filter
 Preview 2

edge enhancement

detail
Preview 3

Contrast enhancement
 Preview 4

deblurring

detail
 Preview 5

• Inpainting: automated texture generation to fill in regions

Textbooks

• R.C. Gonzalez and R.E. Woods, “Digital Image

Processing”, Prentice Hall, 2007 (3rd Edition).
Covers all the course material. Excellent.

• R.N. Bracewell, “The Fourier Transform and its

Applications”, McGraw-Hill, 1993 (3rd edition). For Fourier
transforms, but comprehensive …

• Further reading (www addresses) and the lecture notes

are on http://www.robots.ox.ac.uk/~az/lectures/ia
Textbooks

• Covers most of the material

• Is available free online at

http://szeliski.org/Book/

• Also excellent
 Lecture 1: Image Enhancement

image operation image

• Image basics
• Two types of quantization

• Point operations
• Histogram equalization
• Temporal averaging

• Spatial filters (linear)

• Noise reduction, sharpening
• Separability
• Matched filters
 What is an image?

A monochrome image is an array of values.

There are two types of discretization involved:
1. Spatial sampling (pixels -‘picture elements’), and
2. Intensity quantization (grey level value).

(0,0)

84 133 226 212 218 218 222 212 218 222 226 218
75 156 177 218 212 218 218 218 218 222 218 218
96 84 133 203 218 218 218 222 212 218 222 218
123 75 111 156 212 218 212 212 218 218 218 226
93 75 71 133 185 231 226 226 222 212 218 218
51 75 75 75 156 206 218 218 218 222 212 222
44 110 75 65 143 194 231 218 218 218 218 218
52 123 69 84 60 156 199 231 231 222 226 226
52 75 84 81 65 69 150 231 231 226 231 231
36 36 84 93 84 71 156 160 240 240 231 231
36 40 113 75 69 75 71 133 194 240 240 240
52 52 105 85 69 75 75 123 111 222 231 231
69 44 69 93 81 75 75 69 150 177 247 240
73 44 40 96 101 75 75 75 84 133 231 240
Intensity Quantization

2 levels - binary 4 levels

16 levels 256 levels – 1 byte

Spatial Sampling

384 x 288 pixels 192 x 144 pixels

92 x 72 pixels 48 x 36 pixels
Colour Sensing in Camera (RGB)
3-chip vs. 1-chip: quality vs. cost

http://www.cooldictionary.com/words/Bayer-filter.wikipedia
Slide by Steve Seitz
Practical Colour Sensing: Bayer Grid

Estimate RGB
at ‘G’ cells from
neighbouring values

http://www.cooldictionary.com/
words/Bayer-filter.wikipedia
Slide by Steve Seitz
 Image statistics

Quantization:
• Often 8 bits per pixel (0-255, 28 = 256 levels) for monochrome
• 24 bits per pixel for colour (8 bits for each of Red Green Blue)
• Medical images 12 bits (4096 levels) or 16 bits (65536 levels)

Size
• Cameras typically 4K x 3K pixels or far more
• Satellite images 10 -100K pixels width

• 3D images
• e.g. Magnetic Resonance Images
• Videos (2D + time)
Shanghai Skyline - Stitched from 12,000 photos

273 G pixels

http://gigapan.com/
 Google Art Project

resolution 30,000 × 23,756 pixels

What is image enhancement?
Improve the quality of an image or accentuate particular features.
Techniques are more general purpose and, unlike restoration, a strong
model of the degradation process is not assumed.

Examples
• How to improve contrast?

• How to sharpen edges?

• How to reduce noise?

 Point Operations

in = function(i)

new original
intensity intensity

• Change contrast and brightness by transforming pixel intensities

• Assign the same new intensity value to all pixels having a given
original value
• e.g. an affine (linear) map: in

i
Point operations – Negative
 Point operations – power-law transformations

1.00

in = iγ
0.75

in
0.50

0.25

0 0.25 0.5 0.75 1.0

i
Image Enhancement

original γ=3

γ=4 γ=5
Contrast Stretching
Image Histograms

Cumulative Histograms

s = T(r)
Histogram equalization (or flattening)

Objective: apply a monotonic map T(i) to the intensities so

that the intensity histogram is less peaked (flattened)

new

original
frequency
original goal
histogram

intensity
• use cumulative distribution as a measure

cumulative
Flattening map

original goal
N N

i in
 N

in i Proof: exercise

Algorithm:

• Compute the cumulative probability distribution C(i) from the

intensity histogram
• Map pixel intensities as

MATLAB equalized_image = hist_eq(original_image)

Histogram Equalization – Example 1
Example 2

• valueswhich are quite close together are spread out so

that minor variations are more visible
• note, histogram is not flat
Example 3

• intervals without many corresponding pixels are compressed

Signal model

Signals can be loosely modelled as

• piecewise constant or piecewise continuous
with additive noise which is
• Gaussian or at least mean zero and uncorrelated
This model is used to guide the design of enhancement
algorithms.
Temporal filtering

• Temporal average to suppress mean zero additive noise

• 100 frame low light sequence

Time averaged frames
histogram equalized

20
80
5
10
40
100 frames
0 frames
frames

• Average N noise samples with zero mean and variance

result has zero mean and variance /N
 Spatial Operations
in(x, y) = function(i ∈ spatial neighbourhood(x, y))

new original
intensity intensity
 Image filtering

• Image filtering: compute function of local neighborhood

at each position

• Linear filtering: function is a weighted sum/difference of

pixel values

• Many applications:
– Enhance images
• Denoise, resize, increase contrast, etc.
– Extract information from images
• Texture, edges, distinctive points, etc.
– Detect patterns
• Template matching
Image filtering 1 1 1

h[ , ] 1 1 1

1 1 1

f [.,.] g[.,.]
0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0
0 0
0 0
0 90
90 0
0 90
90 90
90 90
90 0
0 0
0

0
0 0
0 0
0 90
90 90
90 90
90 90
90 90
90 0
0 0
0

0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0

0
0 0
0 90
90 0
0 0
0 0
0 0
0 0
0 0
0 0
0

0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0

g[m, n]   h[k , l ] f [m  k , n  l ]
k ,l Credit: S. Seitz
Image filtering 1 1 1

h[ , ] 1 1 1

1 1 1

f [.,.] g[.,.]
0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0
?
0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0 50

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

g[m, n]   h[k , l ] f [m  k , n  l ]
k ,l Credit: S. Seitz
Image filtering 1 1 1

h[ , ] 1 1 1

1 1 1

f [.,.] g[.,.]
0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10

0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20

0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30

0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20

0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10

0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

g[m, n]   h[k , l ] f [m  k , n  l ]
k ,l Credit: S. Seitz
Box Filter

What does it do?

• Replaces each pixel with 1 1 1
an average of its
neighborhood 1 1 1

1 1 1
• Achieve smoothing effect
(remove sharp features)

Slide credit: David Lowe (UBC)

Smoothing with box filter
Practice with linear filters

0 0 0
0 1 0 ?
0 0 0

Original

Source: D. Lowe
Practice with linear filters

0 0 0
0 0 1 ?
0 0 0

Original

Source: D. Lowe
Practice with linear filters

0 0 0 1 1 1
0
0
2
0
0
0
- 1
1
1
1
1
1
?
(Note that filter sums to 1)
Original

Source: D. Lowe
Sharpening

Source: D. Lowe
Other filters

1 0 -1
2 0 -2
1 0 -1
Sobel

Vertical Edge
(absolute value)
Other filters

1 2 1
0 0 0
-1 -2 -1
Sobel

Horizontal Edge
(absolute value)
Rectangular filter

=
*
h[m,n]

f[m,n] g[m,n]
Rectangular filter

=
*
h[m,n]

f[m,n] g[m,n]
 Properties of linear filters
• Use ∗ for the action of the filter
• Linear filtering is a linear operation

• e.g. filtering the sum of two signals produces the same result
as filtering each signal individually and summing the result
• and is shift invariant, e.g. shift the input image two pixels to
the left, then the output is shifted two pixels to the left
• We will see later:
• the connection between filters and convolution
• non-linear filters
 Gaussian noise

>> noise = randn(size(im)).*sigma;

MATLAB >> output = im + noise;
Fig: M. Hebert
Effects of noise
Consider a single row or column of the image
• Plotting intensity as a function of position gives a signal

How to compute a derivative?

Where is the edge?

Solution: smooth first

Where is the edge? Look for peaks in

Noise reduction using linear filters

Suppose signal consists of a step edge plus Gaussian noise

f(x)

f(x) + n(x)

• averageN pixel values in spatial neighbourhood to

reduce noise to variance /N
Spatial averaging – smoothing filter

Use Gaussian kernel for a centre weighted filter

 small  large
e.g. if  = 1, k = 3
 f(x) + n(x)

G(x,y) * f(x,y)
Gaussian filter  = 2
6

Gaussian filter  = 4 6

• spatial averaging removes noise, blurs edges

Example

Gaussian smoothing
 Matlab
>> hsize = 10;
>> sigma = 5;
>> h = fspecial(‘gaussian’ hsize, sigma);

>> mesh(h);

>> imagesc(h);

>> outim = imfilter(im, h);

>> imshow(outim);

outim
Example

image gradient magnitude

 Separability
• In some cases, a filter is separable, and we can factor
the operation into two steps:
– filter all rows
– filter all columns
Separability example

2D filtering
(centre location only)

The filter factors *

into a product of 1D
filters:

Perform filtering =
along rows: *

Followed by filtering =
along the remaining column: *

Source: K. Grauman
Separability example
*
=

gv * gh * f = (gv * gh) * f

= =
*
What is the computational
complexity advantage for a
separable filter of size k x k,
in terms of number of
= gv * (gh * f)
operations per output pixel?
 Summary

1 1 1
Linear filtering is a weighted sum/difference of 1 1 1

pixel values 1 1 1

• Can smooth, sharpen, translate (among many

other uses)
• Filtering in Matlab, e.g. to filter image f with h

g = filter2( h, f );

h=filter f=image
e.g. h = fspecial('gaussian');
How to determine the action of a filter

derivative of Gaussian Laplacian of Gaussian

 h[ , ]
1 1 1

f [.,.]
1

1
1

1
1

1
g[.,.]
0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

g[m, n]   h[k , l ] f [m  k , n  l ] Note, similarity to the dot

k ,l product between h and f
 Rearrange regions as vectors
image f template h

Write regions as vectors

f
h

vector f vector h
• g is max when f and h are parallel
• maximum response when filter matches signal
Template matching example
• Goal: find       in image

Slide: Derek Hoiem

Matching with filters
• Goal: find       in image
• filter the image with eye patch
g[m, n]   h[k , l ] f [m  k , n  l ]
k ,l
f = image
h = filter

What went wrong?

Input Filtered Image

Matching with filters
• Goal: find       in image
• filter the image with zero‐mean eye
g[m, n]   (h[k , l ] h ) ( f [m  k , n  l ] )
k ,l mean of template h

True detections

False
detections

Input Filtered Image (scaled) Thresholded Image

 Matching with filters
• Goal: find       in image
• Normalized cross‐correlation
mean template mean image patch

 (h[k , l ]  h )( f [m  k , n  l ]  f m,n )
g[m, n]  k ,l
0.5
 2
  (h[k , l ]  h )  ( f [m  k , n  l ]  f m ,n ) 
2

 k ,l k ,l 

Matlab: normxcorr2(template, im)

Matching with filters
• Goal: find       in image
• Normalized cross‐correlation

True detections

Input Normalized X-Correlation Thresholded Image

What is next?

• Spatial frequency analysis of images

• 2D Fourier transforms, the convolution theorem