CS376 Computer Vision

Lecture 2: Linear Filters

Qixing Huang
January 28th 2019
 Announcements
• Piazza for assignment questions

• A0 due tomorrow. Submit on Canvas.

• Office hours posted on class website

 Plan for today
• Image noise

• Linear filters
– Examples: smoothing filters
 Images as matrices
Result of averaging 100 similar snapshots

Little Leaguer Kids with Santa The Graduate Newlyweds

From: 100 Special Moments, by Jason Salavon (University of Chicago) (2004)

http://salavon.com/SpecialMoments/SpecialMoments.shtml
Image Formation

Source: https://slideplayer.com/slide/13474214/
Images as functions
 Digital camera

A digital camera replaces film with a sensor array

• Each cell in the array is light-sensitive diode that converts
photons to electrons
• http://electronics.howstuffworks.com/digital-camera.htm
Digital images

Slide credit: Derek Hoiem

 Digital images
• Sample the 2D space on a regular grid
• Quantize each sample (e.g., round to nearest integer)
• Image thus represented as a matrix of integer values.

10 10 10 10 24 20
21 21 70 10 30 40
00 05 10 15 30 10 2D
05 15 10 16 17 28
75 50 30 20 25 30
98 78 77 77 75 46

1D
Digital color images
 Slide Credit:
Digital color images Kristen Grauman

Color images,
RGB color
space

R G B
 Images in Matlab
• Images represented as a matrix
• Suppose we have a NxM RGB image called “I”
– I(1,1,1) = top-left pixel value in R-channel
– I(y, x, b) = y pixels down, x pixels to right in the bth channel
– I(N, M, 3) = bottom-right pixel in B-channel
• imread(filename) returns a uint8 image (values 0 to 255)
– Convert to double format (values 0 to 1) with im2double
 Main idea: image filtering
• Aggregate the local neighborhood at each pixel in the
image
– Function specified a pattern saying how to aggregate
values from neighbors

• Uses of filtering:
– Enhance an image (denoise, resize, level-of-details, etc)
– Extract information (texture, edges, features, etc)
– Detect patterns (template matching)
 Motivation: noise reduction

• Even multiple images of the same static scene will not be

identical.

Slide credit:
Kristen Grauman
 Common types of noise
– Salt and pepper noise:
random occurrences of
black and white pixels

– Impulse noise: random

occurrences of white pixels

– Gaussian noise: variations

in intensity drawn from a
Gaussian normal
distribution

Source: S. Seitz
 Gaussian noise

>> noise = randn(size(im)).*sigma;

>> output = im + noise;

What is impact of the sigma? Fig: M. Hebert

Effect of
sigma on
Gaussian
noise:

Image
shows the
noise values
themselves.
Effect of
sigma on
Gaussian
noise:

Image
shows the
noise values
themselves.
 Effect of
sigma on
Gaussian
noise:

sigma=1 This shows

the noise
values
added to the
raw
intensities of
an image.
Effect of
sigma on
Gaussian
noise:

Image
shows the
noise values
themselves.
 Effect of
sigma on
Gaussian
noise

sigma=16 This shows

the noise
values
added to the
raw
intensities of
an image.
 Motivation: noise reduction

• Even multiple images of the same static scene will not be

identical.
• How could we reduce the noise, i.e., give an estimate of the
true intensities?
• What if there’s only one image?

Slide credit:
Kristen Grauman
 First attempt at a solution
• Let’s replace each pixel with an average of all
the values in its neighborhood
• Assumptions:
– Expect pixels to be like their neighbors
– Expect noise processes to be independent from
pixel to pixel
 First attempt at a solution
• Let’s replace each pixel with an average of all
the values in its neighborhood
• Moving average in 1D:

Source: S. Marschner
 Weighted Moving Average
• Can add weights to our moving average
• Weights [1, 1, 1, 1, 1] / 5

Source: S. Marschner
 Weighted Moving Average
• Non-uniform weights [1, 4, 6, 4, 1] / 16

Source: S. Marschner
 Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
 Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
 Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
 Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
 Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 90 0 90 90 90 0 0

0 0 0 90 90 90 90 90 0 0

0 0 0 0 0 0 0 0 0 0

0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
 Moving Average In 2D

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 20 10

0 0 0 90 90 90 90 90 0 0 0 20 40 60 60 60 40 20

0 0 0 90 90 90 90 90 0 0 0 30 60 90 90 90 60 30

0 0 0 90 90 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 0 90 90 90 0 0 0 30 50 80 80 90 60 30

0 0 0 90 90 90 90 90 0 0 0 20 30 50 50 60 40 20

0 0 0 0 0 0 0 0 0 0 10 20 30 30 30 30 20 10

0 0 90 0 0 0 0 0 0 0 10 10 10 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0

Source: S. Seitz
 Correlation filtering
Say the averaging window size is 2k+1 x 2k+1:

Attribute uniform Loop over all pixels in neighborhood

weight to each pixel around image pixel F[i,j]

Now generalize to allow different weights depending on

neighboring pixel’s relative position:

Non-uniform weights
 Correlation filtering

This is called cross-correlation, denoted

Filtering an image: replace each pixel with a linear

combination of its neighbors.

The filter “kernel” or “mask” H[u,v] is the prescription for the

weights in the linear combination.
 Averaging filter
• What values belong in the kernel H for the moving average
example?

0 0 0 0 0 0 0 0 0 0
0

0
0

0
0

0
0

90
0

90
0

90
0

90
0

90
0

0
0

0
1 1 1 0 20 40 60 60

0
0
0
0
0
0
90
90
90
90
90
90
90
90
90
90
0
0
0
0 ?
1 1 1
0 0 0 90 0 90 90 90 0 0
1 1 1
0 0 0 90 90 90 90 90 0 0

“box filter”
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
 Boundary Issues
• What is the size of the output?

imfilter
 Boundary Issues
• What is the size of the output?

filter2
Smoothing with a Gaussian
 Gaussian filters
• What parameters matter here?
• Size of kernel or mask
– Note, Gaussian function has infinite support, but discrete filters
use finite kernels

σ = 5 with σ = 5 with
10 x 10 30 x 30
kernel kernel Slide credit:
Kristen Grauman
 Gaussian filters
• What parameters matter here?
• Variance of Gaussian: determines extent of
smoothing

σ = 2 with σ = 5 with
30 x 30 30 x 30
kernel kernel
Slide credit:
Kristen Grauman
 Matlab
>> hsize = 10;
>> sigma = 5;
>> h = fspecial(‘gaussian’ hsize, sigma);

>> mesh(h);

>> imagesc(h);

>> outim = imfilter(im, h); % correlation

>> imshow(outim);

Slide credit: outim

Kristen Grauman
 Smoothing with a Gaussian
Parameter σ is the “scale” / “width” / “spread” of the Gaussian
kernel, and controls the amount of smoothing.

for sigma=1:3:10
h = fspecial('gaussian‘, fsize, sigma);
out = imfilter(im, h);
imshow(out);
pause;
Slide credit:
end Kristen Grauman
Keeping the two Gaussians in play straight…
More noise →

Wider smoothing kernel →

 Properties of smoothing filters
• Smoothing
– Values positive
– Sum to 1 → constant regions same as input
– Amount of smoothing proportional to mask size
– Remove “high-frequency” components; “low-pass” filter

Slide credit:
Kristen Grauman
Predict the outputs using correlation
filtering

0 0 0 0 0 0
* 0 1 0 =? * 0 0 1 =?
0 0 0 0 0 0

-
0 0 0 1 1 1
* 0 2 0 1 1 1 =?
0 0 0 1 1 1

Slide credit:
Kristen Grauman
 Practice with linear filters

?
0 0 0
0 1 0
0 0 0

Original

Source: D. Lowe
 Practice with linear filters

0 0 0
0 1 0
0 0 0

Original Filtered
(no change)

Source: D. Lowe
 Practice with linear filters

?
0 0 0
0 0 1
0 0 0

Original

Source: D. Lowe
 Practice with linear filters

0 0 0
0 0 1
0 0 0

Original Shifted left

by 1 pixel
with
correlation
Source: D. Lowe
 Practice with linear filters

1 1 1
1 1 1 ?
1 1 1

Original

Source: D. Lowe
 Practice with linear filters

1 1 1
1 1 1
1 1 1

Original Blur (with a

box filter)

Source: D. Lowe
 Practice with linear filters

-
0 0 0 1 1 1
0 2 0
0 0 0
1 1 1
1 1 1
?
Original

Source: D. Lowe
 Practice with linear filters

-
0 0 0 1 1 1
0 2 0 1 1 1
0 0 0 1 1 1

Original Sharpening filter:

accentuates differences
with local average

Source: D. Lowe
Filtering examples: sharpening
More Examples
1 1 1
1 1 1
1 1 1
 -
0 0 0 1 1 1
0 2 0 1 1 1
0 0 0 1 1 1
 Convolution
• Convolution:
– Flip the filter in both dimensions (bottom to top, right to left)
– Then apply cross-correlation

Notation for H
convolution
operator

Slide Credit: Kristen Grauman

 Properties of convolution
• Shift invariant:
– Operator behaves the same everywhere, i.e. the value of
the output depends on the pattern in the image
neighborhood, not the position of the neighborhood.
• Superposition:
– h * (f1 + f2) = (h * f1) + (h * f2)

Slide Credit: Kristen Grauman

 Properties of convolution
• Commutative:
f*g=g*f
• Associative
(f * g) * h = f * (g * h)
• Distributes over addition
f * (g + h) = (f * g) + (f * h)
• Scalars factor out
kf * g = f * kg = k(f * g)
• Identity:
unit impulse e = […, 0, 0, 1, 0, 0, …]. f * e = f

Slide Credit: Kristen Grauman

 Separability
• In some cases, filter is separable, and we can factor into two
steps:
– Convolve all rows with a 1D filter
– Convolve all columns with a 1D filter

Slide Credit: Kristen Grauman

 Effect of smoothing filters

Additive Gaussian noise Salt and pepper noise

Slide Credit: Kristen Grauman

 Median filter

• No new pixel values

introduced
• Removes spikes: good
for impulse, salt &
pepper noise
• Non-linear filter

Slide Credit: Kristen Grauman

 Slide Credit: Kristen Grauman
Median filter
Salt and
Median
pepper
filtered
noise

Plots of a row of the image

Matlab: output im = medfilt2(im, [h w]); Source: M. Hebert

 Slide Credit: Kristen Grauman
Median filter
• Median filter is edge preserving
Filtering application: Hybrid Images

Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006
Application: Hybrid Images
Gaussian Filter A. Oliva, A. Torralba, P.G. Schyns,
“Hybrid Images,” SIGGRAPH 2006

Laplacian Filter

unit impulse Gaussian Laplacian of Gaussian

Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006
 wwww

Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006
 Summary
• Image “noise”
• Linear filters and convolution useful for
– Enhancing images (smoothing, removing noise)
• Box filter
• Gaussian filter
• Impact of scale / width of smoothing filter
– Detecting features (next time)
• Separable filters more efficient
• Median filter: a non-linear filter, edge-preserving
 Coming up
• Wednesday:
– Filtering part 2: filtering for features (edges,
gradients, seam carving application)

• Tomorrow:
– Assignment 0 is due on Canvas 11:59 PM
•