Bilişim Enstitüsü

Bilgisayar Bilimleri ABD

Görüntü Analizi
Week 8: Filtering and Edges
For shrinking interpolation bad, averaging good

“interpolation averaging

X
”
Convolution: Weighted sum over pixels
Filters
Boundary issues
• What about near the edge?
– the filter window falls off the edge of the image
– need to extrapolate
– methods:
• clip filter (black)
• wrap around
• copy edge
• reflect across edge

Slide credit: S. Marschner

Boundary issues
• What is the size of the output?
• MATLAB: output size / “shape” options
– shape = ‘full’: output size is sum of sizes of f andg
– shape = ‘same’: output size is same as f
– shape = ‘valid’: output size is difference of sizes of f and g

full same valid

g g g g
g g

f f f

g g g
g g g

Slide credit: S. Lazebnik

Boundary issues
• What about near the edge?
– the filter window falls off the edge of the image
– need to extrapolate
– methods (MATLAB):
• clip filter (black): imfilter(f, g, 0)
• wrap around: imfilter(f, g, ‘circular’)
• copy edge: imfilter(f, g, ‘replicate’)
• reflect across edge: imfilter(f, g, ‘symmetric’)

Slide credit: S. Marschner

Cross-Correlation vs Convolution
Cross-Correlation vs Convolution
Do this
Cross-Correlation vs Convolution
These are nicer mathematically
So what can we do with these convolutions anyway?
Mathematically: all the nice things
- Commutative
- A*B = B*A
- Associative
- A*(B*C) = (A*B)*C
- Distributes over addition
- A*(B+C) = A*B + A*C
- Plays well with scalars
- x(A*B) = (xA)*B = A*(xB)
So what can we do with these convolutions anyway?
This means some convolutions decompose:
- 2d gaussian is just composition of 1d gaussians
- Faster to run 2 1d convolutions
So what can we do with these convolutions anyway?
- Blurring
- Sharpening
- Edges
- Features
- Derivatives
- Super-resolution
- Classification
- Detection
- Image captioning
- ...
So what can we do with these convolutions anyway?
- Blurring
- Sharpening All of computer vision
- Edges
- Features is convolutions
- Derivatives (basically)
- Super-resolution
- Classification
- Detection
- Image captioning
- ...
Image Filtering
• Image filtering: computes a function of a local neighborhood at each pixel
position
• Called “Local operator,” “Neighborhood operator,” or “Window operator”
• f: image à image
• Uses:
– Enhance images
• Noise reduction, smooth, resize, increase
contrast, recolor, artistic effects, etc.
– Extract features from images
• Texture, edges, distinctive points, etc.
– Detect patterns
• Template matching, e.g., eye template
Slide credit: D. Hoiem
Common types of noise
– Salt and pepper noise:
random occurrences of
black and white pixels
– Impulse noise:
random occurrences
of white pixels
– Gaussian noise:
variations in intensity
drawn from a Gaussian
normal distribution

Slide credit: S. Seitz

Gaussian noise

>> noise = randn(size(im)).*sigma;

>> output = im + noise;

What is the impact of the sigma? Slide credit: M. Hebert

Motivation: noise reduction

• Make multiple observations of the same static scene

• Take the average
• Even multiple images of the same static scene will not be
identical.
Adapted from: K. Grauman
Motivation: noise reduction

• Make multiple observations of the same static scene

• Take the average
• Even multiple images of the same static scene will not be
identical.
• What if we can’t make multiple observations?
Adapted from: K. Grauman
W h a t if there’s only one image?
Median filter

• No new pixel values

introduced
• Removes spikes: good for
impulse, salt & pepper
noise
• Non-linear filter

Slide credit: K. Grauman

Median filter

Salt and Median

pepper noise filtered

Plots of a row of the image

Matlab: output im = medfilt2(im, [h w]);
Slide credit: M. Hebert
Median filter
• What advantage does median filtering have over
Gaussian filtering?
– Robustness to outliers
– Median filter is edge preserving

Slide credit: K. Grauman

So what can we do with these convolutions anyway?
- Blurring
- Sharpening All of computer vision
- Edges
- Features is convolutions
- Derivatives (basically)
- Super-resolution
- Classification
- Detection
- Image captioning
- ...
Levels of reasoning in vision
Scenes

Objects

Lines

Edges

Pixels

Images
[Slide by Neeraj Kumar]
Edge detection

• Convert a 2D image into a set of curves

• Extracts salient features of the scene
• More compact than pixels
What’s an edge?
- Image is a function
- Edges are rapid changes in this function
What’s an edge?
- Image is a function
- Edges are rapid changes in this function
Finding edges
- Could take derivative
- Edges = high response
Image derivatives
- Recall:
-
- We don’t have an “actual”
Function, must estimate
- Possibility: set h = 1
- What will that look like?
Image derivatives
- Recall:
-
- We don’t have an “actual”
Function, must estimate
- Possibility: set h = 1
- What will that look like?
Image derivatives
- Recall:
-
- We don’t have an “actual”
Function, must estimate
- Possibility: set h = 2
- What will that look like?
Image derivatives
- Recall:
-
- We don’t have an “actual”
Function, must estimate
- Possibility: set h = 2
- What will that look like?
Images are noisy!
Effects of noise
• Consider a single row or column of the image
• Plotting intensity as a function of position gives a signal

Derivatives amplify noise!

Where is the edge?

But we already know how to smooth

* =
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Smooth first, then derivative
Sobel filter! Smooth & derivative
Solution: smooth first

Where is the edge? Look for peaks in

Derivative theorem of convolution

• This saves us one operation:

Image derivatives
- Recall:
-
- Want smoothing too!
Finding edges
- Could take derivative
- Find high responses
- Sobel filters!
- But…
Finding edges
- Could take derivative
- Find high responses
- Sobel filters!
- But…
- Edges go both ways
- Want to find extrema
2nd derivative!
- Crosses zero at extrema
Laplacian (2nd derivative)!
- Crosses zero at extrema
- Recall:
-
- Laplacian:
-
- Again, have to
estimate f’’(x):
Laplacians
- Laplacian:
-
Laplacians
- Laplacian:
-
Laplacians
- Laplacian:
-
Laplacians
- Laplacian:
-
Laplacians
- Laplacian:
-
- Negative Laplacian, -4 in middle
- Positive Laplacian --->
Laplacians also sensitive to noise
- Again, use gaussian smoothing
- Can just use one kernel since convs commute
- Laplacian of Gaussian, LoG
- Can get good approx. with
5x5 - 9x9 kernels
Another edge detector:
- Image is a function:
- Has high frequency and low frequency components
- Think in terms of fourier transform
- Edges are high frequency changes
- Maybe we want to find edges of a specific size (i.e. specific
frequency)
Difference of Gaussian (DoG)
- Gaussian is a low pass filter
- Strongly reduce components with frequency f < σ
- (g*I) low frequency components
- I - (g*I) high frequency components
- g(σ1)*I - g(σ2)*I
- Components in between these frequencies
- g(σ1)*I - g(σ2)*I = [g(σ1) - g(σ2)]*I
Difference of Gaussian (DoG)
- g(σ1)*I - g(σ2)*I = [g(σ1) - g(σ2)]*I

- =
σ = 2 σ = 1
Difference of Gaussian (DoG)
- g(σ1)*I - g(σ2)*I = [g(σ1) - g(σ2)]*I
- This looks a lot like our LoG!
- (not actually the same but similar)
DoG (1 - 0)
DoG (2 - 1)
Another approach: gradient magnitude
- Don’t need 2nd derivatives
- Just use magnitude of gradient
Another approach: gradient magnitude
- Don’t need 2nd derivatives
- Just use magnitude of gradient
- Are we done? No!
What we really want: line drawing
Canny Edge Detection
- Your first image processing pipeline!
- Old-school CV is all about pipelines

Algorithm:
- Smooth image (only want “real” edges, not noise)
- Calculate gradient direction and magnitude
- Non-maximum suppression perpendicular to edge
- Threshold into strong, weak, no edge
- Connect together components

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Smooth image
- You know how to do this, gaussians!

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Gradient magnitude and direction
- Sobel filter

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Non-maximum suppression
- Want single pixel edges, not thick blurry lines
- Need to check nearby pixels
- See if response is highest

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Non-maximum suppression
Non-maximum suppression
Non-maximum suppression
Non-maximum suppression
Non-maximum suppression
Non-maximum suppression
Non-maximum suppression
Non-maximum suppression

• Check if pixel is local maximum along gradient direction

• requires checking interpolated pixels p and r
Non-maximum suppression

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Threshold edges
- Still some noise
- Only want strong edges
- 2 thresholds, 3 cases
- R > T: strong edge
- R < T but R > t: weak edge
- R < t: no edge
- Why two thresholds?

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Connect ‘em up!
- Strong edges are edges!
- Weak edges are edges
iff they connect to strong
- Look in some neighborhood
(usually 8 closest)

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Canny Edge Detection
- Your first image processing pipeline!
- Old-school CV is all about pipelines

Algorithm:
- Smooth image (only want “real” edges, not noise)
- Calculate gradient direction and magnitude
- Non-maximum suppression perpendicular to edge
- Threshold into strong, weak, no edge
- Connect together components
- Tunable: Sigma, thresholds

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Canny Edge Detection

http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/
Effect of σ (Gaussian kernel spread/size)

original Canny with Canny with

The choice of σ depends on desired behavior

• large σ detects large scale edges
• small σ detects fine features
Demo

• http://bigwww.epfl.ch/demo/ip/demos/edgeDetector/