EE5811

Topics in Computer Vision

Dr. LI Haoliang
Department of Electrical Engineering
Recap of Week 3
• Color Space: RGB, HSV, etc.
• Histogram: as image feature
• Histogram equalization: contrast enhancement
• Texture: LBP, Co-occurrence, Filter bank
 Edge detection

• Convert a 2D image into a set of curves

– Extracts salient features of the scene
– More compact than pixels
 Edge detection

• Goal: Identify visual changes

(discontinuities) in an image.

• Intuitively, semantic information is

encoded in edges.

• What are some ‘causes’ of

visual edges?

Source: D. Lowe
Origin of Edges
surface normal discontinuity

depth discontinuity

surface color discontinuity

illumination discontinuity

• Edges are caused by a variety of factors

Images as functions…

• Edges look like steep

cliffs
Closeup of edges

Source: D. Hoiem
Closeup of edges

Source: D. Hoiem
Closeup of edges

Source: D. Hoiem
Closeup of edges

Source: D. Hoiem
 Characterizing edges
• An edge is a place of rapid change in the image
intensity function

intensity function
image (along horizontal scanline) first derivative

edges correspond to
Source: L. Lazebnik extrema of derivative
Intensity profile

Source: D. Hoiem
 Image derivatives
• How can we differentiate a digital image F[x,y]?
– Option 1: reconstruct a continuous image, f, then
compute the derivative
– Option 2: take discrete derivative (finite difference)

How would you implement this as a linear filter?

: 1 -1 : -1
1

Source: S. Seitz
Image gradient
• The gradient of an image:
The gradient points in the direction of most rapid increase in intensity

The edge strength is given by the gradient magnitude:

The gradient direction is given by:

Source: Steve Seitz

Image gradient

Source: L. Lazebnik
Effects of noise

Noisy input image

Where is the edge?

Source: S. Seitz
Solution: smooth first
f

f*h

To find edges, look for peaks in

Source: S. Seitz
Associative property of convolution
• Differentiation is convolution, and convolution is
associative:

• This saves us one operation:

Source: S. Seitz
The 1D Gaussian and its
derivatives
2D edge detection filters

derivative of Gaussian (x)

Gaussian
Derivative of Gaussian filter

x-direction y-direction
 The Sobel operator
• Common approximation of derivative of Gaussian

-1 0 1 1 2 1
-2 0 2 0 0 0
-1 0 1 -1 -2 -1

• The standard defn. of the Sobel operator omits the 1/8 term
– doesn’t make a difference for edge detection
– the 1/8 term is needed to get the right gradient magnitude
Sobel operator: example

Source: Wikipedia
Sobel operator

Original Magnitude Orientation

Laplacian of Gaussian (LoG)
• Consider

Laplacian of Gaussian
operator

Where is the edge? Zero-crossings of bottom graph 25

2D edge detection filters

Laplacian of Gaussian

Gaussian derivative of Gaussian

is the Laplacian operator:

26
LoG Operator
Using the LoG Function
• The LoG function will be
• Zero far away from the edge
• Positive on one side
• Negative on the other side
• Zero just at the edge
• It has simple digital mask implementation(s)
• So it can be used as an edge operator
• BUT, is this good enough?
Designing an edge detector
• Criteria for a good edge detector:
– Good detection: the optimal detector should find all real
edges, ignoring noise or other artifacts
– Good localization
• the edges detected must be as close as possible
to the true edges
• the detector must return one point only for each
true edge point
• Cues of edge detection
– Differences in color, intensity, or texture across the boundary
– Continuity and closure
– High-level knowledge

Source: L. Fei-Fei
 Canny edge detector
• This is probably the most widely used edge
detector in computer vision

J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern

Analysis and Machine Intelligence, 8:679-714, 1986.

Source: L. Fei-Fei
Example

• original image (Lena)

Finding edges

gradient magnitude
Finding edges

where is the edge?

thresholding
 Get Orientation at Each Pixel
• Threshold at minimum level
• Get orientation
theta = atan2(gy, gx)
Non-maximum suppression

• Check if pixel is local maximum along gradient

direction

35
Picture from Prem K Kalra
Before Non-max Suppression
After Non-max Suppression
 Finding edges

thresholding

Threshold the gradient magnitude image such that strong edges are
kept and noise is suppressed
Finding edges

thinning
(non-maximum suppression)
Canny edge detector
MATLAB: edge(image,‘canny’)

1. Filter image with derivative of Gaussian

2. Find magnitude and orientation of gradient

3. Non-maximum suppression

4. Linking and thresholding (hysteresis):

• Define two thresholds: low and high
• Use the high threshold to start edge curves and
the low threshold to continue them

Source: D. Lowe, L. Fei-Fei

‘Hysteresis’ thresholding
• Two thresholds – high and low
• Grad. mag. > high threshold? = strong edge
• Grad. mag. < low threshold? noise
• In between = weak edge

• ‘Follow’ edges starting from strong edge pixels

• Continue them into weak edges
• Connected components (Szeliski 3.3.4)

Source: S. Seitz
Final Canny Edges
𝜎 = 2, 𝑡𝑙𝑜𝑤 = 0.05, 𝑡ℎ𝑖𝑔ℎ = 0.1
Effect of  (Gaussian kernel spread/size)

Original 𝜎= 2 𝜎=4 2

The choice of  depends on desired behavior

• large  detects large scale edges
• small  detects fine features
Source: S. Seitz
Canny edge detector

original Canny with Canny with

• The choice of depends on desired behavior

– large detects “large-scale” edges
– small detects fine edges

Source: S. Seitz
 Scale space (Witkin 83)
first derivative peaks

larger

Gaussian filtered signal

• Properties of scale space (w/ Gaussian smoothing)

• edge position may shift with increasing scale ()
• two edges may merge with increasing scale
• an edge may not split into two with increasing scale
Canny Characteristics

• The Canny operator gives single-pixel-wide images

with good continuation between adjacent pixels

• It is probably the most widely used edge operator

today; no one has done better since it came out in
the late 80s. Many implementations are available.

• It is very sensitive to its parameters, which need to

be adjusted for different application domains.
J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern
Analysis and Machine Intelligence, 8:679-714, 1986.
What is a good edge detector?

Do we lose information when we look at edges?

Are edges ‘complete’ as a representation of images?

Where do humans see boundaries?
image human segmentation gradient magnitude

Potential project topic

Berkeley segmentation database:
http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/
Summary
• Image filtering
• Blurring: box filter, Gaussian filter,…
• Resampling: bilinear interpolation, …
• Edge detection
• Derivating filters (Sobel)
• Combined derivative and smoothing filters (deriv. of Gaussian)
• Laplacian of Gaussian
• Canny edge detector
Feature extraction: Corners and blobs
Motivation: Automatic panoramas

Credit: Matt Brown

Why extract features?
• Motivation: panorama stitching
• We have two images – how do we combine them?
Why extract features?
• Motivation: panorama stitching
• We have two images – how do we combine them?

Step 1: extract features

Step 2: match features
Why extract features?
• Motivation: panorama stitching
• We have two images – how do we combine them?

Step 1: extract features

Step 2: match features
Step 3: align images
Image matching

by Diva Sian

by swashford
Harder case

by Diva Sian by scgbt

Harder still?
Answer below (look for tiny colored squares…)

NASA Mars Rover images

with SIFT feature matches
Feature Matching
Feature Matching
 Invariant local features
Find features that are invariant to transformations
• geometric invariance: translation, rotation, scale
• photometric invariance: brightness, exposure, …

Feature Descriptors
Advantages of local features
Locality
• features are local, so robust to occlusion and clutter
Quantity
• hundreds or thousands in a single image
Distinctiveness:
• can differentiate a large database of objects
Efficiency
• real-time performance achievable
More motivation…
Feature points are used for:
• Image alignment (e.g., mosaics)
• 3D reconstruction
• Motion tracking
• Object recognition
• Indexing and database retrieval
• Robot navigation
• … other
Approach
Feature detection: find it
Feature descriptor: represent it
Feature matching: match it

Feature tracking: track it, when motion

Local features: main components
1) Detection: Identify the interest
points

2) Description: Extract vector

feature descriptor surrounding
each interest point. x1 = [ x1
(1)
, , x (1)
d ]

3) Matching: Determine
correspondence between
descriptors in two views x 2 = [ x1( 2) ,, xd( 2) ]

Kristen Grauman
What makes a good feature?

Snoop demo
Want uniqueness
Look for image regions that are unusual
• Lead to unambiguous matches in other images

How to define “unusual”?

Local measures of uniqueness
Suppose we only consider a small window of pixels
• What defines whether a feature is a good or bad
candidate?

Credit: S. Seitz, D. Frolova, D. Simakov

 Local measure of feature
uniqueness
• How does the window change when you shift it?
• Shifting the window in any direction causes a big
change

“flat” region: “edge”: “corner”:

no change in all no change along the significant change in
directions edge direction all directions
Credit: S. Seitz, D. Frolova, D. Simakov
 Harris corner detection: the math

Consider shifting the window W by (u,v)

• how do the pixels in W change?
• compare each pixel before and after by (u,v)
W

summing up the squared differences (SSD)

• this defines an SSD “error” E(u,v):

• We are happy if this error is high

• Slow to compute exactly for each pixel
and each offset (u,v)
Small motion assumption
Taylor Series expansion of I:

If the motion (u,v) is small, then first order approximation is good

Plugging this into the formula on the previous slide…

 Corner detection: the math

Consider shifting the window W by (u,v) W

(u,v)

• define an SSD “error” E(u,v):

 Corner detection: the math

Consider shifting the window W by (u,v) W

(u,v)

• define an SSD “error” E(u,v):

• Thus, E(u,v) is locally approximated as a quadratic error function

The second moment matrix
The surface E(u,v) is locally approximated by a quadratic form.

Let’s try to understand its shape.

 E(u,v)

Horizontal edge:
v
u
 E(u,v)

Vertical edge:
v
u
 General case
We can visualize H as an ellipse with axis lengths
determined by the eigenvalues of H and orientation
determined by the eigenvectors of H

min, max : eigenvalues of H

Ellipse equation: direction of the
smaller variance
u 
[u v] H   = const direction of the
larger variance
v 
(min)-1/2
(max)-1/2
Intuitive way of understanding Harris detector

Image patch

X derivative

Y derivative
Intuitive way of understanding Harris detector
Intuitive way of understanding Harris detector
 Quick eigenvalue/eigenvector review
The eigenvectors of a matrix A are the vectors x that satisfy:

The scalar  is the eigenvalue corresponding to x

• The eigenvalues are found by solving:

• In our case, A = H is a 2x2 matrix, so we have

• The solution:

Once you know , you find x by solving

Corner detection: the math

xmin

xmax

Eigenvalues and eigenvectors of H

• Define shift directions with the smallest and largest change in error
• xmax = direction of largest increase in E
• max = amount of increase in direction xmax
• xmin = direction of smallest increase in E
• min = amount of increase in direction xmin
Corner detection: the math
How are max, xmax, min, and xmin relevant for feature detection?
• What’s our feature scoring function?
Corner detection: the math
How are max, xmax, min, and xmin relevant for feature detection?
• What’s our feature scoring function?
Want E(u,v) to be large for small shifts in all directions
• the minimum of E(u,v) should be large, over all unit vectors [u v]
• this minimum is given by the smaller eigenvalue (min) of H
 Interpreting the eigenvalues
Classification of image points using eigenvalues of M:

2 “Edge”
2 >> 1 “Corner”
1 and 2 are large,
1 ~ 2 ;
E increases in all
directions

1 and 2 are small;

E is almost constant “Flat” “Edge”
in all directions region 1 >> 2

1
Corner detection summary
Here’s what you do
• Compute the gradient at each point in the image
• Create the H matrix from the entries in the gradient
• Compute the eigenvalues.
• Find points with large response (min > threshold)
• Choose those points where min is a local maximum as features
Corner detection summary
Here’s what you do
• Compute the gradient at each point in the image
• Create the H matrix from the entries in the gradient
• Compute the eigenvalues.
• Find points with large response (min > threshold)
• Choose those points where min is a local maximum as features
The Harris operator
min is a variant of the “Harris operator” for feature detection

• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
• Very similar to min but less expensive (no square root)
• Called the “Harris Corner Detector” or “Harris Operator”
• Lots of other detectors, this is one of the most popular
The Harris operator

Harris
operator
Harris detector example
f value (red high, blue low)
Threshold (f > value)
Find local maxima of f
Harris features (in red)
Weighting the derivatives
• In practice, using a simple window W doesn’t work
too well

• Instead, we’ll weight each derivative value based

on its distance from the center pixel
Harris Detector [Harris88]
• Second moment matrix
 I x2 ( D ) I x I y ( D )
 ( I ,  D ) = g ( I )    1. Image Ix Iy
 I x I y ( D ) I y ( D ) 
2
derivatives
(optionally, blur first)

Ix2 Iy2 IxIy

2. Square of
derivatives
det M = 12
trace M = 1 + 2
3. Gaussian g(Ix2) g(Iy2) g(IxIy)
filter g(I)

4. Cornerness function – both eigenvalues are strong

5. Non-maxima suppression 96
har
Exercise
• Suppose that an image 𝑓(𝑥, 𝑦) is convolved with a mask of size n*n
ҧ 𝑦).
(with coefficients 1/𝑛2 ) to produce a smoothed image 𝑓(𝑥,
(a) Derive an expression of edge strength (edge magnitude) of the
smoothed image as a function of mask size. Assume for simplicity that n is
odd and that edges are obtained using the partial derivatives.

(b) Show that the ratio of the maximum edge strength of the smoothed
image to the maximum edge strength of the original image is 1/n.
Solution

Similarly, we can have

Solution
Coding exercise
• Using convolution filtering for edge detection (butterfly
image as input)

Roberts
operator

Laplacian
operator

• OpenCV
• cv.Canny( image, threshold1, threshold2)