CS5670: Computer Vision

Local features & Harris corner detection

Announcements
• Project 1 code due this Friday, February 9 at 8pm
– Turnin via Github Classroom
• Project 1 artifact due Monday, 2/12 at 8pm
• Quiz this Thursday via Canvas
• Project 2 will be released next Tuesday
Reading
• Szeliski: 7.1
Today: Feature extraction—Corners and
blobs
Motivation: Automatic panoramas

Credit: Matt Brown

Panorama stitching

Panorama captured by Perseverence Rover, Feb. 20, 2021

https://www.space.com/nasa-perseverance-rover-first-panorama-mars
Motivation: Automatic panoramas

GigaPan:
http://gigapan.com/
Also see Google Zoom Views:
https://www.google.com/culturalinstitute/beta/project/gigapixels
Why extract features?
• Motivation: panorama stitching
– We have two images – how do we combine them?
Why extract features?
• Motivation: panorama stitching
– We have two images – how do we combine them?

Step 1: extract features

Step 2: match features
Why extract features?
• Motivation: panorama stitching
– We have two images – how do we combine them?

Step 1: extract features

Step 2: match features
Step 3: align images
Application: Visual SLAM
• (aka Simultaneous Localization and Mapping)
Do these images overlap?
Answer below (look for tiny colored squares…)

NASA Mars Rover images

with SIFT feature matches
Feature matching for object search
Feature matching
Invariant local features
Find features that are invariant to transformations
– geometric invariance: translation, rotation, scale
– photometric invariance: brightness, exposure, …

Feature Descriptors
Advantages of local features
Locality
– features are local, so robust to occlusion and clutter
Quantity
– hundreds or thousands in a single image
Distinctiveness:
– can differentiate a large database of objects
Efficiency
– real-time performance achievable
More motivation…
Feature points are used for:
– Image alignment (e.g., mosaics)
– 3D reconstruction
– Motion tracking (e.g. for AR)
– Object recognition
– Image retrieval
– Robot/car navigation
– … other
Local features: main components
1) Detection: Identify the
interest points

2) Description: Extract vector x1 = [ x1(1) ,  , xd(1) ]

feature descriptor
surrounding each interest
point

x 2 = [ x1( 2) ,, xd( 2) ]

3) Matching: Determine
correspondence between
descriptors in two views
Credit: Kristen Grauman
What makes a good feature?

Snoop demo
Want uniqueness
Look for image regions that are unusual
– Lead to unambiguous matches in other images

How to define “unusual”?

Local measures of uniqueness
Suppose we only consider a small window of pixels
– What defines whether a feature is a good or bad candidate?

Credit: S. Seitz, D. Frolova, D. Simakov

Local measures of uniqueness
• How does the window change when you shift it?
• Shifting the window in any direction causes a big change

“flat” region: “edge”: “corner”:

no change in all no change along significant change
directions the edge direction in all directions
Credit: S. Seitz, D. Frolova, D. Simakov
Harris corner detection: the math
Consider shifting the window W by (u,v)
• how do the pixels in W change?
• compare each pixel before and after by
summing up the squared differences (SSD) (u,v)
W

• this defines an SSD “error” E(u,v):

• We are happy if this error is high

• We are very happy if this error is high for all offsets (u,v)
• Slow to compute exactly for each pixel and each offset
(u,v) Chris Harris and Mike Stephens (1988). "A Combined
Corner and Edge Detector". Alvey Vision Conference.
Small motion assumption
Taylor Series expansion of I:

If the motion (u, v) is small, then first order approximation is good

Plugging this into the formula on the previous slide…

Corner detection: the math

Consider shifting the window W by (u,v) (u,v)

W

• define an SSD “error” E(u,v):

Corner detection: the math

Consider shifting the window W by (u,v) (u,v)

W

• define an SSD “error” E(u,v):

• Thus, E(u,v) is locally approximated as a quadratic error function

The second moment matrix
The surface E(u,v) is locally approximated by a quadratic form.

Let’s try to understand its shape.

 E(u,v)

Horizontal edge:
v
u
 E(u,v)

Vertical edge:
v
u
General case
We can visualize H as an ellipse with axis lengths
determined by the eigenvalues of H and orientation
determined by the eigenvectors of H

lmax, lmin : eigenvalues of H

Ellipse equation: direction of the
fastest change
éu ù direction of the
[u v] H ê ú = const slowest change
ëv û
(lmax)-1/2
(lmin)-1/2
Quick eigenvalue/eigenvector review
The eigenvectors of a matrix A are the vectors x that satisfy:

The scalar l is the eigenvalue corresponding to x

– The eigenvalues are found by solving:

– In our case, A = H is a 2x2 matrix, so we have

– The solution:

Once you know l, you find x by solving

Corner detection: the math

xmin

xmax

Eigenvalues and eigenvectors of H

• Define shift directions with the smallest and largest change in error
• xmax = direction of largest increase in E
• lmax = amount of increase in direction xmax
• xmin = direction of smallest increase in E
• lmin = amount of increase in direction xmin
Corner detection: the math
How are lmax, xmax, lmin, and xmin relevant for feature detection?
• What’s our feature scoring function?
Corner detection: the math
How are lmax, xmax, lmin, and xmin relevant for feature detection?
• What’s our feature scoring function?
Want E(u,v) to be large for small shifts in all directions
• the minimum of E(u,v) should be large, over all unit vectors [u v]
• this minimum is given by the smaller eigenvalue (lmin) of H
Interpreting the eigenvalues
Classification of image points using eigenvalues of M:

l2 “Edge”
l2 >> l1 “Corner”
l1 and l2 are large,
l1 ~ l2;
E increases in all
directions

l1 and l2 are small;

E is almost constant “Flat” “Edge”
in all directions region l1 >> l2

l1
Corner detection summary
Here’s what you do:
• Compute the gradient at each point in the image
• For each pixel:
• Create the H matrix from nearby gradient values
• Compute the eigenvalues.
• Find points with large response (lmin > threshold)
• Choose those points where lmin is a local maximum as features
Corner detection summary
Here’s what you do:
• Compute the gradient at each point in the image
• For each pixel:
• Create the H matrix from nearby gradient values
• Compute the eigenvalues.
• Find points with large response (lmin > threshold)
• Choose those points where lmin is a local maximum as features
The Harris operator
lmin is a variant of the “Harris operator” for feature detection

• The trace is the sum of the diagonals, i.e., trace(H) = h11 + h22
• Very similar to lmin but less expensive (no square root)
• Called the Harris Corner Detector or Harris Operator
• Lots of other detectors, this is one of the most popular
Alternate Harris operator
• For Project 2, you will use an alternate definition of the
Harris operator:
The Harris operator

Harris
operator
Harris detector example
f value (red high, blue low)
Threshold (f > value)
Find local maxima of f (non-max suppression)
Harris features (in red)
Weighting the derivatives
• In practice, using a simple window W doesn’t work too
well

• Instead, we’ll weight each derivative value based on its

distance from the center pixel
Harris Detector – Recap [Harris88]
0. Input image
Second moment matrix
é I x2 (s D ) I x I y (s D )ù 1. Image Ix Iy
µ (s I , s D ) = g (s I ) * ê ú derivatives
ëê I x I y (s D ) I y (s D ) ûú
2

Ix2 Iy2 IxIy

2. Square of
derivatives
det M = l1l2
trace M = l1 + l2 g(IxIy)
3. Gaussian g(Ix2) g(Iy2)
filter g(sI)

4. Cornerness function – both eigenvalues are strong

5. Non-maxima suppression har 48

Harris Corners – Why so complicated?
• Can’t we just check for regions with lots of gradients in
the x and y directions?
– No! A diagonal line would satisfy that criteria
Current
Window
Questions?