Lecture

6:
Finding Features (part 1/2)
Professor Fei-Fei Li
Stanford Vision Lab

Fei-Fei Li! Lecture 6 - !1 2-Oct-13

!
!
 What we will learn today?
Local invariant features
MoHvaHon
Requirements, invariances
Keypoint localizaHon
Harris corner detector
Scale invariant region selecHon
AutomaHc scale selecHon
Next lecture (#7)
Dierence-of-Gaussian (DoG) detector
SIFT: an image region descriptor

Fei-Fei Li! Lecture 6 - !2 2-Oct-13

!
!
 What we will learn today?
Local invariant features
MoHvaHon
Requirements, invariances
Keypoint localizaHon
Harris corner detector
Scale invariant region selecHon
AutomaHc scale selecHon
Dierence-of-Gaussian (DoG) detector
SIFT: an image region descriptor

Fei-Fei Li! Lecture 6 - !3 2-Oct-13

!
!
 Image matching:
a challenging problem

Fei-Fei Li! Lecture 6 - !4 2-Oct-13

!
!
 Image matching:
a challenging problem

Slide credit: Steve Seitz

by Diva Sian

by swashford

Fei-Fei Li! Lecture 6 - !5 2-Oct-13

!
!
 Harder Case

Slide credit: Steve Seitz

by Diva Sian by scgbt

Fei-Fei Li! Lecture 6 - !6 2-Oct-13

!
!
 Harder SHll?

Slide credit: Steve Seitz

NASA Mars Rover images

Fei-Fei Li! Lecture 6 - !7 2-Oct-13

!
!
 Answer Below (Look for Hny colored squares)

Slide credit: Steve Seitz

NASA Mars Rover images with SIFT feature matches
(Figure by Noah Snavely)

Fei-Fei Li! Lecture 6 - !8 2-Oct-13

!
!
 MoHvaHon for using local features
Global representaHons have major limitaHons
Instead, describe and match only local regions
Increased robustness to
Occlusions

ArHculaHon
d
dq


q

Intra-category variaHons

Fei-Fei Li! Lecture 6 - !9 2-Oct-13

!
!
 General Approach
1. Find a set of
distinctive key-
points
A1 B 3 2. Define a region
around each
keypoint
A2 A3
B 2 3. Extract and
B 1 normalize the
region content

Slide credit: Bastian Leibe

fA Similarity fB 4. Compute a local
measure
descriptor from the
N pixels

normalized region
e.g. color e.g. color

N pixels d( f A, fB ) < T 5. Match local

descriptors

Fei-Fei Li! Lecture 6 - !10 2-Oct-13

!
!
 Common Requirements
Problem 1:
Detect the same point independently in both images

Slide credit: Darya Frolova, Denis Simakov

No chance to match!

We need a repeatable detector!

Fei-Fei Li! Lecture 6 - !11 2-Oct-13
!
!
 Common Requirements
Problem 1:
Detect the same point independently in both images

Problem 2:
For each point correctly recognize the corresponding one

Slide credit: Darya Frolova, Denis Simakov

?

We need a reliable and distinctive descriptor!

Fei-Fei Li! Lecture 6 - !12 2-Oct-13
!
!
 Invariance: Geometric TransformaHons

Slide credit: Steve Seitz

Fei-Fei Li! Lecture 6 - !13 2-Oct-13
!
!
 Levels of Geometric Invariance

Slide credit: Bastian Leibe

CS131 CS231a

Fei-Fei Li! Lecture 6 - !14 2-Oct-13

!
!
Invariance: Photometric TransformaHons

Slide credit: Tinne Tuytelaars

Ocen modeled as a linear
transformaHon:
Scaling + Oset

Fei-Fei Li! Lecture 6 - !15 2-Oct-13

!
!
 Requirements
Region extracHon needs to be repeatable and accurate
Invariant to translaHon, rotaHon, scale changes
Robust or covariant to out-of-plane (ane) transformaHons
Robust to lighHng variaHons, noise, blur, quanHzaHon
Locality: Features are local, therefore robust to occlusion
and clufer.
QuanHty: We need a sucient number of regions to cover
the object.

Slide credit: Bastian Leibe

DisHncHveness: The regions should contain interesHng
structure.
Eciency: Close to real-Hme performance.

Fei-Fei Li! Lecture 6 - !16 2-Oct-13

!
!
 Many ExisHng Detectors Available

Hessian & Harris [Beaudet 78], [Harris 88]

Laplacian, DoG [Lindeberg 98], [Lowe 99]
Harris-/Hessian-Laplace [Mikolajczyk & Schmid 01]
Harris-/Hessian-Ane [Mikolajczyk & Schmid 04]
EBR and IBR [Tuytelaars & Van Gool 04]
MSER [Matas 02]
Salient Regions [Kadir & Brady 01]

Slide credit: Bastian Leibe

Others

Those detectors have become a basic building block for

many recent applica8ons in Computer Vision.

Fei-Fei Li! Lecture 6 - !17 2-Oct-13

!
!
 Keypoint LocalizaHon

Slide credit: Bastian Leibe

Goals:
Repeatable detecHon
Precise localizaHon
InteresHng content
Look for two-dimensional signal changes

Fei-Fei Li! Lecture 6 - !18 2-Oct-13

!
!
 Finding Corners

Key property:

Slide credit: Svetlana Lazebnik

In the region around a corner, image gradient has two
or more dominant direcHons
Corners are repeatable and dis8nc8ve

C.Harris and M.Stephens. "A Combined Corner and Edge Detector.

Proceedings of the 4th Alvey Vision Conference, 1988.

Fei-Fei Li! Lecture 6 - !19 2-Oct-13

!
!
 Corners as DisHncHve Interest Points
Design criteria
We should easily recognize the point by looking through a
small window (locality)
Shicing the window in any direc8on should give a large
change in intensity (good localiza8on)

Slide credit: Alyosha Efros

flat region: edge: corner:
no change in all no change along significant change
directions the edge direction in all directions
Fei-Fei Li! Lecture 6 - !20 2-Oct-13
!
!
 Harris Detector FormulaHon
Change of intensity for the shic [u,v]:
2
E (u, v) = w( x, y) [ I ( x + u, y + v) I ( x, y) ]
x, y

Window Shifted Intensity

function intensity

Slide credit: Rick Szeliski

Window function w(x,y) = or

1 in window, 0 outside Gaussian

Fei-Fei Li! Lecture 6 - !21 2-Oct-13

!
!
 Harris Detector FormulaHon
This measure of change can be approximated by:
u
E (u, v) [u v] M
v
where M is a 22 matrix computed from image derivaHves:
I x2 I x I y Gradient with
M = w( x, y) 2
x, y I x I y I y respect to x,
times gradient
Slide credit: Rick Szeliski

with respect to y
Sum over image region the area we are
checking for corner

Fei-Fei Li! Lecture 6 - !22 2-Oct-13

!
!
 Harris Detector FormulaHon

Image I Ix Iy IxIy
where M is a 22 matrix computed from image derivaHves:
I x2 I x I y Gradient with
M = w( x, y) 2
x, y I x I y I y respect to x,
times gradient
Slide credit: Rick Szeliski

with respect to y
Sum over image region the area we are
checking for corner

Fei-Fei Li! Lecture 6 - !23 2-Oct-13

!
!
 What Does This Matrix Reveal?
First, lets consider an axis-aligned corner:

I x2 I I x y
1 0
M = 2 =
I x I y I y 0 2

This means:
Dominant gradient direcHons align with x or y axis
If either is close to 0, then this is not a corner, so look for

Slide credit: David Jacobs

locaHons where both are large.
What if we have a corner that is not aligned with the
image axes?

Fei-Fei Li! Lecture 6 - !24 2-Oct-13

!
!
 What Does This Matrix Reveal?
First, lets consider an axis-aligned corner:

I x2 I I x y
1 0
M = 2 =
I x I y I y 0 2

This means:
Dominant gradient direcHons align with x or y axis
If either is close to 0, then this is not a corner, so look for

Slide credit: David Jacobs

locaHons where both are large.
What if we have a corner that is not aligned with the
image axes?

Fei-Fei Li! Lecture 6 - !25 2-Oct-13

!
!
 General Case
1 0
1
Since M is symmetric, we have M = R
0 R
2
(Eigenvalue decomposition)
We can visualize M as an ellipse with axis lengths determined
by the eigenvalues and orientaHon determined by R

adapted from Darya Frolova, Denis Simakov

Direction of the
fastest change

Direction of the
slowest change

(max)-1/2
(min)-1/2

Fei-Fei Li! Lecture 6 - !26 2-Oct-13

!
!
 InterpreHng the Eigenvalues
ClassicaHon of image points using eigenvalues of M:
2 Edge
2 >> 1
Corner
1 and 2 are large, 1 ~ 2;
E increases in all direcHons
Slide credit: Kristen Grauman

1 and 2 are small;

E is almost constant in Flat Edge
all direcHons region 1 >> 2
1
Fei-Fei Li! Lecture 6 - !27 2-Oct-13
!
!
 Corner Response FuncHon
= det(M ) trace(M )2 = 12 (1 + 2 )2
2 Edge
<0
Corner
> 0
Slide credit: Kristen Grauman

Fast approximaHon
Avoid compuHng the
eigenvalues
: constant Flat Edge
(0.04 to 0.06) region <0
1
Fei-Fei Li! Lecture 6 - !28 2-Oct-13
!
!
 Window FuncHon w(x,y)
I x2 I x I y
M = w( x, y) 2
x, y I x I y I y
OpHon 1: uniform window
Sum over square window
I x2 I x I y
M =
I x I y
x , y I y2
Problem: not rotaHon invariant 1 in window, 0 outside

OpHon 2: Smooth with Gaussian

Slide credit: Bastian Leibe

Gaussian already performs weighted sum
I x2 I x I y
M = g ( ) 2
I I
x y I y Gaussian
Result is rotaHon invariant

Fei-Fei Li! Lecture 6 - !29 2-Oct-13

!
!
 Summary: Harris Detector [Harris88]
Compute second moment matrix
(autocorrelaHon matrix)
I x2 ( D ) I x I y ( D ) 1. Image
Ix Iy
M ( I , D ) = g ( I )
2
I x I y ( D ) I y ( D ) derivatives

2. Square of Ix2 Iy2 IxIy

derivatives
Slide credit: Krystian Mikolajczyk

3. Gaussian
filter g(I)
g(Ix2) g(Iy2) g(IxIy)
4. Cornerness function two strong eigenvalues
= det[M ( I , D )] [trace(M ( I , D ))]2
= g ( I x2 ) g ( I y2 ) [ g ( I x I y )]2 [ g ( I x2 ) + g ( I y2 )]2

5. Perform non-maximum suppression

R
Fei-Fei Li! Lecture 6 - !30 2-Oct-13
!
!
 Harris Detector: Workow

Slide adapted from Darya Frolova, Denis Simakov

Fei-Fei Li! Lecture 6 - !31 2-Oct-13
!
!
 Harris Detector: Workow
- computer corner responses

Slide adapted from Darya Frolova, Denis Simakov

Fei-Fei Li! Lecture 6 - !32 2-Oct-13
!
!
 Harris Detector: Workow
- Take only the local maxima of , where >threshold

Slide adapted from Darya Frolova, Denis Simakov

Fei-Fei Li! Lecture 6 - !33 2-Oct-13
!
!
 Harris Detector: Workow
- ResulHng Harris points

Slide adapted from Darya Frolova, Denis Simakov

Fei-Fei Li! Lecture 6 - !34 2-Oct-13
!
!
 Harris Detector Responses [Harris88]

Slide credit: Krystian Mikolajczyk

Effect: A very precise
corner detector.

Fei-Fei Li! Lecture 6 - !35 2-Oct-13

!
!
 Harris Detector Responses [Harris88]

Slide credit: Krystian Mikolajczyk

Fei-Fei Li! Lecture 6 - !36 2-Oct-13
!
!
 Harris Detector Responses [Harris88]

Slide credit: Kristen Grauman

Results are well suited for nding stereo correspondences

Fei-Fei Li! Lecture 6 - !37 2-Oct-13

!
!
 Harris Detector: ProperHes

TranslaHon invariance?

Slide credit: Kristen Grauman

Fei-Fei Li! Lecture 6 - !38 2-Oct-13
!
!
 Harris Detector: ProperHes

TranslaHon invariance
RotaHon invariance?

Slide credit: Kristen Grauman

Ellipse rotates but its shape (i.e.
eigenvalues) remains the same

Corner response is invariant to image rotation

Fei-Fei Li! Lecture 6 - !39 2-Oct-13

!
!
 Harris Detector: ProperHes

TranslaHon invariance
RotaHon invariance
Scale invariance?

Slide credit: Kristen Grauman

Corner All points will be
classified as edges!
Not invariant to image scale!

Fei-Fei Li! Lecture 6 - !40 2-Oct-13

!
!
 What we have learned today?
Local invariant features
MoHvaHon
Requirements, invariances
Keypoint localizaHon
Harris corner detector
Scale invariant region selecHon
AutomaHc scale selecHon
Next lecture (#7)
Dierence-of-Gaussian (DoG) detector
SIFT: an image region descriptor

Fei-Fei Li! Lecture 6 - !41 2-Oct-13

!
!