Lecture 9

Optical Flow, Feature Tracking,

Normal Flow
Gary Bradski
Sebastian Thrun

http://robots.stanford.edu/cs223b/index.html 1
* Picture from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Q from stereo about
Essential
(from Trucco P-153)
Matrix
• Equation of the epipolar plane
– Co-planarity condition of vectors Pl, T and Pl-T

(Pl  T)T T Pl 0

• Essential Matrix E = RS
– 3x3 matrix constructed from R and T (extrinsic only)
• Rank (E) = 2, two equal nonzero singular values Pr R(Pl  T)

 r11 r12 r13   0  Tz Ty  T

  Pr EPl 0
R  r21 r22 r23  S  Tz 0  Tx 
 r31 r32 r33    Ty Tx 0  fl f
 pl  Pl p r  r Pr
Zl Zr
Rank (R) =3 Rank (S) =2

Why is this zero if it’s not T 2

orthogonal? p r Ep l 0
Question: p r TEp l 0 Why is this zero if it’s not orthogonal?
Answer: We’re dealing with equations of lines in
homogeneous coordinates.
Remember from Sebastian’s lecture, projective equations
are nonlinear because of the scale factor (1/Z). By
adding a generic scale, we get simple linear equations.
Thus, a point in the image plane is expressed as:

 x For a line:  x  a
 x
p    p  y  p  y  ; line parameters   b 
 y  1   1   c 
Equation
p T  0, or ax  by  c 0
T
Thus, p r Ep l 0 represents the projection of the line pl onto the right image
plane.
T
So, p r Ep l 0 is the equation of the line in the right image written in terms
of the point pl. That is, a statement that the point pl lies on
the that line.

3
 Optical Flow

Image tracking
3D computation

Image sequence Tracked sequence 3D structure

(single camera)
+
3D trajectory

4
 What is Optical Flow?
p2 
v2
p3 
 v3
p1 v1
Optical Flow
p4 
v4

I (t  1)
I (t ), { pi } 
Velocity vectors i }
{v

Optical flow is the relation of the motion field

• the 2D projection of the physical movement of points relative to the observer
to 2D displacement of pixel patches on the image plane.

Common assumption:
The appearance of the image patches do not change (brightness constancy)

I ( pi , t )  I ( pi  vi , t  1)
5
Note: more elaborate tracking models can be adopted if more frames are process all at once
 What is Optical Flow?
Optical flow is the relation of the motion field
• the 2D projection of the physical movement of points relative to the observer
to 2D displacement of pixel patches on the image plane.

When/where does this break down?

E.g.: In what situations does the displacement of pixel patches
not represent physical movement of points in space?

1. Well, TV is based on illusory motion

– the set is stationary yet things seem to move

2. A uniform rotating sphere

– nothing seems to move, yet it is rotating

3. Changing directions or intensities of lighting can make things seem to move

– for example, if the specular highlight on a rotating sphere moves.

4. Muscle movement can make some spots on a cheetah move opposite direction of motion.
– And infinitely more break downs of optical flow.
6
 Optical Flow Break Down

Perhaps an aperture problem discussed later.

7
 * From Marc Pollefeys COMP 256 2003
Optical Flow Assumptions:
Brightness Constancy

8
* Slide from Michael Black, CS143 2003
Optical Flow Assumptions:

9
* Slide from Michael Black, CS143 2003
Optical Flow Assumptions:

10
* Slide from Michael Black, CS143 2003
 Optical Flow: 1D Case
Brightness Constancy Assumption:

f (t )  I ( x (t ), t )  I ( x (t  dt ), t  dt )


f ( x )
0 Because no change in brightness with time
t
I  x  I
  0
x t  t  t x(t )

Ix v It

It
 v 
Ix

11
Tracking in the 1D case:

I ( x, t ) I ( x, t  1)


v ?
p

12
 Tracking in the 1D case:

I ( x, t ) I ( x, t  1)

It
Temporal derivative
p 
v

x
Ix

Spatial derivative
Assumptions:
I I  I
Ix  It  v  t • Brightness constancy
x t t x p
Ix • Small motion 13
 Tracking in the 1D case:
Iterating helps refining the velocity vector

I ( x, t ) I ( x, t  1)
Temporal derivative at 2nd iteration

It
p

x
Ix

Can keep the same estimate for spatial derivative

  I
v  v previous  t Converges in about 5 iterations
Ix 14
 Algorithm for 1D tracking:
For all pixel of interest p:
 Compute local image derivative at p: I x
 Initialize velocity vector: v  0
 Repeat until convergence:
 Compensate for current velocity vector: I ' ( x, t  1)  I ( x  v, t  1)
 Compute temporal derivative: I t  I ' ( p, t  1)  I ( p, t )
  I
 Update velocity vector: v v t
Ix

Requirements:
 Need access to neighborhood pixels round p to compute I
x
 Need access to the second image patch, for velocity compensation:
 The pixel data to be accessed in next image depends on current
velocity estimate (bad?)
 Compensation stage requires a bilinear interpolation (because v is
not integer)
 The image derivative I x needs to be kept in memory throughout the 15
iteration process
From 1D to 2D tracking
I  x  I
1D:   0
x t  t  t x(t )

I  x  I  y  I
2D:     0
x t  t  y t  t  t x(t )

I I I
u v 0
x t y t t x(t )

Shoot! One equation, two velocity (u,v) unknowns…

16
 From 1D to 2D tracking

We get at most “Normal Flow” – with one point we can only detect movement
perpendicular to the brightness gradient. Solution is to take a patch of pixels
Around the pixel of interest. 17
* Slide from Michael Black, CS143 2003
How does this show up visually?
Known as the “Aperture Problem”

18
Aperture Problem Exposed

Motion along just an edge is ambiguous 19

Aperture Problem in Real Life

20
 From 1D to 2D tracking
I ( x, y , t  1)
y

v2
I ( x, t ) I ( x, t  1) 
 v3
 v1
v

v4
x I ( x, y , t )
x

The Math is very similar:


v  G  1b
 It
v   I x2 IxIy 
Ix G  
window around p  I x I y

I y2 
Aperture problem
 I x It 
b   
window around p  I y I t 
21
Window size here ~ 11x11
 More Detail:
Solving the aperture problem
• How to get more equations for a pixel?
– Basic idea: impose additional constraints
• most common is to assume that the flow field is smooth locally
• one method: pretend the pixel’s neighbors have the same (u,v)
– If we use a 5x5 window, that gives us 25 equations per pixel!

22
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 RGB version
• How to get more equations for a pixel?
– Basic idea: impose additional constraints
• most common is to assume that the flow field is smooth locally
• one method: pretend the pixel’s neighbors have the same (u,v)
– If we use a 5x5 window, that gives us 25*3 equations per pixel!

23
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Lukas-Kanade flow
• Prob: we have more equations than unknowns

• Solution: solve least squares problem

– minimum least squares solution given by solution (in d) of:

– The summations are over all pixels in the K x K window

– This technique was first proposed by Lukas & Kanade (1981)
• described in Trucco & Verri reading 24
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Conditions for solvability
– Optimal (u, v) satisfies Lucas-Kanade equation

When is This Solvable?

• ATA should be invertible
• ATA should not be too small due to noise
– eigenvalues 1 and 2 of ATA should not be too small
• ATA should be well-conditioned
– 1/ 2 should not be too large (1 = larger eigenvalue)

25
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Eigenvectors of ATA

• Suppose (x,y) is on an edge. What is ATA?

– gradients along edge all point the same direction
– gradients away from edge have small magnitude

– is an eigenvector with eigenvalue

– What’s the other eigenvector of ATA?
• let N be perpendicular to

• N is the second eigenvector with eigenvalue 0

• The eigenvectors of ATA relate to edge direction and magnitude 26
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Edge

– large gradients, all the same

– large1, small 2
27
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Low texture region

– gradients have small magnitude

– small1, small 2
28
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 High textured region

– gradients are different, large magnitudes

– large1, large 2
29
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Observation
• This is a two image problem BUT
– Can measure sensitivity by just looking at one of the
images!
– This tells us which pixels are easy to track, which are
hard
• very useful later on when we do feature tracking...

30
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Errors in Lukas-Kanade
What are the potential causes of errors in this procedure?
– Suppose ATA is easily invertible
– Suppose there is not much noise in the image

• When our assumptions are violated

– Brightness constancy is not satisfied
– The motion is not small
– A point does not move like its neighbors
• window size is too large
• what is the ideal window size?

31
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Improving accuracy
• Recall our small motion assumption
It-1(x,y)
It-1(x,y)
• This is not exact
– To do better, we need to add higher order terms back in:
It-1(x,y)
• This is a polynomial root finding problem
– Can solve using Newton’s method
• Also known as Newton-Raphson method

– Lukas-Kanade method does one iteration of Newton’s method

• Better results are obtained via more iterations

32
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Iterative Refinement
• Iterative Lukas-Kanade Algorithm
1. Estimate velocity at each pixel by solving Lucas-Kanade
equations
2. Warp I(t-1) towards I(t) using the estimated flow field
- use image warping techniques
3. Repeat until convergence

33
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Revisiting the small motion assumption

• Is this motion small enough?

– Probably not—it’s much larger than one pixel (2nd order terms dominate)
– How might we solve this problem? 34
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Reduce the resolution!

35
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
 Coarse-to-fine optical flow
estimation

u=1.25 pixels

u=2.5 pixels

u=5 pixels

u=10 pixels
image It-1 image I
36
Gaussian pyramid of image It-1 Gaussian pyramid of image I
 Coarse-to-fine optical flow
estimation

run iterative L-K

warp & upsample

run iterative L-K

.
.
.

image JIt-1 image I

37
Gaussian pyramid of image It-1 Gaussian pyramid of image I
Multi-resolution Lucas Kanade
Algorithm

38
Optical Flow Results

39
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Optical Flow Results

40
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
41
 42
* From Marc Pollefeys COMP 256 2003
 Generalization

 1 x
 
 y 1

  / 0    / 0   cos   sin  
 0  /    0  /   sin  cos  
  
43
 44
* From Marc Pollefeys COMP 256 2003
 45
* From Marc Pollefeys COMP 256 2003
Affine Flow

46
* Slide from Michael Black, CS143 2003
 Horn & Schunck algorithm

Additional smoothness constraint :

es 
((u x2  u 2y )  ( v x2  v 2y ))dxdy ,

besides Opt. Flow constraint equation term

ec 
( I x u  I y v  I t ) 2 dxdy ,

minimize es+ec
47
 * From Marc Pollefeys COMP 256 2003
 Horn & Schunck algorithm
In simpler terms: If we want dense flow, we need to regularize what happens
in ill conditioned (rank deficient) areas of the image. We take the old cost function:

 I ( x, t )  I ( x  d , t  1) 
2
d arg min d
xN

And add a regularization term to the cost:

 I ( x, t )  I ( x  d , t  1)    d
2
d arg min d
xN
where ||d|| is some length metric, typically Euclidian length. When you solve, what
happens to our former solution
 2


 Ix IxIy   I x It 
v  G  1b G 
window around p  I x I y

I y2 
b   
window around p  I y I t 
?

The above solution requires that G be of full rank, that is, on a corner. Simplified,
what basically happens for the solution in Horn and Schunck is that:
  I x2 I x I y    0 
G  G    
 window 
around p  I x I y
2 
I y    0   which is always full rank.48

What does the regularization do for you?
• It’s a sum of squared terms (a Euclidian distance measure).
• We’re putting it in the expression to be minimized.
• => In texture free regions, v = 0
• => On edges, points will flow to nearest points.

Regularized flow

Optical flow

49
 Dense Optical Flow ~ Michael Black’s method
Michael Black took this one step further, starting from the regularized cost:

 I ( x, t )  I ( x  d , t  1)    d
2
d arg min d
xN

He replaced the inner distance metric, a quadradic:

with something more robust: d arg min d   I ( x, t ), I ( x  d , t 1)  d

xN

Where  looks something like

Basically, one could say that Michael’s method adds ways to handle 50
occlusion, non-common fate, and temporal dislocation
 Other Kinds of Flow
• Feature based – E.g.
– Will not say anything more than identifiable
features just lead to a search strategy.
– Of course, search and gradient flow can be
combined in the cost term distance measure.
• Normal Flow by motion templates
• …many others….

51
 Normal Flow by Motion Templates
Davis, Bradski, WACV 2000
• Object silhouette
Bradski Davis, Int. Jour. of Mach.
• Motion history images Vision Applications 2001

• Motion history gradients

• Motion segmentation algorithm

MHG
silhouette MHI

52
Motion Template Idea

53
 Motion Segmentation Algorithm
• Stamp the current motion history template with the
system time and overlay it on top of the others:

54
 Motion Segmentation Algorithm
• Measure gradients of the overlaid motion history
templates:

55
 Motion Segmentation Algorithm
• Threshold large gradients to get rid of motion template
edges resulting from too large of a time delay:

56
 Motion Segmentation Algorithm
• Find boundaries of most recent motions
• “Walk around boundary
• If drop not too high, Flood fill downwards to segment motions

Segmented Segmented
Motion Motion

57
 Motion Segmentation Algorithm
Actually need a two-pass algorithm for labeling all motion segments:
1. Fill downwards; At bottom, turn around and fill upwards.
2. Keep the union of these fills as the segmented motion.

58
 Motion Template for Motion
Segmentation and Gesture
Overlay silhouettes, take gradient for normal optical
flow. Flood fill to segment motions.

Motion Motion
Segmentation Segmentation

Pose Gesture
Recognition Recognition

59
Human Motion System
Illusory Snakes

60