Scribd
Upload
18 views27 pages

cs429 s04 Lecture05a Motion

Uploaded by

Shehan Fernando
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
18 views27 pages

cs429 s04 Lecture05a Motion

Uploaded by

Shehan Fernando
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 27

Motion and Optical Flow

Key Problem

• Main problem in most multiple-image


methods: correspondence
Correspondence

• Small displacements
– Differential algorithms
– Based on gradients in space and time
– Dense correspondence estimates
– Most common with video
• Large displacements
– Matching algorithms
– Based on correlation or features
– Sparse correspondence estimates
– Most common with multiple cameras / stereo
Result of Correspondence

• For points in image i displacements to


corresponding locations in image j
• In stereo, usually called disparity
• In video, usually called motion field
Computing Motion Field

• Basic idea: a small portion of the image


(“local neighborhood”) shifts position
• Assumptions
– No / small changes in reflected light
– No / small changes in scale
– No occlusion or disocclusion
– Neighborhood is correct size: aperture
problem
Actual and Apparent Motion

• If these assumptions violated, can still


use the same methods – apparent
motion
• Result of algorithm is optical flow
(vs. ideal motion field)
• Most obvious effects:
– Aperture problem: can only get motion
perpendicular to edges
– Errors near discontinuities (occlusions)
Aperture Problem

• Too big:
confused by
multiple motions

• Too small:
only get motion
perpendicular
to edge
Computing Optical Flow:
Preliminaries
• Image sequence I(x,y,t)
• Uniform discretization along x,y,t –
“cube” of data
• Differential framework: compute partial
derivatives along x,y,t by convolving
with derivative of Gaussian
Computing Optical Flow:
Image Brightness Constancy
• Basic idea: a small portion of the image
(“local neighborhood”) shifts position,
but
still looks the same
• Brightness constancy
dI assumption
0
dt
Computing Optical Flow:
Image Brightness Constancy
• This does not say that the image
remains
the I
dI same brightness!
dt t
• vs. : total vs. partial derivative
• Use chain rule
dI x(t ), y (t ), t  I dx I dy I
  
dt x dt y dt t
Computing Optical Flow:
Image Brightness Constancy
• Given optical flow v(x,y)

dI x(t ), y (t ), t 
0
dt
I dx I dy I
  0
x dt y dt t
(I ) T v  I t  0
Image brightness constancy equation
Computing Optical Flow:
Discretization
• Look at some neighborhood N:
want
 I (i, j ) v  I (i, j )  0
T
t
( i , j )N
want
Av  b  0

 I (i1 , j1 )   I t (i1 , j1 ) 
 I (i , j )   I (i , j ) 
A  2 2 
b  t 2 2 
     
   
 I (in , jn )  I t (in , jn )
Computing Optical Flow:
Least Squares
• In general, overconstrained linear
system
• Solve by least squares
want
Av  b  0
 ( A T A) v  A Tb
T 1 T
v  ( A A) A b
Computing Optical Flow:
Stability
• Has a solution unless C = ATA is
singular
T
C A A
 I (i1 , j1 ) 
 I (i , j ) 
C I (i1 , j1 ) I (i2 , j2 )  I (in , jn ) 2 2 

  
 
 I (in , jn )
  I x2  I x I y 
C   N N
2 


 N
IxIy  Iy
N

Computing Optical Flow:
Stability
• Where have we encountered C before?
• Corner detector!
• C is singular if constant intensity or edge
• Use eigenvalues of C:
– to evaluate stability of optical flow
computation
– to find good places to compute optical flow
(finding good features to track)
– [Shi-Tomasi]
Computing Optical Flow:
Improvements
• Assumption that optical flow is constant
over neighborhood not always good
• Decreasing size of neighborhood 
C more likely to be singular
• Alternative: weighted least-squares
– Points near center = higher weight
– Still use larger neighborhood
Computing Optical Flow:
Weighted Least Squares
• Let W be a matrix of weights

A  WA
b  Wb

v   ( A T A)  1 A T b
T 2 1 T 2
 v w   ( A W A) A W b
Computing Optical Flow:
Improvements
• What if windows are still bigger?
• Adjust motion model: no longer
constant within a window
• Popular choice: affine model
Computing Optical Flow:
Affine Motion Model
• Translational model
 x2   x1   t x 
 y   y    t 
 2   1  y 
• Affine model
 x2   a b   x1   t x 
 y   c d   y    t 
 2    1  y 
• Solved as before, but 6 unknowns instead
of 2
Computing Optical Flow:
Improvements
• Larger motion: how to maintain
“differential” approximation?
• Solution: iterate
• Even better: adjust window / smoothing
– Early iterations: use larger Gaussians to
allow more motion
– Late iterations: use less blur to find exact
solution, lock on to high-frequency detail
Iteration

• Local refinement of optical flow


estimate
• Sort of equivalent to multiple iterations
of Newton’s method
Computing Optical Flow:
Lucas-Kanade
• Iterative algorithm:
1. Set = large (e.g. 3 pixels)
2. Set I’  I1
3. Set v  0
4. Repeat while SSD(I’, I2) > 
1. v += Optical flow(I’  I2)
2. I’  Warp(I1, v)

5. After n iterations,
set  = small (e.g. 1.5 pixels)
Computing Optical Flow:
Lucas-Kanade

• I’ always holds warped version of I1


– Best estimate of I2
• Gradually reduce thresholds
• Stop when difference between I’ and I2
small
– Simplest difference metric = sum of
squared differences (SSD) between pixels
Optical Flow Applications

Video Frames

[Feng & Perona]


Optical Flow Applications

Optical Flow Depth Reconstruction

[Feng & Perona]


Optical Flow Applications

Obstacle Detection: Unbalanced Optical Flow

[Temizer]
Optical Flow Applications

• Collision avoidance:
keep optical flow
balanced between
sides of image

[Temizer]

You might also like