Stereo Vision

Source: S. Lazebnik
34 Computer Vision: Algorithms and Applicati

(a) (b)
2
 Amount of horizontal movement is
534
…
Computer Vision: Algorithms and Applications (September 3, 2010 draft)
…inversely proportional to the distance from the camera

(a) (b) (c)

3
 Depth from Stereo
• Goal: recover depth by finding image coordinate x’
that corresponds to x
X
X

x z
x x’

f f
x'
C Baseline C’
B

4
Optic axes of 2 cameras are parallel

image planes z
camera L f
O Z
xl
baseline b
f
camera R xr
x-b

P=(x,z)
X
Y-axis is
perpendicular
z x z x-b z = y = y to the page.
= =
f xl f xr f yl yr
(from similar triangles)
 3D from Stereo Images
For stereo cameras with parallel optical axes,
focal length f, baseline b, corresponding image
points (xl,yl) and (xr,yr), the location of the 3D
point can be derived from previous slide’s
equations:
Depth z = f*b / (xl - xr) = f*b/d

x = xl*z/f or b + xr*z/f

y = yl*z/f or yr*z/f

Note that depth is inversely proportional to disparity

 Depth from Stereo
• Goal: recover depth by finding image coordinate x’ that
corresponds to x
• Sub-Problems
1. Calibration: How do we recover the relation of the cameras (if
not already known)?
2. Correspondence: How do we search for the matching point x’?

x'
7
 Correspondence Problem

x ?

• We have two images taken from cameras with different

intrinsic and extrinsic parameters

• How do we match a point in the first image to a point in the

second? How can we constrain our search?

8
Key idea: Epipolar constraint
X

x x’

x’

x’

Potential matches for x have to lie on the corresponding line l’.

Potential matches for x’ have to lie on the corresponding line l.

9
 Epipolar geometry: notation X

x x’

• Baseline – line connecting the two camera centers

• Epipoles
= intersections of baseline with image planes
= projections of the other camera center
• Epipolar Plane – plane containing baseline (1D family)

10
 Epipolar geometry: notation X

x x’

• Baseline – line connecting the two camera centers

• Epipoles
= intersections of baseline with image planes
= projections of the other camera center
• Epipolar Plane – plane containing baseline (1D family)
• Epipolar Lines - intersections of epipolar plane with image
planes (always come in corresponding pairs)
11
 Epipolar constraint
X

x x’

• If we observe a point x in one image, where

can the corresponding point x’ be in the
other image?

12
 Epipolar constraint
X

x x’

x’

x’

• Potential matches for x have to lie on the corresponding

epipolar line l’.

• Potential matches for x’ have to lie on the corresponding

epipolar line l.
13
 Moving on to stereo…
Fuse a calibrated binocular stereo pair to
produce a depth image
image 1 image 2

Dense depth map

Many of these slides adapted from

14
Steve Seitz and Lana Lazebnik
 Basic stereo matching algorithm

• If necessary, rectify the two stereo images to transform

epipolar lines into scanlines
• For each pixel x in the first image
– Find corresponding epipolar scanline in the right image
– Search the scanline and pick the best match x’
– Compute disparity x-x’ and set depth(x) = fB/(x-x’)
15
Stereo image rectification

16
 Example
Unrectified

Rectified

17
 Left Right

scanline

Matching cost
disparity

• Slide a window along the right scanline and

compare contents of that window with the
reference window in the left image
• Matching cost: SSD, SAD, or normalized cross
correlation
18
 Correspondence search
Left Right

scanline

SSD 19
 Correspondence search
Left Right

scanline

NCC 20
 Effect of window size

W=3 W = 20
• Smaller window
+ More detail
– More noise

• Larger window
+ Smoother disparity maps
– Less detail
– Fails near boundaries 21
 Results with window search
Data

Window-based matching Ground truth

22
 How can we improve window-based matching?

• So far, matches are independent for each

point

• What constraints or priors can we add?

23
 Stereo constraints/priors
• Uniqueness
– For any point in one image, there should be at
most one matching point in the other image

24
 Stereo constraints/priors
• Uniqueness
– For any point in one image, there should be at most
one matching point in the other image
• Ordering
– Corresponding points should be in the same order in
both views

25
 Stereo constraints/priors
• Uniqueness
– For any point in one image, there should be at most
one matching point in the other image
• Ordering
– Corresponding points should be in the same order in
both views

Ordering constraint doesn’t 26hold

 Priors and constraints
• Uniqueness
– For any point in one image, there should be at most one
matching point in the other image
• Ordering
– Corresponding points should be in the same order in both
views
• Smoothness
– We expect disparity values to change slowly (for the most
part)

27
Stereo as energy minimization

• What defines a good stereo correspondence?

1. Match quality
• Want each pixel to find a good match in the other image
2. Smoothness
• If two pixels are adjacent, they should (usually) move about
the same amount
28
 Matching windows:
Similarity Measure Formula

Sum of Absolute Differences (SAD)

Sum of Squared Differences (SSD)

Zero-mean SAD

Normalized Cross Correlation (NCC)

SAD SSD NCC Ground truth

29
http://siddhantahuja.wordpress.com/category/stereo-vision/
 Stereo reconstruction pipeline
Steps
– Calibrate cameras
– Rectify images
– Compute disparity
– Estimate depth

What will cause errors?

• Camera calibration errors
• Poor image resolution
• Occlusions
• Violations of brightness constancy
• Low-contrast image regions

30
Multi-view stereo ?

31
 Using more than two images

Multi-View Stereo for Community Photo Collections

M. Goesele, N. Snavely, B. Curless, H. Hoppe, S. Seitz
Proceedings of ICCV 2007,
32