Sampling and Reconstruction

Most slides from Steve Marschner

15-463: Computational Photography Alexei Efros, CMU, Fall 2007

Sampling and Reconstruction

1D Example: Audio

low frequencies

high

Sampled representations
How to store and compute with continuous functions? Common scheme for representation: samples
write down the functions values at many points

2006 Steve Marschner 4

[FvDFH fig.14.14b / Wolberg]

Reconstruction
Making samples back into a continuous function
for output (need realizable method) for analysis or processing (need mathematical method) amounts to guessing what the function did in between

2006 Steve Marschner 5

[FvDFH fig.14.14b / Wolberg]

Sampling in digital audio

Recording: sound to analog to samples to disc Playback: disc to samples to analog to sound again
how can we be sure we are filling in the gaps correctly?

2006 Steve Marschner 6

Sampling and Reconstruction

Simple example: a sign wave

2006 Steve Marschner 7

Undersampling
What if we missed things between the samples? Simple example: undersampling a sine wave
unsurprising result: information is lost

2006 Steve Marschner 8

Undersampling
What if we missed things between the samples? Simple example: undersampling a sine wave
unsurprising result: information is lost surprising result: indistinguishable from lower frequency

2006 Steve Marschner 9

Undersampling
What if we missed things between the samples? Simple example: undersampling a sine wave
unsurprising result: information is lost surprising result: indistinguishable from lower frequency also was always indistinguishable from higher frequencies aliasing: signals traveling in disguise as other frequencies

2006 Steve Marschner 10

Aliasing in video

Slide by Steve Seitz

Aliasing in images

Whats happening?
Input signal:

Plot as image:

x = 0:.05:5; imagesc(sin((2.^x).*x)) Alias! Not enough samples

Antialiasing
What can we do about aliasing? Sample more often
Join the Mega-Pixel crazy of the photo industry But this cant go on forever

Make the signal less wiggly

Get rid of some high frequencies Will loose information But its better than aliasing

Preventing aliasing
Introduce lowpass filters:
remove high frequencies leaving only safe, low frequencies choose lowest frequency in reconstruction (disambiguate)

2006 Steve Marschner 15

Linear filtering: a key idea

Transformations on signals; e.g.:
bass/treble controls on stereo blurring/sharpening operations in image editing smoothing/noise reduction in tracking

Key properties
linearity: filter(f + g) = filter(f) + filter(g) shift invariance: behavior invariant to shifting the input
delaying an audio signal sliding an image around

Can be modeled mathematically by convolution

2006 Steve Marschner 16

Moving Average
basic idea: define a new function by averaging over a sliding window a simple example to start off: smoothing

2006 Steve Marschner 17

Weighted Moving Average

Can add weights to our moving average Weights [, 0, 1, 1, 1, 1, 1, 0, ] / 5

2006 Steve Marschner 18

Weighted Moving Average

bell curve (gaussian-like) weights [, 1, 4, 6, 4, 1, ]

2006 Steve Marschner 19

Moving Average In 2D
What are the weights H?
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 0 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Slide by Steve Steve Seitz 2006 Marschner 20

Cross-correlation filtering
Lets write this down as an equation. Assume the averaging window is (2k+1)x(2k+1):

We can generalize this idea by allowing different weights for different neighboring pixels:

This is called a cross-correlation operation and written: H is called the filter, kernel, or mask.
Slide by Steve Steve Seitz 2006 Marschner 21

Gaussian filtering
A Gaussian kernel gives less weight to pixels further from the center of the window
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 0 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 1 2 4 2 1 2 1

22

Slide by Steve Seitz

Thi k

li

ti

ti

Mean vs. Gaussian filtering

23

Slide by Steve Seitz

Convolution
cross-correlation:

A convolution operation is a cross-correlation where the filter is flipped both horizontally and vertically before being applied to the image:

It is written:

Suppose H is a Gaussian or mean kernel. How does convolution differ from cross-correlation?
Slide by Steve Seitz

Convolution is nice!
Notation: Convolution is a multiplication-like operation
commutative associative distributes over addition scalars factor out identity: unit impulse e = [, 0, 0, 1, 0, 0, ]

Conceptually no distinction between filter and signal Usefulness of associativity

often apply several filters one after another: (((a * b1) * b2) * b3) this is equivalent to applying one filter: a * (b1 * b2 * b3)
2006 Steve Marschner 25

Tricks with convolutions

Image half-sizing

This image is too big to fit on the screen. How can we reduce it? How to generate a halfsized version?

Image sub-sampling

1/8 1/4

Throw away every other row and column to create a 1/2 size image - called image sub-sampling
Slide by Steve Seitz

Image sub-sampling

1/2

1/4

(2x zoom)

1/8

(4x zoom)

Aliasing! What do we do?

Slide by Steve Seitz

Gaussian (lowpass) pre-filtering

G 1/8 G 1/4

Gaussian 1/2

Solution: filter the image, then subsample

Filter size should double for each size reduction. Why?
Slide by Steve Seitz

Subsampling with Gaussian pre-filtering

Gaussian 1/2

G 1/4

G 1/8

Slide by Steve Seitz

Compare with...

1/2

1/4

(2x zoom)

1/8

(4x zoom)

Slide by Steve Seitz

Gaussian (lowpass) pre-filtering

G 1/8 G 1/4

Gaussian 1/2

Solution: filter the image, then subsample

Filter size should double for each size reduction. Why? Slide by Steve Seitz How can we speed this up?

Image Pyramids

Known as a Gaussian Pyramid [Burt and Adelson, 1983]

In computer graphics, a mip map [Williams, 1983] A precursor to wavelet transform
Slide by Steve Seitz

A bar in the big images is a hair on the zebras nose; in smaller images, a stripe; in the smallest, the animals nose

Figure from David Forsyth

What are they good for?

Improve Search
Search over translations
Like project 1 Classic coarse-to-fine strategy

Search over scale

Template matching E.g. find a face at different scales

Pre-computation
Need to access image at different blur levels Useful for texture mapping at different resolutions (called mip-mapping)

Gaussian pyramid construction

filter mask

Repeat
Filter Subsample

Until minimum resolution reached

can specify desired number of levels (e.g., 3-level pyramid)

The whole pyramid is only 4/3 the size of the original image!
Slide by Steve Seitz

Continuous convolution: warm-up

Can apply sliding-window average to a continuous function just as well
output is continuous integration replaces summation

2006 Steve Marschner 38

Continuous convolution
Sliding average expressed mathematically:

note difference in normalization (only for box)

Convolution just adds weights

weighting is now by a function weighted integral is like weighted average again bounds are set by support of f(x)

2006 Steve Marschner 39

One more convolution

Continuousdiscrete convolution

used for reconstruction and resampling

2006 Steve Marschner 40

Reconstruction

2006 Steve Marschner 41

Resampling
Changing the sample rate
in images, this is enlarging and reducing

Creating more samples:

increasing the sample rate upsampling enlarging

Ending up with fewer samples:

decreasing the sample rate downsampling reducing

2006 Steve Marschner 42

Resampling
Reconstruction creates a continuous function
forget its origins, go ahead and sample it

2006 Steve Marschner 43

Cont.disc. convolution in 2D
same convolutionjust two variables now

loop over nearby pixels, average using filter weight looks like discrete filter, but offsets are not integers and filter is continuous remember placement of filter relative to grid is variable

2006 Steve Marschner 44

A gallery of filters
Box filter
Simple and cheap

Tent filter
Linear interpolation

Gaussian filter
Very smooth antialiasing filter

B-spline cubic
Very smooth

2006 Steve Marschner 45

Box filter

2006 Steve Marschner 46

 2006 Steve Marschner 47

 2006 Steve Marschner 48

 2006 Steve Marschner 49

Effects of reconstruction filters

For some filters, the reconstruction process winds up implementing a simple algorithm Box filter (radius 0.5): nearest neighbor sampling
box always catches exactly one input point it is the input point nearest the output point so output[i, j] = input[round(x(i)), round(y(j))] x(i) computes the position of the output coordinate i on the input grid

Tent filter (radius 1): linear interpolation

tent catches exactly 2 input points weights are a and (1 a) result is straight-line interpolation from one point to the next

2006 Steve Marschner 50

Properties of filters
Degree of continuity Impulse response Interpolating or no Ringing, or overshoot

interpolating filter used for reconstruction

2006 Steve Marschner 51

Ringing, overshoot, ripples

Overshoot
caused by negative filter values

Ripples
constant in, non-const. out ripple free when:

2006 Steve Marschner 52

Yucky details
What about near the edge?
the filter window falls off the edge of the image need to extrapolate methods:
clip filter (black) wrap around copy edge reflect across edge vary filter near edge

2006 Steve Marschner 53

Median filters
A Median Filter operates over a window by selecting the median intensity in the window. What advantage does a median filter have over a mean filter? Is a median filter a kind of convolution?

Slide by Steve Steve Seitz 2006 Marschner 54

Comparison: salt and pepper noise

Mean Gaussian Median

3x3

5x5

7x7

Slide by Steve Steve Seitz 2006 Marschner 55