Computational Imaging

Lecture 13: Computing Toolbox: Image Blurry

 Q:Why are Our Images
Blurry?

GAMES204 Computational Imaging, Qilin Sun

  Lens Imperfections and Physical Limit.
 Camera Shake.
 Scene Motion.
Today’s
Topic  Depth Defocus.
 Lens Imperfections and
Physical Limit

GAMES204 Computational Imaging, Qilin Sun

Lens Imperfections

 Ideal lens: A point maps to a point at a certain plane.

1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓

object distance S sensor distance S’

GAMES204 Computational Imaging, Qilin Sun

Lens Imperfections

 Ideal lens: A point maps to a point at a certain plane.

 Real lens: A point maps to a circle that has non-zero minimum radius among all planes.

1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓

object distance S sensor distance S’

What is the effect of this on the images we capture?
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections

 Ideal lens: A point maps to a point at a certain plane.

 Real lens: A point maps to a circle that has non-zero minimum radius among all planes.

1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓
blur kernel

object distance S sensor distance S’

Shift-invariant blur.
GAMES204 Computational Imaging, Qilin Sun
Lens Imperfections
(Important note: Oblique aberrations like coma and distortion are
 What causes lens imperfections? not shift-invariant blur and we do not consider them here!)
 Aberrations.

 Diffraction

small large
aperture aperture

GAMES204 Computational Imaging, Qilin Sun

 Lens: An Optical Low-pass Filter

Point spread function (PSF): The blur kernel of a lens.

 “Diffraction-limited” PSF: No aberrations, only diffraction. Determined by aperture shape.

1 1 1
+ =
𝑆𝑆 ′ 𝑆𝑆 𝑓𝑓
blur kernel

diffraction-limited PSF
of a circular aperture
object distance S sensor distance S’ (Airy pattern)

GAMES204 Computational Imaging, Qilin Sun

 Basics of Diffraction Theory

Assume that we can use:

 Fraunhofer diffraction
i.e., distance of sensor and aperture is large relative to wavelength.

 Incoherent illumination
i.e., the light we are measuring is not laser light.

Ignore various scale factors. Different functions are not drawn to scale.

GAMES204 Computational Imaging, Qilin Sun

 Basics of Diffraction Theory

aperture:
rect 𝑥𝑥

The 1D case
?
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

coherent point spread

function: sinc 𝑥𝑥
aperture:
rect 𝑥𝑥

The 1D case ?
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

coherent point spread

function: sinc 𝑥𝑥
aperture:
rect 𝑥𝑥 ?

optical transfer
The 1D case
function: tent 𝑥𝑥
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

coherent point spread

function: sinc 𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥 function: sinc 2 𝑥𝑥

why do we get the

same result?
optical transfer
The 1D case
function: tent 𝑥𝑥
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

coherent point spread

function: sinc 𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥 function: sinc 2 𝑥𝑥

what happens if we
increase the aperture
size?
optical transfer
The 1D case
function: tent 𝑥𝑥
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

coherent point spread

function: sinc 2𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥/2 function: sinc 2 2𝑥𝑥

optical transfer
The 1D case
function: tent 𝑥𝑥/2
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

coherent point spread

function: sinc 10𝑥𝑥
aperture: incoherent point spread
rect 𝑥𝑥/10 function: sinc 2 10𝑥𝑥

As the aperture size … point spread function

increases… becomes smaller
optical transfer
The 1D case function: tent(𝑥𝑥/
10)
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

aperture incoherent point spread

function

As the aperture size … point spread function

increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

aperture incoherent point spread

function

As the aperture size … point spread function

increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

Why do we prefer circular apertures?

aperture incoherent point spread

function

As the aperture size … point spread function

increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
 Basics of Diffraction Theory

Other shapes produce very

anisotropic blur.

aperture incoherent point spread

function

As the aperture size … point spread function

increases… becomes smaller
optical transfer
The 2D case
function
GAMES204 Computational Imaging, Qilin Sun
 Lens: An Optical Low-pass Filter

Point spread function (PSF): The blur kernel of a lens.

 “Diffraction-limited” PSF: No aberrations, only diffraction. Determined by aperture shape.

blur kernel

aperture

diffraction-limited PSF
of a circular aperture
optical transfer function object distance S sensor distance S’ (Airy pattern)
(OTF)
GAMES204 Computational Imaging, Qilin Sun
 Lens: An Optical Low-pass Filter

* =

image from a perfect lens imperfect lens PSF image from imperfect lens

x * c = b
If we know b and c, can we recover x?
GAMES204 Computational Imaging, Qilin Sun
 Camera Shake

GAMES204 Computational Imaging, Qilin Sun

 Camera Shake: Motion PSF

Motion PSF is a
H Function of:
Energy ~ 1/ speed

 Motion path
 Motion speed

Y
X
Spatial spread

GAMES204 Computational Imaging, Qilin Sun

Camera Shake as A Filter

* =
PSF
image from static camera from camera motion image from shaky camera

x * c = b
If we know b and c, can we recover x?
GAMES204 Computational Imaging, Qilin Sun
 Sharp image Blur kernel
Multiple Possible Solutions

= *

= *
Blurry image
How do we
= * detect this
PSF?

GAMES204 Computational Imaging, Qilin Sun

Use Prior Information

Among all the possible pairs of images and blur kernels, select the ones where:

 The image “looks like” a natural image.

 The kernel “looks like” a motion PSF.

GAMES204 Computational Imaging, Qilin Sun

Shake Kernel Statistics

Gradients in natural images follow a

characteristic “heavy-tail” distribution.
sharp
natural
image

blurry
natural
image

Can be approximated by‖∇x ‖0.8

GAMES204 Computational Imaging, Qilin Sun
Use Prior Information

Among all the possible pairs of images and blur kernels, select the ones where:

 The image “looks like” a natural image.

Gradients in natural images follow a

characteristic “heavy-tail” distribution.

 The kernel “looks like” a motion PSF.

Shake kernels are very sparse, have

continuous contours, and are always positive

How to use this information for blind deconvolution?

GAMES204 Computational Imaging, Qilin Sun
 Three Sources of Information

b = observed image c = blur kernel x = sharp image

𝑝𝑝(𝑐𝑐, 𝑥𝑥|𝑏𝑏) = 𝑘𝑘 𝑝𝑝(𝑏𝑏|𝑐𝑐, 𝑥𝑥) 𝑝𝑝(𝑥𝑥) 𝑝𝑝(𝑏𝑏)

GAMES204 Computational Imaging, Qilin Sun

 Regularized Blind Deconvolution

Solve regularized least-squares optimization

minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1

What does each term in this summation correspond to?

GAMES204 Computational Imaging, Qilin Sun

 Regularized Blind Deconvolution

Solve regularized least-squares optimization

minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1

* =
Estimated
Sharp image Blur kernel Blurry image

Note: Solving such optimization problems is complicated (no longer linear least squares).
GAMES204 Computational Imaging, Qilin Sun
 A Demonstration

input deconvolved image and kernel

This image looks worse

than the original…

This doesn’t look like a

plausible shake kernel…

GAMES204 Computational Imaging, Qilin Sun

 Regularized Blind Deconvolution

Solve regularized least-squares optimization

minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1

cost function

GAMES204 Computational Imaging, Qilin Sun

 Regularized Blind Deconvolution

Solve regularized least-squares optimization

minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1

cost function

inverse loss

Where in this graph is the

solution we find?

pixel intensity
GAMES204 Computational Imaging, Qilin Sun
 Regularized Blind Deconvolution

Solve regularized least-squares optimization

minx,c ‖b – c ∗ x‖2 + ‖∇x‖0.8 + ‖ c ‖1

cost function

inverse loss many plausible solutions here

Rather than keep just

maximum, do a weighted
average of all solutions

optimal solution pixel intensity

GAMES204 Computational Imaging, Qilin Sun
 A Demonstration

input maximum-only average

GAMES204 Computational Imaging, Qilin Sun

 Image Artifacts & Estimated Kernels

Blur kernels

Image patterns

Note: blur kernels were inferred from large image patches,NOT the image patterns shown
GAMES204 Computational Imaging, Qilin Sun
 Scene Motion

GAMES204 Computational Imaging, Qilin Sun

Scene Motion Blur

Moshe Ben-Ezra and Shree K. Nayar Motion Deblurring Using Hybrid Imaging
GAMES204 Computational Imaging, Qilin Sun
 Depth Defocus

GAMES204 Computational Imaging, Qilin Sun

 Recall: Out of Focus Blur
Out of focus blur

circle of confusion
Focal plane

GAMES204 Computational Imaging, Qilin Sun

 Recall: Out of Focus Blur
Out of focus blur

circle of confusion
Focal plane

GAMES204 Computational Imaging, Qilin Sun

 Recall: Out of Focus Blur
Out of focus blur

circle of confusion
Focal plane

GAMES204 Computational Imaging, Qilin Sun

 PSF Behavior of Different Depths

0.5m 0.7m 1.0m 1.5m

AL2550

0.5m 0.7m 1.0m 1.5m

ACA254

0.5m 0.7m 1.0m 1.5m

Ours

Qilin Sun et.al End-to-end Complex Lens Design with Differentiable Ray Tracing
GAMES204 Computational Imaging, Qilin Sun
 Depth Defocus Examples

https://digital-photography-school.com/out-of-focus-photos/
GAMES204 Computational Imaging, Qilin Sun
  Lens Imperfections and Physical Limit.
 Camera Shake.
 Scene Motion.
Today’s
Topic  Depth Defocus.
Thank You!