Computational Imaging

Lecture 14: Image Deconvolution with the Half Quadratic

Splitting (HQS) Method
  Image Deconvolution – Brief Review
 A Bayesian Perspective of Inverse Problems
 Image Priors/Regularization and Total Variation (TV)
Today’s  Half-Quadratic Splitting (HQS) Method
Topic  Image Deconvolution with HQS
 Outlook on Unrolled Optimization
 Image Deconvolution

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution – Brief Review

Given: blurry & noisy image Desired: sharp & noise-free image

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution – Brief Review

 Image formation model:

𝑏𝑏 = 𝑐𝑐 ∗ 𝑥𝑥 + 𝜂𝜂

2D measurements additive noise

known 2D convolution kernel 2D target image

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution – Brief Review

 Image formation model:

 Convolution theorem:

 Inverse filtering:

 Wiener filtering:

 Duality of “signal processing” and “algebraic” interpretation:

𝑏𝑏 = 𝑐𝑐 ∗ 𝑥𝑥 𝒃𝒃 = 𝑪𝑪𝑪𝑪

GAMES204 Computational Imaging, Qilin Sun

Image Deconvolution – Inverse Filtering

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution – Wiener Filtering

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution

Problem: this is an ill-posed inverse problem, i.e., there are infinitely many
solutions that satisfy the measurements

Need some way to determine how “desirable” any one of these feasible solutions
is -> need an image prior

GAMES204 Computational Imaging, Qilin Sun

 A Bayesian Perspective of
Inverse Problems

GAMES204 Computational Imaging, Qilin Sun

 A Bayesian Perspective of Inverse Problems

Image formation model:

Interpret as random variables:

Probability of observation i:

Joint probability of all observations:

GAMES204 Computational Imaging, Qilin Sun

 A Bayesian Perspective of Inverse Problems

Bayes’ rule: 𝑝𝑝(𝒃𝒃|𝒙𝒙, 𝜎𝜎)𝒑𝒑(𝒙𝒙)

𝑝𝑝(𝒙𝒙|𝒃𝒃, 𝜎𝜎) = ∝ 𝑝𝑝(𝒃𝒃|𝒙𝒙, 𝜎𝜎)𝑝𝑝(𝒙𝒙)
𝑝𝑝(𝒃𝒃)

posterior image formation model prior

Maximum-a-posterior (MAP) solution:

GAMES204 Computational Imaging, Qilin Sun

 A Bayesian Perspective of Inverse Problems

Terminology: prior
regularizer

𝛹𝛹 (𝒙𝒙) = −𝑙𝑙𝑙𝑙𝑙𝑙(𝑝𝑝(𝑥𝑥))

data fidelity term regularization term

GAMES204 Computational Imaging, Qilin Sun

 Image Priors/Regularization
and Total Variation (TV)

GAMES204 Computational Imaging, Qilin Sun

 Examples of Image Priors / Regularizers

Laplace operator
GAMES204 Computational Imaging, Qilin Sun
 Total Variation (TV)

 Express (forward finite difference)

gradient as convolution! 0 0 0 0 0 0
𝑫𝑫𝑥𝑥 𝒙𝒙 = 𝑑𝑑𝑥𝑥 ∗ 𝑥𝑥, 𝑑𝑑𝑥𝑥 = 0 −1 1 𝑫𝑫𝑦𝑦 𝒙𝒙 = 𝑑𝑑𝑦𝑦 ∗ 𝑥𝑥, 𝑑𝑑𝑦𝑦 = 0 −1 0
𝒙𝒙 0 0 0 0 1 0

GAMES204 Computational Imaging, Qilin Sun

 Total Variation (TV)

better: isotropic easier: anisotropic

𝒙𝒙 𝑫𝑫𝑥𝑥 𝒙𝒙)2𝑖𝑖 + 𝑫𝑫𝑦𝑦 𝒙𝒙)2𝑖𝑖 𝑫𝑫𝑥𝑥 𝒙𝒙)2𝑖𝑖 + 𝑫𝑫𝑦𝑦 𝒙𝒙)2𝑖𝑖

GAMES204 Computational Imaging, Qilin Sun

 Total Variation (TV)

 Examples are mostly black, indicating that gradient magnitudes are close to 0 -> natural
images have sparse gradients!
 This intuition is well-captured by the TV pseudo-norm, either anisotropic or isotropic:

GAMES204 Computational Imaging, Qilin Sun

 Total Variation (TV)

The TV pseudo-norm is one of the most popular regularization schemes for natural
images!

Extensions to make it more general or applicable for other data:

 Hyper-Laplacian: Levin et al. 2009, Krishnan & Fergus 2009
 Total generalized variation: Bredies et al. 2009
 Frobenius norm of Hessian: Lefkimmiatis et al. 2003
 …

GAMES204 Computational Imaging, Qilin Sun

 How to solve inverse problem that
use these regularizers?

GAMES204 Computational Imaging, Qilin Sun

 Solving Regularized Inverse Problem

 Objective or “loss” function

of general inverse problem:

weight of regularizer

 Practical #1 go-to solution: Adam solver implemented in PyTorch

 3 simple steps, will explore in problem session & homework:

1. Implement evaluation of loss function
2. Set hyperparameters, including learning rate
3. Run

 The “fine print”: convenient but doesn’t always converge well

GAMES204 Computational Imaging, Qilin Sun

 Half-quadratic Splitting (HQS)
Method

GAMES204 Computational Imaging, Qilin Sun

 The Half-quadratic Splitting (HQS) Method

 Objective or “loss” function of

general inverse problem:
weight of regularizer

 Reformulate as:

𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑡𝑡𝑡𝑡 𝑫𝑫𝑫𝑫 − 𝒛𝒛 = 0

 Remove constraints using penalty term (equivalent for large ρ):

GAMES204 Computational Imaging, Qilin Sun

 The Half-quadratic Splitting (HQS) Method

 Alternating gradient descent approach to solving penalty

formulation leads to following iterative algorithm:

while not converged:

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with TV

 Generic:

 Deconv:

𝐱𝐱𝐱ℝN unknown sharp image

𝑪𝑪𝑪ℝNxN circulant convolution matrix for known kernel c
𝒛𝒛𝒛ℝ2N slack variable, twice the size of x!
𝑫𝑫𝒙𝒙 finite difference gradients, horizontal & vertical
𝑫𝑫 = 𝑫𝑫 ϵℝ2NxN
𝒚𝒚

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with TV

while not converged:

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with TV

x - update:

reformulate

find solution by setting gradient to 0

close-form solution

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with TV
x - update:

exploit duality of algebraic & signal processing interpretation

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with TV

x - update:

 Efficient x–update operates purely on 2D images with FFTs and

element-wise multiplications and divisions:

can pre-compute most parts

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with TV
x - update:

 Efficient z–update operates purely on 2D images with FFTs and

element-wise multiplications and divisions:

𝑘𝑘 = 𝜆𝜆�𝜌𝜌
This element-wise soft thresholding is the proximal operator for anisotropic TV, see
course notes on block soft thresholding for isotropic TV. 𝒗𝒗 = 𝑫𝑫𝑫𝑫

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with Denoiser
x - update:

𝑧𝑧𝑧𝑧ℝ𝑁𝑁

no matrix D!

 Efficient x–update operates purely on 2D images with FFTs and

element-wise multiplications and divisions:

GAMES204 Computational Imaging, Qilin Sun

 HQS for Image Deconvolution with Denoiser
x - update:

 Efficient z–update uses arbitrary denoiser D(⋅) , such as DnCNN and 𝜆𝜆

non-local means, using noise variance 𝜎𝜎 2 =
𝜌𝜌

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution with HQS

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution with HQS

GAMES204 Computational Imaging, Qilin Sun

 Image Deconvolution with HQS
HQS for deconvolution with denoiser

HQS for deconvolution with TV

GAMES204 Computational Imaging, Qilin Sun

 HQS - Convergence Criterion
 Run or “unroll” HQS for K iterations
 Run until change in residual between iterations is < threshold

...

GAMES204 Computational Imaging, Qilin Sun

 Outlook on Unrolled Optimization

GAMES204 Computational Imaging, Qilin Sun

 Outlook on Unrolled Optimization
b
 Run or “unroll” HQS for K iterations
 Interpret as unrolled feedforward network:

...
...

Credit: Diamond et al. 2017

GAMES204 Computational Imaging, Qilin Sun
 Outlook on Unrolled Optimization
b
 Run or “unroll” HQS for K iterations
 Interpret as unrolled feedforward network:

Benefits over unrolled optimization

 Learnable parameters: λ(k), ρ(k) denoiser
 DenseNet-like skip connections D(k)
 Denoiser/regularizer can adapt to matrix C
 Can train with advanced loss functions (perceptual,

...
...

...
adversarial, other network, …)

Credit: Diamond et al. 2017

GAMES204 Computational Imaging, Qilin Sun
  Image Deconvolution – Brief Review
 A Bayesian Perspective of Inverse Problems
 Image Priors/Regularization and Total Variation (TV)
Today’s  Half-Quadratic Splitting (HQS) Method
Topic  Image Deconvolution with HQS
 Outlook on Unrolled Optimization
Thank You!