Introduction to Medical Image Registration

Tom Vercauteren, PhD

Learn2Reg 2019: Tutorial on Deep Learning in Medical Image Registration

MICCAI 2019
Includes slides adapted from Marc Modat, Matt Clarkson and more

?
What is medical image registration?
• Aim: Establishing spatial correspondences between images
• Also referred to as Spatial Normalisation

• Image registration focuses on finding a spatial transformation (or mapping) between the
spaces of the images

• Registration is an important question but typically only one component of a clinically relevant
solution

Learn2Reg 2019 - Vercauteren 2

Why do we need it? Fuse, track, control
• Patient motion (alignment of temporal series)
• Patient changes (pre- / post-treatment images)
• Patient comparison (atlas-based analysis)
• Information fusion (complementary modalities or timings, planning transfer)
• Motion compensation for improved reconstruction
• Field-of view enlargement (mosaicking)
• Localisation and visual servoing
• Etc.

Learn2Reg 2019 - Vercauteren 3

Some key components
• Spatial transformation model
• Rigid, affine, local basis function-based, displacement field, etc.
• Image matching driver
• Sparse paired features, image intensity comparisons, etc.
• Regularisation
• Spatial transformation smoothness prior, etc.
• Data-driven priors
• Optimisation strategy
• Gradient descent, derivative-free continuous optimiser, discrete graph-based optimiser, etc.
• Statistical, machine-learning based

Learn2Reg 2019 - Vercauteren 4

Two main approaches to image registration
• Feature-based matching
• Identify paired corresponding points, lines or other geometric primitives
• Fit a spatial transformation model to these correspondences

• Intensity-based registration
• Define a cost function to compare the similarity between images
• Warp the images with a spatial transformation
• Find the spatial transformation that optimises the cost function between the warped images

• In this introduction, we focus on intensity-based registration

Learn2Reg 2019 - Vercauteren 5

Classical intensity-based registration: An optimization problem
• Note that notations vary widely in the field

• F : Fixed image
• Spatial transformation leaves from its domain ΩF
• M : Moving image
• Spatial transformation arrives to its domain ΩM
• s* : Sought spatial transformation
• From ΩF to ΩM
• x ∈ ΩF
• G : Transformation model
• Space of the spatial transformations s
• Sim(·, ·) : Image similarity measure
• Reg(·) : (Optional) regularization term

Learn2Reg 2019 - Vercauteren 6

Examples of spatial transformations spaces and related complexity
Rigid body Projective Diffeomorphism

• Spatial transformations do not necessarily form vector spaces

• Addition: no geometric meaning
• s1,s2ÎG Þ s = s1+s2ÏG
+ = ??

• Natural operation: composition

• s1,s2ÎG Þ s = s1◦s2ÎG, s: pÎW, s(p)=s1(s2(p))

○ =
• Lie group structure
• Need to find a good intrinsic parameterization of the space or use constrained optimisation

Learn2Reg 2019 - Vercauteren 7

Classical intensity-based registration: An optimization problem
• Optimisation problem to solve (omitting weighting factors)
• s* = arg mins Sim(F, M ◦ s) + Reg(s)
= arg mins E(s)
• Typically solved iteratively

Spatial
Transformation s

Moving Image M Iterations

Optimisation
Fixed Image F

Energy Model E Transformation Model G

Similarity Measure Sim Regularization Term Reg

Learn2Reg 2019 - Vercauteren 8

From classical to learning-based approaches
• Classical approach
• Rely on one pair or images F and M
• s* = arg mins Sim(F, M ◦ s) + Reg(s)

• Extension to learning-based approach (omitting weighting factors)

• Exploit many image pairs Fi and Mi
• Potentially use extra information E available only at training time (annotations, simulated ground-truth)
• Define a supervised loss SuppLoss
• Use a function approximator (e.g. a CNN) f θ (∙) parameterised by θ to encode s
• s = f θ ( F, M )
• Optimise over all training set
• θ * = arg minθ EP(F, M)[ Sim(F, M ◦ fθ(F,M) ) + Reg( fθ(F,M) ) + SupLoss( E, F, M, fθ(F,M) ) ]
• Many other learning-based options possible (e.g. learn Sim or learn the update step)

Learn2Reg 2019 - Vercauteren 9

Intensity-based registration: hidden difficulties
• Many choices to make
• Sim, Reg, G, optimiser, parameters, weighting, etc.
• Most choices should be problem specific
• Many of these can be difficult to learn from the data
• Little ground truth available
• No magic recipe!

• Classical approaches pose difficult optimisation problems

• High dimensionality
• Strongly non-linear problem
• Strongly non-convex energy function with many local minima
• Shares many features of deep learning!

Learn2Reg 2019 - Vercauteren 10

From discrete samples to a continuous problem
• Digital images are typically a collection of samples on a regular grid of voxels
• Spatial transformations cannot be accurately represented by exact (discrete) voxel matches
• Need to be a continuous representation
• Image similarity measure typically rely on continuous integrals
• Interpolation and resampling used to “work-around” this conundrum
• For deep learning-based approaches, it is important to make sure the steps are differentiable

Learn2Reg 2019 - Vercauteren 11

Interpolation and resampling
• For a voxel position x∈ΩF, s(x) does not in general fall on
a voxel position in ΩM
Interpolation schemes
• What does M ◦ s(x) = M(s(x)) mean? ï4.9

• For Nyquist–Shannon sampling theorem band- ï4.91

limited continuous function can be exactly recovered ï4.92

from a set of discrete samples by using a sinc kernel ï4.93

ï4.94
• In practice, due to the sinc computational complexity

Value
ï4.95
and the related artefacts (e.g. ringing), nearest
ï4.96
neighbour, linear or cubic interpolation is used
ï4.97

ï4.98

ï4.99

ï5
0 10 20 30 40 50 60 70 80 90 100
Point indices

Learn2Reg 2019 - Vercauteren 12

Interpolation and resampling (2)

Talk about direction of transform

s: 20° Resampling:
(clockwise) -20°
rotation (anti-clockwise) M ◦s
s(Ω M rotation
ΩF F)

Learn2Reg 2019 - Vercauteren 13

Impact of the interpolation strategy
• Resampling many times with a small rotation
• Avoid resampling many times! Linear interpolation is fine if used once

Using nearest-neighbor Using linear interpolation Using cubic interpolation

interpolation
Learn2Reg 2019 - Vercauteren 14
Basic registration walk-through (1)
• s(x) = S·x
• May use homogeneous coordinates
• Sim(F, M ◦ s) = ∫|F(x) – M(S·x)|² dx
• Reg(s) = 0
• Where do we take the integral on?
• x ∈ ΩF
• s(x) ∈ ΩM
• Ωs = ΩF ∩ s-1(ΩM)

s(Ω
F)
ΩF ΩM
Learn2Reg 2019 - Vercauteren 15
Basic registration walk-through (2)
• How to compute the integral?
• ∫Ωs|F(x) – M(S·x)|² dx ≈ ∑ x∈Ωs|F(x) – M(S·x)|²
• Sim(F, M ◦ s) ≜ ∑ x∈Ωs|F(x) – M(S·x)|²
• Didn’t we introduce aliasing? See Cardoso et al. MICCAI’15

High Res Resampling Resampled to lution Grid

~ →
Image Grid High Res Grid Gaussian sfPSF

~ →

Learn2Reg 2019 - Vercauteren 16

Basic registration walk-through (3)
• Sim(F, M ◦ s) ≜ ∑ x∈Ωs|F(x) – M(S·x)|²
• How to optimise it?
• We probably need the gradient of Sim (or half of it…)
• Use the chain rule
• ∇Sim(S) = -∑ x∈Ωs (F(x) – M(S·x)) · (∇M)(S·x) T · (xT⨂IdN)
• Didn’t we miss the fact that Ωs depends on S?
• Let’s neglect this…
• Also, what is ∇M? Does it need to be compatible with the interpolation strategy? How do we
evaluate it at non-grid points?

Learn2Reg 2019 - Vercauteren 17

Basic registration walk-through (4)
• We can now do a gradient descent
• Sn+1 = Sn + λ ∇Sim(Sn)
• We need a rule for the step length. Ad-hoc? Line search?
• We only get first order convergence. Can we get better?
• Newton-Raphson (using the Hessian) could be an option
• How do we get a stable estimate of the Hessian of the cost function?
• Can this scale for deformable registration?

Learn2Reg 2019 - Vercauteren 18

Measures of image similarity – Examples
• Sum of squared differences (SSD) for identical intensities + Gaussian noise
• Normalised Cross-correlation (NCC) for affine relationships
• Illumination differences, small contrast changes
• Local version (LNCC) provides good versatility
• Joint entropy, the dispersion in the joint image histogram and (normalised) mutual information
for non-parametric statistical relationship between intensities
• Multimodal registration

Learn2Reg 2019 - Vercauteren 19

Joint histogram

Initial joint
histogram

Learn2Reg 2019 - Vercauteren 20

Spatial multiresolution pyramids
• Multiscale representation of images
• Capture large-scale information efficiently
on small images
• Propagate to finer grained levels and refine
• Simple downsampling /subsampling would
introduce aliasing
• Apply a low-pass first and then downsample
• Gaussian smoothing + downsampling by a
factor 2 is the most classical approach

Learn2Reg 2019 - Vercauteren 21

Gaussian pyramids for image registration
Often improves speed,
accuracy and robustness of
registration

Learn2Reg 2019 - Vercauteren 22

Do we need all this sophistication?
• Let’s look at a very simple deformable registration algorithm: CURT
• Rohlfing IEEE TMI 2012 Feb; 31(2)

nf
nm = bNm c
Nf
Learn2Reg 2019 - Vercauteren 23
CURT
• From the paper
• “We introduce a new non-rigid registration algorithm based on a closed-form solution to
maximizing the Rank Correlation criterion. The new algorithm produces more accurate
registrations than two state-of-the-art methods as judged by image similarity and tissue
overlap scores. It is also two to three orders of magnitude faster, requires no affine pre-
registration, and has no tunable parameters.”

Learn2Reg 2019 - Vercauteren 24

 CURT:
Completely
Useless
Registration
Learn2Reg 2019 - Vercauteren
Tool 25
Important definitions for validation (see Fitzpatrick & West, 2001)

Fiducial Fiducial Target Registration

Registration Error Localisation Error Error (TRE): the
(FRE): root- mean (FLE): the error in distance between
square distance locating the fiducial corresponding points
between corresponding points other than the fiducial
fiducial points after points after registration
registration
This is what really
matters!

Learn2Reg 2019 - Vercauteren 26

TRE increases as the distance of the target from the fiducial centroid

Learn2Reg 2019 - Vercauteren 27

FRE is not a reliable indicator of registration accuracy (!!)
• FRE is independent of fiducial configuration
• FRE is independent of bias errors (e.g., MRI gradient, digitizer camera malalignment, bent
handheld probe)

• TRE has an approximate N-1/2 dependence

- Mean square fiducial registration error

- Number of fiducials

- Distance (squared) between target point and fiducial

configuration principle axis k.
- RMS (squared) between fiducial points and fiducial
configuration principle axis k.

Learn2Reg 2019 - Vercauteren 28

Considerations remain valid for learning-based image registration

Image-intensity-based similarity measure

e.g. de Vos et al 2017, Yang et al 2017, Cao et al 2017

Moving Image
Convolutional
Neural Resampler
Network
Output DDF

Fixed Image

Deformation Warped
Overall
Regularisation Moving
Training
Loss Similarity Image
Measure

Learn2Reg 2019 - Vercauteren 29

Take Home Messages
• For good image registration performance
• Study application constraints
• Pick most suitable tools w.r.t constraints
• Understand the numerics involved
• Evaluation is more than important
• There is room for research!

Learn2Reg 2019 - Vercauteren 30

Introduction to Medical Image Registration

Tom Vercauteren, PhD

Learn2Reg 2019: Tutorial on Deep Learning in Medical Image Registration

MICCAI 2019
Includes slides adapted from Marc Modat, Matt Clarkson and more