Caroline PETITJEAN

¡ Segmentation
¡ Semantic segmentation
¡ Instance segmentation
 retinal vessels from retinal fundus images
partition of an
image into several
"coherent" parts

(without any attempt at

understanding what these
parts represent)

Cf binary segmentation
https://www.groundai.com/project/retinal-vessel-segmentation-
based-on-fully-convolutional-networks/1
C. Petitjean LITIS - Medical Image Acquisition and Processing 3
¡ semantic segmentation: partition the image into semantically
meaningful parts and to classify each part into one of the pre-
determined classes (add a label)

C. Petitjean LITIS - Medical Image Acquisition and Processing

https://ai-pool.com/d/could-you-explain-me-how-instance-segmentation-works 4
State of the art in image segmentation
(somehow in chronological order)

¡ Active contours/deformable models (1990’s)

¡ Graph cut segmentation (2000’s)

¡ Multi atlas registration based: especially for

medical images

¡ Deep (machine) learning based (from 2015)

C. Petitjean LITIS - Medical Image Acquisition and Processing 5
 Kass, Witkin, Terzopoulos 1988
¡ A deformable contour or curve is attracted by the
object boundary
¡ An energy functional E is associated with the curve.
¡ The problem of finding object boundary is cast as an
energy minimization problem.

http://xphilipp.developpez.com/contribuez/
C. Petitjean LITIS - Medical ImageXu and Prince,
Acquisition Gradient Vector Flow, John Hopkins
and Processing 6 U
 Initialisation : Iterative optimisation: the active
The curve must be close to contour is deformed until it
the object to be reaches a position of MINIMUM
segmented ENERGY
http://personal.ee.surrey.ac.uk/Personal/R.Bowden/publications/bmvc97/image4.gif

C. Petitjean LITIS - Medical Image Acquisition and Processing 8

Source : O. Lauchaud,
Bordeaux
C. Petitjean LITIS - Medical Image Acquisition and Processing 10
 Snake definition
C = {v( s) = ( x( s), y ( s)), s Î [0,1]} P
9 P8
P10
P7

P11 P6

P3

P1 P5
P2 P4

Regularization term
smoothness +
geometric properties
Esnake = Einterne + Eexterne Data-driven term
based on image content:
of the contour gradient….

• Propriétés intrinsèques • Propriétés locales de l’image

• Longueur, courbure… autour du snake

GOAL: find C such that Esnake(C) is minimum

 A word about curvature
¡ Curvature: measures at what rate the tangent is
changing along the contour
¡ Second derivative of v(s)

12
¡ Internal energy : measures the curve smoothness
1
Eint = ò (a v' ( s ) + b v' ' ( s ) 2 )ds
2
0
Elasticity Curvature
n n -1

å P -P åP
2 2
Eelastic = i i -1
Erigidité = i -1 - 2 Pi + Pi +1
i =1 i =1

Curve evolution when Curve evolution when

elastic energy stifness energy
is minimized is minimized
¡ Related to the image content
¡ How designing this energy such that it’s minimum on
the area of the interest, ie the object boundaries?
 1
¡ Related to the image content Eext = l ò g 2 ( I (v( s )))ds
0

g is a decreasing function of the image gradient

¡ On a contour : high gradient à g ≈ 0

¡ In an homogeneous area: low gradient à high g
16
¡ How to find C that minimizes E = Eint + Eext ?

C = {v( s) = ( x( s), y ( s)), s Î [0,1]}

1 1
Eint = ò (a v' ( s ) + b v' ' ( s ) )ds
2 2
Eext = l ò g 2 ( I (v( s )))ds
0 0
¡ How to find C that minimizes E = Eint + Eext ?

C = {v( s) = ( x( s), y ( s)), s Î [0,1]}

1 1
Eint = ò (a v' ( s ) + b v' ' ( s ) )ds
2 2
Eext = l ò g 2 ( I (v( s )))ds
0 0

¡ Find the C=v(s) that minimizes:

E = ò f ( s, v( s ), v' ( s ))ds
¡ How to find C that minimizes E = Eint + Eext ?

C = {v( s) = ( x( s), y ( s)), s Î [0,1]}

1 1
Eint = ò (a v' ( s ) + b v' ' ( s ) )ds
2 2
Eext = l ò g 2 ( I (v( s )))ds
0 0

¡ Find the C=v(s) that minimizes:

E = ò f ( s, v( s ), v' ( s ))ds

Euler-Lagrange to the rescue !!

¡ Calculus of variation E = ò f ( s, v( s ), v' ( s ))ds
¶f d ¶f
In order for E to reach an extremum, v(s) needs to verify: - ( )=0
¶v ds ¶v'
Euler-Lagrange equation

Gradient descent — time evolution converges to a solution :

Evolution equation of the snake

¶v( s, t )
- av' ' ( s) - bv ( s) + lÑg ( I (v( s))) =
( 4) 2

¶t

This was the original solution of Kass et al 1987.

But Gradient descent can provide local minima…
¡ Find the contour C (xk,yk) that minimises E = Eint + Eext
¡ Contour C has n points

¡ à It is an optimization problem for function of 2n

variables (x1,y1…xn,yn)

¡ Several other ways to solve this problem:

§ More robust option than Euler Lagrange : dynamic
programming. But computationnally costly L (Amini 1988)

§ Greedy algorithm (Williams & Shah 1991)

¡ Initialization :
§ For each point Pi : Ei = Val_max

¡ For each point i

§ For all points k in the neighborhood V(Pi) compute
energy Ek
▪ If Ek < Ei then Ei = Ek and move contour point i to point k

¡ Keep on looping as long as the nb of moved

points is superior to a threshold
http://web.cs.hacettepe.edu.tr/~erkut/bil717
.s12/w08-snakes.pdf

C. Petitjean LITIS - Medical Image Acquisition and Processing 23

Limitations
Source : A. Yezzi, Georgia Tech Univ.

Iterations

¡ Pros: Flexibility with the energy design

¡ Cons:
¡ S = arg min D(S) + l R(S) Parameter value?
¡ Initialisation : contour must be close enough to the borders
¡ Cannot segment multiple instance object
C. Petitjean LITIS - Medical Image Acquisition and Processing 24
C. Petitjean LITIS - Medical Image Acquisition and Processing 25
http://web.cs.hacettepe.edu.tr/~erkut/bil717.s12/w08-snakes.pdf
C. Petitjean LITIS - Medical Image Acquisition and Processing 26
¡ A way to solve the topology problem!

¡ Let’s consider the active contour problem

from another perspective:

¡ evolution of a curve:

C. Petitjean 27
¡ Another way to represent to curve:

¡ The curve is represented

Ψ is the distance to the curve

C. Petitjean 28
¡ 3d Function Ψ such that C = (Ψ = 0)
(Osher et Sethian, 1988)

¡ Ψ is a level set

¡Distance to the
Contour C

http://zoi.utia.cas.cz/files/segmentace.pdf
C. Petitjean LITIS - Medical Image Acquisition and Processing 29
 y
• 3D Function 3D : distance map to C
Définition du contour Calcul de y
* * * * Ö8 Ö5 2 2
* * * * Ö5 Ö2 1 1
* * 0 0 2 1 0 0
* * 0 * 2 1 0 -1

Représentation 3D de y
 y
¡ Fonction 3D : carte de distance à C
C. Petitjean 32
¡ Consider the evolution of a curve:

¡ Instead of evolving C, Ψ will be the one

evolving!

¡ New PDE for psi…

C. Petitjean 33
¡ Consider the evolution of a curve:

¡ Instead of evolving C, Ψ will be evolving

¡ Final model (Caselles 1993)

C. Petitjean 34
¡ Discretization of the evoluation equation of Ψ
y ijn +1 - y ijn
= Fij Ñ ijy ijn + Ñ ij g ij ( I ).Ñ ijy ijn
Dt
¡ Handles topology changes

C. Petitjean LITIS - Medical Image Acquisition and Processing 35

¡ Instead of evolving C, Ψ will be evolving
y ijn +1 - y ijn
= Fij Ñ ijy ijn + Ñ ij g ij ( I ).Ñ ijy ijn
Dt

¡ Handles topology changes

C. Petitjean LITIS - Medical Image Acquisition and Processing 36

https://www.spiedigitallibrary.org/journals/Optical-Engineering/volume-50/issue-10/107001/Statistical-approaches-to-automatic-
level-set-image-segmentation-with-multiple/10.1117/1.3631042.short
C. Petitjean LITIS - Medical Image Acquisition and Processing 38
¡ Spatial discretization issues

¡ The psi function needs to be re-initialized to

the distance function from time to time

C. Petitjean 39
¡ Idea: use a model
of the shape

Vu, CVPR’08
C. Petitjean LITIS - Medical Image Acquisition and Processing 40
¡ Medical images contain:
§ Noise
§ Occlusion
§ Fuzziness (PVE)

¡ The shape of the object to be

segmented is known (up to some variation)
Shape modelling can improve image segmentation

C. Petitjean LITIS - Medical Image Acquisition and Processing 41

 No shape model With shape model

Occlusion

High level
of noise
Xavier Bresson, Active contours, EPFL in French
C. Petitjean LITIS - Medical Image Acquisition and Processing 42