Today

Variational image segmentation: Part three (Level

sets)

Some slides borrowed from Nikos Paragios, Christophe Restife, Chuck Dyer,
Guillermo Sapiro
 Level Sets

• Proposed by Osher-Sethian (1987). Caselles, Catte, Dibos (1992) and

many others have suggested improvements.
• Consider a curve or shape or a border which will move (similar to active
contours but with important differences)
• Will be able to handle topological changes (shortly!)
Level Sets

C ( p) : R  R 2

Implicit

C  {( x, y ) |  ( x, y )  0}
Level Sets

C ( p) : R  R 2

Implicit

C  {( x, y ) |  ( x, y )  0}
Represent curves as all
points in the plane that
satisfy a certain equation
 Level Sets

C ( p) : R  R 2

Implicit

C  {( x, y ) |  ( x, y )  0}
0 Only those points that
0 satisfy a certain equation
are curve points (zero dist)
 Level Sets

C ( p) : R  R 2

Implicit

C  {( x, y ) |  ( x, y )  0}
0 Zero level set!
0
 Level Sets

C ( p) : R  R 2

 1

0   1

0 Every time the profile cuts the curve

 Level Sets

0
0
To evolve this curve: we need to be able to
characterize tangents, normal and curvature with the
“implicit” representation. Else, its useless
 Level Sets

0
0
To evolve this curve: we need to be able to
characterize tangents, normal and curvature with the
“implicit” representation. Else, its useless

Cannot use “curve” parameterization. Possible?

 Normals
 
Show : N  
|  |

Remember: All points which belong to the curve we have is 0. So

along the curve when we travel is NOT changing.
So “along” the arc length, we have 0. But….

s ( x, y )  x xs   y ys   , xs ys 
T
 Normals
 
Show : N  
|  |

Remember: All points which belong to the curve we have is 0. So

along the curve when we travel is NOT changing.
So “along” the arc length, we have 0. But….

s ( x, y )  x xs   y ys   , xs ys 
T
  , T
Tangent


But  , T  0
 Normals
 
Show : N  
|  |


 , T  0

 
, T  0 (perpendic ular stays with change in magnitude)
|  |

If something is perpendicular to the tangent, it must be the

normal, so
  Important: does not deal with
N curve points! Only with
|  |
 Curvature

  
Show :   div   div (a b)  a x  by
 |  | 
Remember: Curvature is second derivative due to arc.
Problem: We don’t have the explicit parameterization anymore; we
only have ϕ.
Let us go along the zero level set…where 0
d d  d  
ss ( x , y )  ( x x s   y y s )   , T   , T   ,  N
ds ds ds

d  
So ,   ,T    , N  0
ds
 Curvature

  
Show :   div   div (a b)  a x  by
 |  | 
 d 
  , N     ,T
ds
 d 
   ,     ,T
| | ds

 
   ,   [  xx x s   xy y s ,  xy x s   yy y s ], T
| |

   Don’t need C! Only with

  div  
 |  | 
 Level Sets

• Proposed by Osher-Sethian (1987). Caselles, Catte, Dibos (1992) and

many others have suggested improvements.
• Consider a curve or shape or a border which will move (similar to active
contours but with important differences)
• Will be able to handle topological changes (shortly!)

• The interface moves normal to

itself to a subsequent position at
time point 1
• Assume the speed of movement
did not change with position
• Assume that the speed is 1
 Level Sets

• Proposed by Osher-Sethian (1987). Caselles, Catte, Dibos (1992) and

many others have suggested improvements.
• Consider a curve or shape or a border which will move (similar to active
contours but with important differences)
• Will be able to handle topological changes (shortly!)

• The interface moves normal to

itself to a subsequent position at
time point 1
• Assume the speed of movement
did not change with position
• Assume that the speed is 1
• This consists of all the points
reached in unit time (that is all
points at distance 1)
 Level Sets

• Proposed by Osher-Sethian (1987). Caselles, Catte, Dibos (1992) and

many others have suggested improvements.
• Consider a curve or shape or a border which will move (similar to active
contours but with important differences)
• Will be able to handle topological changes (shortly!)

How to describe the curve/interface?

Idea 1: similar to active contours?
Via discrete vertices or particles!
 Level Sets

• Proposed by Osher-Sethian (1987). Caselles, Catte, Dibos (1992) and

many others have suggested improvements.
• Consider a curve or shape or a border which will move (similar to active
contours but with important differences)
• Will be able to handle topological changes (shortly!)

How to describe the curve/interface?

Idea 1: similar to active contours?
Via discrete vertices or particles!
Decide on the number of particles.
Write an ordinary differential equation
for each one
Solve!
Reach new positions at time point 1
Interpolate new curve (just like snakes)
 Level Sets

• Proposed by Osher-Sethian (1987). Caselles, Catte, Dibos (1992) and

many others have suggested improvements.
• Consider a curve or shape or a border which will move (similar to active
contours but with important differences)
• Will be able to handle topological changes (shortly!)

This is NOT the Level Sets procedure!

This is a simple example. Particles may

be nice and separated.
Otherwise:
Particles can spread far apart.
Interpolation problems

Can come too close together.

Numerical problems
 Level Sets

• Instead of specifying where the particles are or how many there are, the
curve/interface is given implicitly by

 ( x, y )  0
 Level Sets

• Instead of specifying where the particles are or how many there are, the
curve/interface is given implicitly by

 ( x, y )  0
We can also do

 ( x, y )  1,  ( x, y )  2,...,  ( x, y )  42
These are level sets of p and are in fact
solutions to

 ( x, y )  c (a constant)
 Level Sets

C=2

C=1

C=0

C=1 C=2 C=0

 Level Sets

Consider a more precise form of the previous example

 ( x, y )  x  y2 2

What are the level sets of this function?

 Level Sets

Consider a more precise form of the previous example

 ( x, y )  x  y
2 2

What are the level sets of this function? Set

 ( x, y )  x  y  c
2 2

To get the c-th level set.

 Level Sets

Consider a more precise form of the previous example

 ( x, y )  x  y2 2

What are the level sets of this function?

 ( x, y )  x  y  c
2 2

A circle assuming c>0

 Level Sets

Consider a more precise form of the previous example

 ( x, y )  x  y2 2

What are the level sets of this function?

 ( x, y )  x  y  c
2 2

We are introducing a new dimension

 Level Sets

Consider a more precise form of the previous example

 ( x, y )  x  y2 2

What are the level sets of this function?

 ( x, y )  x  y  c
2 2

The circle is the level set at height “c”

A way to describe the curve implicitly

via one dimension higher

Advantages?
 Level Sets

When connected becomes unconnected and when unconnected becomes

connected, NO problem!
The function  is still nice and easy to work with
 Level Sets

The third dimension (t) set to c cuts the function in two different places: two
circles

If t increases, the circles come closer, they merge, ϕ still remains a nice and
smooth function
 Level Sets

The third dimension (t) set to c cuts the function in two different places: two
circles

If t increases, the circles come closer, they merge, ϕ still remains a nice
and smooth function
 Level Sets

The third dimension (t) set to c cuts the function in two different places: two
circles

If t increases, the circles come closer, they merge,  still remains a nice and
smooth function

 will be the unknown we will solve for. This will be function of x, y, and t
 Differences relative to particles

Looking for all points at time 1, i.e., unit distance

Interpolation gap: no particles Corners: particles converging

reach the smooth part of curve to a single point. Where to go
next?
 Formalization

Recall: gradient of the function points in its normal direction


N  
For a unit normal, use  ( )
N 
| ( ) |
Easy case is when the gradient’s magnitude is 1.

These are called distance functions and very important in level

sets (coming up shortly).
 Formalization

dC 
V N
Recall: for curve evolution we had

dt
d
Here we will get (shortly)  V |  |
dt
 Formalization

dC 
V N
Recall: for curve evolution we had

dt
d
Here we will get (shortly)  V |  |
dt
C  {( x, y ) |  ( x, y )  0}
We take derivative of ϕ according to t

 x xt   y yt  t  0
 t   x xt   y yt
 t   , Ct
 Formalization

dC 
V N
Recall: for curve evolution we had

 t   , Ct dt

 t   ,V N
  
 t  V  , N But N  
|  |


 t  V  ,
|  |
t  V |  |
 Formalization: Level set equation

Evolving the interface 

 v.( )  0
t
(one dimensional wave equation
or convection equation)

v may depend on , and if so the equation becomes non-linear

( ) 
v.  v | ( ) | v N | ( ) |
| ( ) |
We call the speed in the normal direction as F


 F | ( ) | 0
t
Level Sets equation (look up Hamilton Jacobi)
 Formalization: Setting up F

F can be a function of grad()

 ( ) 
For example, curvature in 2-D is given as   div  
 | ( ) | 
Curvature based evolution

Small curvature

High curvature
 Level Sets: distance function

Note that  is defined for all points, not just the ones on the boundary (unlike
snakes)
Signed distance function has the property that
Speed as a function of curvature given as
 Level Sets: distance function
Usual choice for : signed distance to the front (0)
 - d(x,y, ) if (x,y) inside the front
(x,y,0) =  0 “ on “
 d(x,y, ) “ outside “

7 6 5 4 4 4 3 2 1 1 1 2 3 4 5
5
6 5 4 3 3 3 2 1 0 0 0 1 2 3 4
5 4 3 2 2 2 1 0 -1 -1 -1 0 1 2 3
4 3 2 1 1 1 0 -1 -2 -2 -2 -1 0 1 2
3 2 1 0 0 0 -1 -2 -3 -3 -2 -1 0 1 2
2 1 0 -1 -1 -1 -2 -3 -3 -2 -1 0 1 2 3 0
2 1 0 -1 -2 -2 -3 -3 -2 -1 0 1 2 3 4
2 1 0 -1 -2 -2 -2 -2 -1 0 1 2 3 4 5
3 2 1 0 -1 -1 -1 -1 -1 0 1 2 3 4 5
4 3 2 1 0 0 0 0 -1 -1 0 1 2 3 4 -2
5 4 3 2 1 1 1 1 0 0 1 2 3 4 5
6 5 4 3 2 2 2 2 1 1 2 3 4 5 6

(x,y,t) (t)
 Level Sets: distance function
7 6 5 4 4 4 3 2 1 1 1 2 3 4 5
6 5 4 3 3 3 2 1 0 0 0 1 2 3 4
5 4 3 2 2 2 1 0 -1 -1 -1 0 1 2 3
• no movement, only change of values
4 3 2 1 1 1 0 -1 -2 -2 -2 -1 0 1 2
3 2 1 0 0 0 -1 -2 -3 -3 -2 -1 0 1 2 • the front may change its topology
2 1 0 -1 -1 -1 -2 -3 -3 -2 -1 0 1 2 3
2 1 0 -1 -2 -2 -3 -3 -2 -1 0 1 2 3 4 • the front location may be
2 1 0 -1 -2 -2 -2 -2 -1 0 1 2 3 4 5 between samples
3 2 1 0 -1 -1 -1 -1 -1 0 1 2 3 4 5
4 3 2 1 0 0 0 0 -1 -1 0 1 2 3 4
5 4 3 2 1 1 1 1 0 0 1 2 3 4 5
6 5 4 3 2 2 2 2 1 1 2 3 4 5 6
7 6 5 4 4 4 3 2 1 1 1 2 3 4 5
(x,y,t) 6 5 4 3 3 3 2 0 -1 0 0 1 2 3 4
5 4 3 2 2 2 1 -1 -2 -1 -1 0 1 2 3
4 3 2 1 1 1 0 -1 -2 -2 -2 -1 0 1 2
3 2 1 0 0 0 -1 -2 -3 -3 -2 -1 0 1 2
2 1 0 -1 -1 -1 -2 -3 -3 -2 -1 0 1 2 3
2 1 0 -1 -2 -2 -3 -3 -2 -1 0 1 2 3 4
2 1 1 0 -2 -2 -2 -2 0 0 1 2 3 4 5

(x,y,t+1) = (x,y,t) + (x,y,t) 3 2 1 0 -1 -3 -1 0 1 1 1 2 3 4 5

4 3 2 0 -1 -2 0 1 1 0 0 2 2 3 4
5 4 3 2 0 0 1 1 0 -1 0 1 2 4 5
6 5 4 3 2 1 2 2 0 0 1 2 4 5 6
 Level Sets: distance function

Segmentation with LS:

 Level Sets: distance function

Segmentation with LS:

• Initialise the front (0)
 Level Sets: distance function

Segmentation with LS:

• Initialise the front (0)
• Compute (x,y,0)
 Level Sets: distance function

Segmentation with LS:

• Initialise the front (0)
• Compute (x,y,0)
• Iterate:
(x,y,t+1) = (x,y,t) + (x,y,t)
until convergence
 Level Sets: distance function

Segmentation with LS:

• Initialise the front (0)
• Compute (x,y,0)
• Iterate:
(x,y,t+1) = (x,y,t) + (x,y,t)
until convergence
• Mark the front (tend)
Level Sets equation setup

 ˆ
 k I  FA  FG ( )     0
t
 Level Sets equation setup

• Equation 17 p162:
 ˆ
 k I  FA  FG ( )     0
t

(x,y,t+1) - (x,y,t)
 Level Sets equation setup

 ˆ
 k I  FA  FG ( )     0
t

(x,y,t+1) - (x,y,t)

extension of the
speed function kI
(image influence)
 Level Sets equation setup

 ˆ
 k I  FA  FG ( )     0
t

(x,y,t+1) - (x,y,t)

extension of the
speed function kI
(image influence)

constant “force”
(balloon pressure)
 Level Sets equation setup

 ˆ
 k I  FA  FG ( )     0
t

(x,y,t+1) - (x,y,t)
smoothing “force”
depending on the
extension of the local curvature 
speed function kI (contour influence)
(image influence)

constant “force”
(balloon pressure)
 Level Sets equation setup

 ˆ
 k I  FA  FG ( )     0
t

(x,y,t+1) - (x,y,t)
smoothing “force”
depending on the
extension of the local curvature 
speed function kI (contour influence)
(image influence)

constant “force”   
(balloon pressure)
  div 

  
 Level Sets equation setup

 ˆ
 k I  FA  FG ( )     0
t
spatial
(x,y,t+1) - (x,y,t) derivative
smoothing “force”
of 
depending on the
extension of the local curvature 
speed function kI (contour influence)
(image influence)

constant “force”   
(balloon pressure)
  div 

  
 Level Sets equation setup

product of influences

 ˆ
 k I  FA  FG ( )     0
t
spatial
(x,y,t+1) - (x,y,t) derivative
smoothing “force”
of 
depending on the
extension of the local curvature 
speed function kI (contour influence)
(image influence)

constant “force”   
(balloon pressure)
  div 

  
 Level Sets equation setup

product of influences

 ˆ
 k I  FA  FG ( )     0
t
spatial
(x,y,t+1) - (x,y,t) derivative
smoothing “force”
of 
depending on the
extension of the local curvature 
speed function kI (contour influence)
(image influence)

constant “force”   
1   div 
kI  (balloon pressure) 
  
1  Gσ  I  x,y 
- Gσ  I  x,y 
kI  e
Narrow-band extension
• Weaknesses o algorithm 1
– update of all (x,y,t): inefficient, only care about the front
– speed extension: computationally expensive
• Improvement:
– narrow band: only update a few level sets around 

2 1 1 1 2
2 1 0 0 0 1 2
2 2 2 1 0 -1 -1 -1 0 1 2
2 1 1 1 0 -1 -2 -2 -2 -1 0 1 2
2 1 0 0 0 -1 -2 -2 -1 0 1 2
2 1 0 -1 -1 -1 -2 -2 -1 0 1 2
2 1 0 -1 -2 -2 -2 -1 0 1 2
2 1 0 -1 -2 -2 -2 -2 -1 0 1 2
2 1 0 -1 -1 -1 -1 -1 0 1 2
2 1 0 0 0 0 -1 -1 0 1 2
2 1 1 1 1 0 0 1 2
2 2 2 2 1 1 2
Fast Marching Method

(i) (ii)

(iii) (iv)
Examples
Examples