CS7.

404: Digital Image Processing

Monsoon 2023: Image Segmentation - 2

Anoop M. Namboodiri
Biometrics and Secure ID Lab, CVIT,
IIIT Hyderabad
 Types of Segmentation
• Classification-based
• Label pixels based on region properties
• Label each pixel based on object models
• Region-based
• Region growing and splitting
• Boundary-based
• Find edges in the image and use them as region boundary
• Motion-based
• Group pixels that have consistent motion (e.g., move in the same
direction)
Is Intensity Histogram Sufficient?
 Segmentation by Pixel Classification
Two Primary Challenges:
1. How to use object / background properties to decide on
pixel label?
• e.g., Ducks are white and yellow, while background is green and
brown
2. How to ensure that regions are continuous regions?
• Avoid fragmentation of object regions
 Segmentation as Optimal Labeling
• Model knowledge about the world
• Classify each pixel as belonging to a specific object
• Independent classification does not work
• Need to incorporate neighborhood information
• Consider a graph over the image
• Each node in the graph need to be labeled
• Edges in the graph represent neighborhood constraints
• Define a cost function, Q( f ), using the above
• Compute the optimal labeling wrt Q( f ).
Binary Image Segmentation

How ?

Cost function Models our knowledge about natural images

Optimize cost function to obtain the segmentation

Binary Image Segmentation

Object - white, Background - green/grey Graph G = (V,E)

Each vertex corresponds to a pixel

Edges define a 4-neighbourhood grid graph

Assign a label to each vertex from L = {obj,bkg}

Binary Image Segmentation

Object - white, Background - green/grey Graph G = (V,E)

Cost of a labelling f : V  L Per Vertex Cost

Cost of label ‘obj’ low Cost of label ‘bkg’ high

Binary Image Segmentation

Object - white, Background - green/grey Graph G = (V,E)

Cost of a labelling f : V  L Per Vertex Cost

Cost of label ‘obj’ high Cost of label ‘bkg’ low

UNARY COST
Binary Image Segmentation

Object - white, Background - green/grey Graph G = (V,E)

Cost of a labelling f : V  L Per Edge Cost

Cost of same label low

Cost of different labels high
Binary Image Segmentation

Object - white, Background - green/grey Graph G = (V,E)

Cost of a labelling f : V  L Per Edge Cost

Cost of same label high PAIRWISE

Cost of different labels low COST
Binary Image Segmentation

Object - white, Background - green/grey Graph G = (V,E)

Problem: Find the labeling with minimum cost f*

Binary Image Segmentation

Graph G = (V,E)
Vertex corresponds to a pixel
L = {fg, bg} Edges define grid graph

Problem: Find the labeling with minimum cost f*

The General Problem
1 2

b c Graph G = ( V, E )
1

3
Discrete label set L = {1,2,…,h}
a
d

Assign a label to each vertex

f
e
2 2 f: V  L

Cost of a labelling Q(f)

Unary Cost Pairwise Cost

Find f* = arg min Q(f)

Formulation: Energy Function
Label l1

Label l0

Va Vb Vc Vd

Da Db Dc Dd

Random Variables V = {Va, Vb, ….}

Labels L = {l0, l1, ….} Data D
Labelling f: {a, b, …. }  {0,1, …}
Energy Function
Label l1 2 4 6 3

Label l0
5 2 3 7
Va Vb Vc Vd

Da Db Dc Dd

Easy to minimize
Q(f) = ∑a qa;f(a)
Unary Potential Neighbourhood
Energy Function
Label l1 2 4 6 3

Label l0
5 2 3 7
Va Vb Vc Vd

Da Db Dc Dd

E : (a,b) Î E iff Va and Vb are neighbours

E = { (a,b) , (b,c) , (c,d) }
Energy Function
Label l1 2 0 4 0 6 1 3
1 1 2 3 4 1
Label l0
5 0 2 1 3 0 7
Va Vb Vc Vd

Da Db Dc Dd
Pairwise Potential
Q(f) = ∑a qa;f(a) +∑(a,b) qab;f(a)f(b)
Energy Function
Label l1 2 0 4 0 6 1 3
1 1 2 3 4 1
Label l0
5 0 2 1 3 0 7
Va Vb Vc Vd

Da Db Dc Dd

Q(f; q) = ∑a qa;f(a) +∑(a,b) qab;f(a)f(b)

Parameter
MAP Estimation
Label l1 2 0 4 0 6 1 3
1 1 2 3 4 1
Label l0
5 0 2 1 3 0 7
Va Vb Vc Vd

Q(f; q) = ∑a qa;f(a) + ∑(a,b) qab;f(a)f(b)

MAP Estimation
Label l1 2 0 4 0 6 1 3
1 1 2 3 4 1
Label l0
5 0 2 1 3 0 7
Va Vb Vc Vd

Q(f; q) = ∑a qa;f(a) + ∑(a,b) qab;f(a)f(b)

2 + 1 + 2 + 1 + 3 + 1 + 3 = 13
MAP Estimation
Label l1 2 0 4 0 6 1 3
1 1 2 3 4 1
Label l0
5 0 2 1 3 0 7
Va Vb Vc Vd

Q(f; q) = ∑a qa;f(a) + ∑(a,b) qab;f(a)f(b)

MAP Estimation
Label l1 2 0 4 0 6 1 3
1 1 2 3 4 1
Label l0
5 0 2 1 3 0 7
Va Vb Vc Vd

Q(f; q) = ∑a qa;f(a) + ∑(a,b) qab;f(a)f(b)

5 + 1 + 4 + 0 + 6 + 4 + 7 = 27
MAP Estimation
Label l1 2 0 4 0 6 1 3
1 1 2 3 4 1
Label l0
5 0 2 1 3 0 7
Va Vb Vc Vd

q* = min Q(f; q) = Q(f*; q)

Q(f; q) = ∑a qa;f(a) + ∑(a,b) qab;f(a)f(b)

f* = arg min Q(f; q)

MAP Estimation
16 possible labellings f* = {1, 0, 0, 1}
q* = 13
f(a) f(b) f(c) f(d) Q(f; q) f(a) f(b) f(c) f(d) Q(f; q)
0 0 0 0 18 1 0 0 0 16
0 0 0 1 15 1 0 0 1 13
0 0 1 0 27 1 0 1 0 25
0 0 1 1 20 1 0 1 1 18
0 1 0 0 22 1 1 0 0 18
0 1 0 1 19 1 1 0 1 15
0 1 1 0 27 1 1 1 0 23
0 1 1 1 20 1 1 1 1 16
Computational Complexity

Segmentation

2|V|

|V| = number of pixels ≈ 320 * 480 = 153600

Can we do better than brute-force?

MAP Estimation is NP-hard !!

Computational Complexity

Segmentation

2|V|

|V| = number of pixels ≈ 320 * 480 = 153600

Exact algorithms do exist for special cases

Good approximate algorithms for general case
Questions?
 Watershed Segmentation
• Consider the intensity profile of an
image as a topographical surface
• We can identify three types of points:
1. Local minima, where water collects
2. Points where water flows to a single
minima (catchment basin)
3. Points where water can flow to any one
of multiple minima (watershed lines)
• Principal Goal: Find the watershed lines
(object boundaries)
 Watershed Segmentation
• Original Image, Topographical view
• Various stages of flooding.

Water Water

Water
 The Watershed Algorithm
• Let M1, M2, .., MR be the regional minima.
• Let C(Mi) be points in the catchment area of Mi.
• Let T[n] represent the set of co-ordinates for which g(s,t) < n.
• Flood the topography in integer increments
• From 𝑛 = min + 1 to 𝑛 = max + 1.
• At each stage, we compute Cn(Mi) from Cn-1(Mi) and T[n]
using morphological operators
• Essentially, add pixels of value n to the corresponding C(Mi)
 Watershed Example
• Watershed is often performed
on the gradient image (top
right).
Over-segmentation from Watershed
Over-segmentation from Watershed
Questions?