Image Segmentation

(Optimization Problem)

1
 Labeling & Segmentation
• Labeling is a common way for modeling
various computer vision problems (e.g.
optical flow, image segmentation, stereo
matching, etc)
• The set of labels can be discrete (as in
image segmentation)
L = {l1, . . . , l m } with L = m

• Or continuous (as in optical flow)

L⊂ Rn for n ≥ 1 2
 Labeling is a function
• Labels are assigned to sites (pixel locations)
• For a given image, we have 𝛺 =N c o l s .N ro w s
• Identifying a labeling function
(with segmentation) is f : 𝛺 →L

We aim at calculating a labeling function that

minimizes a given (total) error or energy

3
 Energy Function
The role of an energy function in
minimization-based vision is twofold:

1. as the quantitative measure of the global

solution
2. as a guide to the search for a minimal solution.

Correct solution is embedded as the

minimum. 4
 Segmentation as an Energy Minimization Problem

• Edata assigns non-negative penalties to a pixel

location p when assigning a label to this location.

• Esmooth assigns non-negative penalties by

comparing the assigned labels fp and fq at
adjacent positions p and q.

5
 Segmentation as an Energy Minimization Problem

• Edata assigns non-negative penalties to a pixel

location p when assigning a label to this location.
• Esmooth assigns non-negative penalties by
comparing the assigned labels fp and fq at
adjacent positions p and q.

6
 Segmentation as an Energy Minimization Problem

• Edata assigns non-negative penalties to a pixel

location p when assigning a label to this location.
• Esmooth assigns non-negative penalties by
comparing the assigned labels fp and fq at
adjacent positions p and q.

This optimization model is characterized by local

interactions along edges between adjacent pixels, and
often called MRF (Markov Random Field) model.

7
Energy Function-Details

8
 Energy Function-Details

To solve this problem, transform the energy

functional into min-cut/max-flow problem and
solve it!

9
 Energy Function
Unary Cost
(data term)

Discontinuity
Cost

p q 2
1
 Recap: Image as a Graph

p q Edg
e Node/verte
pixe x
l

2
2
 Graph Cuts for Optimal Boundary Detection
(Boykov ICCV 2001)
F

• Binary label: foreground vs.

background B

• User labels some pixels F F B

• Exploit F F B
– Statistics of known Fg & Bg F B B
– Smoothness of label
• Turn into discrete graph
optimization
– Graph cut (min cut / max flow) 2
3
 Graph-Cut

Each pixel is connected to its neighbors in an undirected graph

Goal: split nodes into two sets A and B based on pixel values, and try to
13
classify Neighbors in the same way too!
 Graph-Cut

Each pixel is connected to its neighbors in an undirected graph

Goal: split nodes into two sets A and B based on pixel values, and try to
14
classify Neighbors in the same way too!
 Graph-Cut
• Each pixel = node
• Add two nodes F & B
• Labeling: link each pixel to either F
or B F Desired
result

B 15
 Cost Function: Data term
• Put one edge between each pixel and both
F &G
• Weight of edge = minus data term
F

B 16
 Cost Function: Smoothness term
• Add an edge between each neighbor
pair
• Weight = smoothness term
F

B 17
 Min-Cut
• Energy optimization equivalent to graph min
cut
• Cut: remove edges to disconnect F from B
• Minimum: minimize sum of cut edge weight
F cut

B 2
9
 Min-Cut

Source

2 9 Graph (V, E, C)
1 Vertices V = {v1, v2 ... vn}

v1 v2 Edges E = {(v1, v2) ....}

2 Costs C = {c(1, 2) ....}
5 4

Sink

19
 Min-Cut
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source

2 9 What is the cost of a st-cut?

1
Sum of cost of all edges going
v1 v2
from S to T
2
5 4

Sink

5 + 2 + 9 = 16 20
 Min-Cut
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source

2 9 What is the cost of a st-cut?

1
Sum of cost of all edges going
v1 v2
from S to T
2
5 4
What is the st-mincut?
Sink st-cut with the
minimum cost

5 + 2 + 9 = 16 21
 How to compute min-cut?
Solve the dual maximum flow problem
Compute the maximum flow between
Source Source and Sink
Constraints
2 9
Edges: Flow < Capacity
1
v1 v2 Nodes: Flow in & Flow out
2
5 4
Min-cut\Max-flow Theorem
Sink
In every network, the maximum flow
equals the cost of the st-mincut

22
 Max-Flow Algorithms
Flow = 0
Augmenting Path Based
Algorithms
Source

2 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
5 4 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 23

 Max-Flow Algorithms
Flow = 0
Augmenting Path Based
Algorithms
Source

2 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
5 4 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 24

 Max-Flow Algorithms
Flow = 0 + 2
Augmenting Path Based
Algorithms
Source

2-2 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
4 through this path
5-2
3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity 25

 Max-Flow Algorithms
Flow = 2
Augmenting Path Based
Algorithms
Source

0 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 4 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 26

 Max-Flow Algorithms
Flow = 2
Augmenting Path Based
Algorithms
Source

0 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 4 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 27

 Max-Flow Algorithms
Flow = 2
Augmenting Path Based
Algorithms
Source

0 9 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 4 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 28

 Max-Flow Algorithms
Flow = 2 +
Augmenting Path Based
4
Algorithms
Source

0 5 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 0 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 29

 Max-Flow Algorithms
Flow = 6
Augmenting Path Based
Algorithms
Source

0 5 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 0 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 30

 Max-Flow Algorithms
Flow = 6
Augmenting Path Based
Algorithms
Source

0 5 1. Find path from source to sink

with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 0 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 31

 Max-Flow Algorithms
Flow = 6 + 1
Augmenting Path Based
Algorithms
Source

0 4 1. Find path from source to sink

with positive capacity
1-1
v1 v2
2. Push maximum possible flow
2+1
2 0 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 32

 Max-Flow Algorithms
Flow = 7
Augmenting Path Based
Algorithms
Source

0 4 1. Find path from source to sink

with positive capacity
0
v1 v2
2. Push maximum possible flow
3
2 0 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 33

 Max-Flow Algorithms
Flow = 7
Augmenting Path Based
Algorithms
Source

0 4 1. Find path from source to sink

with positive capacity
0
v1 v2
2. Push maximum possible flow
3
2 0 through this path

3. Repeat until no path can be

Sink
found

Algorithms assume non-negative capacity 34

 Example Segmentation in a Video
n- s = 0 st-cut
links

E(x) x*
wpq

t=1

Image Flow Global Optimum35