Image Segmentation

Gray-level thresholding
Supervised vs. unsupervised
thresholding
Binarization using Otsus method
Locally adaptive thresholding
Maximally stable extremal regions
Color-based segmentation
Region labeling and counting
Region moments

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 1
 Gray-level thresholding
How can holes be filled?

Original image Thresholded f [x, y ] m [x, y ]

Peter f [x,y] Peter m [x,y]

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 2
 How to choose the threshold?
pdf
Background

Foreground

Gray level

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 3
 Unsupervised thresholding
Idea: find threshold T that minimizes within-class variance of both foreground and
background (same as k-means)
N Fgrnd (T ) N Bgrnd (T )

= (T ) 2
within 2
Fgrnd (T ) + Bgrnd
2
(T )
N N
Equivalently, maximize between-class variance
between
2
(T=) 2 within
2
(T )
1 2
N Fgrd 1 N Bgrnd 1
= f [ x, y ]
2
f [ x, y ] Fgrnd
2 2
f [ x, y ] Bgrnd
2 2

x, y
N N N
Fgrd x , yFgrnd N N Bgrnd x , yBgrnd

( Fgrnd ) + N ( Bgrnd )
N Fgrnd 2 N Bgrnd 2 N Fgrnd 2 N Bgrnd 2
= +
2
Fgrnd + Bgrnd =
N N N
N Fgrnd (T ) N Bgrnd (T )
( Fgrnd ( ) Bgrnd ( ))
2
T T
N2
[Otsu, 1979]

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 4
 Unsupervised thresholding (cont.)
Algorithm: Search for threshold T to maximize
N Fgrnd (T ) N Bgrnd (T )
(T ) ( (T ) (T ) )
2
= 2
between Fgrnd Bgrnd
N2
Useful recursion for sweeping T across histogram:
N Fgrnd (=
T + 1) N Fgrnd (T ) + nT
N Bgrnd (=
T + 1) N Bgrnd (T ) nT
Fgrnd (T ) N Fgrnd (T ) + nT T
Fgrnd (T + 1) =
N Fgrnd (T + 1)
Bgrnd (T ) N Bgrnd (T ) nT T
Bgrnd (T + 1) =
N Fgrnd (T + 1)

[Otsu, 1979]

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 5
 Unsupervised thresholding (cont.)
T = 112

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 6
 Unsupervised thresholding (cont.)
T = 56

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 7
 Unsupervised thresholding (cont.)
T = 168

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 8
 Unsupervised thresholding (cont.)

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 9
 Sometimes, a global threshold does not work

Thresholded with
Original image
Otsus Method

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 11
 Locally adaptive thresholding
Slide a window over the image
For each window position, decide whether to
perform thresholding
Thresholding should not be performed in uniform
areas
Use variance or other suitable criterion
Non-uniform areas: apply Otsus method (based on
local histogram)
Uniform areas: classify the entire area as
foreground or background based on mean value

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 12
 Locally adaptive thresholding (example)

Non-uniform areas Local threshold values Locally thresholded result

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 13
 Maximally stable extremal regions
Extremal region: any connected region in an image with all pixel values above (or
below) a threshold
Observations:
Nested extremal regions result when the threshold is successively raised (or lowered).
The nested extremal regions form a component tree.
Key idea: choose thresholds such that the resulting bright (or dark) extremal
regions are nearly constant when these thresholds are perturbed by +/-

maximally stable extremal regions (MSER)

[Matas, Chum, Urba, Pajdla, 2002]

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 14
 MSERs: illustration

A+
A
A-
+

-

A A +
Local minimum of MSER
A

[Matas, Chum, Urba, Pajdla, 2002]

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 15
 Level sets of an image
1 1 1 1 1 1 1 1 1 1 1 5 4 4 8
1 7 6 4 2 2 3 3 3 3 1 5 4 4 8
1 7 6 4 2 2 3 3 3 3 1 5 4 4 8
1 7 6 4 2 2 3 3 3 3 1 5 4 4 8
1 7 6 4 2 2 5 5 5 5 1 5 4 4 8
1 6 6 4 2 2 5 5 5 6 1 5 4 4 4
1 6 6 4 2 2 6 6 6 6 1 5 5 5 5
1 4 4 4 2 2 6 6 6 6 1 5 5 5 5
1 1 1 1 1 2 6 1 1 1 1 2 2 2 2
1 8 8 5 1 2 6 1 7 7 1 2 2 2 2
1 8 8 5 1 1 1 1 7 7 1 1 1 1 2
1 8 8 5 5 5 3 3 7 7 1 1 1 1 2
1 8 8 5 5 3 3 3 7 7 7 1 1 1 2
1 8 8 5 5 3 3 3 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

f [ x, y ] f [fxf[,[xxy,,]yy>]]>
8> 07642531

Image Level Set

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 16
 Component tree of an image
a A0=225
Local minima of sequence
A1=43
A A +
A1=67 A1=36

b c d
A
A2=21 A2=30 A2=32 A2=36
e f g d
= , + 1, K MSERs

A3=21 A3=18 A3=32 A3=19 A3=9

e h g i j
A4=12 A4=18 A4=14 A4=5 A4=19
k h l m i
A5=12 A5=11 A5=10
k n o
A6=4
p

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 17
 MSER: examples

Dark MSERs, =15 Original image Bright MSERs, =15

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 18
 MSER: examples

Dark MSERs, =15 Original image Bright MSERs, =15

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 19
 Supervised thresholding
pdf
Background

Foreground

error Gray level

probability

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 21
 Supervised thresholding
pdf MAP (Maximum A Posteriori) detector
Background

Foreground

error Gray level

probability

If errors BGFG and FGBG are associated with different costs:

Bayes minimum risk detector is optimal.

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 22
 Multidimensional MAP detector
Training
Provide labelled set of training data
Subdivide n-dimensional space into small bins
Count frequency of occurrence for each bin and class
in training set, label bin with most probable class
(Propagate class labels to empty bins)
For test data: identify bin, look up the most
probable class

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 23
 MAP detector in RGB-space
Non-skin Samples

Skin Samples

Original image Skin color detector

Five training images

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 24
 Linear discriminant function
To segment image with n components fi, i=1,2,,n
into two classes, perform test

w f
i
i i + w0 0 ?

Categories are separated by hyperplane in n-space

Numerous techniques to determine weights
wi, i=0,1,2,,n, see, e.g., [Duda, Hart, Stork, 2001]
Can be extended to the intersection of several linear
discrimant functions
Can be extended to multiple classes

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 26
 Chroma keying

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 27
 Landsat image processing

Original Landsat image false color picture out Water area segmented and enhanced to show
of bands 4,5,6 sediments

Source: US Geological Survey USGS, http://sfbay.wr.usgs.gov/

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 28
 Region labeling and counting
How many fish in this picture?

Original Fish image after thresholding

Which pixels belong to the same object (region labeling)?

How large is each object (region counting)?

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 29
 4-connected and 8-connected neighborhoods
Definition: a region is a set of pixels, where each pixel can be reached from
any other pixel in the region by a finite number of steps, with each step starting
at a pixel and ending in the neighborhood of the pixel.

4-neighborhood 8-neighborhood

Typically, either definition leads to the same regions, except when

a region is only connected across diagonally adjacent pixels.

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 30
 Region labeling algorithm (4-neighborhood)
Loop through all pixels f [x,y], left to right, top to bottom
If f [x,y]=0, do nothing.
If f [x,y]=1, distinguish 4 cases

Generate new Copy label Copy label Copy label from the left. If
region label from above from the left labels above and to the left are
different, store equivalence.

Second pass through image to replace equivalent label by the

same label.

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 31
 Region labeling example (4-neighborhood)
List of Region Labels
1
1 2 3
1 All three labels are
equivalent, so merge
2 2 2 into single label.

2 2

3 3 3 3 3

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 32
 Region labeling example (4-neighborhood)
List of Region Labels
1
1
1

1 1 1

1 1

1 1 1 1 1

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 33
 Example: region labeling

Thresholded image 20 labeled regions

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 34
 Region counting algorithm
Measures the size of each region
Initialize counter[label]=0 for all label
Loop through all pixels f [x,y], left to right, top to bottom
If f [x,y]=0, do nothing.

If f [x,y]=1, increment counter[label[x,y]]

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 35
 Small region removal

Loop through all pixels f [x,y], left to right, top to bottom

If f [x,y]=0, do nothing.

If f [x,y]=1 and counter[label[x,y]]<S, set f [x,y]=0

Removes all regions smaller than S pixels

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 36
 Hole filling as dual to small region removal
After NOT operation, (background)
Mask with holes region labeling, small region removal,
and second NOT operation

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 37
 Region moments
Raw moments M pq =
x , yRegion
x p yq

Central moments
M 10 M 01
pq= (x x) ( y y) with = and =
p q
x y
x , yRegion M 00 M 00

Region orientation and eccentricity:

calculate eigenvectors of covariance
matrix 20 11

02
11

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 39
 Example: Detecting bar codes

Original Image

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 40
 Example: Detecting bar codes
Locally adaptive
thresholding

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 41
 Example: Detecting bar codes
Locally adaptive
thresholding

Filtering by
eccentricity

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 42
 Example: Detecting bar codes
Locally adaptive
thresholding

Filtering by
eccentricity

Filtering by major
axis length

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 43
 Example: Detecting bar codes
Locally adaptive
thresholding

Filtering by
eccentricity

Filtering by major
axis length

Filtering by
orientation

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Image Segmentation 44