Image Segmentation - I

Lecture 19
15

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 4

Agenda
• Image segmentation basics
• Thresholding based
• Binarization
• Otsu
• Region based
• Merging
• Splitting
• Clustering based
• K-means (SLIC)
• Mean-shift

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 6

Image Segmentation
• Partition an image into a collection of set of pixels
• Meaningful regions (coherent objects)
• Linear structures (line, curve, …)
• Shapes (circles, ellipses, …)

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 7

Image Segmentation
• Partition an image into a collection of set of pixels
• Meaningful regions (coherent objects)
• Linear structures (line, curve, …)
• Content based image retrieval
• Shapes (circles, ellipses, …) • Machine Vision
• Medical Imaging applications (tumor delineation,..)
• Object detection (face detection,…)
• 3D Reconstruction
• Object/Motion Tracking
• Object-based measurements such as size and shape
• Object recognition (face recognition,…)
• Fingerprint recognition,
• Video surveillance
…
11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 8
Image Segmentation
• In computer vision, image segmentation is one of the oldest and most
widely studied problems
• Early techniques -> region splitting or merging
• Recent techniques -> Energy minimization, hybrid methods, and deep learning

Inference car

Higher level
Low level features
inference
11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 9
 Thresholding
Machine Region based
Learning methods
based (region
methods growing,..)

Image
Segmentation Clustering
Energy
Methods
minimization (k-means,
methods mean
(MRF,..) shift,..)

Shape based Graph-based

methods
methods (level
(graph-cut,
set, active random
contours) walk,…)
11/16/2021 CAP5415 - Lecture 15 Image Segmentation -I 10
Basics of Image Segmentation
• Definition: Image segmentation partitions an image into regions
called segments.

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 11

Basics of Image Segmentation
• Definition: Image segmentation partitions an image into regions called
segments.

Image segmentation creates segments of connected pixels by analyzing

some similarity criteria:
11/16/2021intensity, color, texture, histogram, features,
CAP5415 - Lecture …
15 Image Segmentation -I 12
Agenda
• Image segmentation basics
• Thresholding based
• Binarization
• Otsu
• Region based
• Merging
• Splitting
• Clustering based
• K-means (SLIC)
• Mean-shift

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 13

Image Binarization
• Image binarization applies often just one global threshold
T for mapping a scalar image I into a binary image

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 14

Image Binarization
• Image binarization applies often just one global threshold
T for mapping a scalar image I into a binary image

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 15

Image Binarization
• Image binarization applies often just one global threshold T
for mapping a scalar image I into a binary image

• The global threshold can be identified by an optimization

strategy aiming at creating “large” connected regions and at
reducing the number of small-sized regions, called artifacts.

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 16

Image Binarization
• Thresholding: Most frequently employed method for
determining threshold is based on histogram analysis of
intensity levels.
Peak on the left of the histogram corresponds
to dark objects
Brighter objects
Peak on the right of the histogram corresponds
to brighter objects

DIFFICULTIES
Darker objects
1. The valley may be so broad that it is difficult
to locate a significant minimum
2. Number of minima due to type of details in
the image
3. Noise
4. No visible valley
5. Histogram may be multi-modal
11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 17
Thresholding Example

Original Image Thresholded Image

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 18

Thresholding Example 2

Threshold Too Low Threshold Too High

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 19

Thresholding Example 3

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 20

Thresholding Example-4

110 147 185

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 21

Agenda
• Image segmentation basics
• Thresholding based
• Binarization
• Otsu
• Region based
• Merging
• Splitting
• Clustering based
• K-means (SLIC)
• Mean-shift

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 22

Otsu Thresholding
• Definition: The method uses grey-value histogram of the given image I as
input and aims at providing the best threshold

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 23

Otsu Thresholding
• Definition: The method uses grey-value histogram of the given image I as
input and aims at providing the best threshold
• Otsu’s algorithm selects a threshold that maximizes the between-class
variance .

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 24

Otsu Thresholding
• Definition: The method uses grey-value histogram of the given image I as
input and aims at providing the best threshold
• Otsu’s algorithm selects a threshold that maximizes the between-class
variance .
• In the case of two classes,

where P1 and P2 denote class probabilities, and μi the means of object and
background classes.
u=p1u1+p2u2
p1+p2=1

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 25

Otsu Thresholding
• Definition: The method uses grey-value histogram of the given image I as
input and aims at providing the best threshold
• Otsu’s algorithm selects a threshold that maximizes the between-class
variance .
• In the case of two classes,

where P1 and P2 denote class probabilities, and μi the means of object and
background classes.
• Let be the relative cumulative histogram of an image I, then P1 and P2 are
approximated by and , respectively.
• u is assumed to be the chosen threshold.
11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 26
Otsu Thresholding Algorithm

11/16/2021 probabilities ClassCAP5415

means- Lecture 15 Image Segmentation - I 27
 Example: Otsu Thresholding

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 28

 Example: Otsu Thresholding

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 29

Agenda
• Image segmentation basics
• Thresholding based
• Binarization
• Otsu
• Region based
• Merging
• Splitting
• Clustering based
• K-means (SLIC)
• Mean-shift

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 30

 Region Based Segmentation-Basics
Region:
A group of connected pixels
with similar properties

Closed boundaries

Computation of regions is
based on similarity

Regions may correspond to Objects

in a scene or parts of objects

Spatial proximity + similarity

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 31
Region Growing
• For segment generation in grey-level or color images, we may start
at one seed pixel (x,y,I(x,y)) and add recursively adjacent pixels that
satisfy a “similarity criterion” with pixels contained in the so-far
grown region around the seed pixel.

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 33

Region Growing
• For segment generation in grey-level or color images, we may start
at one seed pixel (x,y,I(x,y)) and add recursively adjacent pixels that
satisfy a “similarity criterion” with pixels contained in the so-far
grown region around the seed pixel.

• Defining similarity criteria alone is not an effective basis for

segmentation
• It is necessary to consider the adjacency spatial relationship
between pixels

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 34

Region Growing
Algorithm

1. The absolute intensity difference between candidate pixel and the

seed pixel must lie within a specified range
2. The absolute intensity difference between a candidate pixel and the
running average intensity of the growing region must lie within a
specified range;
3. The difference between the standard deviation in intensity over a
specified local neighborhood of the candidate pixel and that over a
local neighborhood of the candidate pixel must (or must not) exceed a
certain threshold

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 35

Seeded Segmentation (Region Growing)

1. Choose the seed pixel

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 36

Seeded Segmentation (Region Growing)

1. Choose the seed pixel

2. Check the neighboring pixels and add them to the region if they are
similar to the seed

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 37

Seeded Segmentation (Region Growing)

1. Choose the seed pixel

2. Check the neighboring pixels and add them to the region if they are
similar to the seed
3. Repeat step 2 for each of the newly added pixels; stop if no more
pixels can be added

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 38

Ex: Muscle/Bone Segmentation in CT Scans

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 39

Ex: Muscle/Bone Segmentation in CT Scans

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 40

Ex: Muscle/Bone Segmentation in CT Scans

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 41

Ex: Muscle/Bone Segmentation in CT Scans

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 42

Region Growing Implementation

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 43

Comparison of Thresholding and Region Growing

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 44

Agenda
• Image segmentation basics
• Thresholding based
• Binarization
• Otsu
• Region based
• Merging
• Splitting
• Clustering based
• K-means (SLIC)
• Mean-shift

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 45

Region splitting and Merging Segmentation

• Region splitting:
• Unlike region growing, which starts from a set of seed points, region splitting
starts with the whole image as a single region and subdivides it into
subsidiary regions recursively while a condition of homogeneity is not
satisfied.

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 46

Region splitting and Merging Segmentation

• Region splitting:
• Unlike region growing, which starts from a set of seed points, region splitting
starts with the whole image as a single region and subdivides it into
subsidiary regions recursively while a condition of homogeneity is not
satisfied.
• Region merging:
• Region merging is the opposite of splitting, and works as a way of avoiding
over-segmentation

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 47

Region splitting and Merging Segmentation

• Region splitting:
• Unlike region growing, which starts from a set of seed points, region splitting
starts with the whole image as a single region and subdivides it into
subsidiary regions recursively while a condition of homogeneity is not
satisfied.
• Region merging:
• Region merging is the opposite of splitting, and works as a way of avoiding
over-segmentation
• Start with small regions (2x2 or 4x4 regions) and merge the regions that have
similar characteristics (such as gray level, variance).

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 48

Region splitting and Merging Segmentation

• RAG: region
adjacency graph

• Quadtree for
splitting (top-down)
procedure

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 49

Region splitting and Merging Segmentation

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 50

Region splitting and Merging Segmentation

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 51

Depth of RAG – how many levels?
2 2 2 2 1 1 1 1

2 2 2 2 1 1 1 1

2 2 3 3 2 2 2 2

2 2 3 3 3 2 2 2

1 1 1 1 2 2 2 2

1 1 1 1 2 2 2 2

1 1 1 1 2 2 2 2

1 1 1 1 2 2 2 2

11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 53

Region Growing Implementation

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 9

Region splitting and Merging Segmentation

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 10

Agenda
• Image segmentation basics
• Thresholding based
• Binarization
• Otsu
• Region based
• Merging
• Splitting
• Clustering based
• K-means (SLIC)
• Mean-shift

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 11

 Clustering Based Segmentation Methods

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 12

What is Clustering?
• Organizing data into classes such that:
• High intra-class similarity
• Low inter-class similarity

• Finding the class labels and the number of classes directly from the
data (as opposed to classification tasks)

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 13

What is a natural grouping?

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 14

What is a natural grouping?

Clustering is subjective

11/22/2021Simpson's Family School Employees Females

CAP5415 - Lecture 16 Image Segmentation - II Males 15
What is similarity ?

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 16

What is similarity ?
Cluster by features

• Color
• Intensity
• Location
• Texture
• ….

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 17

Distance metrics

Peter Piotr

0.23 3 342.7

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 18

Agenda
• Image segmentation basics
• Thresholding based
• Binarization
• Otsu
• Region based
• Merging
• Splitting
• Clustering based
• K-means (SLIC)
• Mean-shift

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 19

K-means
Most well-known and popular clustering algorithm:

Start with some initial cluster centers

Iterate:
• Assign/cluster each example to closest center
• Recalculate centers as the mean of the points in a cluster

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 20

K-means: an example

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 21

K-means: Initialize centers randomly

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 22

K-means: assign points to nearest center

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 23

K-means: readjust centers

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 24

K-means: assign points to nearest center

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 25

K-means: readjust centers

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 26

K-means: assign points to nearest center

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 27

K-means: readjust centers

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 28

K-means: assign points to nearest center

No changes: Done
11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 29
K-means
Iterate:
• Assign/cluster each example to closest center
• Recalculate centers as the mean of the points in a cluster

How do we do this?
11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 30
K-means
Iterate:
◼ Assign/cluster each example to closest center
iterate over each point:
- get distance to each cluster center
- assign to closest center (hard cluster)
◼ Recalculate centers as the mean of the points in a cluster

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 31

K-means
Iterate:
◼ Assign/cluster each example to closest center
iterate over each point:
- get distance to each cluster center
- assign to closest center (hard cluster)
◼ Recalculate centers as the mean of the points in a cluster

11/22/2021 What distance measure

CAP5415 - Lecture should
16 Image Segmentation - II we use? 32
Distance measures
Euclidean:

n
d(x, y) = å i=1(xi - yi ) 2

good for spatial data

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 33

K-means
Iterate:
• Assign/cluster each example to closest center
• Recalculate centers as the mean of the points in a cluster

11/22/2021
Where are the cluster centers?
CAP5415 - Lecture 16 Image Segmentation - II 34
K-means
Iterate:
• Assign/cluster each example to closest center
• Recalculate centers as the mean of the points in a cluster

11/22/2021
How do we calculate these?
CAP5415 - Lecture 16 Image Segmentation - II 35
K-means
Iterate:
• Assign/cluster each example to closest center
• Recalculate centers as the mean of the points in a cluster

Mean of the points in the cluster:

1
m (C) = å
|C | xÎC
x

where:

x xi
å i=1 C
n
å
n
x+y= x + yi =
11/22/2021
i=1 i
CAP5415 - Lecture 16 Image Segmentation - II
C 36
K-means loss function
K-means tries to minimize what is called the “k-means”
loss function:

n
loss = å d(x , m )
i k
2
where mk is cluster center for xi
i=1

that is, the sum of the squared distances from

each point to the associated cluster center

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 37

K-means variations/parameters
Start with some initial cluster centers

Iterate:
◼ Assign/cluster each example to closest center
◼ Recalculate centers as the mean of the points in a cluster

What are some other

variations/parameters we haven’t
specified?
11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 38
K-means variations/parameters
Initial (seed) cluster centers

Convergence
• A fixed number of iterations
• partitions unchanged
• Cluster centers don’t change

11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 39

K-means: Initialize centers randomly

What would happen here?

11/22/2021
Seed selection ideas?
CAP5415 - Lecture 16 Image Segmentation - II 40
Seed choice
Results can vary drastically based on random seed selection

Some seeds can result in poor convergence rate, or

convergence to sub-optimal clusterings

Common heuristics
• Random centers in the space
• Randomly pick examples
• Points least similar to any existing center (furthest centers
heuristic)
• Try out multiple starting points
• Initialize with the results of another clustering method
11/22/2021 CAP5415 - Lecture 16 Image Segmentation - II 41
SLIC (Achanta et al. PAMI 2012)
1. Initialize cluster centers on pixel
grid in steps S
- Features: Lab color, x-y position
2. Move centers to position in 3x3
window with smallest gradient
3. Compare each pixel to cluster
center within 2S pixel distance
and assign to nearest
4. Recompute cluster centers as + Fast 0.36s for 320x240
mean color/position of pixels + Regular superpixels
+ Superpixels fit boundaries
belonging to each cluster - May miss thin objects
- Large number of superpixels
5. Stop when residual error is
11/22/2021 small CAP5415 - Lecture 16 Image Segmentation - II 42
Slide Credits and References
• Ulas Bagci
• David Kauchak (k-means)
• R.Klette, Concise Computer Vision, 2014.
• Y. Ukrainitz & B. Sarel, Weizmann
• Thu Huong Nhuyen, TU Dresden
• Jens N. Kaftan and Andr`e A. Bell and Til Aach . Mean Shift Segmentation Evaluation of Optimization Techniques
• Cheng, Y. (1995). Mean Shift, Mode Seeking, and Clustering. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 17(8):790–799.
• KEINOSUKE FUKUNAGA, AND LARRY D. HOSTETLER. The Estimation of the Gradient of a Density Function, with Applications
in Pattern – Recognition. IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. IT-21, NO. 1, JANUARY 1975
• Comaniciu, D. and Meer, P. (1997). Robust Analysis of Feature Spaces: Color Image Segmentation.
• In IEEE Conference on Computer Vision and Pattern Recognition. CVPR 1997, pages 750–755.
• Comaniciu, D. and Meer, P. (1999). Mean Shift Analysis an Applications. In International Conference on Computer Vision.
ICCV 1999, volume 2, pages 1197–1203.
• Comaniciu, D. and Meer, P. (2002). Mean Shift: A Robust Approach Toward Feature Space Analysis. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 24(5):603–619.
• K. Fukunaga, L.D. Hostetler, The estimation of the gradient of a density function, with applications in pattern recognition,
IEEE Trans. Inf. Theory 21 (1975) 3240.
• Yiping Hong , Jianqiang Yi and Dongbin Zhao . Improved mean shift segmentation approach for natural images
11/16/2021 CAP5415 - Lecture 15 Image Segmentation - I 54