2/20/24

Computer Vision
Course
School of Information and
Communication Technology

Computer Vision
Chapter 5: Image segmentation

Computer Vision Group

School of Information and Communications Technology

1
 2/20/24

Chapter 5. Segmentation

• Introduction to image segmentation

• Segmentation based on pixel classification
• Thresholding
• Clustering techniques
• Region-based segmentation
• Region growing algorithm,
• Split and merge algorithm
• Edge-based segmentation
Segmentation based on active contours

2/20/24 CV Group – School of Information and Communication and Technology 3

Introduction

• Purpose:
• to partition an image into meaningful regions with respect
to a particular application
• Goal:
• to cluster pixels into salient image regions, i.e., regions
corresponding to individual surfaces, objects, or natural
parts of objects.
• The segmentation is based on the feature measurements
taken from the image:
• grey level, color, texture, depth or motion…

2/20/24 CV Group – School of Information and Communication and Technology 4

2
 2/20/24

Introduction

Source : Jean-Christophe Baillie, ENSTA, uei.ensta.fr/baillie/assets/ES322%20-%20Segmentation.ppt

2/20/24 CV Group – School of Information and Communication and Technology 5

Introduction

• Entity can be extracted from images using mask

Source : Pascal Bertolino, Cours de Traitement d'images. LIS, INPG (France)

2/20/24 CV Group – School of Information and Communication and Technology 6

3
 2/20/24

Applications

• Image segmentation
• is usually an initial and vital step in a series of processes
aimed at overall image understanding of computer vision
• Segmentation applications:
• Object recognition;
• Image retrieval;
• Medical image analysis;
• Boundary estimation within motion or stereo systems;
• Tracking of objects in a video;
• Classification of terrains visible in satellite images
• …

2/20/24 CV Group – School of Information and Communication and Technology 8

Applications

2/20/24 CV Group – School of Information and Communication and Technology 9

4
 2/20/24

Approaches for image segmentation

• Segmentation is usually based on:
• discontinuities: edges
• sudden changes, borders (frontiers) between regions…
• homogeneous zones: regions
• same color, texture, intensity, ...

2/20/24 CV Group – School of Information and Communication and Technology 10

10

Approaches for image segmentation

• Pixel-based approach
• Region-based approach:
• look for homogeneous areas in the image
• Edge-based approach :
• look for discontinuities in the image
• A closed edge is equivalent to a region
• Hybrid (Dual) approach (region + edge)

2/20/24 CV Group – School of Information and Communication and Technology 11

11

5
 2/20/24

Examples

Original images Segmented images

2/20/24 CV Group – School of Information and Communication and Technology 12

12

Pixel-based approach

• Pixel-based approach
• Thresholding
• Clustering

• It is not a region segmentation technique

• But we often in segmentation looking for regions
• Need some post-processing

2/20/24 CV Group – School of Information and Communication and Technology 13

13

6
 2/20/24

Thresholding

• Thesholding is a simple and popular method for

object segmentation in digital images

• Thresholding can be
• Global: one threshold for the whole image
• Local: one threshold for a part of the image
• Adaptive: one threshold ajusted according to each image
or each image part

2/20/24 CV Group – School of Information and Communication and Technology 15

15

Basic global thresholding

• Basic thresholding (2 classes) – main idea :

• IF value(pixel) >= threshold THEN value(pixel) = 1 (or 255)
• IF value(pixel) < threshold THEN value(pixel) = 0

• The result is a binary image

• It is also possible to use n thresholds to split the

image in n+1 classes

• Problem: choosing the threshold(s)!

2/20/24 CV Group – School of Information and Communication and Technology 16

16

7
 2/20/24

Basic global thresholding

• Find the threshold on histogram of gray level

intensity (histogram thresholding)

0, 𝑓(𝑥, 𝑦) < 𝑇
𝑔 𝑥, 𝑦 = &
1, 𝑓(𝑥, 𝑦) ≥ 𝑇

2/20/24 CV Group – School of Information and Communication and Technology 17

17

Basic global thresholding

• Threshold value: not difficile if

• Controled environment
• Industrial applications

2/20/24 CV Group – School of Information and Communication and Technology 18

18

8
 2/20/24

Multi-thresholds

• n thresholds to split the image in n+1 classes:

• IF value(pixel) < threshold_1
• THEN value(pixel) = 0
• IF value(pixel) >= threshold_1 && value(pixel) <
threshold_2
• THEN value(pixel) = 1
• …
• IF value(pixel) >= threshold_n
• THEN value(pixel) = n
• Problems: How many thresholds?

2/20/24 CV Group – School of Information and Communication and Technology 19

19

Multi-thresholds

0, 𝑓 𝑥, 𝑦 < 𝑇1
𝑔 𝑥, 𝑦 = /1, 𝑇2 > 𝑓 𝑥, 𝑦 ≥ 𝑇1
2, 𝑓 𝑥, 𝑦 ≥ 𝑇2

2/20/24 CV Group – School of Information and Communication and Technology 20

20

9
 2/20/24

Threshold value

• Global thresholding: How to find the value of the

threshold T ?
• Value obtained by tests
• The mean value of gray values
• The median value between the min gray level and the max
one
• One value balancing both sections of the histogram
• automatic thresholding

2/20/24 CV Group – School of Information and Communication and Technology 21

21

Choice of thresholds (optimal)

• 2 surfaces (background and object) in an image

• We suppose mathematical models for distributions
(gaussians, etc.)
• We determine the probability of error in the classes 1 and
2 (surfaces 1 et 2)
• We search for a threshold T resulting in a minimum error
• Several methods for achieving this

Source : www.iro.umontreal.ca/~dift2730/

2/20/24 CV Group – School of Information and Communication and Technology 22

22

10
 2/20/24

Example: Global automatic thresholding

• One possible algorithm:

• Choose an initial value for the threshold T (mean, median, …)
• We obtain 2 groups of pixels
• G1 where f(x,y) >= T and G2 where f(x,y) < T
• Compute the gray level means for G1 and G2 -> μ1 and μ2
• Compute a new value for T
• T = 1/2 (μ1 + μ2)
• Repeat until T is ~ constant

• There exist many other global automatic methods

• Otsu, Kittler, K-means, …
• No solution on which one to use
• Must be tested for each new application

2/20/24 CV Group – School of Information and Communication and Technology 23

23

Example: Otsu algorithm

• Sweep all possible threshold value

for T
• For each value of T:
• Compute the mean and variance for
each class
• We look for the intraclass variance
• Means: 𝜇!, 𝜇"
• Variances: 𝜎!", 𝜎""
• Intra-class variance:
𝜎#" = 𝑃! ∗ 𝜎!" + 𝑃" ∗ 𝜎""

• The optimal threshold is the one

with the minimum value for 𝜎!"

• It is based on the idea that classes

are well defined and well grouped
Source : www.iro.umontreal.ca/~dift2730/
2/20/24 CV Group – School of Information and Communication and Technology 24

24

11
 2/20/24

Example: Otsu algorithm

• Threshold found by the algorithm:

• T = 125

2/20/24 CV Group – School of Information and Communication and Technology 25

25

Global threshold: problem

• Problem:
• Global thresholding cannot be used in that case
• Solution: adaptive local thresholding

2/20/24 CV Group – School of Information and Communication and Technology 26

26

12
 2/20/24

Example of adaptive thresholding

• Split the image in sub-images and process each sub-image with its own threshold
• The main decision is to choose the size of the sub-images
• Before processing each sub-image, check the variance to make sure that the sub-
image contains two regions, and not only one.
• Example: no thresholding for a sub-image if variance<100

2/20/24 CV Group – School of Information and Communication and Technology 27

27

Example of adaptive thresholding

Bimodal

Bimodal ?

2/20/24 CV Group – School of Information and Communication and Technology 28

28

13
 2/20/24

Example of adaptive thresholding

2/20/24 CV Group – School of Information and Communication and Technology 29

29

Clustering-based segmentation
• Image is considered as a set of N image pixels.
• Attributes (property) of the pixels
• gray level of single-band gray tone images,
• color values of three-band color images: (r, g, b)
• values of multi-band images, …
• Based on the similar attribute, pixels classification operators
partition an image into homogeneous regions.
• Clustering provides a grouping of the pixels that is dependent on their values
in the image but not necessarily on their locations in the image unless
location is a specified property
• Classifier provide the pixel classes which should be homogeneous regions.

2/20/24 CV Group – School of Information and Communication and Technology 30

30

14
 2/20/24

Clustering algorithms

• Image segmentation approaches including:

• Feature space clustering approaches
• Graph-based approaches

• Clustering algorithms:
• K-Means clustering
• Mean-Shift Clustering
• Expectation-Maximization Clustering
• Watershed Segmentation
• Graph Cuts (Spectral clustering)
• Normalized cuts
• …

2/20/24 CV Group – School of Information and Communication and Technology 31

31

K-means Clustering

• Let X = {p1, . . . , pN} be a set of N image pixels:

• V(pi): the property vector associated with pixel pi
• The clustering algorithm is to partition the image into K clusters (K
regions)

• The K-means algorithm:

• Initialization step: An initial property vector of each class Ck is chosen
randomly from the set of all property vector, note μk(0)
• Interactive step: Assignment of image pixels to K clusters
• Pixel pi is assigned to the closest cluster, using a
distance between 2 property vectors.
• Update the property vector of each class: μ k(t) is
computed as the mean of {V(pi) | xi ∈ Ck}.
• 2 Steps above are repeated until algorithm convergence.
each class Ck should be a region Rk
2/20/24 CV Group – School of Information and Communication and Technology 32

32

15
 2/20/24

K-means Clustering

2/20/24 CV Group – School of Information and Communication and Technology 33

33

K-means Clustering

Input image K-means on gray level K-means on color

Source : D.A. Forsyth and J. Ponce. Computer Vision : A Modern Approach. Prentice-Hall, 2002.

2/20/24 CV Group – School of Information and Communication and Technology 34

34

16
 2/20/24

K-means Clustering

K-means on color for

11 groups

Source : D.A. Forsyth and J. Ponce. Computer Vision : A Modern Approach. Prentice-Hall, 2002.

2/20/24 CV Group – School of Information and Communication and Technology 35

35

Example

2/20/24 CV Group – School of Information and Communication and Technology 36

36

17
 2/20/24

Example

2/20/24 CV Group – School of Information and Communication and Technology 37

37

Pixel-based approach: Pros & cons

• Pros
• Simple, fast

• Cons: thresholding is mainly an operation on pixels

• It does not give connected regions è can add more
features
• we need to « clean » the results
• erase lonely pixels, keep regions

• Other segmentation methods exist

• that can keep the integrity of regions (connected pixels)

2/20/24 CV Group – School of Information and Communication and Technology 38

38

18
 2/20/24

Features for segmentation

• Intensity, Color?
• Position
• Texture
•…

2/20/24 CV Group – School of Information and Communication and Technology 39

39

Segmentation as clustering

Depending on what we choose as the feature space, we

can group pixels in different ways.

Grouping pixels based

on intensity similarity

Feature space: intensity value (1-d)

Slide credit: Kristen Grauman
2/20/24 CV Group – School of Information and Communication and Technology 40

40

19
 2/20/24

Segmentation as clustering

Depending on what we choose as the feature space, we can

group pixels in different ways.

Grouping pixels based on

intensity+position similarity
Intensity

Both regions are black, but if we also

include position (x,y), then we could group
X the two into distinct segments; way to
encode both similarity & proximity.
Slide credit: Kristen Grauman

2/20/24 CV Group – School of Information and Communication and Technology 41

41

Segmentation as clustering

Depending on what we choose as the feature space, we can

group pixels in different ways.

Grouping pixels based

on texture similarity
F1

F2
Filter bank
of 24 filters
…

F24
Feature space: filter bank responses (e.g., 24-d)
Slide credit: Kristen Grauman
2/20/24 CV Group – School of Information and Communication and Technology 42

42

20
 2/20/24

Segmentation with texture features

• Find “textons” by clustering vectors of filter bank outputs

• Describe texture in a window based on texton histogram

Count
Image Texton map
Texton index

Count

Count
Malik, Belongie, Leung and Shi. IJCV 2001. Texton index Texton index

Adapted from Lana Lazebnik

2/20/24 CV Group – School of Information and Communication and Technology 43

43

Image segmentation example

Slide credit: Kristen Grauman

2/20/24 CV Group – School of Information and Communication and Technology 44

44

21
 2/20/24

Region-based segmentation

• Finding region based on the criterion of homogeneity

and connectivity of pixels (region)
• Each region is homogeneous (i.e., uniform in terms of the
pixel attributes such as intensity, color, range, or texture,
etc.)
• and connected
• Algorithms:
• Region growing
• Split and merge algorithm
• Hierarchical clustering
• …

2/20/24 CV Group – School of Information and Communication and Technology 45

45

Region-based segmentation

• Region-based approaches provide :

• All pixels must be assigned to regions
• Each pixel must belong to a single region only
• Each region must be uniform
• Any merged pair of adjacent regions must be non-uniform
• Each region must be a connected set of pixels
• Region-based approaches:
• Different methods
• Common point: homogeneity criteria

2/20/24 CV Group – School of Information and Communication and Technology 46

46

22
 2/20/24

Region growing

• Start from a point (seed) and add neighbor pixels following a

given criteria
• The seeds can be manually or automatically chosen
• automatic seeds in very homogeneous zones for example

image Seed Region growing Final result

2/20/24 CV Group – School of Information and Communication and Technology 47

47

Region growing algorithm

• Algorithm:
• Choose K random pixels in K regions
• Use 8-connected and threshold to determine
• Repeat a and b until almost points are K classified.
• Example illustrated:

2/20/24 CV Group – School of Information and Communication and Technology 48

48

23
 2/20/24

Region growing with multi-seeds

2/20/24 CV Group – School of Information and Communication and Technology 49

49

Example

Simulation
of region
growing
(90% pixels )

2/20/24 CV Group – School of Information and Communication and Technology 50

50

24
 2/20/24

Split-and-merge

• Split (step 1)
• Recursively split all non-homogeneous
regions following a given criteria
• variance, max-min, …
• Dividing one region gives 4 subregions
• Subregion attributes are re-computed
• Merge (step 2)
• Group all homogeneous adjacent
regions following a given criteria

2/20/24 CV Group – School of Information and Communication and Technology 51

51

Split-and-merge: split

Image initiale
Split1
Split 2
split 3

Homogeneity = criteria on the variance

(or max-min <= 1)

Source : Jean-Christophe Baillie. Cours de segmentation. ENSTA ParisTech (France).

2/20/24 CV Group – School of Information and Communication and Technology 52

52

25
 2/20/24

Split-and-merge: merge

Phase 1: Create homogeneous zones (split)

Phase 2: Group homogeneous zone (merge)

Quadtree

Connect homogeneous adjacent regions

Source : Jean-Christophe Baillie. Cours de segmentation. ENSTA ParisTech (France)

2/20/24 CV Group – School of Information and Communication and Technology 53

53

Split-and-merge

2/20/24 CV Group – School of Information and Communication and Technology 54

54

26
 2/20/24

Edge-based segmentation

• Finding region based on edges

• Algorithms:
• Basic Edge Detection
• The Marr-Hildreth edge detector (LoG)
• Short response Hilbert transform (SRHLT)
• Watersheds

2/20/24 CV Group – School of Information and Communication and Technology 55

55

Watershed segmentation

• We consider the image as a 3D shape using the gray

level as the third dimension

2D image Visualization in 3D
Source : http://www.gpa.etsmtl.ca/cours/gpa669/
2/20/24 CV Group – School of Information and Communication and Technology 56

56

27
 2/20/24

Watershed segmentation

• After we reverse (upside down) the values to create

« holes » in the shape

2/20/24 CV Group – School of Information and Communication and Technology 57

57

Watershed segmentation

• Next we fill in the holes with water

2/20/24 CV Group – School of Information and Communication and Technology 58

58

28
 2/20/24

Watershed segmentation

2/20/24 CV Group – School of Information and Communication and Technology 59

59

Watershed segmentation

https://docs.opencv.org/master/d3/db4/tutorial_py_watershed.html
2/20/24 CV Group – School of Information and Communication and Technology 60

60

29
 2/20/24

Watershed segmentation

2/20/24 CV Group – School of Information and Communication and Technology 61

61

Segmentation – advices

• Image segmentation
• No method works for all images
• No miracle, no warranty!

• One of the main problem is to define the goal of

segmentation:
• What exactly are we looking for in the image?
• Global regions or small details?
• Presence or not of persons details in the face?
• It is good to think in advance what we will do with
this segmentation results
• This helps to define the level of precision needed

2/20/24 CV Group – School of Information and Communication and Technology 62

62

30
 2/20/24

Segmentation – advices

• Image Pre-processing:
• good selection of sensors and energy source, and
controled image acquisition conditions help to make
segmentation easier and more efficient
• For some applications, we realize today that we can
avoid to segment the image. It is often better like
this.

2/20/24 CV Group – School of Information and Communication and Technology 63

63

Limits of segmentation

Image segmentation alone

cannot find all image
objects as we can interpret
them

2/20/24 CV Group – School of Information and Communication and Technology 64

64

31
 2/20/24

Segmentation vs. grouping

• Term 'segmentation' :
• less used
• segmentation, which let think about an exact image splitting into regions

• 'Pixel grouping'
• which refers only to a notion of similarity between pixels without relation on the
content of regions.

Source : [Malik 2001].

2/20/24 CV Group – School of Information and Communication and Technology 65

65

Motion segmentation

Input sequence Image Segmentation Motion Segmentation

Input sequence Image Segmentation Motion Segmentation

A.Barbu, S.C. Zhu. Generalizing Swendsen-Wang to sampling arbitrary posterior probabilities,

IEEE Trans. PAMI, August 2005.
Credit: Kristen Grauman, UT Austin

2/20/24 CV Group – School of Information and Communication and Technology 66

66

32
 2/20/24

THANK YOU !

CV Group – School of Information and Communication and Technology

71
71

33