Department of Mechatronics

Image Acquisition and Pre-processing

MACHINE VISION AND IMAGE PROCESSING
(MTE 4075)

Year/ Semester : IVth / VII th

Name of Faculty:
Dr. Umesh Sahu
(Assistant Professor)
Manipal Institute of Technology, Manipal, India
−1−
Syllabus
Image acquisition and pre-processing: [15]
Vision system components, Image acquisition and analysis, Image
formation - Sampling & Quantization, simple operations on image,
Image enhancement Spatial Domain and frequency domain methods,
Image noise, image restoration, morphological operations,
Segmentation, image analysis, representation of regions, Feature
extraction, fundamentals of color image processing, conversion of
color spaces.

Manipal Institute of Technology, Manipal, India

−2−
Content:
▪Image Segmentation
–Introduction
–Discontinuity based Image Segmentation
•Edge detection
–Segmentation based on property of similarity
–Thresholding Based segmentation
–Region based segmentation
–Segmentation Quality Indexes

Manipal Institute of Technology, Manipal, India

−3−
Problem

Original image
Ideal image is
often has
composed of the
inconsistencies
fewest possible
due to light and
distinct intensities.
noise.

Manipal Institute of Technology, Manipal, India

−4−
Solution

Segment original image.

▪ Represent array of related intensities with
one intensity.
▪ Allows user or machine to more easily
determine which pixels belongs to which object.

Manipal Institute of Technology, Manipal, India

−5−
Segmentation
▪ Image segmentation is defined as the
partitioning of an image into non-overlapped,
consistent regions which are homogeneous
in respect to some characteristics such as
intensity, color, tone, texture, etc.

▪ The major ways to do the segmentation:

– Thresholding
– Clustering
– Edge-based segmentation
– Region-based segmentation

Manipal Institute of Technology, Manipal, India

−6−
 Image Segmentation
❑ Applications of image segmentation includes:
• Identifying objects in a scene for object-based measurements
❑ Usually image such as size and shape.
segmentation is • Identifying objects which are at different distances from a sensor
an initial and using depth measurements from a laser range finder enabling
vital step in a path planning for a mobile robot.
series of • Automated target detection, ex: infrared image segmentation in
processes military application.
aimed at overall • Automated Industrial Application.
image • Remote sensing: Satellite image segmentation for detecting the
crop area, vegetation area, urban area etc.
understanding
• Bio-medical: MRI, Ultrasound, CT image segmentation for
diagnosis of diseases and disorders.

Manipal Institute of Technology, Manipal, India

−7−
 Segmentation

Discontinuity Similarity of pixels

Variation in pixel intensities. All neighbouring pixels

Appropriate grouping of Have similarity in pixel
edge pixels. intensities are grouped
into a region.

Manipal Institute of Technology, Manipal, India

−8−
1. Discontinuity based Image Segmentation

Edge detection Techniques are applied.

What is
Edge
A set of edge pixels are obtained detection?

Some edge pixels Some edge pixels

correspond to are isolated.
boundary of objects.

Manipal Institute of Technology, Manipal, India

−9−
Two edge pixels are linked up based upon two
criteria

➢ Edge strength- gradient

➢ Edge angle- direction of the gradient

Boundary detection is done by examining the links of

edge pixels.

Manipal Institute of Technology, Manipal, India

− 10 −
Gradient of an image f(x,y) at location (x,y)
 f 
Gx   x 
f =   =  f 
G y   
 y 

Magnitude of gradient vector

f = 
Gx2 + Gy 
2 1/ 2

Direction of the gradient vector

−1  G y

 ( x, y ) = tan  
 Gx 
Manipal Institute of Technology, Manipal, India
− 11 −
An edge pixel at (x', y') is similar in magnitude
to the pixel at (x , y) if

f ( x, y) − f ( x, y)  E

where E is a non-negative threshold.

An edge pixel at (x' , y') has an angle similar

to the pixel at (x , y) if
 ( x, y) −  ( x, y)  A

Manipal Institute of Technology, Manipal, India

− 12 −
Similarity of edge pixels are determined by taking a
neighborhoods of (3X3) or (5X5).

0 0 0 0 0 0 Link 2

0 0 0
0 0 0

Link 1

Manipal Institute of Technology, Manipal, India

− 13 −
 ➢ Local edge linking approach: A local neighborhood
is used.

➢ Global edge linking approach: each edge pixel is

linked with other edge pixels.

Hough transform

More about edge linking approach: https://users.cs.cf.ac.uk/dave/Vision_lecture/node30.html

Manipal Institute of Technology, Manipal, India
− 14 −
 2. Image Segmentation based on property of similarity

f1 f3
f4

f2

Extract the regions or object of interest based on their properties of

similarity.

Manipal Institute of Technology, Manipal, India

− 15 −
Segmentation based on Thresholding
• Based on similarity of intensity value.
• Above threshold ➔ Binary 1
• Below Threshold ➔ Binary 0
• Threshold selection:
Ex. With dark object and bright background
• Image intensity:
f (x. y )  T : object point

f (x. y )  T: : background

Manipal Institute of Technology, Manipal, India

− 16 −
 Thresholding Histogram
Analyze Histogram.
• Peak and Valley method.
1.Find two most prominent peak in histogram.
2.Find deepest valley in histogram.
3.Use gray value of deepest valley.

Manipal Institute of Technology, Manipal, India

− 17 −
Manipal Institute of Technology, Manipal, India
− 18 −
 Object Background

No. of Pixels

0 255
T Gray Values
Dark Bright
Manipal Institute of Technology, Manipal, India
− 19 −
Manipal Institute of Technology, Manipal, India
− 20 −
 Object Background

No. of Pixels

MATLAB Demo

0 Gray Values 255

Tutorial_3_10_1.m

Manipal Institute of Technology, Manipal, India

− 21 −
Manipal Institute of Technology, Manipal, India
− 22 −
Threshold operator
▪Factors on which T depends:
T = f1 x, y, p(x, y ), f (x, y )
➢Depends on local neighbourhood : f(x, y)
➢Vary from region to region : (x, y)
➢Depends on local property: P(x, y)
▪A thresholded image is defined as

1 if f ( x, y )  T
g ( x, y ) =  f(x,y) = intensity at (x,y)
0 if f ( x, y )  T
P(x,y) = some property at (x,y)

Manipal Institute of Technology, Manipal, India

− 23 −
 Threshold
Types

Global Threshold Local Threshold

T only depends T is a function of
on f(x, y). both P(x, y) and f(x, y).

Dynamic threshold
T is a function
of [x, y, P(x, y) f(x, y)].

Manipal Institute of Technology, Manipal, India

− 24 −
➢ In a controlled environment ( proper lighting
condition) Global thresholding is successful.

➢ In an uncontrolled environment, Local and

Dynamic thresholding are successful.

Manipal Institute of Technology, Manipal, India

− 25 −
Basic Global Thresholding
• A heuristic algorithm
1. Select an initial estimate for threshold T
2. Segment the image using T
• G1 is all pixels with intensities > T
• G2 is all pixels with intensities <= T

3. Compute averages m1 and m2 for the pixels in G1 and G2

4. Let T = (m1+m2)/2
5. Repeat steps 1-4 until no more change
Manipal Institute of Technology, Manipal, India
− 26 −
 MATLAB Demo

Example 10.15
Manipal Institute of Technology, Manipal, India
− 27 −
 Manual
Thresholding

Global
Thresholding

Tutorial_3_10_1.m

Manipal Institute of Technology, Manipal, India

− 28 −
 Optimal Global Thresholding
▪ Any threshold separates the histogram into 2 groups with each
group having its own statistics (mean, variance)
▪ The homogeneity of each group is measured by the within group
variance.
▪ This threshold is determined by minimizing intra-class intensity
variance, or equivalently, by maximizing inter-class variance.
▪ The optimum threshold is that threshold which
– minimizes the within group variance thus maximizing the
homogeneity of each group

Manipal Institute of Technology, Manipal, India

− 30 −
Optimal Global Thresholding
▪ po(z): pdf of dark region.
▪ pb(z): pdf of bright region.

Overall probability density condition,

h(i)
2500.00

2000.00

Background
1500.00

1000.00

500.00
Pb(z)
▪ Po + P b = 1 0.00
0.00 50.00
Object
100.00 150.00 200.00 250.00
i

Po(z)

T
Manipal Institute of Technology, Manipal, India
− 31 −
 Optimal Global Thresholding

–Minimisation of the within group variance

• Let group o (object) be those pixels with graylevel <=T
• Let group b (background) be those pixels with graylevel >T
• The prior probability of group o is po(T)
• The prior probability of group b is pb(T)

– Robot Vision, Haralick & Shapiro, volume 1, page 20

Manipal Institute of Technology, Manipal, India
− 32 −
 Optimal Global Thresholding
▪The following expressions can easily be derived for prior
probabilities of object and background
T
po ( T ) =  P( i )
i =0
255
pb ( T ) =  P( i )
i = T +1

P(i ) = h( i ) / N
▪where h(i) is the histogram of an N pixel image

Manipal Institute of Technology, Manipal, India

− 33 −
 Optimal Global Thresholding
• The mean and variance of each group are
as follows :
T
 o ( T ) =  iP(i ) / po ( T )
i=0
255
 b ( T ) =  iP( i ) / pb ( T )
i = T +1

 ( T ) =  i −  o ( T ) P(i ) / po ( T )
T 2
2
o
i=0

 ( T ) =  i −  b ( T ) P( i ) / pb ( T )
255 2
2
b
i = T +1

Manipal Institute of Technology, Manipal, India

− 34 −
 Optimal Global Thresholding

• The within group variance is defined as :

 W2 ( T ) =  2o ( T ) po ( T ) +  b2 ( T ) pb ( T )

• We determine the optimum T by minimizing this

expression with respect to T

Manipal Institute of Technology, Manipal, India

− 35 −
Optimal Global Thresholding
h(i)
2500.00

2000.00

Histogram
1500.00
Within group variance

1000.00

500.00

0.00 i
0.00 50.00 100.00 150.00 200.00 250.00

Manipal Institute of Technology, Manipal, India

− 36 −
 Optimal Global Thresholding
• We can examine the performance of this algorithm on low and high
noise image
▪ For the low noise case, it gives an optimum threshold of T=33
▪ Almost exactly halfway between the object and background peaks
▪ We can apply this optimum threshold to both the low and high
noise images

High noise image Thresholded at

Low noise image Thresholded at
T=69
T=33
Manipal Institute of Technology, Manipal, India
− 37 −
 Optimal Global Thresholding (Limitation)

–Otsu's method performs well when the histogram has

a bimodal distribution with a deep and sharp valley
between the two peaks.
–Like all other global thresholding methods, Otsu's
method performs badly in case of heavy noise, small
objects size, inhomogeneous lighting and larger intra-
class than inter-class variance.
• In those cases, local adaptations of the Otsu method have
been developed.

Manipal Institute of Technology, Manipal, India

− 39 −
 Otsu’s Method for Global Thresholding
Used in Matlab’s
• Choose threshold to minimize the “graythresh” function

within group variance

W = Po o + Pb  b
2 2 2
0.03
Summary

0.025

• Or equivalently, maximize the

0.02
variance between groups
 B2 = P0 (μ o− μ G ) + Pb ( μb − μ G )
2 2 0.015

0.01

• where 0.005

k L−1
Po =  pi , Pb =  pi 0
0 50 100 150 200 250 300
i=0 i=k+1

• μG is the global mean; μk is the mean of class k

Manipal Institute of Technology, Manipal, India

− 40 −
 Otsu Algorithm
Algorithm:
1. Compute histogram and probabilities of each intensity level
2. Set up initial Pi (0) and 𝜇𝑖 (0)
3. Step through all possible thresholds t=1,…… maximum
intensity
1. Update Pi and 𝜇𝑖
2. Compute 𝜎𝐵2
4. Desired threshold corresponds to the maximum

Manipal Institute of Technology, Manipal, India

− 41 −
 Example 10.16
Manipal Institute of Technology, Manipal, India
− 42 −
 Manual
Thresholding

Original Image

Global
Optimal Thresholding
Thresholding

MATLAB Inbuilt
Function:
level = graythresh(x);

Tutorial_3_10_1.m

Manipal Institute of Technology, Manipal, India

− 43 −
 Adaptive (dynamic) thresholding - how it works?
There are two main approaches to finding the threshold:
(i) the Chow and Kaneko approach and
(ii) local thresholding.
The assumption behind both methods is that smaller image regions are more likely to
have approximately uniform illumination, thus being more suitable for thresholding.
Chow and Kaneko divide an image into an array of overlapping subimages and then
find the optimum threshold for each subimage by investigating its histogram. The
threshold for each single pixel is found by interpolating the results of the subimages.
Limitation: computational expensive and, therefore, is not appropriate for real-time
applications.

Manipal Institute of Technology, Manipal, India

− 44 −
Adaptive (dynamic) thresholding
T12 T13
T11

Image

T22 T23
T21

Manipal Institute of Technology, Manipal, India

− 45 −
 Variable Thresholds
• Global thresholding may not work if illumination or reflectance varies across image
• Can partition image into subimages, compute threshold for each

Example 10.20
Manipal Institute of Technology, Manipal, India
− 46 −
 MATLAB Demo

Tutorial_3_10_2.m

Manipal Institute of Technology, Manipal, India

− 47 −
 Adaptive thresholding - Local thresholding
An alternative approach to find the local threshold is to statistically examine the intensity
values of the local neighborhood of each pixel. The statistic which is most appropriate
depends largely on the input image.

The size of the neighborhood has to be large enough to cover sufficient foreground and
background pixels, otherwise a poor threshold is chosen.
On the other hand, choosing regions which are too large can violate the assumption of
approximately uniform illumination.
This method is less computationally intensive than the Chow and Kaneko approach and
produces good results for some applications.
Manipal Institute of Technology, Manipal, India
− 48 −
Adaptive thresholding - Local thresholding

MATLAB Demo

Tutorial_3_10_2.m

Manipal Institute of Technology, Manipal, India

− 49 −
 Multiple Thresholds
• Otsu’s method can be generalized to find multiple thresholds
• In the case of K classes, we
K
maximize the between-class variance
 B2 =  Pk (mk − mG )
2

k =1

Pk =  p i
where
mk =
1
i pi
Pk iCk
iCk

Manipal Institute of Technology, Manipal, India

− 50 −
 MATLAB Demo
Multiple Thresholds

MATLAB Inbuilt Function: THRESH = multithresh(A, N)

Tutorial_3_10_2.m

Manipal Institute of Technology, Manipal, India

− 51 −
Segmentation in OpenCV/ MATLAB
• MATLAB
• Imbinarize()
• Otsu Algorithm : graythresh()
• Global Threshold: gbt1()
• Multiple Thresholding: multithresh()

• OpenCV Python\
• Simple thresholding: cv2.threshold()
• Adaptive thresholding: cv2.adaptiveThreshold()

Manipal Institute of Technology, Manipal, India

− 52 −
Practice Problem
Gonzalez, Rafael C., and Richard E. Woods. "Digital Image Processing,
New Jersey.", 2007. 3rd edition.

Example: 10.10 to 10.22

Manipal Institute of Technology, Manipal, India

− 53 −
 Region-Based Segmentation

▪ The fundamental drawback of histogram-based region detection is that histograms

provide no spatial information (only the distribution of gray levels).
▪ Edges and thresholds sometimes do not give good results for segmentation.
▪ Region-based segmentation is based on the connectivity of similar pixels in a region.
• Each region must be uniform.
• Connectivity of the pixels within the region is very important.

Manipal Institute of Technology, Manipal, India

− 54 −
 Region Growing
1. Define the set of seed points.
2. Define the similarity function = P (Ri)
3. Append the neighboring pixels based upon the similarity
function.
1 0 0 5 6 7 P(Ri) = pixels within Ri
1 2 1 7 6 5 must not have difference
0 1 2 1 4 6 in intensity more than 3
2 1 0 6 7 7 Difficulties:
2 2 1 4 5 6 1. How to start the seed point
2. What is the similarity function
Manipal Institute of Technology, Manipal, India
− 55 −
How do we choose the seed(s) in practice ?
• It depends on the nature of the problem.

• If targets need to be detected using infrared images for example, choose

the brightest pixel(s).

• Without a-priori knowledge, compute the histogram and choose the gray-
level values corresponding to the strongest peaks.

Manipal Institute of Technology, Manipal, India

− 56 −
How do we choose the similarity criteria (predicate)?

➢ The homogeneity predicate can be based on any characteristic of the

regions in the image such as:
Mean intensity variance color texture
Motion shape size

Manipal Institute of Technology, Manipal, India

− 57 −
 3

Region Based Segmentation

Region:
• A group of connected pixels with similar
properties

• Closed boundaries

• Computation of regions is based on

similarity

• Regions may correspond to Objects in a

scene or parts of objects

• Spatial proximity + similarity

Manipal Institute of Technology, Manipal, India
− 58 −
 4
Region Growing
• Perhaps, one of the simplest approaches among region based
methods.
1. Select a seed point/points
2. Define a growth criteria
3. Join all pixels connected to the seed that follow the
growth criteria
4. Stop when no adjacent pixel agree with the growth
criteria

4-connectivity 8-connectivity

Manipal Institute of Technology, Manipal, India

− 59 −
Question 1 Question 2

▪ In the following arrangement of ▪ In the following arrangement of

pixels, are the two regions (of 1s) pixels, are the two parts (of 1s)
adjacent? (if 8-adjacency is used) adjacent? (if 4-adjacency is used)

1 1 1
Region 1
1 1 1 Part 1

1 0 1 1 0 1
0 1 0 Region 2 0 1 0 Part 2
0 0 1 0 0 1
1 1 1 1 1 1
1 1 1 1 1 1

Manipal Institute of Technology, Manipal, India

− 60 −
 Region-Based Method: Region Growing
• Region Growing
– Start from a seed, and let it grow (include similar neighborhood)

Key:
similarity
measure

Example 10.23
Manipal Institute of Technology, Manipal, India
− 64 −
 Original figure The Seed Points

Result of region growing Boundaries of segmented defective welds

Manipal Institute of Technology, Manipal, India

− 65 −
What can go wrong with region growing ?

Region growth may “leak”

through a single weak spot
in the boundary

Manipal Institute of Technology, Manipal, India

− 66 −
 Region growing

Sea region leaks into sky

due to a weak boundary
between them

Manipal Institute of Technology, Manipal, India

− 67 −
Example:
For a given image perform segmentation by region growing with 8-connectivety, threshold ≤ 2 and seed points as
shown by bold number(0, 3 and 7).

seed point 7 3 0
label a b c

5 7 7 5 6 7 7 6 a a a a a a a a
6 7 6 5 2 2 7 5 a a a a c c a a
5 6 7 5 0 2 5 6 a a a a c c a a
5 6 0 0 1 1 7 7 a a c c c c a a
2 0 2 0 1 3 7 6 c c c c c b a a
0 2 1 4 3 4 4 4 c c c b b b b b
2 0 2 0 4 3 5 5 c c c c b b b b
5 5 4 4 5 5 4 5 b b b b b b b b

Manipal Institute of Technology, Manipal, India

− 68 −
Exercise:

Manipal Institute of Technology, Manipal, India

− 69 −
▪Advantages：
1. Region growing methods can correctly separate the regions that have the same
properties we define.
2. The concept is simple. We only need a small number of seed points to represent the
property we want, then grow the region.
3. We can determine the seed points and the criteria we want to make.
4. We can choose the multiple criteria at the same time.

▪Disadvantages：
1. Computationally expensive
2. It is a local method with no global view of the problem.
3. Sensitive to noise.

Manipal Institute of Technology, Manipal, India

− 70 −
Region Split-Merge Approaches

Manipal Institute of Technology, Manipal, India

− 72 −
Region-Based Method: Split and Merge
• Split and Merge

The main problem with region splitting is determining where to split a region.

One method to divide a region is to use a quadtree structure.

Quadtree: a tree in which nodes have exactly four descendants

Manipal Institute of Technology, Manipal, India

− 73 −
 Region Splitting and Merging

▪The split and merge procedure:

–Split into four disjoint quadrants any region Ri for which P(Ri) = FALSE.

–Merge any adjacent regions Rj and Rk for which P(RjURk) = TRUE.

(the quadtree structure may not be preserved)

–Stop when no further merging or splitting is possible.

Manipal Institute of Technology, Manipal, India

− 74 −
Split and Merge Algorithm

• Unlike RG, region splitting starts with the whole image as a single region
and subdivides it into subsidiary regions recursively while a condition of
homogeneity is not satisfied.
• Region merging is the opposite of splitting, and works as a way of
avoiding over-segmentation

Manipal Institute of Technology, Manipal, India

− 75 −
 Region-Based Method: Split and Merge
• Example: Quadtree Split and Merge Procedure

Iteration 1

split merge

original image 4 regions 4 regions

(nothing to merge)
Split Step → split every non-uniform region to 4
Merge Step → merge all uniform adjacent regions
Manipal Institute of Technology, Manipal, India
− 76 −
Region-Based Method: Split and Merge
• Example: Quadtree Split and Merge Procedure

Iteration 2

split merge

from Iteration 1 13 regions 4 regions

Split Step → split every non-uniform region to 4

Merge Step → merge all uniform adjacent regions
Manipal Institute of Technology, Manipal, India
− 77 −
 Region-Based Method: Split and Merge
• Example: Quadtree Split and Merge Procedure

Iteration 3

split merge

from Iteration 2 10 regions 2 regions

Split Step → split every non-uniform region to 4 final segmentation

Merge Step → merge all uniform adjacent regions result

Manipal Institute of Technology, Manipal, India

− 78 −
Hard Problem: Textures

Similarity measure makes the difference

Manipal Institute of Technology, Manipal, India

− 79 −
 The Split-and-Merge Algorithm

1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 2
1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0
3 1 4 9 9 8 1 0 3 1 4 9 9 8 1 0
1 1 8 8 8 4 1 0 1 1 8 8 8 4 1 0
1 1 6 6 6 3 1 0 1 1 6 6 6 3 1 0
1 1 5 6 6 3 1 0 1 1 5 6 6 3 1 0
1 1 5 6 6 2 1 0 1 1 5 6 6 2 1 0
1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0

Sample image First split

8
0

Manipal Institute of Technology, Manipal, India

− 80 −
 The Split-and-Merge Algorithm

1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 2
1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 0
3 1 4 9 9 8 1 0 3 1 4 9 9 8 1 0
1 1 8 8 8 4 1 0 1 1 8 8 8 4 1 0
1 1 6 6 6 3 1 0 1 1 6 6 6 3 1 0
1 1 5 6 6 3 1 0 1 1 5 6 6 3 1 0
1 1 5 6 6 2 1 0 1 1 5 6 6 2 1 0
1 1 1 1 1 1 0 0 1 1 1 1 1 1 0 0

Final segmented
Sample image
result
8
1

Manipal Institute of Technology, Manipal, India

− 81 −
 Example 10.24
Manipal Institute of Technology, Manipal, India
− 82 −
 Self learning Topic
Segmentation Quality Indexes
• Jaccard Coefficient (Intersection-Over-Union)
• Dice coefficient (F1 Score)
• Summary

References:
1. https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2
2. Zou, K. H., Warfield, S. K., Bharatha, A., Tempany, C. M., Kaus, M. R., Haker, S. J., Wells, W. M., 3rd, Jolesz, F. A., & Kikinis, R. (2004).
Statistical validation of image segmentation quality based on a spatial overlap index. Academic radiology, 11(2), 178–189.
https://doi.org/10.1016/s1076-6332(03)00671-8

Manipal Institute of Technology, Manipal, India

− 84 −
Jaccard Coefficient (Intersection-Over-Union)
• The Intersection-Over-Union (IoU), also known as the Jaccard Index, is one
of the most commonly used metrics in segmentation.
• The IoU is a very straightforward metric that’s extremely effective.

• This metric ranges from 0–1 (0–100%) with 0 signifying no overlap

and 1 signifying perfectly overlapping segmentation.

Manipal Institute of Technology, Manipal, India

− 85 −
IoU: Case Study
• For binary (two classes) or multi-class segmentation, the mean IoU of the
image is calculated by taking the IoU of each class and averaging them.
• Let’s assume all the ships (colored boxes) are part of the same class.

Image from Vlad Shmyhlo in article: Image Segmentation: Kaggle experience (Part 1 of 2) in TDS

Manipal Institute of Technology, Manipal, India

− 86 −
IoU: Case Study

Predicted segmentation (Left) Ground truth annotation (Right)

• Let’s calculate the ship IoU first.

• We assume the total area of the image is 100 (100 pixels).
• First, let’s think about the ships’ overlap. We can pretend that we move the predicted
segmentation (left) directly above the ground truth (right), and see if there are any ship pixels
that overlap.
• Since there are no pixels that are classified as ships by the model, there are 0 overlapping
ship pixels.
Manipal Institute of Technology, Manipal, India
− 87 −
IoU: Case Study
• Union consists of all of the pixels classified as ships from both images, minus the
overlap/intersection. In this case, there are 5 pixels that are classified as ships total. Subtract the
overlap/intersection which is 0 to get 5 as the area of union.

• Ships:
• Area of Overlap = 0, Area of Union = (5+0)-0 =5
• Area of Overlap/Area of Union = 0%
• Background:
• Area of Overlap = 95, Area of Union = (95+100)–95 = 100
• Area of Overlap/Area of Union =95%
• Mean IoU = (Ships + Background)/2 = (0%+95%)/2 = 47.5%

ThIt is obvious that 47.5 is a much better indication of the success of segmentation.

Manipal Institute of Technology, Manipal, India

− 88 −
 Dice coefficient (F1 Score)
Dice Coefficient is 2 * the Area of Overlap divided by the total number of
pixels in both images.

Calculations (based on previous case study):

• Total Number of Pixels for both images combined = 200
• Ships:
• Area of Overlap = 0
• (2 * Area of Overlap)/(total pixels combined) = 0/200 = 0
• Background:
• Area of Overlap = 95
• (2 * Area of Overlap)/(total pixels combined) = 95*2/200 = 0.95
• Dice = (Ships + Background)/2 = (0%+95%)/2 = 47.5%

Manipal Institute of Technology, Manipal, India

− 89 −
Summary
• The Dice coefficient is very similar to the IoU.
• They are positively correlated, meaning if one says model A is better
than model B at segmenting an image, then the other will say the
same.
• Like the IoU, they both range from 0 to 1, with 1 signifying the
greatest similarity between predicted and truth.

Manipal Institute of Technology, Manipal, India

− 90 −
Summary
▪We have looked at:
❑Image Segmentation
–Introduction
–Discontinuity based Image Segmentation
•Edge detection
–Segmentation based on property of similarity
–Thresholding Based segmentation
–Region based segmentation
–Segmentation Quality Indexes (self learning)
Manipal Institute of Technology, Manipal, India
− 91 −
Thank you
for your attention

Manipal Institute of Technology, Manipal, India

− 92 −