Image Segmentation

• Image analysis:
– Extracting information from an image.

• First step:
– To segment the image
– i.e. to subdivide an image into its constituent
regions or objects.
– Sub division is carried depends on the problem
– Segmentation should stop when the objects or
regions of interest have been detected.
 Image Segmentation
• Segmentation is based on two basic
properties of gray-level values:
– Discontinuity, i.e. to partition the image based
on abrupt changes in intensity (gray levels), e.g.
edges

– Similarity, i.e. to partition the image into similar

(according to predefined criteria) regions, e.g.
thresholding, region growing, region splitting and
merging
 Detection of Discontinuities
• 3 basic types of gray-level discontinuities:

Isolated Points
Lines
Edges

• Edge pixels are pixels at which intensity of

an image function changes abruptly.
• Common method of detection: run a mask
through the image.
Image Segmentation
 Point Detection
• Point detection based on the second
derivative. (Laplacian operator)
• T: nonnegative threshold: | R |≥ T

9
R = w1 z1 + w2 z 2 + ... + w9 z9 = ∑ wi zi
i =1
 Point Detection
• A point has been detected at the location on
which the mask is centered if: |R|>T

• The gray level of an isolated point will be

quite different from the gray levels of its
neighbors

– measure the weighted differences between the

center point and its neighbors
 Line Detection

R1 R2 R3 R4

• If at a certain point |Ri|>|Rj|, this point is

more likely associated with a line in the
direction of mask i.
 Edge Detection
• Edge (a set of connected pixels):
– the boundary between two regions with
relatively distinct gray-level properties.
– Note: edge vs. boundary

• Assumption:
– the regions are sufficiently homogeneous, so that
the transition between two regions can be
determined on the basis of gray-level
discontinuities alone.
Image Segmentation
Image Segmentation
 Edge Detection
• Basic Idea:
– A profile is defined perpendicularly to the edge direction
and the results are interpreted.
– The magnitude of the first derivative is used to detect an
edge (if a point is on a ramp)
– The sign of the second derivative can determine whether
an edge pixel is on the dark or light side of an edge.

• Remarks on second derivative:

– It produces two responses for every edge
– The line that can be formed joining its positive and
negative values crosses zero at the mid point of the edge
(zero-crossing)
 Edge Detection
• Computation of a local derivative operator
– A profile is defined perpendicularly to the edge
direction and the results are interpreted.

– The first derivative is obtained by using the

magnitude of the gradient at that point.

– The second derivative is obtained by using the

Laplacian.
 Three fundamental steps
performed in edge detection
• Image smoothing for noise reduction
• Detection of edge points : local operation
that extracts from an image all points that
are potential candidates to become edge
points.
• Edge localization : To select from the
candidate edge points only the points that
are true members of the set of points
comprising an edge.
 Gradient Operators

 ∂f 
Gx   ∂x 
∇F =   =  ∂f 
G y   
 ∂y 

The gradient vector points in the direction of

maximum rate of change of f at (x,y).
 Gradient Operators

Gradient: ∇f = mag (∇F ) = [Gx2 + G y2 ]1/ 2

(maximum rate of increase of f(x,y) per unit distance)

∇f ≈| Gx | + | G y |

 Gy 
−1
Direction angle of ∇f at (x,y): a( x, y ) = tan  
 Gx 
Image Segmentation
Image Segmentation
Image Segmentation
Image Segmentation
Image Segmentation
 Gradient Operators
• Computation of the gradient of an image:

– Soebel operators provide both a differencing & a

smoothing effect:

Gx = ( z7 + 2 z8 + z9 ) − ( z1 + 2 z 2 + z3 )

G y = ( z3 + 2 z6 + z9 ) − ( z1 + 2 z 4 + z7 )
 Summary: Gradient Operators

Smooth edges due to

blurring (result of sampling)

Positive: leading
Negative: trailing
Zero: in constant gray levels

Positive: from dark side

Negative: from light side
Zero: in constant gray levels
 Summary
• The magnitude of the first derivative detects
the presence of an edge and the sign of the
second detects whether the edge pixel lies
on the dark or light side of an edge.

• The second derivative has a zero-crossing at

the mid-point of a transition.
 Laplacian
∂ 2
∂
• (of a 2-D function f(x,y)): ∇ 2 f = 2 + f2
f 2

∂x ∂y

• A 3 x 3 discrete mask based on the above is:

∇ 2 f = 4 z 5 − ( z 2 + z 4 + z 6 + z8 )
 Laplacian
• The idea:

– Coefficient of center pixel should be positive

– Coefficients of outer pixels should be negative
– Sum of coefficients should be zero
(the Laplacian is a derivative)
Image Segmentation
 Laplacian
• The Laplacian is seldom used in practice,
because:

– It is unacceptably sensitive to noise (as second-

order derivative)
– It produces double edges
– It is unable to detect edge direction
 Laplacian
• An important use of the Laplacian:

– To find the location of edges using its zero-

crossings property.

• Plus, the Laplacian plays only the role of

detector of whether a pixel is on the dark or
light side of an edge.
 Laplacian
• Convolve an image with the Laplacian of a
2D Gaussian function of the form:
 x2 + y2 
h(x, y) = −exp− 2 
 2σ 

where σ is the standard deviation.

 Laplacian
• The Laplacian of the above Gaussian is:

 r2 − σ 2   r2 
∇ h = −
2
 exp− 2 
 σ   2σ 
4

where r2 = x2 + y2.

σ determines the degree of blurring that occurs.

Image Segmentation
Image Segmentation
 Edge Linking &
Boundary Detection
• Ideal case:
– Techniques detecting intensity discontinuities
should yield pixels lying only on edges ( or the
boundary between regions).

• Real life:
– The detected set of pixels very rarely describes a
complete edge due to effects from: noise, breaks
in the edge due to non-uniform illumination.
 Edge Linking &
Boundary Detection
• Solution:

– Edge-detection techniques are followed by

linking and other boundary detection procedures
which assemble edge pixels into meaningful
boundaries.
 Local Processing
• Analyze the pixel characteristics in a small
neighborhood (3x3, 5x5) about every (x,y) in
an image.

• Link similar points to form a edge/boundary

of pixels sharing common properties.
 Local Processing
• Criteria used/Properties:

1. The strength of the response of the gradient

operator that produced the edge pixel.

2. The direction of the gradient vector.

 Local Processing
• In other words:
1. ∇f = mag (∇F ) = [G 2
x +G y]
2 1/ 2
or ∇f ≈| Gx | + | G y |

(x’,y’) and (x,y) are similar if:

∇f ( x, y ) − ∇f ( x′, y′) ≤ T

where T is a nonnegative threshold.

 Local Processing
• In other words (cont.):
−1  Gy 
2. α ( x, y ) = tan  
 Gx 
(x’,y’) and (x,y) are similar if:

α ( x, y ) − α ( x′, y′) < A

where A is an angle threshold.
Image Segmentation
 Global Processing via
the Hough Transform
• Points are linked by determining whether
they lie on a curve of specified shape.

• Problem:

– Find subsets of n points that lie on straight lines.

 Global Processing via
the Hough Transform
• Solution:
– Find all lines determined by every pair of points
– Find all subsets of points close to particular lines

– Involves: n(n-1)/2 ~ n2 lines

n(n(n-1))/2 ~ n3 computations
for comparing every point to all lines.
 Global Processing via
the Hough Transform
• Better solution: Hough Transform

– Equation of line passing through point (xi,yi):

yi = axi + b (a,b varies)

– But: b = -xia + yi
equation of single line on ab plane
Global Processing via
the Hough Transform
 Global Processing via
the Hough Transform
• A line in the (x,y) plane passes through
several points of interest and has a set of
specific (a,b) values.

• A line in parameter space [(a,b) plane]

denotes all lines that pass through a certain
point (xi,yi) and has an infinite number of
(a,b) values.
 Global Processing via
the Hough Transform
• A specific line is represented by a point in
the (a,b) plane.

• Two lines in parameter space that meet at a

certain point show points belonging to the
same line (in x,y plane).
 Global Processing via
the Hough Transform
• Since a,b approach infinity as a line
approaches the vertical, we can use the
normal representation of a line:

x cosθ + y sin θ = ρ
 Global Processing via
the Hough Transform
• Hough transform is applicable to any
function of the form g(v,c) = 0.
– v: vector of coordinates, c: coefficients.

• e.g. points lying on a circle:

( x − c1 ) 2 + ( y − c2 ) 2 = c32
 Morphology
• Morphology is a branch of biology , deals with form
and structure of animals and plants

• Mathematical morphology is a tool for extracting

image components useful in:

– representation and description of region shape (e.g.

boundaries)
– pre- or post-processing (filtering, thinning, etc.)
• Based on set theory
 Morphology
• Sets represent objects in images
• Sets in binary images (x,y)
• Sets in gray scale images (x,y,g)
• Some morphological operations:
Dilation & Erosion
Opening & Closing
Hit-or-Miss Transform
Basic Algorithms
Morphological Image Processing
Morphological Image Processing
Morphological Image Processing
 Dilation & Erosion
• Basic operations
• Are dual to each other
- Erosion shrinks foreground and enlarges
background
- Dilation enlarges foreground and shrinks
background
 Dilation & Erosion
• Dilation:
“grows’ or “thickens” objects in an
image. One of the simplest application of dilation
is bridging gaps.
• Erosion:
Shrinking or thinning operation.
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0 0
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0 0 0
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0 0 0 0
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0 0 0 0 1
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0 0 0 0 1 0
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0 0 0 0 1 0 0
 Example for Erosion
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 0 0 0 0 1 0 0 0
 Erosion
• Erosion is the set of all points in the image ,
where the structuring element “ fits into”

• Consider each foreground pixel in the input image

- if the structuring element fits in, write a “1” at
the origin of the structuring element
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1 0
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1 0 1
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1 0 1 1
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1 0 1 1 1
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1 0 1 1 1 1
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1 0 1 1 1 1 1
 Example for Dilation
• Input image 1 0 0 0 1 1 1 0 1 1

• Structuring element

1 1 1

• Output image 1 0 1 1 1 1 1 1
 Dilation
• Dilation is the set of all points in the image ,
where the structuring element “touches” the
foreground
• Consider each pixel in the input image
- if the structuring element touches the foreground
image, write a “1” at the origin of the structuring
element
 Dilation & Erosion
• Basic definitions:

– A,B: sets in Z2 with components a=(a1,a2)

and b=(b1,b2)

– Translation of A by x=(x1,x2), denoted by (A)x is

defined as:
(A)x = {c| c=a+x, for a∈A}
 Dilation & Erosion
• More definitions:

Reflection of B: B̂ = {x|x=-b, for b∈B}

Complement of A: Ac = {x|x∉A}

Difference of A & B: A-B = {x|x∈A, x ∉ B} = A∩Bc

 Dilation & Erosion
• Dilation:

– ∅: empty set; A,B: sets in Z2

– Dilation of A by B:

A ⊕ B = {x | ( Bˆ ) x ∩ A ≠ ∅}
 Dilation & Erosion
• Dilation:

– Obtaining the reflection of B about its origin and

then shifting this reflection by x

– The dilation of A by B then is the set of all x

displacements such that B̂ and A overlap by at
least one nonzero element…
 Dilation & Erosion
• Dilation:

A ⊕ B = {x | [( Bˆ ) x ∩ A] ⊆ A}

B is the structuring element in dilation.

Morphological Image Processing
Morphological Image Processing
 Dilation & Erosion
• Erosion:
A B = {x | ( B ) x ⊆ A}
i.e. the erosion of A by B is the set of all points x
such that B, translated by x, is contained in A.

In general:
( A B)c = Ac ⊕ Bˆ
Morphological Image Processing
Morphological Image Processing
 Thresholding
• Foundation:
 Thresholding
• In A: light objects in dark background

• To extract the objects:

– Select a T that separates the objects from the

background
– i.e. any (x,y) for which f(x,y)>T is an object point.
 Thresholding
• In B: a more general case of this approach
(multilevel thresholding)

• So: (x,y) belongs:

– To one object class if T1<f(x,y)≤T2

– To the other if f(x,y)>T2
– To the background if f(x,y)≤T1
 Thresholding
• A thresholded image:

1 if f ( x, y ) > T (objects)
g ( x, y ) = 
0 if f ( x, y ) ≤ T (background)
 Thresholding
• Thresholding can be viewed as an operation
that involves tests against a function T of
the form:

T = T [ x, y, p ( x, y ), f ( x, y )]
where p(x,y) denotes
some local property of this point.
 Thresholding
• If T is constant applicable over whole image
global thresholding

• If T changes over an image

Variable (local/Regional) thresholding

• When T depends on x and y (spatial

coordinates) dynamic (adaptive)
thresholding
 Simple Global Thresholding
• To partition the image histogram by using a single
threshold T.

• Then the image is scanned and labels are assigned.

• This technique is successful in highly controlled

environments.
Iterative algorithm for estimation
of threshold T
(1)Select an initial estimate for T
(2)Segment image using T−
→ Group G1(values>T)
Group G2(values<T)
(3)Compute average intensity values for G1,G2−
→m1,m2
(4)Compute a new threshold value
T=(1/2)(m1+m2)
(5)Repeat (2) through (4) until the difference in T in
successive iterations is smaller than ∆T
Image Segmentation
 Chapter 10
Image Segmentation
Image Segmentation
 Region-Oriented Segmentation
• Segmentation is a process that partitions R
into n subregions R1, R2, …, Rn such that:
n
– UR i =R
i =1

–Ri is a connected region, i = 1, 2, …, n

–Ri ∩ Rj = 0 for all i and j, i≠j
–P(Ri) = TRUE for i = 1, 2, …, n
P(Ri): logical predicate
–P(Ri ⋃ Rj) = FALSE for i≠j
 Region Growing by
Pixel Aggregation
• Start with a set of “seed” points and from
these grow regions by appending to each
seed point those neighboring pixels that
have similar properties.
 Region Growing by
Pixel Aggregation
• Problems:

– Seed selection
– Selection of suitable properties for including
points in the various regions

• Descriptors
• Local vs. general criteria
 Region Splitting and Merging
• Subdivide an image initially into a set of
arbitrary, disjointed regions and then merge
and/or split the regions in an attempt to
satisfy the conditions of region-oriented
segmentation.

• Quadtree-based algorithm
 Region Splitting and Merging
• Procedure:

– Split into 4 disjointed quadrants any region Ri

where P(Ri) = FALSE
– Merge any adjacent regions Rj and Rk for which
P(Rj ∪ Rk) = TRUE
– Stop when no further splitting or merging is
possible.
Image Segmentation
 Morphological Watersheds
• Visualize an image topographically in 3D
– The two spatial coordinates and the intensity

• Three types of points

– Points belonging to a regional minimum.
– Points ta which a drop of water would fall certainly to
a regional minimum (catchment basin).
– Points at which the water would be equally likely to
fall to more than one regional minimum (crest lines
or watershed lines).
• Objective: find the watershed lines.
Morphological Watersheds (cont.)

Image • Topographic representation.

• The height is proportional to
the image intensity.
• Backsides of structures are
shaded for better visualization.
 Morphological Watersheds (cont.)

• A hole is punched in each regional minimum and the topography is

flooded by water from below through the holes.
• When the rising water is about to merge in catchment basins, a dam is
built to prevent merging.
• There will be a stage where only the tops of the dams will be visible.
• These continuous and connected boundaries are the result of the
segmentation.
 Morphological Watersheds (cont.)
Regional
minima

• Topographic representation of the image.

• A hole is punched in each regional minimum (dark
areas) and the topography is flooded by water (at
equal rate) from below through the holes.
 Morphological Watersheds (cont.)

• Before flooding.
• To prevent water from spilling through the image
borders, we consider that the image is surrounded
by dams of height greater than the maximum image
intensity.
 Morphological Watersheds (cont.)

• First stage of flooding.

• The water covered areas corresponding to the dark
background.
 Morphological Watersheds (cont.)

• Next stages of flooding.

• The water has risen into the other catchment basin.
 Morphological Watersheds (cont.)

• Further flooding. The water has risen into the third

catchment basin.
 Morphological Watersheds (cont.)

Short dam

• Further flooding.
• The water from the left basin overflowed into the
right basin.
• A short dam is constructed to prevent water from
merging.
 Morphological Watersheds (cont.)

Longer dam New dams

• Further flooding.
• The effect is more pronounced.
• The first dam is now longer.
• New dams are created.
 Morphological Watersheds (cont.)

Final watershed lines

superimposed on the
image.

• The process continues until the maximum level of flooding is

reached.
• The final dams correspond to the watershed lines which is
the result of the segmentation.
• Important: continuous segment boundaries.
 Morphological Watersheds (cont.)
• Dams are constructed by morphological dilation.
Cn-1(M1) Cn-1(M2)

Flooding step n-1.

Regional minima: M1 and M2.

Catchment basins associated: Cn-1(M1) and Cn-1(M2).
C[n −1] = Cn−1(M1) ∪Cn−1(M2 )
C[n-1] has two connected components.
 Morphological Watersheds (cont.)

Cn-1(M1) Cn-1(M2) q

Flooding step n-1. Flooding step n.

• If we continue flooding, then we will have one connected

component.
• This indicates that a dam must be constructed.
• Let q be the merged connected component if we perform
flooding a step n.
 Morphological Watersheds (cont.)
q

• Each of the connected components is dilated by

the SE shown, subject to:
1. The center of the SE has to be contained in q.
2. The dilation cannot be performed on points that
would cause the sets being dilated to merge.
 Morphological Watersheds (cont.)
Conditions
1. Center of SE in q.
2. No dilation if merging.

• In the first dilation, condition 1 was satisfied by every

point and condition 2 did not apply to any point.
• In the second dilation, several points failed condition 1
while meeting condition 2 (the points in the perimeter
which is broken).
 Morphological Watersheds (cont.)
Conditions
1. Center of SE in q.
2. No dilation if merging.

• The only points in q that satisfied both conditions form

the 1-pixel thick path.
• This is the dam at step n of the flooding process.
• The points should satisfy both conditions.
 Morphological Watersheds (cont.)
Image Gradient magnitude
• A common application
is the extraction of
nearly uniform, blob-
like objects from their
background.
• For this reason it is
generally applied to the
gradient of the image
and the catchment
basins correspond to
the blob like objects.

Watersheds Watersheds
on the image
 Morphological Watersheds (cont.)

• Noise and local minima lead generally to oversegmentation.

• The result is not useful.
• Solution: limit the number of allowable regions by additional
knowledge.