Image segmentation 1.

Introduction
• The objective is to subdivide an image into its
constituent parts or objects for subsequent
• Introduction processing such as recognition.
• Detection of Discontinuities • It is one of the most important steps leading to the
• Point detection analysis of processed image data.
• Line detection
• Edge detection
• Combined detection Complete v.s. patrial segmentation
• Edge linking and boundary detection • In complete segmentation,
• Thresholding ‰ Disjoint regions segmented are uniquely
• Adaptive thresholding corresponding with objects in the input image.
• Threshold selection based on boundary ‰ Cooperation with higher processing levels which

characteristics use specific knowledge of the problem domain is

necessary.
• Region-oriented segmentation
• Region growing by pixel aggregation • In partial segmentation,
• Region splitting and merging ‰ Regions segmented do not correspond directly

with image objects.

• Totally correct and complete segmentation of

complex scenes usually can't be achieved.
• A reasonable aim is to use partial segmentation as
an input to higher level processing.
&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
Applications: 2. Detection of Discontinuities:
• Simple segmentation problems:
• There are 3 basic types of discontinuities: points,
1. Contrasted objects on a uniform background lines and edges.
2. Simple assembly tasks, blood cells, printed
characters, etc. • The detection is based on convoluting the image
with a spatial mask.
How to achieve segmentation?
 w−1, −1 w−1, 0 w−1,1 
• Image is divided into separate regions that are • A general 3x3 mask  w0, −1 w0, 0 w0,1 

homogeneous with respect to a chosen property
 w1, −1 w1, 0 w1,1 
such as color, brightness, texture, etc.
• Segmentation algorithms generally are based on 2
basic properties of gray level values: • The response of the mask at any point (x,y) in the
1 1
1. Discontinuity - isolated points, lines and edges image is R x , y = ∑ ∑ p( x − i, y − j ) w(i, j )
i = −1 j = −1
of image.
2. Similarity - thresholding, region growing, region
splitting and merging.
• Segmentation methods:
1. Global approaches such as thresholding
2. Edge-based segmentation
3. Region-based segmentation

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
2.1 Point detection 2.2 Line detection
• A point has been detected at the location p(i,j) on
which the mask is centered if |R |>T, where T is a
• Line masks
nonnegative threshold, and R is obtained with the
− 1 − 1 − 1
following mask. 2
Horizontal line 2 2
 
− 1 − 1 − 1 − 1 − 1 − 1
− 1 8 − 1
  − 1 −1 2 
− 1 − 1 − 1 45$ line − 1 2 − 1
 
• The idea is that the gray level of an isolated point  2 − 1 − 1
will be quite different from the gray level of its − 1 2 − 1
neighbors. Vertical line − 1 2 − 1
 
− 1 2 − 1
2 − 1 − 1
- 45$ line − 1 2 − 1
 
− 1 − 1 2 

2ULJLQDO 1RLVHDGGHG • If, at a certain point in the image, |Ri|>|Rj| for all
j ≠ i , that point is said to be more likely associated
with a line in the direction of mask i.

)LOWHUHGRS 7KUHVKROGHGRS

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
 2.3 Edge detection
• It locates sharp changes in the intensity function.
• Edges are pixels where brightness changes abruptly.
• A change of the image function can be described by
a gradient that points in the direction of the largest
growth of the image function.
2ULJLQDO
• An edge is a property attached to an individual pixel
and is calculated from the image function behavior
in a neighborhood of the pixel.
• Magnitude of the first derivative detects the
presence of the edge.
• Sign of the second derivative determines whether
the edge pixel lies on the dark sign or light side.
+RUL]RQWDOOLQH ROLQH

9HUWLFDOOLQH ROLQH

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
 (a) Gradient operator

• For a function f(x,y), the gradient of f at coordinates

(x',y') is defined as the vector
 ∂f 
 
∇f ( x ' , y ' ) =  ∂∂x 
f
 
 ∂y  ( x' , y ' )

• Magnitude of vector ∇f ( x' , y ' ) :

1

  ∂f  2  ∂f  2 
2

∇f ( x ' , y ' ) =    +   
  ∂x   ∂y  
)LJ (GJHGHWHFWLRQE\GHULYDWLYHRSHUDWRUVDOLJKW
  ( x ', y ' )
VWULSHRQDGDUNEDFNJURXQGEGDUNVWULSHRQD
OLJKWEDFNJURXQG
• Direction of the vector ∇f ( x' , y ' ) :
α ( x' , y ' ) = tan −1 (∂∂fy ∂f
∂x
) ( x ', y ' )

• Its magnitude can be approximated in the digital

domain in a number of ways, which result in a
number of operators such as Roberts, Prewitt and
Sobel operators for computing its value.

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
Sobel operator: (b) Laplacian Operator

• It provides both a differentiating and a smoothing • The Laplacian of a 2D function f(x,y) is a 2nd-order
effect, which is particularly attractive as derivatives derivative defined as
typically enhance noise. ∂2 f ∂2 f
∇ f ( x' , y ' ) = 2 + 2
2

∂x ∂y ( x' , y ' )
− 1 − 2 − 1  − 1 0 1
Gx :  0 0 0  Gy : − 2 0 2 • The Laplacian has the same properties in all
    directions and is therefore invariant to rotation in
 1 2 1   − 1 0 1 the image.

• It can also be implemented in digital form in various

ways.

• For a 3x3 region, the mask is given as

 0 −1 0 
− 1 4 − 1
 
 0 − 1 0 
2ULJLQDO 3URFHVVHGLPDJH

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
 2.4 Combined Detection:

• Detection of combinations of points, lines and edges

can be achieved by using sets of orthogonal masks.

• A set of 9 3X3 masks were proposed by Frei and

Chen (1977)
2ULJLQDO 3URFHVVHGLPDJH
Basis of edge subspace :
1 2 1 1 0 −1 
1   1 
• It is seldom used in practice for edge detection for W1= 0 0 0 W = 2 0 − 2
2 2  2 2 
2
the following reasons: − 1 − 2 − 1  1 0 − 1 
1. As a 2nd-order derivative, it is unacceptably
 0 −1 2  2 −1 0 
sensitive to noise. 1   1  
W3= 1 0 − 1 W = −1 0 1 
2. It produces double edges and is unable to detect 2 2  2 2
4

edge direction. − 2 1 0   0 1 − 2 

• The Laplacian usually plays the secondary role of Basis of line subspace :
detector for establishing whether a pixel is on the − 1 0 1 0 1 0
W5=  0 0 0 W6= − 1 − 1
dark or light side of an edge. 1 1
0
2  2 
 1 0 − 1  0 1 0 
1 −2 1  − 2 1 − 2
1
4 − 2
1
W7= − 2 W8= 1 4 1
6  6 
 1 − 2 1  − 2 1 − 2

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
 1 1 1
W9= 1 1 1
1
"Average" subspace : • Example:
3 
4 7 1 
1 1 1
What's the attribute of the center of 3 5 2 ?
 
• Given a 3x3 region represented by {f(i,j)|-2<i,j<2}, 2 0 0
we have
R1 = 4.5607 R2 = 2.2678
1 1
Rm = ∑ ∑ f (i, j ) wm (i, j ) R3 = -2.6213 R4 = -0.8284
i = −1 j = −1 R5 = -0.5000 R6 = 1.0000
1/ 2
Pline =  ∑ Rm  R7 = 0.5000 R8 = 3.0000
8 2

 m =5  R9 = 8.0000
1/ 2
 4
Pedge =  ∑ Rm 
2

 m =1  Pedge = 5.7879
Paverage = R9 Pline = 3.2404
Paver = 8.0000
where Pline , Paverage and Pedge are the magnitudes of the
projections onto edge, line and average subspaces
respectively, which tell how likely it is associated
with either an edge, a line or nothing.

Conclusion: It's likely not to be an edge or a line.

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
2.5 Edge linking and boundary detection • A point (x',y') in the neighborhood of (x,y) is linked
to the pixel at (x,y) if both the following magnitude
• The techniques of detecting intensity discontinuities
and direction criteria are satisfied.
yield pixels lying only on the boundary between
regions.
∇f ( x' , y ' ) − ∇f ( x, y ) ≤ Threshold Tm
• In practice, this set of pixels seldom characterizes a α ( x' , y ' ) − α ( x, y ) ≤ Threshold Td
boundary completely because if noise, breaks in
boundary from nonuniform illumination, and other
effects that introduce spurious intensity
discontinuities.
• Edge detection algorithms are typically followed by
linking and other boundary detection procedures
designed to assemble edge pixels into meaningful
boundaries.
D
(a) Local processing

• Two principal properties used for establishing

similarity of edge pixels in this kind of analysis are:
1. The strength of the response of the gradient
operator used to produce the edge pixel.
2. The direction of the gradient. E F
)LJ D2ULJLQDOLPDJHEGHWHFWLRQUHVXOWZLWKRXWORFDO
• In a small neighborhood, e.g. 3x3, 5x5, all points SURFHVVLQJFGHWFWLRQUHVXOWZLWKORFDOSURFHVVLQJ
with common properties are linked: 7P [PD[_∇ I_DQG7G SL

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
3. Thresholding • Special cases:
If T depends on
• Thresholding is one of the most important 1. f(x,y) only - global threshold
approaches to image segmentation. 2. Both f(x,y) & p(x,y) - local threshold
3. (x,y) - dynamic threshold
• If background and object pixels have gray levels
grouped into 2 dominant modes, they can be • Multilevel thresholding is in general less reliable as
separated with a threshold easily. it is difficult to establish effective thresholds to
isolate the regions of interest.

• Thresholding may be viewed as an operation that

involves tests against a function T of the form 2ULJLQDO 7KUHVKROGUHVXOW7 
T=T[x,y,p(x,y),f(x,y)], where f(x,y) is the gray level
of point (x,y), and p(x,y) denotes some local
property of this point such as the average gray level
of a neighborhood centered on (x,y).

+LVWRJUDP
)LJ1RQDGDSWLYHWKUHVKROGLQJUHVXOW
&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
3.1 Adaptive thresholding

• The threshold value varies over the image as a

function of local image characteristics.
• Image f is divided into subimages.
• A threshold is determined independently in each
subimage.
• If a threshold can't be determined in a subimage, it
can be interpolated with thresholds obtained in
neighboring subimages.
• Each subimage is then processed with respect to its
local threshold. )LJ+LVWRJUDPRIWKHVXELPDJHV

)LJ $GDSWLYHWKUHVKROGLQJUHVXOW78/ 785 

7'/ 7'5 

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
3.2 Threshold selection based on boundary
0 if G[ f ( x, y )] < T
characteristics 
s ( x, y ) =  1 if G[ f ( x, y )] ≥ T and L[ f ( x, y )] ≥ 0
• A reliable threshold must be selected to identify the − 1 if G[ f ( x, y )] ≥ T and L[ f ( x, y )] < 0

mode peaks of a given histogram.
• This capability is very important for automatic where T is a threshold.
threshold selection in situations where image
characteristics can change over a broad range of
intensity distributions.

• We can consider only those pixels that lie on or near

the boundary between objects and the background
such that the associated histogram is well-shaped to
provide a good chance for us to select a good
D
threshold.

• The gradient can indicate if a pixel is on an edge or

not.
• The Laplacian can tell if a given pixel lies on the
dark or light (background or object) side of an edge.
E F
)LJ D2ULJLQDOESURFHVVHGUHVXOWZLWKRXWXVLQJ
ERXQGDU\FKDUDFWHULVWLFDQGFSURFHVVHGUHVXOW
• The gradient and laplacian can produce a 3-level ZLWKXVLQJERXQGDU\FKDUDFWHULVWLF
image
&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
4. Region-oriented segmentation ‰ The regions must be disjoint.
• In previous methods, we partition an image into ‰ It deals with the properties that must be satisfied
regions by finding boundaries between regions by the pixels in a segmented region - for example
based on intensity discontinuities. P(Ri)=true if all pixels in Ri have the same
intensity.
• Here, segmentation is accomplished via thresholds
‰ Regions Ri and Rj are different in the sense of
based on the distribution of pixel properties, such as
predicate P.
intensity or color.
• Basic formulation:
Let R represents the entire image which is
partitioned into subregions R1, R2,...Rn such that
n
‰  Ri = R
i =1

‰ Ri is a connected region, i=1,2...n

‰ Ri ∩ R j = {} for all i ≠ j ,
‰ P( Ri ) =true for i=1,2..n
‰ P( Ri ∪ R j ) =false for i ≠ j
where P(Ri) is a logical predicate over the points in
set Ri.

• Physical meaning of the formulation:

‰ The segmentation must be complete, i.e. every
point must be in a region.
‰ Points in a region must be connected.

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
4.1 Region growing by pixel aggregation
• Region growing is a procedure that groups pixels or
subregions into larger regions.
• Pixel aggregation starts with a set of "seed" points
from those grows by appending to each seed point
those neighboring pixels that have similar properties D E F
such as gray level, texture and color. )LJ 2ULJLQDOLPDJHZLWKVHHGSRLQWEHDUO\VWDJHRI
UHJLRQJURZWKFILQDOUHJLRQ
0 0 5 6 7 a a b b b
1 1 5 8 7 a a b b b
0 1 6 7 7 a a b b b
2 0 7 6 6 a a b b b
0 1 5 6 5 a a b b b • Problems have to be resolved:
Original intensity array Result of threshold=3
1. Selection of initial seeds that properly represent
a
a
a
a
a
a
b
b
b
b
a
a
a
a
a
a
a
a
a
a
regions of interest.
a a b b b a a a a a
a a b b b a a a a a 2. Selection of suitable properties for including
a a a b ? a a a a a
points in the various regions during the growing
Result of Threshold=5.5 Result of threshold=9
process.
([DPSOHRIUHJLRQJURZLQJXVLQJNQRZQVWDUWLQJSRLQWV 3. The formulation of stopping rule.

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
4.2 Region splitting and merging • Example:
• To subdivide an image initially into a set of
arbitrary, disjointed regions and then merge and/or
split the regions in an attempt to satisfy the
conditions stated above.
• A split and merge algorithm is summarized by the (a) (b) (c) (d)
following procedure in which, at each step, we:
(1) split into 4 disjointed quadrants any regions
([DPSOHRIVSOLWDQGPHUJHDOJRULWKP
Ri where P( Ri ) =false;
(2) merge any adjacent regions Rj and Rk for
which P( Ri ∪ R j ) =true; and
(3) stop when no further merging or splitting is
possible. (a) The entire image is split into 4 quadrants.
(b) Only the top left region satisfies the predicate
so it is not changed, while the other 3 quadrants
R
are split into subquadrants.
R1 R2

R1 R2 R3 R4 (c) At this point several regions can be merged,

R 41 R 42
R3 with the exception of the 2 subquadrants that
R 43 R 44
R 41 R 42 R 43 R 44 include the lower part of the object; these do not
satisfy the predicate and must be split further.
(a) (b)

)LJ3DUWLWLRQHGLPDJHDQGFRUUHVSRQGLQJTXDGWUHH

&<+,PDJH6HJPHQWDWLRQS &<+,PDJH6HJPHQWDWLRQS
 D E F
)LJ D2ULJLQDOLPDJHE5HVXOWRIVSOLWDQGPHUJH
DOJRULWKPF5HVXOWRIWKUHVKROGLQJE

• Image segmentation is a preliminary step in most

automatic pictorial pattern-recognition and scene-
analysis problems.
• The choice of one segmentation technique over
another is dicated mostly by the peculiar
characteristics of the problem being considered.

&<+,PDJH6HJPHQWDWLRQS