Lecture 16

Image Segmentation
1. The basic concepts of segmentation
2
2. P i t liline, edge
Point, d d detection
t ti
3. Thresh holding
4
4. Region based segmentation
Region-based
5. Segmentation with Matlab
What is segmentation
• What is segmentation
– Segmentation subdivides an image into its constituent regions or
objects until
objects, ntil the objects of interest in an application ha
have
e been
isolated.
• Segmentation conditions
R0 ,..., Rn
n
(a) UR i =R
i =0

(b) Ri is a connected set i = 1, 2,..., n

(c) Ri ∩ R j = ∅ for all i and j , i ≠ j
(d ) Q(Ri ) = true for i = 1, 2,..., n
(e) Q(Ri ∪ R j ) = false
f l ffor any adjacent
dj t regions
i Ri andd R j
where Q( Ri ) is a logical predicate defined over the points in set Rk .

• Segmentation problem: to partition the image into

regions satisfying above conditions
Two principal approaches
• Edge-based segmentation
– partition an image based on abrupt changes in intensity (edges)

• Region-based segmentation
– partition an image into regions that are similar according to a set
off predefined
d fi d criteria.
it i
Detection of Discontinuities
• Detect the three basic types of gray-level discontinuities
– points , lines , edges
• Use the image sharpening techniques
– The first order derivatives produce thicker edges
– The second order derives have a strong response to fine detail,
such as thin lines and isolated points, and noise
– Laplasian operation
• Can be done by running a mask through the image

R = w1 z1 + w2 z2 + ... + w9 z9
9
= ∑ wk zk
k =1

4
Point Detection
Steps for point dection
1. Apply
pp y Laplacian
p filter to the image
g
to obtain R(x, y)
2. Create binary image by threshold

⎧1, if | R( x, y ) |≥ T
g ( x, y ) = ⎨
⎩0,, otherwise
where T is a nonnegative threshold

5
Example

6
Line Detection
• A special mask is needed to detect a special type of line
• Examples:
– H
Horizontal
i t l maskkh
has hi
high
h response when
h a liline passed
d th
through
h
the middle row of the mask.

7
 Multiple Lines Detection
• Apply every masks on the image, find the maximum response.

Example:

Let R1, R2, R3, R4 denotes the response of the horizontal, +45 degree,
vertical and -45 degree masks, respectively. if, at a certain point in the
image |Ri| > |Rj|, for all j≠i, that point is said to be more likely associated
with a line in the direction of mask i.

8
Edge Detection

• Edge detection is the approach for segmenting

images based on abrupt changes in intensity
intensity.
• What is an edge
– an edge is a set of connected pixels that lie on the boundary
between two regions.
– an edge is a “local” concept whereas a region boundary, owing
to the way
y it is defined,, is a more global
g idea.
• Edge models
– Step edge (Idea edge), ramp edge (thick edge), and roof edge

9
Example of edges in an image
• An image may all the three types of edges
First and Second derivatives at the edge

1. The magnitude of the first derivative can be used to detect the

presence of an edge at a point
2. Second derivative produces two values for every edge in an image.
An imaginary straight line joining the extreme positive and negative
values of the second derivative would cross zero near the midpoint
of the edge. zero
zero--crossing point: the center of thick edges

11
Noise Images
• First column: images and gray-
level profiles of a ramp edge
p
corrupted by
y random Gaussian
noise of mean 0 and σ = 0.0,
0.1, 1.0 and 10.0, respectively.

• Second column: first

first-derivative
derivative
images and gray-level profiles.

• Third column : second-

derivative images and gray
gray-
level profiles.

12
Steps in edge detection
1. Image smoothing for noise reduction
2. Detection of edge points. Points on an edge
3. Edge localization

13
Image gradient
• Gradient is a vector ⎡ ∂f ⎤
⎡ g x ⎤ ⎢ ∂x ⎥
∇f = ⎢ ⎥ = ⎢ ∂f ⎥
⎣g y ⎦ ⎢ ⎥
⎢⎣ ∂y ⎥⎦
• The magnitude of the gradient
2
⎛ ∂f ⎞ ⎛ ∂f ⎞
2

M ( x, y ) = mag (∇f ) = g x2 + g y2 = ⎜ ⎟ + ⎜⎜ ⎟⎟
⎝ ∂x ⎠ ⎝ ∂y ⎠
• The direction of the gradient vector
⎡ gy ⎤
α ( x, y ) = tan ⎢ ⎥
−1

⎣ gx ⎦
• The direction of an edge at (x
(x, y) is perpendicular to the direction
of the gradient vector at that point
14
Example

⎡ ∂f ⎤
⎢ ∂x ⎥ ⎡ −2 ⎤
∇f = ⎢ ⎥ = ⎢ ⎥ , M ( x, y ) = 2 2,
⎢ ∂f ⎥ ⎣ 2 ⎦
⎢⎣ ∂y ⎥⎦
α ( x, y ) = tan −1 (−2 / 2) = −45o
Gradient Operators and Masks
∂f
gx = = f ( x + 1, y ) − f ( x, y )
∂x
∂f
gy = = f ( x, y + 1) − f ( x, y )
∂y
Roberts cross-gradient operators
∂f ( x, y )
gx = = z9 − z5
∂x
∂f ( x, y )
gy = = z8 − z6
∂y
Prewitt operators
∂f
gx = = ( z7 + z8 + z9 ) − ( z1 + z2 + z3 )
∂x
∂f
gy = = ( z3 + z6 + z9 ) − ( z1 + z4 + z7 )
∂y

Sobel operators
∂f
gx = = ( z7 + 2 z8 + z9 ) − ( z1 + 2 z2 + z3 )
∂x
∂f
gy = = ( z3 + 2 z6 + z9 ) − ( z1 + 2 z4 + z7 )
∂y
16
Prewitt and Sobel masks for Diagonal edges

Sobel masks have

slightly superior
noise-suppression
characteristics which
is an important issue
when dealing with
derivatives.

17
Example

18
Example
Example

20
Example
Laplacian

Laplacian operator ∂ 2
f ( x , y ) ∂ 2
f ( x, y )
∇ f =
2
+
∂x 2
∂y 2

∇ 2 f = [ f ( x + 1, y ) + f ( x − 1, y )
+ f ( x, y + 1) + f ( x, y − 1) − 4 f ( x, y )]

22
Laplacian of Gaussian
• Laplacian combined with smoothing as a precursor to
find edges via zero-crossing.

x2 + y2
−
G ( x, y ) = e 2σ 2

∂ 2G ∂ 2G
∇ G ( x, y ) =
2
+
∂x ∂y
x2 + y2
⎡ x + y − 2σ ⎤
2 2 2 −
= ⎢ 2σ 2

⎥e
⎣ σ 4
⎦

g ( x, y ) = ∇ 2G ( x, y ) ∗ f ( x, y 0

23
Marr-Hildreth edge detection algorithm
1. Filter the input with an n by an Gaussian lowpass filter
2. Compute the Laplacian of the image of step 1
3. Find the zero crossing of the image from step
Approximate the zero crossing from LoG image to threshold the LoG image by
setting all its positive values to white and all negative values to black
black. the zero
crossing occur between positive and negative values of the thresholded LoG.
Zero crossing vs. Gradient
• Attractive
– Zero crossing produces thinner edges
– Noise reduction
• Drawbacks
– Zero crossing creates closed loops. (spaghetti effect)
– sophisticated computation.
• Gradient is more frequently used.

25
Example

a). Original image

b). Sobel Gradient
c). Spatial Gaussian
smoothing function
d). Laplacian mask
e).
) LoG
L G
f). Threshold LoG
g). Zero crossing

26
Edge Linking and Boundary Detection
• An edge detection algorithm are followed by linking
procedures to assemble edge pixels into meaningful
edges.
d
• Basic approaches
– Local Processing
– Global Processing via the Hough Transform
– Global Processing via Graph-Theoretic Techniques

27
Local Processing
• Analyze the characteristics of pixels in a small neighborhood Sxy
(say, 3x3, 5x5) about every edge pixels (x, y) in an image.
• All points
i t th
thatt are similar
i il according di tto a sett off predefined
d fi d criteria
it i are
linked, forming an edge of pixels that share those criteria.
• Criteria | M ( s, t ) − M ( x, y ) |≤ E
where E is a positive threshold
| α ( s, t ) − α ( x, y ) |≤ A
where A is a positive angle threshold
• Algorithm steps
1. compute ∇f , M ( x, y ), α ( x, y )
⎧1, M ( x, y ) > TM and α ( x, y ) = A ± TA
2. Form a binary image g ( x, y ) = ⎨
⎩0, otherwise
3. Scan the rows of g and fill all gaps ineach tow that do not exceed a specified
length K
4. Detect the gap in any direction θ , rotate g by this angle and apply the horizontal
scanning procedure in Step 3. Rotate the result by - θ
28
Example of local precessing

TM =30%,A =90,TA = 45, Gap =25

 Hough Transformation (Line)

yi =ax
axi + b b = - axi + yi
all points (xi ,yi) contained on the same line must have lines in
parameter space that intersect at (a (a’,b’)
,b )

30
ρθ-plane

• problem of using equation y = ax + b is that value of a is infinite for a

vertical line.
• To avoid the problem, use equation x cos θ+ y sin θ = ρ to represent a
line instead.
• vertical has θ = 90° with
ti l liline h ith ρ equals
l tto th
the positive
iti y-intercept
i t t or θ =
-90° with ρ equals to the negative y-intercept
31
Generalized Hough Transformation
The method can be used for any function of the form
g(v, c) = 0, where v is a vector of coordinates, c is a vector
of coefficients

Example: circles, (x-c1)2 + (y-c2)2 = c32

(c1, c2, c3) cube like cells, accumulators of the form A(i, j, k)
increment c1 and c2 , solve of c3 that satisfies the equation
update the accumulator corresponding to the cell associated
with triplet (c1, c2, c3)

33
Edge-linking based on Hough Transformation
1. Compute the gradient of an image and threshold it to
obtain a binary image.
2. Specify subdivisions in the ρθ-plane.
3. Examine the counts of the accumulator cells for high
pixel concentrations.
p
4. Examine the relationship (principally for continuity)
between pixels in a chosen cell.
Continuity:
1. based on computing the distance between
disconnected p pixels identified during
g traversal of the
set of pixels corresponding to a given accumulator cell.
2. a gap at any point is significant if the distance between
that point and its closet neighbor exceeds a certain
threshold.
34
Example
link criteria:
1). the pixels belonged to one of the set of pixels linked
according
di tto ththe hi
highest
h t countt
2). no gaps were longer than 5 pixels 36
Image Segmentation II

1. Threshold
2
2. R i b
Region-based
d segmentation
t ti
3. Segmentation using watersheds
4
4. Segmentation with Matlab
Thresholding
• Problem: how to determine the threshold value of
segmentation?
• Histogram

⎧ a, if f ( x, y ) > T2
⎧1, if f ( x, y ) > T ⎪
g ( x, y ) = ⎨ g ( x, y ) = ⎨b, if T1 < f ( x, y ) ≤ T2
0 if f ( x, y ) ≤ T
⎩0, ⎪ c, if f ( x, y ) ≤ T
⎩ 2
The role of noise in image thresholding
• When noise is small, the method work, otherwise it may
not work. Noise reduction has to be done before
choosing
h i th threshold
h ld value
l
The Role of Illumination
Illumination plays an important role
f(x,y)
f( y) = i(x,y)
( y) r(x,y)
( y)
Approach:
1.g(
g(x,y)
,y) = ki(x,y)
( ,y)
2.h(x,y) = f(x,y)/g(x,y) = r(x,y)/k

40
Basic Global Thresholding
Based on visual inspection of histogram
1. Select an initial estimate for T.
2 Segment the image using T.
2. T This will produce two
groups of pixels: G1 consisting of all pixels with gray
level values > T and G2 consisting of pixels with gray
level values ≤ T
3. Compute the average gray level values m1 and m2 for
the pixels in regions G1 and G2
4 Compute a new threshold value T = 0.5
4. 0 5 (m1 + m2)
5. Repeat steps 2 through 4 until the difference in T in
successive iterations is smaller than a predefined
parameter To.

41
Basic Global Thresholding
Use T midway between the max and min gray levels
generate binary image

T0 = 1 , 3 iterations, result T = 125

Optimum global thresholding
Choose thresholding values that maximize the between-class variance
Otsu’s method
1 Compute
1. Comp te the normali ed histogram of thek image pi , i = 0,1,2,...,
normalized 0 1 2 L −1
2. Computer the cumulative sums m(k ) = ∑ipi , k = 0,1,2,..., L −1
i =0k
3. Computer the cumulative mean P1 (k ) = ∑ pi , k = 0,1,2,..., L −1
i=0 L−1
4. Computer the global intensity mean, mG = ∑ipi , k = 0,1,2,..., L −1
i =0
5. Compute the between-class variance
[ m G P1 ( k ) − m ( k )]2
σ (k ) =
2
, k = 00,1,
1 2,...,
2 L −1
P1 ( k )[1 − P1 ( k )]
B

6. Obain the Otsu thereshold

k * th e a v e ra g e o f a ll k ' s u c h th a t σ 2
B ( k ' ) = m a x {σ B2 ( k )}
k
7. Compute the separability measure
σ 2
( k *
)
η*= B 2
σG
Example
Example
Example
Using edges to improve global thresholding
1. Compute the edge image g(x, y) of f(x, y)
2. Specify a threshold value T of g(x, y)
3. Threshold the image f(x, y) using T, obtain gt(x, y)
4. Compute a histogram of f(x, y) using pixels (x, y) with
gt(x,
( y)) =1
1
5. Use the above histogram to segment f(x,y) globally.
Example
Multiple thresholding
Otsu’s method can be extended to arbitrary number of thresholds.
1. Compute the normalized histogram of the image pi , i = 0,1,2,..., L −1
2 Computer
2. Comp ter the ccumulative ms Pk = ∑ pi , mk = 1 ∑ ipi , C1,..., CK are classes
m lati e ssums
i∈Ck Pk i∈Ck
3. Computer the cumulative mean
K K
mG = ∑i =1
Pi m i , σ 2
B ( k 1 , .. ., k K ) = ∑
i =1
Pi [ m i − m G ] 2

4. Obain theresold
k 1* , ..., k K* − 1 s u c h th a t σ B2 ( k 1* , ..., k K* − 1 ) = m a x {σ B2 ( k 1 , ..., k K − 1 )}
k

5 Threshold image
5.
⎧ a 1 if f ( x , y ) ≤ k 1*
⎪
⎪ a 2 if k 1 < f ( x , y ) ≤ k 2
* *
g ( x, y) = ⎨
⎪ M
⎪⎩ a K if f ( x , y ) > k K*
Example
Basic Adaptive Thresholding
• Subdivide original image into small areas.

• Utilize a different threshold to segment each subimages.

• The threshold used for each pixel depends on the

location of the pixel in terms of the subimages.

51
Example : Adaptive Thresholding

52
2. Region-Based Segmentation
• Basic Formulation
n
(a ) ∪ Ri = R
i =1

(b) Ri is a connected region, i = 1, 2, ..., n

(c) Ri ∩ R j = φ for all i and j, i ≠ j
(d ) P(Ri ) = TRUE for i = 1, 2, ..., n
(e) P(Ri ∪ R j ) = FALSE for i ≠ j
P(Ri) is a logical predicate property defined over the points in set Ri

ex. P(Ri) = TRUE if all pixel in Ri have the same gray level

53
Two basic approaches
• Region Growing
– start with a set of “seed” points
– growing by appending to each seed those neighbors that have
similar properties such as specific ranges of gray level
e.g
⎧TRUE if theth absolute
b l t difference
diff off the
th
⎪
⎪ intensites between the seed and
Q=⎨
⎪ the pixel at (x,y) is ≤ T
⎪⎩ FALSE otherwise

• Region
g splitting
g and merging
g g
– Iteratively divide a region into smaller regions until all regions
become TRUE
– Merge adjacent regions as along as the resulting region is still
TRUE
54
Region Growing

Select all seed points with gray level 255

criteria:
1. the absolute gray-
level difference
between any pixel
and the seed has to
be less than 65
2. the pixel has to be 8-
connected to at least
one p pixel in that
region (if more, the
regions are merged)

55
Region splitting and merging
Quadtree
1. Split into 4 disjoint quadrants any region Ri for which
P(Ri) = FALSE
2. Merge any adjacent region Rj and Rk for which
P(Ri ∪ Rk ) = TRUE
3. Stop when no further merging or splitting is possible.
Example

⎧ TRUE if σ > a and 0 < m < b

Q=⎨
⎩FALSE otherwise
where m and σ the mean and standard deviation
of pixels in a quadregion
Example

P(Ri) = TRUE if at least 80% of the pixels in Ri have the

property |zj-mi| ≤ 2σi,
where
zj is the gray level of the jth pixel in Ri
mi is the mean gray level of that region
σi is the standard deviation of the gray levels in Ri

58
Segmentation using Watersheds
• The concept of watersheds
– View an image as 3-D graphics
– Three types of points
• Local minimum
• Point at which water will flow to a local minimum
minimum,
also called catchment basin, or watershed.
• Point at which water can equally flow to more than
one local minimum points, also called divide lines,
or watershed lines
• Segmentation by watershed lines
Example
Watershed algorithm
Let M 1 , M 2 ,..., M R be sets denoting the coordinates of the reginoal
minima of image g ( x, y ). C ( M i ) is the set of pixels of catchment
basin of M i
T [n] = {( s, t ) | g ( s, t ) < n}, n = min + 1 to max + 1
Cn ( M i ) = C ( M i ) ∩ T [ n ]
Q[n] denotes the set of connected components of M i
For n = min + 1 to max + 1 do
F each
For h connected
t d componentt q off Q[n]
Q[ ]
if q ∩ C[n − 1] = ∅
Let C[n] ← C[n − 1] ∪ q
else if q ∩ C[ n − 1] contains one connected component of C[n − 1]
C[n] ← C[n − 1] ∪ q
else q ∩ C[n − 1] contains more than one components of C[n − 1]
The build dam within q
Dam construction
Example