Image segmentation

What is Image Segmentation?

There are many definitions:
Segmentation subdivides an image into its
constituent regions or objects ( Gonzales, pp567)

Segmentation is a process of grouping together

pixels that have similar attributes ( Efford , pp250)

Image Segmentation is the process of partitioning

an image into non-intersecting regions such that
each region is homogeneous and the union of no
two adjacent regions is homogeneous ( Pal,
pp1277)
 What is Image Segmentation

Segmentation is typically associated with

pattern recognition problems. It is
considered the first phase of a pattern
recognition process and is sometimes also
referred to as object isolation.

Image Feature
Classification
Segmentation Extraction
Objec
Input
Object Image Feature Vector t Type
Image
Why segmentation is difficult ?

It can be difficult for many reasons:

Non- uniform illumination
No control of the environment
Inadequate model of the object of
interest
Noise
Why segmentation is useful ?

Segmentation algorithms have been used

for a variety of applications. Some examples
are :
Optical character recognition(OCR)
Automatic Target Acquisition
Colorization of Motion Pictures
Detection and measurement of bone,
tissue, etc, in medical images.
 Segmentation Examples
Segmentation attempts to partition the
pixels of an image into groups that strongly
correlate with the objects in an image
Typically the first step in any automated
computer vision application
Segmentation Examples
 Image Segmentation
Image segmentation divides an image into
regions that are connected and have some
similarity within the region and some difference
between adjacent regions.

The goal is usually to find individual objects in an

image

For the most part there are fundamentally two

kinds of approaches to segmentation:
discontinuity and similarity.
Similarity may be due to pixel intensity, color or texture.
Differences are sudden changes (discontinuities) in any
of these, but especially sudden changes in intensity
along a boundary line, which is called an edge.
Detection of Discontinuities

There are three kinds of discontinuities of

intensity: points, lines and edges.
The most common way to look for discontinuities
is to scan a small mask over the image. The
mask determines which kind of discontinuity to
look for. 9
R w1 z1  w2 z 2  ...  w9 z9  wi zi
i 1
 Point Detection

Point detection can be achieved simply

using the mask below:

R T
where T : a nonnegativ e threshold

Points are detected at those pixels in the

subsequent filtered image that are above a
set threshold
 Point Detection

X-ray image Result of Result of

of a turbine point thresholding
blade detection
Detection of Discontinuities
Line Detection
Only slightly more common than point detection is
to find a one pixel wide line in an image

For digital images the only three point straight

lines are only horizontal, vertical, or diagonal (+
or –45)
 Line Detection
Binary image of a wire
bond mask

After
Result of
processing
thresholding
with -45°
filtering
line detector
result
 Edge Detection
An edge is a set of connected pixels that lie
on the boundary between two regions
 Edges & Derivatives

We have already spoken

about how derivatives
are used to find
discontinuities
1st derivative tells us
where an edge is
2nd derivative can
be used to show
edge direction
 Derivatives & Noise

Derivative based edge detectors are

extremely sensitive to noise
We need to keep this in mind
Detection of Discontinuities
Gradient Operators
First-order derivatives:
The gradient of an image f(x,y) at
location (x,y) is defined as the vector:
f
 Gx    x
f     f
 G y    y
The magnitude of this vector:
 2
f mag (f )  G  G
x
2
y 
1
2

The direction of this vector:

 
 1  Gx 
 ( x, y ) tan  
 Gy 
 Edge Detection Example
Horizontal Gradient
Original Image Component

Vertical Gradient Component Combined Edge Image

 Edge Detection Problems

Often, problems arise in edge detection in

that there are is too much detail

For example, the brickwork in the previous

example

One way to overcome this is to smooth

images prior to edge detection
Edge Detection Example With
Smoothing
Original Image Horizontal Gradient Component

Vertical Gradient Combined Edge Image

 Detection of Discontinuities
Gradient Operators
Roberts cross-gradient operators

Prewitt operators

Sobel operators
 Detection of Discontinuities
Gradient Operators

Prewitt masks for

detecting diagonal edges

Sobel masks for

detecting diagonal edges

Prewitt and Sobel masks for

detecting diagonal edges
Detection of Discontinuities
Gradient Operators: Example
Same
sequences
as in
previous
image but
with the
original
image
smoothed
with 5*5
averaging
filter.
Detection of Discontinuities
Gradient Operators: Example
Diagonal
edge
detection
. Result
of using
the mask
given
below
 Detection of Discontinuities
Gradient Operators
• Second-order derivatives: (The Laplacian)
– The Laplacian of an 2D function f(x,y) is defined as
2 2 f 2 f
 f  2  2
x y
– Two forms in practice:
 Detection of Discontinuities
Gradient Operators
• Consider the function: A Gaussian function
r2

h(r )  e 2 2
where r 2  x 2  y 2
and  : the standard deviation
• The Laplacian of h is

r2 The Laplacian of a
2 r    2 2 
2 2 Gaussian (LoG)
 h(r )   e
  4

• The Laplacian of a Gaussian sometimes is called the
Mexican hat function. It also can be computed by
smoothing the image with the Gaussian smoothing
mask, followed by application of the Laplacian mask.
Detection of Discontinuities
Gradient Operators
Edge Detection (overview)

28
 Edge Linking and Boundary
Detection Local Processing
• Two properties of edge points are useful for edge
linking:
– the strength (or magnitude) of the detected edge points
– their directions (determined from gradient directions)
• This is usually done in local neighborhoods.
• Adjacent edge points with similar magnitude and
direction are linked.
• For example, an edge pixel with coordinates (x0,y0) in
a predefined neighborhood of (x,y) is similar to the
pixel at (x,y) if
f ( x, y )  ( x0 , y0 ) E , E : a nonnegativ e threshold
 ( x, y )   ( x0 , y0 )  A, A : a nonegative angle threshold
 Edge Linking and Boundary
Detection Local Processing:
In this example, we canExample
find the license plate candidate after
edge linking process.
 Edge Linking and Boundary
Detection Global Processing via the
Hough Transform
• Hough transform: a way of finding edge points in an
image that lie along a straight line.
• Example: xy-plane v.s. ab-plane (parameter space)

yi axi  b

(a)xy-plane
(b)Parameter
Space

(a) (b)
 Edge Linking and Boundary
Detection Global Processing via the
Hough Transform
• The Hough transform
consists of finding all pairs of
values of  and  which x cos   y sin  
satisfy the equations that
pass through (x,y).
• These are accumulated in
what is basically a 2-
dimensional histogram.
• When plotted these pairs of 
and  will look like a sine
wave. The process is
repeated for all appropriate
(x,y) locations.
 Edge Linking and Boundary
Detection Hough Transform Example
The intersection of the
curves corresponding to
points 1,3,5

2,3,4

1,4
 Edge Linking and Boundary
Detection Hough Transform Example

Infrared Threshold
image. ed
gradient

Hough Linked
transfo pixels.
rm.
 Thresholding

• Assumption: the range of intensity levels covered by

objects of interest is different from the background.
1 if f ( x, y )  T
g ( x, y ) 
0 if f ( x, y ) T

Single threshold Multiple threshold

Gray-level histograms that can be partitioned by
a single threshold and multiple thresholds.
 Thresholding
The Role of Illumine

Computer Histogram of
generated reflectance
reflectance function
function
 Thresholding
The Role of Illumine

(a)Computer
generated
reflectance
function.
(b)Computer
r ( x, y ) (a) (b) i ( x, y ) generated
illumination
(c) (d) function
(c)Product of
(a) and (b)
(d)Histogram
of product
image

f ( x, y ) i ( x, y )r ( x, y )
 Thresholding: Algorithm

An algorithm used to obtain T automatically for global

thresholding

1. Select an initial estimate for T.

2. Segment the image using T. This well produce two groups of pixels: G1
consisting of all pixels with gray level values>T and G2 consisting of pixels with
values <T.
3. Compute the average gray level values 1 and 2 for the pixels in regions G1
and G2.
4. Compute a new threshold value:T = ½[1 + 2 ]
5. Repeat step 2 through 4 until the difference in T in successive iterations is
smaller than a predefined parameter To.
 Thresholding
Basic Global Thresholding

Original
Image

Image Histogram
Result of
global
thresholding
with T midway
between the
maximum and
minimum gray
levels.
 Thresholding
Basic Global Thresholding

Image Histogram
Original
Image

Result of
segmentation
with the
threshold
estimated by
iteration.
Adaptive Thresholding Algorithm

You can simulate the effect with the following steps:

1. Convolve the image with a suitable statistical operator, i.e. the
mean or median.
2. Subtract the original from the convolved image.
3. Threshold the difference image with C.
4. Invert the thresholded image.
 Thresholding
Basic Adaptive Thresholding

Original Result of
Image global
thresholding

Image
Subdivided
into Result of
individual adaptive
subimage thresholding
.
 Thresholding
Basic Adaptive Thresholding
How to solve this problem?

Properly and
Corresponding
improperly
histograms
segmented
subimage
 Thresholding
Basic Adaptive Thresholding
Answer: subdivision

Histogram Further
Result of
of small subdivision of
adaptively
subimage the
segmenting.
at top, left improperly
segmented
subimage
 Thresholding
Optimal Global and Adaptive
•
Thresholding
This method treats pixel values as probability density
functions.
• The goal of this method is to minimize the probability of
misclassifying pixels as either object or background.
• There are two kinds of error:
– mislabeling an object pixel as background, and
– mislabeling a background pixel as object.

Gray-level
probability
density
functions
of two
regions in
an image
 Thresholding
Use of Boundary Characteristics

Original
Image

Image
segmented by
local
thresholding
 Thresholding
Thresholds Based on Several
Variables

Original color Segmentation

Segmentation
image shown as of pixels with
of red
a monochrome colors close to
components
picture. facial tones.
Picking the Threshold is the Hard
Part
• Human operator decided the threshold
• Use mean gray level of the image
• A fixed proportion of pixels are detected
( set to 1) by the thresholding operation
•Analyzing the histogram of an image
 Region Based Segmentation
• Edges and thresholds sometimes do not
give good results for segmentation.
• Region-based segmentation is based on
the connectivity of similar pixels in a
region.
– Each region must be uniform.
– Connectivity of the pixels within the region is
very important.
• There are two main approaches to region-
based segmentation: region growing and
region splitting.
 Region Based Segmentation
Basic Formulation
• Let R represent the entire image region.
• Segmentation is a process that partitions R into
subregions,
n
R1,R2,…,Rn, such that
(a)  Ri R
i 1
(b) Ri is a connected region, i 1,2,..., n
(c) Ri  R j  for all i and j , i  j
(d) P ( Ri ) TRUE for i 1,2,..., n
(e) P ( Ri  R j ) FALSE for any adjacent regions Ri and R j
where P(Rk): a logical predicate defined over the points
in set Rk
For example: P(Rk)=TRUE if all pixels in Rk have the
same gray level.
 Region Based Segmentation
Region Growing

Image
showing Seed
defectiv Points
e welds

Result of Boundaries
region of
growing. segmented
defective
welds (in
black)
 Region Based Segmentation
Region Growing
• Fig. 10.41 shows the histogram of Fig. 10.40 (a). It is
difficult to segment the defects by thresholding
methods. (Applying region growing methods are better
in this case.)

Figure 10.40(a) Figure 10.41

 Region Based Segmentation
Region Splitting and Merging
• Region splitting is the opposite of region
growing.
– First there is a large region (possible the entire
image).
– Then a predicate (measurement) is used to
determine if the region is uniform.
– If not, then the method requires that the region be
split into two regions.
– Then each of these two regions is independently
tested by the predicate (measurement).
– This procedure continues until all resulting regions
are uniform.
 Region Based Segmentation
Region Splitting
• The main problem with region splitting is determining where to
split a region.
• One method to divide a region is to use a quadtree structure.
• Quadtree: a tree in which nodes have exactly four descendants.

Partitioned image Corresponding quardtree

 Region Based Segmentation
Region Splitting and Merging
• The split and merge procedure:
– Split into four disjoint quadrants any region Ri for which
P(Ri) = FALSE.
– Merge any adjacent regions Rj and Rk for which P(RjURk)
= TRUE. (the quadtree structure may not be preserved)
– Stop when no further merging or splitting is possible.

Original image Result of split Result of

and merge thresholding
procedure.
 Region Based Segmentation
SRG
• Region growing: Groups pixels or
sub-region into larger regions.
– step1:
• Start with a set of “seed” points and from
these grow regions by appending to each
seed those neighboring pixels that have
properties similar to the seed.
– step2:
• Region splitting and merging

56
 Region Based Segmentation
SRG
• Advantage:
– With good connectivity

• Disadvantage:
– Initial seed-points:
• different sets of initial seed-point cause
different segmented result
– Time-consuming problem

57
 Region Based Segmentation
USRG
• Unseeded region growing:
– no explicit seed selection is necessary,
the seeds can be generated by the
segmentation procedure automatically.
– It is similar to SRG except the choice of
seed point

58
 Region Based Segmentation
USRG
• Advantage:
– easy to use
– can readily incorporate high level
knowledge of the image composition
through region threshold

• Disadvantage:
– slow speed

59
 Region Based Segmentation
Fast Scanning
• Fast scanning
Algorithm:
– The fast scanning
algorithm somewhat
resembles unseeded
region growing
– the number of clusters
of both two algorithm
would not be decided
before image passing
through them.

60
Region Based Segmentation
Fast Scanning

61
Region Based Segmentation
Fast Scanning

• Last step:

– merge small region to

big region

62
 Region Based Segmentation
Fast Scanning
• Advantage:
– The speed is very fast
– The result of segmentation will be intact
with good connectivity

• Disadvantage:
– The matching of physical object is not
good
• It can be improved by morphology and
geometric mathematic
63
 Segmentation by Morphological
Watersheds
• The concept of watersheds is based on visualizing an
image in three dimensions: two spatial coordinates
versus gray levels.
• In such a topographic interpretation, we consider three
types of points:
– (a) points belonging to a regional minimum
– (b) points at which a drop of water would fall with
certainty to a single minimum
– (c) points at which water would be equally likely to
fall to more than one such minimum
• The principal objective of segmentation algorithms
based on these concepts is to find the watershed lines.
 Segmentation by Morphological
Watersheds
Example
Original image Topographic
view

Two stages of flooding.

 Segmentation by Morphological
Watersheds
Example
Result of Beginning of
further merging of water
flooding. from two
catchment basins
(a short dam was
built between
them)

Final watershed
Longer (segmentation)
dams. lines.
 Segmentation by Morphological
Watersheds
Example
Image gradient
Image of
blobs

Watershed lines
Watershed superimposed
lines on original
image
 The Use of Motion in Segmentation

• ADI: accumulative difference image

ADIs of a rectangular object moving in a
southeasterly direction

Absolute ADI Positive ADI Negative ADI

 The Use of Motion in Segmentation

• Building a static reference image. (a) and (b) Two

frames in a sequence. (c) Eastbound automobile
subtracted from (a) and background restored
from the corresponding area in (b)

(a) (b) (c)

 Document Segmentation

• Document images
consist of texts,
graphics, photos and
so on
• Document
segmentation is
useful for
compression, text
recognition
• Adobe and Xerox are
the major players
70
 Medical Image Segmentation

• Medical image
analysis can be used
as preliminary
screening
techniques to help
doctors
• Partial Differential
Equation (PDE) has
been used for
segmenting medical
images
active contour model (snake)
71
Range Image Segmentation

range intensity ground

truth

72
 Biometric Image Segmentation

• For fingerprint, face

and iris images, we
also need to
segment out the
region of interest
• Various cues can be
used such as ridge
pattern, skin color
and pupil shape
• Robust segmentation
could be difficult for
poor-quality images
73
 Reference

• [1] R.C. Gonzalez, R.E. Woods, Digital Image

Processing second edition, Prentice Hall, 2002
• [2] J.J. Ding, W.W. Hong, Improvement Techniques
for Fast Segmentation and Compression
• [3] J.J. Ding, Y.H. Wang, L.L. Hu, W.L. Chao, Y.W.
Shau, Muscle Injury Determination By Image
Segmentation
• [4] J.J. Ding, W.L. Chao, J.D. Huang, C.J. Kuo,
Asymmetric Fourier Descriptor Of Non-Closed
segments

74