Artificial Intelligence II

Part 2: Lecture 3
Yalda Mohsenzadeh

Slides are adapted from Olga Vesker (UW), Steve Seitz (UW), David Jacobs
(UMD), D. Lowe (UBC), Hong Man
Computer Vision
Image Segmentation
Outline
• Perceptual Grouping in humans
• Gestalt perceptual grouping laws, describe grouping
cues of humans
• Image segmentation (“Pixel Grouping”)
• Clustering
• Simple agglomerative algorithm
• K-means
• Histogram based
• Thresholding
• Mode-finding
• Mean shift
From Images to Objects

• Humans do not perceive the world as a collection of

individual “pixels” but rather as a collection of
objects and surfaces
• For many applications, it is useful to segment or
group image pixels into blobs which are perceptually
meaningful
• Hopefully belong to the same “object” or surface
• How to do this without (necessarily) object
recognition?
• Subjective problem, but has been well-studied
• Gestalt Laws seek to formalize this
• Proximity, similarity, continuation, closure, common fate
Grouping

• Most human observers would report no

particular grouping
Gestalt Principles of Grouping: Common
Form (includes color and texture)
Gestalt Principles of Grouping: Proximity
Gestalt Principles of Grouping: Good Continuation
Gestalt Principles of Grouping: Figure/Ground
Gestalt Principles of Grouping: Symmetry
Gestalt Principles of Grouping: Symmetry
Gestalt Principles of Grouping: Closure
Gestalt Principles of Grouping: Closure
Gestalt Principles of Grouping: Closure
Higher level Knowledge
 Take Home Message

• We perceive the world in terms of objects, not pixels

• What forms an object is determined by regularities and
non-trivial inference
Human perceptual grouping
• Perceptual grouping has been significant
inspiration to computer vision

• Why?
• Perceptual grouping seems to rely partly on the nature
of objects in the world
• This is hard quantity, we hypothesize that human
vision encodes the necessary knowledge
Computer Vision: Image Segmentation

• In vision, we usually refer to perceptual organization problem as

image segmentation or clustering
• Image segmentation is the operation of partitioning an image into a
collection of
• Regions, which usually cover the whole image
• Linear structures, such as
• Line segments
• Curve segments
• Into 2D shapes, such as
• Circles
• Ellipses
• Ribbons (long, symmetric, regions)
• Clustering is a more general term than image segmentation
• Can cluster all sorts of data (usually represented as feature vectors), not just
image pixels
• Web pages, financial records, etc.
• Clustering is a large area of machine learning (not supervised, that is labels of
feature vectors are not known)
Example 1: Region Segmentation
Example 2: Lines and Circular Arcs Segmentation
Image Segmentation
• Image cues are used for grouping/segmentation:
• Pixel-based cues
• Color
• Motion
• Region-based cues
• Texture
• Region shape
• Contour-based cues
• curvature
Image Segmentation Approaches
• Approaches can be roughly divided in two
groups:
1. Parametric: We have a description of what we
want, with parameters:
• Examples: lines, circles, constant intensity regions,
constant intensity regions + Gaussian noise
2. Non-parametric: have constraints the group
should satisfy, or optimally criteria.
• Examples: SNAKEs. Find the closed curve that is
smoothest and that also best follows strong image
gradients.
Clustering Algorithms
• Agglomerative
• Start with each pixel in own cluster
• Iteratively merge clusters together according to some
predefined criterion
• Stop when reached some stopping condition
• Divisive
• Start with all pixels in one cluster
• Iteratively choose and split a cluster into two according to
some predefined criterion
• Stop when reached some stopping condition
• There are clustering methods which are both
agglomerative and divisive
Simplest Agglomerative Clustering based on
Color/Intensity
• Initialize: Each pixel is a cluster (region)
• Loop
• Find two adjacent regions with most similar color (or
intensity)
• Merge to form new region with:
• All pixels of these regions
• Average color (or intensity) of these regions
• Several possibilities for stopping condition
1. No regions similar (color or intensity differences between all
neighboring regions is larger than some threshold, etc.)
Example: Agglomerative Intensity Based
Clustering
Example: Agglomerative Intensity Based
Clustering
Example: Agglomerative Intensity Based
Clustering
Example: Agglomerative Intensity Based
Clustering
Example: Agglomerative Intensity Based
Clustering
Example: Agglomerative Intensity Based
Clustering
Example: Agglomerative Intensity
Based Clustering
Example: Agglomerative Intensity
Based Clustering
Agglomerative Clustering: Discussion

• Start with definition of good clusters

• Simple initialization
• Greedy: take steps that seem to most improve
clustering
• This is a very general, reasonable strategy
• Can be applied to almost any problem
• But, not guaranteed to produce good quality
answer
 Clustering for Image Segmentation
• General clustering problem setting:
• Have samples (or points, or feature vectors)
𝑥1 , … , 𝑥𝑛
• For segmentation 𝑥1 , … , 𝑥𝑛 correspond to n image
pixels
• Each 𝑥1 can be
• Intensity of pixel 𝑥𝑖 (for gray image segmentation)
• Color of pixel 𝑥𝑖 (for color image segmentation)
• For example
Example 1
Example 2
Example 3
Histogram-Based Segmentation
• Segmentation by Histogram Processing
• Given image with N colors, choose k
• Each of K colors defines a region
• not necessarily contiguous
• Performed by computing color histogram, looking for modes

• This is what happens when you downsample image color range,

for instance in Photoshop
Histogram-Based Segmentation
How do we select a Threshold?
• Automatic thresholding
• P-tile method
• Mode method
• Peakiness detection
• Mean-shift
P-Tile Method
• If the size and brightness range of the object is
approximately known, pick T s.t. the area under
the histogram corresponds to the size of the
object:
Mode Method
• Model each region as “constant” + noise
• Usually noise is modeled as 𝑁(0, 𝜎𝑗 )
Example: Image with 3 regions
Finding Modes in a Histogram
Peakiness detection algorithm
• Find the two highest Local Maxima at a minimum
distance apart: 𝑔𝑖 𝑎𝑛𝑑 𝑔𝑗
• Find lowest point between them: 𝑔𝑘
• Measure “peakiness”: min 𝐻 𝑔𝑖 , 𝐻 𝑔𝑗 /𝐻 𝑔𝑘
• Find (𝑔𝑖 , 𝑔𝑗 , 𝑔𝑘 ) with highest peakiness
Mean Shift (Comaniciu & Meer)

• Iterative Mode Search

1. Initialize random seed, and fixed window
2. Calculate center of gravity of the window (the “mean”)
3. Translate the search window to the mean
4. Repeat Step 2 until convergence
Mean Shift (Comaniciu & Meer)
Mean Shift (Comaniciu & Meer)
Mean Shift (Comaniciu & Meer)
Algorithm Mean Shift to find the histogram Peak

1. Choose a Window size (for example 5)

2. Choose the initial location of the search window
3. Compute the mean location in the search
window
4. Center the window at the location computed in 3
5. Repeat steps 3 and 4 until convergence
Algorithm: Mean Shift for Image Segmentation

• Find features (intensity, color, gradients, texture,

etc.)
• Initialize windows at individual feature points
(pixels)
• Perform mean shift at each window (pixel) until
convergence
• Merge windows (pixels) that end up near the
same mode (peak)
Algorithm: Mean Shift for Image Segmentation
Algorithm Mean Shift
Mean Shift Segmentation: Example
The Mean Shift segmentation is a local homogenization
technique that is very useful for damping shading or
tonality differences in localized objects.
Mean Shift Segmentation: Example
Mean Shift Segmentation: Example
Mean shift: Strengths and Weaknesses

Strengths Weaknesses
• Does not assume any prior • The window size is not
shape (e.g. elliptical) on the trivial
data structure • Inappropriate window size
• Can handle arbitrary can cause modes to be
feature spaces merged (giving too few
• Only one parameter to segments) or generate
choose additional shallow modes
(giving too many segments)
• h the window size
• There are adaptive window
size extensions