EECS 274 Computer Vision

Segmentation by Clustering
Segmentation and Grouping
• Motivation: Obtain a compact • Grouping (or clustering)
representation from an – collect together tokens that
image/motion sequence/set of “belong together”
tokens • Fitting
• Should support application – associate a model with tokens
• Broad theory is absent at present – issues
• which model?
• which token goes to which
element?
• how many elements in the
model?
• Reading: FP Chapter 14, S
Chapter 5
How do they assemble?

Why do these tokens belong together?

• because they form surfaces?
• because they form the same texture?
Segmentation: examples
• Summarizing videos: shots
• Finding matching parts
• Finding people
• Finding buildings in satellite images
• Searching a collection of images
• Object detection
• Object recognition
Models problems
• Forming image segments: super pixels, image
regions
• Fitting lines to edge points
• Fitting a fundamental matrix to a set of feature
points:
– If the correspondence is right, there is a
fundamental matrix connecting the points
Segmentation as clustering
• Partitioning
– Decompose into regions that have coherent color and texture
– Decompose into extended blobs, consisting of regions that have
coherent color, texture, and motion
– Take a video sequence and decompose it into shots
• Grouping
– Collecting together tokens that, taken together, form a line
– Collecting together tokens that seem to share a fundamental matrix
• What is the appropriate representation?
General ideas
• Tokens
– whatever we need to group (pixels, points, surface
elements, etc., etc.)
• Top down segmentation
– tokens belong together because they lie on the same object
• Bottom up segmentation
– tokens belong together because they are locally coherent
• These two are not mutually exclusive
Grouping and gestalt
• Gestalt school of psychology
– Studying shape as a whole
instead of just a collection of
simple and curves
– Grouping means the tendency
of visual system to assemble
components together and
perceive them together

Muller-Lyer illusion
Some Gestalt properties

emergence reification
Figure and ground

• White circle/black rectangle: which is figure (foreground) and

which is ground (background)?
Basic ideas of grouping in humans
• Figure-ground discrimination • Gestalt properties
– grouping can be seen in terms of – elements in a collection of
allocating some elements to a elements can have properties that
figure, some to ground result from relationships (Muller-
– impoverished theory Lyer effect)
• gestalt qualitat
(form quality)
– A series of factors affect whether
elements should be grouped
together
• Gestalt factors
Gestalt movement
Examples of Gestalt factors
Example
Occlusion as a cue
Grouping phenomena
Illusory contours

Tokens appear to be grouped together as they provide

a cue to the presence of an occluding object whose
contour has no contrast
Background Subtraction
• If we know what the background • Approach:
looks like, it is easy to identify – use a moving average to estimate
“interesting bits” background image
• Applications – subtract from current frame
– person in an office – large absolute values are
– tracking cars on a road interesting pixels
– surveillance • trick: use morphological
operations to clean up pixels
Results using 80 x 60 frames
(a) Average of all 120 frames. (b) thresholding. (c) results with low threshold (with missing
pixels) . (d) background computed using the EM-based algorithm. (e) background
subtraction results (with missing pixels).
Results using 160 x 120 frames
(a) Average of all 120 frames. (b) thresholding. (c) results with low threshold (with missing
pixels) . (d) background computed using the EM-based algorithm. (e) background
subtraction results (with missing pixels).
Shot boundary detection
• Find the shots in a sequence of • Possible distances
video – frame differences
– shot boundaries usually result in – histogram differences
big differences between – block comparisons
succeeding frames
– edge differences
– allows for objects to move within
a given shot
• Applications:
– representation for movies, or
• Strategy:
video sequences
– compute interframe distances
• find shot boundaries
– declare a boundary where these
• obtain “most representative”
are big
frame
– supports search
Segmentation as clustering
• Cluster together (pixels, tokens, • Point-Cluster distance
etc.) that belong together – single-link clustering (distance
• Agglomerative clustering between two closest elements)
– attach closest to cluster it is – complete-link clustering
closest to (maximum distance between two
– repeat elements)
– group-average clustering
• Divisive clustering
• Dendrograms (tree diagram)
– split cluster along best boundary
– yield a picture of output as
– repeat
clustering process continues
K-Means
• Choose a fixed number of clusters • Algorithm
• Choose cluster centers and point- – fix cluster centers; allocate points
cluster allocations to minimize to closest cluster
error – fix allocation; compute best
• can’t do this by search, because cluster centers
there are too many possible • x could be any set of features for
allocations. which we can compute a distance
(careful about scaling)

 2 
 (clusters, data)     x j  i 
iclusters  jelements of i - th cluster 
Image Clusters on intensity Clusters on color

K-means clustering using intensity alone and color alone

How many clusters?
 Image Clusters on color

K-means using color alone, 11 segments

(6 kinds of vegetables)
K-means using
color alone,
11 segments
(not using texture
or position)
- segments may be
disconnected and
scattered
K-means using color and
position, 20 segments
- peppers are now separated
but, cabbage is still broken up
Graph-theoretic clustering
• Represent tokens using a weighted graph.
– affinity matrix
– similar have large weights
• Cut up this graph to get connected components
– with strong interior links
– by cutting edges with low weights
• A graph is a set of vertices V and edges E, G={V,E}
• A weighted graph is one in which each edge is associated
with a weight
• Every graph consists of a disjoint set of connected
components
Weighted graph

9 points/pixels 9 x 9 Weighted matrix

that looks like block
diagonal

cut ( A, B)  w
i A, jB
ij

A cut with 2 connected components

(Left) a set of points (Right) affinity matrix computed with a
decaying exponential distance (large values  bright pixels,
and small values  dark pixels)
Graph cut and normalized cut

cut ( A, B )  w
iA, jB
ij

cut ( A, B ) cut ( A, B)
Ncut ( A, B)  
assoc( A,V ) assoc( B, V )
assoc( A, V )  assoc( A, A)  cut ( A, B)
sum of all the weights associated with nodes in A
Affinity
Intensity
 1 2
aff ( x, y )  exp 2
I ( x )  I ( y ) 
 2 I 

Distance
 1 2
aff ( x, y )  exp 2
x y 
 2 d 
Texture

 1 2
aff ( x, y )  exp 2
c( x)  c( y ) 
 2 c 
Scale affects affinity

σ=0.1 σ=0.2 σ=1

Extracting a single good cluster
• Simplest idea: we want a vector a • This is an eigenvalue problem -
giving the association between choose the eigenvector of A with
each element and a cluster largest eigenvalue
• We want elements within this
Awn  wn
cluster to, on the whole, have
strong affinity with one another
• We could maximize
T  association of element i with cluster n ×
wn Awn affinity between i and j ×
association of element j and cluster n
• But need the constraint T
wn wn
• Solving constrained optimization 1
with Lagrange multiplier

T T
wn Awn   ( wn wn  1)
Example eigenvector

points

eigenvector

matrix
Clustering with eigenvectors

• If we reorder the nodes, the rows and columns of A are shuffled

• We expect to deal with a few clusters with large association
• We could shuffle the rows and columns of M to form a matrix that is roughly
block diagonal
• Shuffling M simply shuffle the elements of its eigenvectors so
• We can reason about the eigenvectors by working with a shuffled version of M
Eigenvectors and clusters
• Eigenvectors of block-diagonal matrices
consist of eigenvectors of the blocks padded
with zeros
• Each block has an eigenvector corresponding
to a rather large eigenvalue
• Expect that, if there are c significant clusters,
the eigenvectors corresponding to the c largest
eigenvalue
Large eigenvectors and clusters
Eigen gap and clusters
σd=0.1 σd=0.2 σd=1
More than two segments
• Two options
– Recursively split each side to get a tree, continuing
till the eigenvalue are too small
– Use the other eigenvectors
Normalized cuts
• Current criterion evaluates within • Maximize
cluster similarity, but not across
cluster difference  assoc ( A, A)   assoc ( B, B) 
    
• Can also use Cheeger constant  assoc ( A,V )   assoc ( B, V ) 
• Instead, we’d like to maximize
the within cluster similarity
compared to the across cluster
difference • i.e. construct A, B such that their
• Write graph as V, one cluster as within cluster similarity is high
A and the other as B compared to their association with
the rest of the graph
Normalized cuts
• Write a vector y whose elements • This is hard to do, because y’s
are 1 if item is in A, -b if it’s in B values are quantized
• Write the matrix of the graph as
W, and the matrix which has the
row sums of W on its diagonal as
D, 1 is the vector with all ones.
• Criterion becomes

 yT (D  W ) y   y T Ly 
min y  T
  min y  T 
 y Dy   y Dy 
• and we have a constraint
L  D W

y T D1  0
Normalized cuts and spectral graph theory

• Instead, solve the generalized eigenvalue problem

max y ( y T ( D  W ) y )  max y y T Ly
s.t. y T Dy  1
• which gives
( D  W ) y  Dy
Ly  Dy

• Now look for a quantization threshold that maximizes the criterion --- i.e
all components of y above that threshold go to one, all below go to -b
Using affinity based on texture and intensity.
Figure from “Image and video segmentation: the normalized cut framework”,
by Shi and Malik, copyright IEEE, 1998
Using spatio-temoral affinity metric.
Figure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE, 2000
Spectral clustering

Ng, Jordan and Weiss, NIPS 01

Comparisons

3 clusters 2 clusters 2 clusters

8 clusters 3 clusters 2 clusters

4 clusters 2 clusters 2 clusters 3 clusters 3 clusters 8 clusters

Ng, Jordan and Weiss, NIPS 01

Spectral graph theory

Dii   Wij
j

graph Laplacian : D  W
normalized graph Laplacian : L  D 1/ 2 ( D  W ) D 1/ 2  I  D 1/ 2WD 1/ 2
Ly  Dy
generalized eigenvalue problem

• See EECS 275 Lecture 23 for more detail

• See also Laplacian eigenmap, Laplace Beltrami operator,
Laplacian eigenmap, spectral clustering, graph cut,
minimal-cut maximum-flow, normalized cut, random walk,
heat kernel, diffusion map
More information
• G. Scott and H. Longuet-Higgins, Feature grouping by
relocalisation of eigenvectors of the proximity matrix, BMVC,
1990
• F. Chung, Spectral graph theory, AMS 1997
• Y. Weiss, Segmentation using eigenvectors: a unifying view,
ICCV 1999
• L. Saul et al., Spectral methods for dimensionality reduction,
in Semisupervised learning, 2006
• U von Luxburg, A tutorial on spectral clustering, Statistics and
Computing, 2007
Related topics
• Watershed
• Multiscale segmentation
• Mean-shift segmentation
• Parsing tree/stochastic grammar
• Graph cut
• Objet cut
• Grab cut
• Random walk
• Energy-based method