Computer Vision: From

Theory to Applications
 Course Instructor:

Dr. M. Ehatisham-ul-Haq
Email: [email protected])

Instructor
• Research Interests:
– Signal, Image, and Video Processing
– Computer Vision and AI
– Machine Learning and Deep Learning
– Pervasive Computing and AI
Textbook:
• Rick Szeliski, Computer Vision:
Algorithms and Applications
online at http://szeliski.org/Book/
 Segmentation & Clustering

Disclaimer: Many slides have been borrowed from Devi Parikh and Kristen Grauman, who may have borrowed some of them from
others. Any time a slide did not already have a credit on it, I have credited it to Kristen. So there is a chance some of these credits
are inaccurate.
5
Grouping in vision
• Goals:
– Gather features that belong together
– Obtain an intermediate representation that compactly
describes key image or video parts

6
Examples of grouping in vision

[http://poseidon.csd.auth.gr/LAB_RESEARCH/Latest/imgs/Sp
eakDepVidIndex_img2.jpg]

[Figure by J. Shi]
Group video frames into shots
Determine image regions Fg / Bg

[Figure by Wang & Suter]

Figure-ground

[Figure by Grauman & Darrell]

Object-level grouping
7
Grouping in vision
• Goals:
– Gather features that belong together
– Obtain an intermediate representation that compactly
describes key image (video) parts

• Top down vs. bottom up segmentation

– Top down: pixels belong together because they are from the
same object
– Bottom up: pixels belong together because they look similar

• Hard to measure success

– What is interesting depends on the application.

8
The goals of segmentation
• Separate image into coherent “objects”
image human segmentation

9
Source: Lana Lazebnik
The goals of segmentation
• Separate image into coherent “objects”

• Group together similar-looking pixels for

efficiency of further processing

“superpixels”

X. Ren and J. Malik. Learning a classification model for segmentation. ICCV 2003.

10
Source: Lana Lazebnik
Image segmentation: toy example
white
pixels
3

pixel count
black pixels
gray

1 2 pixels

input image
intensity

• These intensities define the three groups.

• We could label every pixel in the image according to
which of these primary intensities it is.
• i.e., segment the image based on the intensity feature.
• What if the image isn’t quite so simple?
11
 pixel count
input image
intensity

pixel count

input image
intensity
12
 pixel count
input image
intensity

• Now how to determine the three main intensities that

define our groups?
• We need to cluster.

13
 0 190 255
intensity

3
1 2

• Goal: choose three “centers” as the representative

intensities, and label every pixel according to which of
these centers it is nearest to.
• Best cluster centers are those that minimize SSD
between all points and their nearest cluster center ci:

14
Segmentation by Thresholding
• Suppose that the gray-level histogram corresponds
to an image f(x,y) composed of dark objects on the
light background, in such a way that object and
background pixels have gray levels grouped into two
dominant modes. One obvious way to extract the
objects from the background is to select a threshold
‘T’ that separates these modes.
• Then any point (x,y) for which f(x,y) < T is called an
object point, otherwise, the point is called a
background point.
Gray Scale Image Example

Image of a Finger Print with light background

Histogram
Segmented Image

Image after Segmentation

Thresholding
• Basic Global Thresholding:
1)Select an initial estimate for T
2)Segment the image using T. This will produce two groups of
pixels. G1 consisting of all pixels with gray level values >T and
G2 consisting of pixels with values <=T.
3)Compute the average gray level values mean1 and mean2 for
the pixels in regions G1 and G2.
4)Compute a new threshold value
T=(1/2)(mean1 +mean2)
5)Repeat steps 2 through 4 until difference in T in successive
iterations is smaller than a predefined parameter T0.
Gray Scale Image - bimodal

Image of rice with black background

Segmented Image

Image histogram of rice Image after segmentation

Clustering
• With this objective, it is a “chicken and egg” problem:
– If we knew the cluster centers, we could allocate points to
groups by assigning each to its closest center.

– If we knew the group memberships, we could get the

centers by computing the mean per group.

22
K-means clustering
• Basic idea: randomly initialize the k cluster centers, and
iterate between the two steps we just saw.
1. Randomly initialize the cluster centers, c1, ..., cK
2. Given cluster centers, determine points in each cluster
• For each point p, find the closest ci. Put p into cluster i
3. Given points in each cluster, solve for ci
• Set ci to be the mean of points in cluster i
4. If ci have changed, repeat Step 2

Properties
• Will always converge to some solution
• Can be a “local minimum”
• does not always find the global minimum of objective function:

23
Source: Steve Seitz
K-means: pros and cons

Pros
• Simple, fast to compute
• Converges to local minimum of
within-cluster squared error

Cons/issues
• Setting k?
• Sensitive to initial centers
• Sensitive to outliers
• Detects spherical clusters
• Assuming means can be computed

24
 Segmentation as clustering
Depending on what we choose as the feature space, we
can group pixels in different ways.

Grouping pixels based

on intensity similarity

Feature space: intensity value (1-d)

25
Slide credit: Kristen Grauman
Segmentation as clustering
Depending on what we choose as the feature space, we can
group pixels in different ways.
R=255
Grouping pixels based G=200
on color similarity B=250

B
R=245
G G=220
B=248

R=15 R=3
G=189 G=12
R B=2 B=2
Feature space: color value (3-d) 26
Segmentation as clustering
Depending on what we choose as the feature space, we can
group pixels in different ways.

Grouping pixels based on

intensity+position similarity

Intensity

X Both regions are black, but if we

also include position (x,y), then
we could group the two into
distinct segments; way to encode
both similarity & proximity. 27
Region-Based Segmentation
Outline

• Region-based Segmentation
• Morphological Filters
Topic: Region-based Segmentation

• Region-based Segmentation
• Morphological Filters
Why Region-Based Segmentation?

• Segmentation
– Edge detection and
Thresholding not always
effective.
• Homogenous regions
– Region-based
segmentation.
– Effective in noisy images.
Definitions

• Based on sets.
• Each image R is a set of R7
R6
regions Ri.
– Every pixel belongs to R1
R5
one region.
– One pixel can only R2 R3
belong to a single region. R4
 Grouping in Vision
Segmentation as Clustering
Mode finding & Mean-Shift
R7

Graph-Based Algorithms R6

Segments as Primitives
CNN-Based Approaches
R1
R5

R2
R3

R4
VC 14/15 - TP9 -
Region-Based
Segmentation
Basic Formulation

Let R represent the entire image region. a) Every pixel must be in a

Segmentation partitions R into n
subregions, R1, R2, ..., Rn, such that: region
a) b) Points in a region must be
connected.
b)
c) Regions must be disjoint.
c) d) All pixels in a region satisfy
specific properties.
d) e) Different regions have
different properties.
e)
How do we form regions?
• Region Growing
• Region Merging
• Region Splitting
• Split and Merge

What a computer sees

Region growing

• Groups pixels into • Iterative process

larger regions. – How to start?
• Starts with a seed – How to iterate?
region. – When to stop?
• Grows region by
merging neighboring Finish
pixels.

Initial Stop
Iterations
Regions Condition
 Grouping in Vision
Segmentation as Clustering
Mode finding & Mean-Shift
Graph-Based Algorithms
Segments as Primitives
CNN-Based Approaches

VC 14/15 - TP9 -
Region-Based
Segmentation
Region merging

• Algorithm
– Divide image into an initial set of regions.
• One region per pixel is the smallest possible region.
– Define a similarity criteria for merging regions.
– Merge similar regions.
– Repeat previous step until no more merge
operations are possible.
Similarity Criteria
• Homogeneity of regions is used as the main
segmentation criterion in region growing.
– gray level
– color, texture
– shape
– model Choice of criteria
– etc. affects segmentation
results dramatically!
Gray-Level Criteria
• Comparing to Original Seed Pixel
– Very sensitive to choice of seed point.
• Comparing to Neighbor in Region
– Allows gradual changes in the region.
– Can cause significant drift.
• Comparing to Region Statistics
– Acts as a drift dampener.
• Other possibilities!
Region splitting

• Algorithm
R1 R2
– One initial set that
R1
includes the whole
R3 R4
image.
– Similarity criteria.
– Iteratively split regions
into sub-regions.
R1 R2
– Stop when no more R1 R2 R3
R R
splittings are possible.
R4 R5 R6 R7 R3 R
4 R
5

6 7
 Grouping in Vision
Segmentation as Clustering
Mode finding & Mean-Shift
Graph-Based Algorithms
Segments as Primitives
CNN-Based Approaches

VC 14/15 - TP9 -
Region-Based
Segmentation [Machine Vision; David Vernon]
Split and Merge

• Combination of both
algorithms.
• Can handle a larger
variety of shapes.
– Simply apply previous
algorithms
consecutively.
Topic: Morphological Filters

• Region-based Segmentation
• Morphological Filters
Mathematical Morphology

• Provides a • What is this used for?

mathematical – Binary images.
description of – Can be used for
geometric structures. post-processing
segmentation results!
• Based on sets.
– Groups of pixels which • Core techniques
define an image region. – Erosion, Dilation.
– Open, Close.
Dilation

• Example using a 3x3 morphological kernel

Erosion

• Example using a 3x3 morphological kernel

Opening
• Example using a 3x3 morphological kernel
Closing
• Example using a 3x3 morphological kernel
Core morphological operators

Dilation Erosion

Closing Opening
Example: Opening

Opening

Tresholding

VC 14/15 - TP9 -
Region-Based
Segmentation
Example: ?

?
Example: Closing

Closing
Questions?