Image classification

16-385 Computer Vision

http://www.cs.cmu.edu/~16385/Spring 2020, Lecture 18
Course announcements

• Programming assignment 4 is due tonight at 23:59.

- Please make sure to download the updated version of
PA4 (last updated Monday, 10 am ET).
- Due Wednesday March 25th.
- Any questions about the homework?

• Programming assignment 5 will be posted tonight and will be due

April 8th.

• Take-home quiz 7 posted and is due Sunday March 29th.

 Overview of today’s
lecture
• Introduction to learning-based vision.

• Image classification.

• Bag-of-words.

• K-means clustering.

• Classification.

• K nearest neighbors.

• Naïve Bayes.

• Support vector machine.

 Slide credits
Most of these slides were adapted from:

• Kris Kitani (16-385, Spring 2017).

• Noah Snavely (Cornell University).

• Fei-Fei Li (Stanford University).

 Course overview
Lectures 1 – 7
1. Image processing. See also 18-793: Image and Video
Processing

Lectures 7 – 13
2. Geometry-based vision. See also 16-822: Geometry-based Methods in
Vision

Lectures 14 – 17
See also 16-823: Physics-based Methods in
3. Physics-based vision.
Vision
See also 15-463: Computational Photography

4. Learning-based vision.
We are starting this part
now

5. Dealing with motion.

 What do we mean by
learning-based vision or
‘semantic vision’?
 Is this a street light?
(Recognition / classification)
Where are the people?
(Detection)
Is that Potala palace?
(Identification)
Sky
What’s in the scene?
(semantic segmentation)

Mountain

Trees

Building

Vendors
People

Ground
Object categorization

mountain

tree
building
banner

street lamp

vendor
people
What type of scene is it?
(Scene categorization)

Outdoor
Marketplace
City
Activity / Event Recognition
what are these
people doing?
 Object recognition
Is it really so hard?
Find the chair in this image Output of normalized correlation

This is a chair
 Object recognition
Is it really so hard?
Find the chair in this image

Pretty much garbage

Simple template matching is not going to make it

A “popular method is that of template matching, by point to point correlation of a model

pattern with the image pattern. These techniques are inadequate for three-dimensional scene
analysis for many reasons, such as occlusion, changes in viewing angle, and articulation of
parts.” Nivatia & Binford, 1977.
 And it can get a lot harder

Brady, M. J., & Kersten, D. (2003). Bootstrapped learning of novel objects. J Vis, 3(6), 413-422
 Why is this hard?

Variability: Camera position

Illumination
Shape parameters
 Challenge: variable viewpoint

Michelangelo 1475-1564
Challenge: variable illumination

image credit: J. Koenderink

Challenge: scale
Challenge: deformation
Deformation
Challenge:
Occlusion

Magritte, 1957
Occlusion
Challenge: background clutter

Kilmeny Niland. 1995

Challenge: Background clutter
Challenge: intra-class variations

Svetlana Lazebnik
Image Classification
Image Classification: Problem
 Data-driven approach
• Collect a database of images with labels
• Use ML to train an image classifier
• Evaluate the classifier on test images
Bag of words
What object do these parts belong to?
Some local feature
are very informative

An object as

a collection of local features

(bag-of-features)

• deals well with occlusion

• scale invariant
• rotation invariant
 (not so) crazy
assumption

spatial information of local features

can be ignored for object recognition (i.e., verification)
 CalTech6 dataset

Works pretty well for image-level classification

Csurka et al. (2004), Willamowski et al. (2005), Grauman & Darrell (2005), Sivic et al. (2003, 2005)
 Bag-of-features

represent a data item (document, texture, image)

as a histogram over features

an old idea
(e.g., texture recognition and information retrieval)
Texture recognition
histogram

Universal texton dictionary

 Vector Space Model
G. Salton. ‘Mathematics and Information Retrieval’ Journal of Documentation,1979

1 6 2 1 0 0 0 1
senso
Tartan robot CHIMP CMU bio soft ankle
r

0 4 0 1 4 5 3 2
senso
Tartan robot CHIMP CMU bio soft ankle
r

http://www.fodey.com/generators/newspaper/snippet.asp
A document (datapoint) is a vector of counts over each word (feature)

counts the number of occurrences just a histogram over words

What is the similarity between two documents?

A document (datapoint) is a vector of counts over each word (feature)

counts the number of occurrences just a histogram over words

What is the similarity between two documents?

Use any distance you want but the cosine distance is

fast.
but not all words are created equal
 TF-IDF
Term Frequency Inverse Document Frequency

weigh each word by a heuristic

inverse document
term frequency
frequency

(down-weights common terms)

Standard BOW
pipeline
(for image classification)
 Dictionary Learning:
Learn Visual Words using clustering

Encode:
build Bags-of-Words (BOW) vectors
for each image

Classify:
Train and test data using BOWs
 Dictionary Learning:
Learn Visual Words using clustering

1. extract features (e.g., SIFT) from images

 Dictionary Learning:
Learn Visual Words using clustering

2. Learn visual dictionary (e.g., K-means clustering)

What kinds of features can we extract?
• Regular grid
• Vogel & Schiele, 2003
• Fei-Fei & Perona, 2005
• Interest point detector
• Csurka et al. 2004
• Fei-Fei & Perona, 2005
• Sivic et al. 2005
• Other methods
• Random sampling (Vidal-Naquet &
Ullman, 2002)
• Segmentation-based patches (Barnard
et al. 2003)
Compute SIFT
descriptor Normalize patch
[Lowe’99]

Detect patches
[Mikojaczyk and Schmid ’02]
[Mata, Chum, Urban & Pajdla, ’02]
[Sivic & Zisserman, ’03]
…
How do we learn the dictionary?
…
…

Clustering
 Visual vocabulary

Clustering
K-means clustering
1. Select initial
centroids at random
1. Select initial 2. Assign each object to
centroids at random the cluster with the
nearest centroid.
1. Select initial 2. Assign each object to
centroids at random the cluster with the
nearest centroid.

3. Compute each centroid as the

mean of the objects assigned to
it (go to 2)
1. Select initial 2. Assign each object to
centroids at random the cluster with the
nearest centroid.

3. Compute each centroid as the 2. Assign each object to

mean of the objects assigned to the cluster with the
it (go to 2) nearest centroid.
1. Select initial 2. Assign each object to
centroids at random the cluster with the
nearest centroid.

3. Compute each centroid as the 2. Assign each object to

mean of the objects assigned to the cluster with the
it (go to 2) nearest centroid.

Repeat previous 2 steps until no change

 K-means Clustering
Given k:

1.Select initial centroids at random.

2.Assign each object to the cluster with the nearest

centroid.

3.Compute each centroid as the mean of the objects

assigned to it.

4.Repeat previous 2 steps until no change.

From what data should I learn the dictionary?

• Codebook can be learned on separate

training set

• Provided the training set is sufficiently

representative, the codebook will be
“universal”
From what data should I learn the dictionary?

• Dictionary can be learned on separate

training set

• Provided the training set is sufficiently

representative, the dictionary will be
“universal”
Example visual dictionary
Example dictionary

…
Appearance codebook
Source: B. Leibe
Another dictionary

…
…
…
…

…
Appearance codebook

Source: B. Leibe
 Dictionary Learning:
Learn Visual Words using clustering

Encode:
build Bags-of-Words (BOW) vectors
for each image

Classify:
Train and test data using BOWs
 1. Quantization: image features
gets associated to a visual word
(nearest cluster center)

Encode:
build Bags-of-Words (BOW) vectors
for each image
 Encode:
build Bags-of-Words (BOW) vectors
for each image 2. Histogram: count
the number of visual
word occurrences
frequency

…..
codewords
 Dictionary Learning:
Learn Visual Words using clustering

Encode:
build Bags-of-Words (BOW) vectors
for each image

Classify:
Train and test data using BOWs
K nearest neighbors

Support Vector Machine

Naïve Bayes
K nearest neighbors
Distribution of data from two classes
Distribution of data from two classes

Which class does q belong too?

Distribution of data from two classes

Look at the neighbors

K-Nearest Neighbor (KNN)
Classifier
Non-parametric pattern
classification approach
Consider a two class problem
where each sample consists of
two measurements (x,y).

For a given query point q, k=1

assign the class of the
nearest neighbor

Compute the k nearest k=3

neighbors and assign the
class by majority vote.
 Nearest Neighbor is
competitive

Test Error Rate (%)

Linear classifier (1-layer NN) 12.0
MNIST Digit Recognition
K-nearest-neighbors, Euclidean 5.0
– Handwritten digits K-nearest-neighbors, Euclidean, deskewed 2.4
– 28x28 pixel images: d = K-NN, Tangent Distance, 16x16 1.1
784 K-NN, shape context matching 0.67
– 60,000 training samples 1000 RBF + linear classifier 3.6
SVM deg 4 polynomial 1.1
– 10,000 test samples
2-layer NN, 300 hidden units 4.7
2-layer NN, 300 HU, [deskewing] 1.6
LeNet-5, [distortions] 0.8
Yann LeCunn Boosted LeNet-4, [distortions] 0.7
What is the best distance metric between data points?

• Typically Euclidean distance

• Locality sensitive distance metrics

• Important to normalize.
Dimensions have different scales

How many K?

• Typically k=1 is good

• Cross-validation (try different k!)

Distance metrics
Euclidean

Cosine

Chi-squared
 Choice of distance metric
• Hyperparameter
Visualization: L2 distance
CIFAR-10 and NN results
 k-nearest neighbor
• Find the k closest points from training data
• Labels of the k points “vote” to classify
 Hyperparameters
• What is the best distance to use?
• What is the best value of k to use?

• i.e., how do we set the hyperparameters?

• Very problem-dependent
• Must try them all and see what works best
Validation
Cross-validation
 How to pick hyperparameters?
• Methodology
– Train and test
– Train, validate, test

• Train for original model

• Validate to find hyperparameters
• Test to understand generalizability
Pros

• simple yet effective

Cons

• search is expensive (can be sped-up)

• storage requirements

• difficulties with high-dimensional data

 kNN -- Complexity and Storage
• N training images, M test images

• Training: O(1)
• Testing: O(MN)

• Hmm…
– Normally need the opposite
– Slow training (ok), fast testing (necessary)
Naïve Bayes
Distribution of data from two classes

Which class does q belong too?

Distribution of data from two classes
• Learn parametric model for each
class
• Compute probability of query
 This is called the posterior.
the probability of a class z given the observed
features X

For classification, z is a
X is a set of observed
discrete random
features
variable (e.g., features from a single image)
(e.g., car, person, building)

(it’s a function that returns a single probability value)

 This is called the posterior:
the probability of a class z given the observed
features X

For classification, z is a
Each x is an observed
discrete random
feature
variable (e.g., visual words)
(e.g., car, person, building)

(it’s a function that returns a single probability value)

Recall:
The posterior can be decomposed
according to Bayes’ Rule
likelihood prior

posterior

In our context…
The naive Bayes’ classifier is solving this optimization

MAP (maximum a posteriori) estimate

Bayes’ Rule

Remove constants

To optimize this…we need to compute this

Compute the likelihood…

 A naive Bayes’ classifier assumes all features are
conditionally independent

Recall:
To compute the MAP estimate

Given (1) a set of known parameters (2) observations

Compute which z has the largest probability

 count 1 6 2 1 0 0 0 1
word Tartan robot CHIMP CMU bio soft ankle sensor
p(x|z) 0.09 0.55 0.18 0.09 0.0 0.0 0.0 0.09

Numbers get really small so use log probabilities

ypically add pseudo-counts (0.001)

this is an example for computing the likelihood, need to multiply times prior to get posterior
 count 1 6 2 1 0 0 0 1
word Tartan robot CHIMP CMU bio soft ankle sensor
p(x|z) 0.09 0.55 0.18 0.09 0.0 0.0 0.0 0.09

log p(X|z=grand challenge) = - 14.58

log p(X|z=bio inspired) = - 37.48

count 0 4 0 1 4 5 3 2
word Tartan robot CHIMP CMU bio soft ankle sensor
http://www.fodey.com/generators/newspaper/snippet.asp p(x|z) 0.0 0.21 0.0 0.05 0.21 0.26 0.16 0.11

log p(X|z=grand challenge) = - 94.06

log p(X|z=bio inspired) = - 32.41

ypically add pseudo-counts (0.001)

this is an example for computing the likelihood, need to multiply times prior to get posterior
Support Vector
Machine
Image Classification
Score function
Linear Classifier

data (histogram)
Convert image to histogram representation
Distribution of data from two classes

Which class does q belong too?

Distribution of data from two classes

Learn the decision boundary

First we need to understand hyperplanes…
Hyperplanes (lines) in 2D

a line can be written

as dot product plus a
bias

another version, add a weight

1 and push the bias inside
Hyperplanes (lines) in 2D
(offset/bias outside) (offset/bias inside)
Hyperplanes (lines) in 2D
(offset/bias outside) (offset/bias inside)

Important property:
Free to choose any normalization of w

The line

and the line

define the same line

 What is the distance
to origin?

(hint: use normal form)

distance to origin

scale by

you get the normal form

 What is the distance
between two parallel lines?
(hint: use distance to origin)
 distance
between two
parallel lines

Difference of distance to origin

Now we can go to 3D …

Hyperplanes (planes) in 3D

what are the

dimensions of this
vector?

What happens if you change b?

Hyperplanes (planes) in 3D
 Hyperplanes (planes) in 3D

What’s the distance

between these
parallel planes?
Hyperplanes (planes) in 3D
What’s the best w?
What’s the best w?
What’s the best w?
What’s the best w?
What’s the best w?
 What’s the best w?

Intuitively, the line that is the

farthest from all interior points
 What’s the best w?

Maximum Margin solution:

most stable to perturbations of data
 What’s the best w?

support vectors

Want a hyperplane that is far away from ‘inner points’

Find hyperplane w such that …

margin

the gap between parallel hyperplanes is maximized

Can be formulated as a maximization problem

What does this constraint mean?

label of the data point

Why is it +1 and -1?

Can be formulated as a maximization problem

Equivalently, Where did the 2 go?

What happened to the labels?

 ‘Primal formulation’ of a linear SVM

Objective Function

Constraints

This is a convex quadratic programming (QP) problem

(a unique solution exists)
‘soft’ margin
What’s the best w?
What’s the best w?

Very narrow margin

Separating cats and dogs

Very narrow margin

 What’s the best w?

Very narrow margin

Intuitively, we should allow for some

misclassification if we can get more robust
classification
 What’s the best w?

Trade-off between the MARGIN and the MISTAKES

(might be a better solution)
 Adding slack variables

misclassifie
d point
 ‘soft’ margin
objective subject to

for
 ‘soft’ margin
objective subject to

for

The slack variable allows for mistakes,

as long as the inverse margin is minimized.
 ‘soft’ margin
objective subject to

for

• Every constraint can be satisfied if slack is large

• C is a regularization parameter
• Small C: ignore constraints (larger margin)
• Big C: constraints (small margin)
• Still QP problem (unique solution)
 References

Basic reading:
• Szeliski, Chapter 14.