Image Processing

COMP4421
Hao CHEN, 陳浩
Dept. of CSE and CBE
HKUST
Image Classification
• Background
• Pattern Classification by Prototype Matching
• Neural Networks and Deep Learning
• Deep Convolutional Neural Networks

2
Image Classification
• Background
• Pattern Classification by Prototype Matching
• Neural Networks and Deep Learning
• Deep Convolutional Neural Networks

3
Background
• Pattern: a spatial arrangement of features.
• Pattern class: a set of patterns that share some common properties.
• Pattern recognition system: to assign a class label to each of its input
patterns, consisting of four main steps:
-Sensing
-Preprocessing
-Feature extraction
-Classification

4
Background
• Labeled (supervised learning), unlabeled patterns (unsupervised
learning).

• Data set generally is subdivided into three subsets: a training set, a

validation set, and a test set.

• Training: the process by which a training set is used to generate classifier

parameters.

5
 Background
• Pattern vectors are represented by lowercase letters, such as x, y, and z,
where n is the total number of such descriptors.

• Vectorize Why not use intensity directly?

• accuracy

Linear indexing

6
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Background
• Discriminant analysis

The length and width of the petal and sepals

7
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Background
• Based on feature descriptors of boundary shape
-Represent an object by its signature

𝜃: the increments of amplitude

8
 Background
• Boundary and regions.

9
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Background
• Texture

10
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Background
• Pattern vectors formed from corresponding pixels in those images.

Extract lower dim space

11
Image Classification
• Background
• Pattern Classification by Prototype Matching
• Neural Networks and Deep Learning
• Deep Convolutional Neural Networks

12
Pattern Classification by Prototype Matching
• Comparing an unknown pattern against a set of prototypes, and
assigning the unknown pattern to the class of the prototype that is the
most “similar” to the unknown.

• What distinguishes one matching method from another is the measure

used to determine similarity.

13
Pattern Classification by Prototype Matching
Minimum-distance classifier
• Minimum-distance classifier computes a distance-based measure
between an unknown pattern vector and each of the class prototypes.

• It then assigns the unknow pattern vector to the class of its closest
prototype.

14
Pattern Classification by Prototype Matching
Minimum-distance classifier
• Compute the Euclidean distances:

where x is an unknown pattern vector, mj is the mean vectors of the various

Center
pattern classes:

• The classifier then assigns an unknown pattern x to class ci if:

15
Pattern Classification by Prototype Matching
Minimum-distance classifier

• Select the smallest distance, which is equivalent to evaluating the

following functions and assigning an unknown pattern x to the class
whose prototype yielded the largest value of d.
Euclidean distance

"
[ ×m
;(
× -
m
,
JT ]
4 =

;
= x xT -
×
2 5
ms
+
m
5
m 25 분쯤 ,

rrin

↑
Eomstaapt I drop .

16
Pattern Classification by Prototype Matching
Minimum-distance classifier
• If:

then, x is assigned to class ci

• Decision boundary separating class ci from cj is given:

• The decision boundaries for a minimum-distance classifier:

17
 Pattern Classification by Prototype Matching
• Example: the means of the two classes are m1=(4.3, 1.3)T and m2=(1.5,
0.3)T, calculate the classification boundary.

18
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Pattern Classification by Prototype Matching
• Example: the means of the two classes are m1=(4.3, 1.3)T and m2=(1.5,
0.3)T, calculate the classification boundary.

• Firstly:

• Then, the equation of the boundary:

19
 Pattern Classification by Prototype Matching
• Example: the means of the two classes are m1=(4.3,1.3)T and
m2=(1.5,0.3)T

20
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Pattern Classification by Prototype Matching
Correlation for prototype matching
• Correlation of a kernel w with an image f(x, y) is given by:

• As you know, correlation has its highest value in the region where f
and w are equal or nearly equal.

• Matching by correlation coefficient:

21
Pattern Classification by Prototype Matching
Correlation for prototype matching
• In image correlation, w is often referred to as a template and
correlation is referred to as template matching.

• Maximum correlation: the maximum value of 𝛾 occurs when the

normalized w and the corresponding normalized region in f are
identical.

• Minimum correlation: the minimum occurs when the two normalized

functions exhibit least similarity.

22
 Pattern Classification by Prototype Matching
Correlation for prototype matching

• Values of the correlation coefficient 𝛾(x, y) are obtained by moving

the center of the template.

Template size: m×n

Center: (x, y)

23
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Pattern Classification by Prototype Matching
• Example: Matching by correlation.

(a) Satellite image (b) 31×31 template (c) Correlation coefficient (d) Best match

24
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Pattern Classification by Prototype Matching
Matching SIFT Features
• SIFT for prototype matching.
• Keypoints found by SIFT:

35 keypoints
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018. 2714 keypoints 25
 Pattern Classification by Prototype Matching
• Matches between keypoints found by SIFT:

41 matches

26
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Image Classification
• Background
• Pattern Classification by Prototype Matching
• Neural Networks and Deep Learning
• Deep Convolutional Neural Networks

27
 Neural Networks and Deep Learning
• Perceptron: learn a linear boundary between two linearly separable
pattern classes.

28
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Neural Networks and Deep Learning

• A schematic diagram of the perceptron

• Form a sum of products of an
input pattern using the weights
and bias.

• If output is +1, classified as c1

• If output is - 1, classified as c2

29
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Neural Networks and Deep Learning
The perceptron training algorithm
• 𝛼 > 0 denotes learning rate, let w(1) be a vector with arbitrary values, and
wn+1(1) be an arbitrary constant. For a pattern vector, x(k), at step k:

S1. If x(k) ∈ c1 and wT (k)x(k)+ wn+1(k) ≤0 let,

S2. If x(k) ∈ c2 and wT (k)x(k)+ wn+1(k) ≥ 0 let,

S3. Otherwise, let,

30
 Neural Networks and Deep Learning

• Example: Using the perceptron training algorithm to learn a decision boundary

• The pattern is x =[3 3 1]T∈ c1 and the other x =[1 1 1]T∈ c2,𝛼 = 1, w(1)=0=[0 0 0] T.

The inner product of w T(1) and x(1),

then,

31
Neural Networks and Deep Learning

• The pattern is x =[3 3 1]T, and the other x =[1 1 1]T, 𝛼 = 1, w(1)=0=[0 0 0] T.

For k = 2, x(2) = [1 1 1]T ∈ c2, w(2) = [3 3 1]T . Their inner product:

The result is positive when it should have been negative, so:

32
Neural Networks and Deep Learning

• The pattern is x =[3 3 1]T, and the other x =[1 1 1]T, 𝛼 = 1, w(1)=0=[0 0 0] T.

For k = 3, x(3) = [3 3 1]T ∈ c1, w(3) = [2 2 0]T . Their inner product is positive as it
should be, The weight vector is not changed:

For k = 4, x(4) = [1 1 1]T ∈ c1, w(4) = [2 2 0]T .Then, Their inner product is
positive and it should have been negative, then:

33
Neural Networks and Deep Learning

• The pattern is x =[3 3 1]T, and the other x =[1 1 1]T, 𝛼 = 1, w(1)=0=[0 0 0] T.

• Following this procedure, you can show that the algorithm converges to the
solution weight vector:

which gives the decision boundary:

34
 Neural Networks and Deep Learning

• Example: limitations of single linear decision boundary for the XOR

classification problem.

35
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Neural Networks and Deep Learning
Multi-layer feedforward neural networks

Non linear transform

36
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Neural Networks and Deep Learning
Forward pass through a feedforward neural network
• The outputs of the layer 1 are the components of input vector x:

• The computation performed by neuron i in layer l is given by:

Offset

• The output (activation value) of neuron i in layer l is given by:

Non linear
where h is an activation function.
• The value of network output (layer L) node i is:

37
 Neural Networks and Deep Learning
• Various activation functions

Sharper around 0

38
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Neural Networks and Deep Learning
Using backpropagation to train neural networks
• This training by backpropagation involves four basic steps:
S1. inputting the pattern vectors;

S2. a forward pass through the network to classify all the patterns of the
training set and determine the classification error;

S3. a backward (backpropagation) pass that feeds the output error back
through the network to compute the changes required to update the
parameters; W and b (weights and biases)

S4. updating the weights and biases in the network.

39
Neural Networks and Deep Learning
Using backpropagation to train neural networks
• The activation values of neuron j in the output layer is aj(L). We define the
error of that neuron as:

j = 1,2,…nL, and rj is the desired response of output neuron aj(L) for a given
pattern x.

• The output error:

40
Neural Networks and Deep Learning
Using backpropagation to train neural networks
• Backpropagation starts with the output:

Express this equation in terms of the output aj(L) using the chain rule:
ㆁ

• Use as our error measure and sigmoid as activation function,

then,

41
Neural Networks and Deep Learning
Using backpropagation to train neural networks
• So,

loprev layer

• We use the chain rule again,

42
Neural Networks and Deep Learning
Using backpropagation to train neural networks
• The last step is to update the network parameters using gradient descent:

and,

for l=L-1, L-2,…2, where the a’s are computed in the forward pass, and the
𝛿’s are computed during backpropagation.
43
 Neural Networks and Deep Learning
Choice of hyperparameter 𝛼

• A reasonable approach is to start with a small value (e.g., 0.01), then experiment on
the training set to determine a suitable value in a given application.
44
https://www.jeremyjordan.me/nn-learning-rate/
Neural Networks and Deep Learning

• Example: Using a fully connected neural network to solve the XOR problem

(a) XOR Problem (b) 2D classification result (a) Decision surface

45
Image Classification
• Background
• Pattern Classification by Prototype Matching
• Neural Networks and Deep Learning
• Deep Convolutional Neural Networks

46
Deep Convolutional Neural Networks
• Neural networks are capable of learning pattern features directly from
training data.

• Convolutional neural networks (CNN) accept images as inputs directly

and are ideally suited for image-based recognition.

• Difference between fully connected nets and CNN:

-CNN inputs are 2-D arrays (images)
-Convolutional and pooling layers

47
Deep Convolutional Neural Networks
LeNet architecture

48
Deep Convolutional Neural Networks
Basic concepts
• Receptive fields: a neighbourhood at one location in the input image.
• Convolution kernel (weight sharing): the set of weights, arranged in the
shape of the receptive field.
• Stride: the number of spatial increments by which a receptive field is
moved.
• Feature map: a 2-D set of values that we store in next layer as a 2-D
array.

49
Deep Convolutional Neural Networks

• Subsampling (or pooling): a reduction in spatial resolution for achieving

translational invariance.
• Pooling is done by subdividing a feature map into a set of small (typically
2 × 2) regions, called pooling neighborhoods, and replacing all elements
in such a neighborhood by a single value.
• Common pooling methods:
Average pooling, max-pooling, and L2 pooling.

50
 Deep Convolutional Neural Networks
• Example: Graphical illustration of the functions performed by the
components of a CNN.
If you want to maintain the size: padding (otherwise shrink)

51
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Deep Convolutional Neural Networks
• Example: Graphical illustration of the functions performed by the
components of a CNN.

52
Deep Convolutional Neural Networks
• Example: Graphical illustration of the functions performed by the
components of a CNN.

53
 Deep Convolutional Neural Networks
• Example: low-level features between natural images and medical images

Ultrasound Natural image

image
Chen, et al.. Standard plane localization in fetal ultrasound via domain transferred deep neural networks. 2015 IEEE JBHI.
54
Krizhevskyet al. Imagenet classification with deep convolutional neural networks. 2021 NIPS.
 Deep Convolutional Neural Networks
• Example: low-level features between natural images and medical images
• The learnt dictionary bases (low-level features) from (a) natural and (b)
ultrasound images.

Chen, et al.. Standard plane localization in fetal ultrasound via domain transferred deep neural networks. 2015 IEEE JBHI.
55
Krizhevskyet al. Imagenet classification with deep convolutional neural networks. 2021 NIPS.
 Deep Convolutional Neural Networks
• Example: Graphical illustration of the functions performed by the
components of a CNN.

56
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Deep Convolutional Neural Networks
• Example: Teaching a CNN to recognize some simple images.

57
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Deep Convolutional Neural Networks
• Example: Teaching a CNN to recognize some simple images.

3 classes

58
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Deep Convolutional Neural Networks
• Example: Teaching a CNN to recognize some simple images.
• A plot of the Mean Square Error (MSE) as a function of epoch.

59
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
 Deep Convolutional Neural Networks
• Example: Using a large training set to teach a CNN to recognize
handwritten numerals, i.e., MNIST Database.

60
Digital Image Processing, by Gonzalez and Woods, Pearson, 2018.
Deep Convolutional Neural Networks
• Example: Using a large training set to teach a CNN to recognize
handwritten numerals.

61
Deep Convolutional Neural Networks
• Example: Using a large training set to teach a CNN to recognize
handwritten numerals.

62
Deep Convolutional Neural Networks
• Example: Using a large training set to teach a CNN to recognize
handwritten numerals.

63
Deep Convolutional Neural Networks
• Example: Using a large training set to teach a CNN to recognize
handwritten numerals.
• Recognition accuracy of training set by image class.

64
Deep Convolutional Neural Networks
• Example: Using a large image database to teach a CNN to recognize
natural images. CIFAR-10 database.

65
Deep Convolutional Neural Networks
• Example: Using a large image database to teach a CNN to recognize
natural images.

Usually smaller than training acc

Training MSE Training accuracy Testing accuracy

66
 Deep Convolutional Neural Networks
• Example: Use CAM (Class Activation Mapping) to visualize discriminative
regions
• Task: Disease classification from Chest X-ray

Zhou et al. Learning deep features for discriminative localization. 2016 CVPR.
67
Luo et al. OXnet: Omni-supervised Thoracic Disease Detection from Chest X-rays. MICCAI 2021.
 Deep Convolutional Neural Networks

X-ray images

CAM heatmaps

Zhou et al. Learning deep features for discriminative localization. 2016 CVPR.
68
Luo et al. OXnet: Omni-supervised Thoracic Disease Detection from Chest X-rays. MICCAI 2021.
Deep Convolutional Neural Networks
Implementation Details (demon is in the details)
• How many layers to specify for the network.
Theoretically, under mild conditions, arbitrarily complex decision functions can
be approximated by a continuous feedforward neural network with a single
hidden layer.

• Vanishing gradients (deep neural networks)

The more layers a network has, the higher the probability that backpropagation
will run into problems.

• Specify the number of neurons per layer

No theoretical ‘best’ answer

69
 Deep Convolutional Neural Networks
• Pretraining: starting point for further fine-tuning (i.e., knowledge transfer).

• Over-fitting
Data augmentation using transformations such as geometric distortions and intensity
variations.
Dropout, a technique that randomly drops nodes with their connections from a neural
network during training.

• Training efficiency
Data shuffling,
Stochastic gradient descent,
Adaptative learning rate, etc.

70
Srivastava et al. Dropout: a simple way to prevent neural networks from overfitting. 2014 JMLR
Summary
• Background
• Pattern Classification by Prototype Matching
• Neural Networks and Deep Learning
• Deep Convolutional Neural Networks

71