Feature extraction and

end analysis
Jayanta Mukhopadhyay
Dept. of CSE,
IIT Kharagpur
 A feature of an image ?
l Characterizes its visual content.
l A part of an organization on describing a region
l A higher level organization than a pixel
l A point in a multidimensional space
l feature vector: n-Dimensional vector: x ϵ Rn
l Represents
l a point within a neighborhood
l a pattern
l a patch
l an object
l the whole image
 Role of features in Image
Analysis

Pre-processing End Analysis

Feature Extraction
 Role of features in learning
visual content

Classification
ML
/ regression
/clustering

DL
Feature
extraction

Input
image Preprocessing
Region descriptors

l Patch descriptors
l Texture descriptors
l Shape descriptors
 Patch Descriptor: Histogram of
Gradients (HoG)
l Compute centered horizontal and vertical
gradients with no smoothing.
l Compute gradient orientation and magnitudes,
l For color image, pick the color channel with the
highest gradient magnitude for each pixel.
l For a 64x128 image, divide the image into 16x16
blocks of 50% overlap. à7x15=105 blocks in
total.
N.Dalal and B. Triggs, Histograms of oriented gradients for
human detection, CVPR-2005
Histogram of Gradients (HoG)

l Each block: 2x2 cells with size 8x8.

l Quantize the gradient orientation into 9 bins.
l The vote is the gradient magnitude.
l Interpolate votes between neighboring bin center.
l The vote can also be weighted with Gaussian to
down-weight the pixels near the edges of the block.
l Concatenate histograms.
l Feature dimension: 105x4x9 = 3,780
N.Dalal and B. Triggs, Histograms of oriented gradients for human
detection, CVPR-2005
 Object detection with patch
descriptors.
l Typical examples:
l Pedestrian detection

l Character recognition

l Detection as a classification task.

l Generate labeled sample feature descriptors.

l Train a classifier.

l NN, SVM, Decision Tree, Random Forest …..

l Label an unknown patch using its descriptor.

Applications
l Pedestrian Detection

N.Dalal and B. Triggs, Histograms of oriented gradients for

human detection, CVPR-2005
 Non-maximal suppression
l Expected to get a high detection score with
neighboring overlapping patches.
l Select the patch with locally maximal score.

l A greedy approach:
l Select the best scoring window

l It is expected to cover the target object.

l Suppress the windows that are too close to the

selected window.
l Search next top-scoring windows out of the rest.
 Character Spotting
(E-PURALEKHAK)
l A tool for digital paleography
l Aids analysis of inscription
l Converts inscribed substrate to editable text
l Processing pipeline
— Preprocessing – denoising and normalization ([0
255])
— Search possible locations of user indicated
character by cross correlation
— Prune the search results by HoG feature matching
— Parse the Unicode file – editable text
Shashaank. M. Aswatha et al., A Method for Extracting Text from Stone Inscriptions using
Character spotting, ACCV, 2014.
 Character Spotting
(E-PURALEKHAK)

Character spotting – process flow

Shashaank. M. Aswatha et al., A Method for Extracting Text from Stone Inscriptions using
Character spotting, ACCV, 2014.
 E-PURALEKHAK

Original
Intermediate
Inscription
Result
Image

Unicode
generation Editable
text
Texture descriptor

l Texture: spatial
arrangement of the
colors or intensities in
an image
l A quantitative measure
of the arrangement of
intensities in the region.

Computer Vision by Shapiro and Stockman, Pearson, (2001)

Texture descriptors

l Edge density and direction

l Local Binary Pattern (LBP).
l Co-occurrence Matrix.
l Laws’ texture energy features.
Edge density and direction
l Compute gradient at each pixel.
l The descriptor: normalized histograms of
magnitudes and directions of gradients over a
region. Normalized histogram
of magnitudes.
l (HR(mag),HR(dir))
Normalized histogram
of directions.
l Numbers of bins in histograms kept small (e.g.
10).
l Use L1 norm between the feature vectors as a distance
measure. Normalized histogram à Area =1;
 Local Binary Pattern (LBP).

3 2 1
4 c 0
You may have
5 6 7 different ordering of
neighbors.
o Values range from 0 to 255.
o Obtain normalized histogram over a region.
o Not rotational invariant.
o Invariant to illumination and contrast.
T. Ojala, M. Pietikainen, and D. Harwood, A Comparative Study of Texture Measures with
Classification Based on Feature Distributions, Pattern Recognition, vol. 29, pp. 51-59, 1996.
Co-occurrence Matrix (Cr)

l Cr(x,y): How many times elements x and y

occur at a pair of pixels related spatially
(designated by r in the notation).
l e.g. p r q denotes q is shifted from p by a
translation of t=(a,b), i.e. q=p+t.
l C(a,b)(x,y): Number of cases in an image
where I(p)=x and I(p+t)=y.
Co-occurrence Matrix (Cr)

0 1 0 1

0 0
0 0 1 1
0 0 1 1 1
1
1 1 0 0 C(1,0)
C(0,1) 0 1
1 1 0 0
0

1
C(0,1)
Co-occurrence Matrix (Cr)

1 0 1
0
4 2 0 4 2
0 0 1 1 0
0 0 1 1 2 4 1 2 4
1
1 1 0 0 C(1,0)
C(0,1) 0 1
1 1 0 0
0 2 2

1 2 3
C(1,1)
Normalized Co-occurrence
Matrix (Nr)
Divide by the sum of frequencies in a matrix.
1 0 1
0
1/3 1/6 0 1/3 1/6
0 0 1 1 0
0 0 1 1 1/6 1/3 1 1/6 1/3
1
1 1 0 0 C(1,0)
C 0 1
1 1 0 0 (0,1)

0 2/9 2/9

1 2/9 1/3
C(1,1)
Symmetric Co-occurrence
Matrix (Sr)

Sr(x,y)=Cr(x,y)+C-r(x,y)
1 0 1
0
0 4+4 2+2 0 4+4 2+2
0 0 1 1
0 0 1 1 1 4+4 2+2
1 2+2 4+4
1 1 0 0
C(0,1)+C(0,-1) 0 1 C(1,0)+C(-1,0)
1 1 0 0
0 2+2 2+2

1 2+2 3+3
C(1,1)+C(-1,-1)
 Features from Normalized Co-
occurrence Matrix

∑3 ∑4 𝑁- 𝑥, 𝑦 𝑥𝑦 − 𝜇3 𝜇4
𝐶𝑜𝑟𝑟𝑒𝑙𝑎𝑡𝑖𝑜𝑛 =
𝜎3 𝜎4
Features from Normalized Co-
occurrence Matrix

Mean and s.d. Mean and s.d.

of row sums of column sums
 Laws’ texture energy features

l A set of 9 5x5 masks used to compute texture

energy.
A mask: Outer
L5 (Level): [1 4 6 4 1]T product of any pair.
E5 (Edge): [-1 -2 0 2 1]T e.g. E5L5: E5.L5T
S5 (Spot): [-1 0 2 0 -1]T
R5 (ripple): [1 -4 6 -4 1]T
Computation with mask:
Convolution
K. Laws, “Rapid Texture Identification”, in SPIE Vol. 238: Image Processing for
Missile Guidance, 1980, pp. 376-380.
 Laws’ texture energy features
lA set of 9 5x5 masks used to compute texture
energy.
L5E5 and E5L5
Take L5R5 and R5L5
L5 (Level): [1 4 6 4 1]T average of E5S5 and S5E5
E5 (Edge): [-1 -2 0 2 1]T responses L5S5 and S5L5
S5 (Spot): [-1 0 2 0 -1]T of two E5R5 and R5E5
R5 (ripple): [1 -4 6 -4 1]T masks. S5R5 and R5S5
S5S5
16 such masks possible. R5R5
Combine a few pairs to make 9 masks. E5E5
K. Laws, “Rapid Texture Identification”, in SPIE Vol. 238: Image Processing for
Missile Guidance, 1980, pp. 376-380.
 Laws’ texture energy
L5E5 and E5L5 9 dimensional
Input image L5R5 and R5L5
E5S5 and S5E5 feature space.
15x15
L5S5 and S5L5
window E5R5 and R5E5
Subtract local S5R5 and R5S5
Combine a few
S5S5
average at each R5R5 symmetric pairs
pixel. E5E5 to 9 channels.
Sum of
absolute
Convolve Compute values
with 16 energy map for in a
each channel. 15x15
masks. window.
Use of texture descriptors

l Detection of object patches represented by

textured patterns.
l Segmentation of images.
l Classification / Matching
l Generate a Library of labelled feature
descriptors.
l Detection of classes (class labels).

l Matching to the nearest texture descriptor.

2-D shape descriptors

l Signature of a contour
l Slope density function
l Features of boundary segments

29
 Courtesy: P.P. Das, Professor, Dept. of CSE, IIT Kharagpur

Signature of a contour
l For a signature convert a 2-D boundary into a
representative 1-D function
l Plot the distance of the boundary from the
centroid as a function of angle

30

Courtesy: R.C. Gonzalez and R.E Woods © 1992-2008

Signature: Transformation
Dependency
l Invariant to location, but will depend on rotation
and scaling.
l Starting at the point farthest from the reference
point or using the major axis of the region can be
used to decrease dependence on rotation.
l Scale invariance can be achieved by either
scaling the signature function to fixed amplitude or
by dividing the function values by the standard
deviation of the function.

31

Courtesy: P.P. Das, Professor, Dept. of CSE, IIT Kharagpur

Contour representation: Slope
density function
l Histogram of the slopes (tangent angles) along a
contour.
l Orientation of the object can be determined using
correlation of slope histograms of model contour
with that of an image contour.
l Can be very useful for object recognition.

32

Courtesy: P.P. Das, Professor, Dept. of CSE, IIT Kharagpur

 Courtesy: P.P. Das, Professor, Dept. of CSE, IIT Kharagpur

Boundary Segments
l Boundary segments: decompose a boundary into
segments.
l Use of the convex hull of the region enclosed by
the boundary is a powerful tool for robust
decomposition of the boundary. A boundary
segment
Convex
deficiency

33

Courtesy: R.C. Gonzalez and R.E Woods © 1992-2008

 Courtesy: P.P. Das, Professor, Dept. of CSE, IIT Kharagpur

Boundary Segments
l The Convex Hull (CH) H of a
set S is defined as the smallest
convex set that contains S
l We define the set of Convex
Deficiencies (CD)
l D = H \ S. Small irregularities lead
l Follow the boundary and mark to tiny meaningless
the points at which transition is convex deficiency
components scattered
made into or out of a all along the boundary.
component of D Apply polygon approximation to find CH.
34

Courtesy: R.C. Gonzalez and R.E Woods © 1992-2008

Region characterization
l CH and CD are useful for entire regions
l Area of the region

l Area of its convex deficiency

l Number of components of convex deficiency

l Relative location of the components of CD

35

Courtesy: P.P. Das, Professor, Dept. of CSE, IIT Kharagpur

 Learning classes from images:
Supervised learning
Supervised learning: exploits knowledge about the
classification problem, such as example instances of
classes.
o Choice of features for discriminating classes.
o Extracting features from images.
o Form training and test data set with class labels.
o Train classifiers and evaluate performance.
o Use for classification of unknown samples.
Classification problem
l Given a labelled data sets:
{(yi,xi)}, i=1,2,…,n such that xi in Rn and yi is
the class of xi, an element of the finite set of
classes.
l yi could be +1 or -1 for a two class
problem.
Design a classifier C which assigns class yi
(output) to xi (input).
Typical examples
(Ravg,Gavg,Bavg)
l Given images of blood samples
determine whether a patient is
anemic or not. +ve
l Classification Problem

l Given images of blood samples ?

-ve
determine amount of hemoglobin
concentration (in gm/dl).
l Regression problem.

? gm/dl
Classification approaches
Classification:
Task of assigning a known category or class to
an object.

n Discriminant analysis
l Probabilistic
n Linear discriminant
l Bayesian classification analysis (LDA)
l Distance based n Artificial neural network
l K-Nearest neighbor (ANN)
n Feed-forward neural
network.
 Perceptron modelling a neuron

bias
x1 w1
Signum Logistic / Sigmoid
x2 w2 f (z) o
wn
xn
A network of perceptrons
provides a powerful
model describing input /
output relations.

https://pixabay.com/vectors/brain-neuron-nerves-cell-science-2022398/
Artificial Neural Network
l A network of perceptrons.
l Input: A vector

l Output: A vector / A scalar

O
I
u
n
t
p
p
u
u
t
t
Feed-forward Network
l No feed back or loop in the network.

X
 Multilayered feed-forward
Network
Hidden n Layer-wise
Input Layer -2 processing
n i th layer takes input
from (i-1)th layer
and forwards its
Output output to the input
of next layer.

Fully connected
Output
(FC) feed-
Layer
forward network.
Hidden
Layer -1
Mathematical description of
the model

l Let j th neuron of i th layer be nej(i).

l Its corresponding weights
l Wj(i)= (wj1(i), wj2(i), …., wjn_(i-1)(i))

l Bias: wj0(i)

l n_(i-1): Dimension of input to the neuron

l n_i: Dimension of output at i th layer

l Output of the neuron:

Mathematical description of
the model

l Output of j th neuron in i th layer:

l Input output relation in i th layer

b(i)
W(i)
 Input output relation
W(i) b(i)
l Output of j th neuron in i th layer:

l Input output relation in i th layer i

i

i
W(1),b(1) W(2),b(2) W(m),b(m)
Input Output
 Input output relation

Input W(1),b(1) W(2),b(2) W(m),b(m)

Output

Input W Output

Parameters
 Optimization problem

Input F(X;W) Output

Given {(Xi , Oi)}, i=1,2,…,N, find W such that it

produces Oi given input Xi for all i.

Minimize:

Apply the same gradient descent procedure to obtain the solution.

 Optimization problem
Training samples:
Input {(Xi , Oi)}, i=1,2,…,N
F(X;W) Output

Apply the same

Minimize:
gradient descent
procedure to obtain
1. Start with an initial W0. the solution.
2. Update W iteratively.

Stochastic gradient descent:

 Chain rule of computing
gradient of a single neuron
x1 w1 Target response: t
x2 w2 f (z) o Error:
wn E=(t-o)2
xn

-2(t-o) xi
f ’(z) Analytical method!
Computed given the
functional values.
f(z)(1-f(z))
Computing gradient: Back
propagation method

l For multi-layered feed forward network.

l Apply chain rule.
l From output to toward input.

l From output layer to toward input layer.

l Compute partial derivatives of weights at (i-1)th

layer from the i th layer.
ANN training

l Initialize W(0) .
l For each training sample (xi, oi) do
l Compute functional values of each neuron
in the forward pass.
l Update weights of each link starting from
the output layer using back propagation.
l Continue till it converges.
 Classification or regression?
l Primarily a regressor.
l Build a model to predict functional value F(x)
given input x.
l Can be converted to a classifier by appropriate
encoding of classes (output vector o).
l Two class problem

l Binary encoding: 0 / 1

l One hot encoding: (1 0) / (0 1)

 Measuring hemoglobin from
images of blood sample

A. Ghosh, et al, A Low-Cost Test for Anemia Using an Artificial Neural Network,
Computer Methods and Programs in Biomedicine, Volume 229, 2023, 107251.
 Measuring hemoglobin from
images of blood sample

ROI
Color correction

A. Ghosh, et al, A Low-Cost Test for Anemia Using an Artificial Neural Network,
Computer Methods and Programs in Biomedicine, Volume 229, 2023, 107251.
 Measuring hemoglobin from
images of blood sample
Data Set: 344 Blood Samples of 86 patients

A. Ghosh, et al, A Low-Cost Test for Anemia Using an Artificial Neural Network,
Computer Methods and Programs in Biomedicine, Volume 229, 2023, 107251.
 Measuring hemoglobin from
images of blood sample

A. Ghosh, et al, A Low-Cost Test for Anemia Using an Artificial Neural Network,
Computer Methods and Programs in Biomedicine, Volume 229, 2023, 107251.
Performance Analysis

l Multiple observations of the same subject.

l Reporting mean and Standard Error.
l S.E. : Estimate of the standard deviation
l Avg. S.E.: 0.22g/dl (range: 0.02 – 0.75 gm/dl)
l Mean absolute error with respect to the gold
standard cyanmethemoglobin measured
hemoglobin values: 0.75 gm/dl
l Correlation Coefficient: 0.91
l Coefficient of regression: 0.82
A. Ghosh, et al, A Low-Cost Test for Anemia Using an Artificial Neural Network,
Computer Methods and Programs in Biomedicine, Volume 229, 2023, 107251.
Performance

A. Ghosh, et al, A Low-Cost Test for Anemia Using an Artificial Neural Network,
Computer Methods and Programs in Biomedicine, Volume 229, 2023, 107251.
 Evaluation of a classifier
l Two class problems. AP AN

l Positive class and Negative class PP TP FP

l TP: Set of +ve samples predicted +ve. PN FN TN
l FP: Set of –ve samples predicted +ve.
l TN: Set of –ve samples predicted –ve. Accuracy:
l FN: Set of –ve samples predicted +ve. (TP+TN)/Total

Sensitivity / Recall: F-Score:

Precision: TP/PP Harmonic mean of
TPR= TP/AP
Recall: TP/AP precision and recall
Specificity:
TNR= TN/AN 2 2 ´ Prec ´ Recall
F= =
1 1 Prec + Recall
+
Prec Recall
Anemia detection

l Normal Population (< 12.5 gm/dl)

l Sensitivity: 82% Specificity: 80%
l Pregnant Women (< 11 gm/dl)
l Sensitivity: 89.4% Specificity: 94%
l On a validation data set of another 64 volunteers
l MAE: 0.78 gm/dl w.r.t. the gold standard measurement
l Correlation Coefficient= 0.88, Coefficient of regression = 0.77
l Normal Population: Sensitivity: 92% Specificity: 85%
l Pregnant Women: Sensitivity: 93% Specificity: 78%
A. Ghosh, et al, A Low-Cost Test for Anemia Using an Artificial Neural Network,
Computer Methods and Programs in Biomedicine, Volume 229, 2023, 107251.
 Summary of Techniques
l Region and texture l Features used for
descriptors. l classifying objects
l HoG l estimating parameters /
l Edge density various measures.
l LBP l Clustering
l Co-occurrence matrix l Use of ANN for
l Laws’ texture energy classification and
regression
Thank You

63