Subject: - Computational Methods

Topic: - Methods for image enhancement and classification Techniques based

on K-nearest neighbor and Linear discrimination Analysis
 A. Introduction:
Digital image processing is meant for processing digital computer. It is the use of computer
algorithm to perform image processing on digital images. It is a technology widely used for
digital image operations like feature extraction, pattern recognition, segmentation, image
morphology etc.

B. Edge Detection
Edge detection is a well developed field on its own within image processing. Edge detection is
basically image segmentation technique, divides spatial domain, on which the image is defined,
into meaningful parts or regions. Edges characterize boundaries and are therefore a problem of
fundamental importance in image processing. Edges typically occur on the boundary between
two different regions in an image. Edge detection allows user to observe those features of an
image where there is a more or less abrupt change in gray level or texture indicating the end of
one region in the image and the beginning of another. It finds practical applications in medical
imaging, computer guided surgery diagnosis, locate object in satellite images, face recognition,
and finger print recognition ,automatic traffic controlling systems, study of anatomical structure
etc. Many edge detection techniques have been developed for extracting edges from digital
images. Gradient based classical operators like Prewitt, Sobel were initially used for edge
detection but they did not give sharp edges and were highly sensitive to noise image. But
algorithm proposed by John F. Canny in 1986 is considered as the ideal edge detection algorithm
for images that are corrupted with noise. Canny's aim was to discover the optimal edge detection
algorithm which reduces the probability of detecting false edge, and gives sharp edges.[1]-[5]

I. First Order Edge Detection or Gradient Based Edge Operator:[1]

It is based on the use of a first order derivative, or can say gradient based. If I (i , j) be the input
image, then image gradient is given by following formula:
∂ I(i , j) ∂ I(i , j)
∇ I (i , j)=i +j
∂i ∂j (1)
∂ I(i , j)
Where: ∂i is the gradient in i direction.
∂I(i, j)
∂j is the gradient in j direction.
The gradient magnitude can be computed by the formula:
∂I 2 ∂I 2
|G|=
√( )( )
+
∂i ∂ j
|G|= G 2 +G 2
√ i j (2)
Or,
The gradient magnitude can be computed by the formula:
G
θ=arctan j
Gi ( ) (3)
The magnitude of gradient computed above gives edge strength and the gradient direction is
always perpendicular to the direction of edge.
I.1. Classical Operators: [1]
Sobel , Prewitt are classified as classical operators which are easy to operate but highly sensitive
to noise
Sobel operator: [1]
Sobel operator is a discrete differentiation operator used to compute an approximation of the
gradient of image intensity function for edge detection. At each pixel of an image, sobel operator
gives either the corresponding gradient vector or normal to the vector. It convolves the input
image with kernel and computes the gradient magnitude and direction. It uses following 3x3 two
kernels:
−1 0 +1 +1 +2 +1

[
Di= −2 0 +2
−1 0 +1
I.2. Canny edge detector: [1]-[5]
] and
[
D j= 0 0 0
−1 −2 −1 ]
Canny edge detector is an optimal edge detection technique as provide good detection, clear
response and good localization. It is widely used in current image processing techniques with
further improvements. The Canny edge detector uses a multi-stage algorithm to detect a wide
range of edges in images. The algorithm runs in 4 separate steps:
1. Smoothing: Noise contained in image is smoothed by convolving the input image I (i, j)
with Gaussian filter G. Mathematically, the smooth resultant image is given by
F(i , j)=G∗I (i, j ) (4)
2. Finding gradients: In this step, edges are detected where the change in grayscale intensity
is maximum. Required areas are determined with the help of gradient of images. Sobel
operator is used to determine the gradient at each pixel of smoothened image. Sobel
operators in i and j directions are given as
−1 0 +1 +1 +2 +1

[
Di= −2 0 +2
−1 0 +1 ] and
[
D j= 0 0 0
−1 −2 −1 ]
These sobel masks are convolved with smoothed image and giving gradients in i and j
directions.
Gi=D i ×F (i , j) and Gi=D j ×F(i , j) (5)
Therefore edge strength or magnitude of a pixel is given by
G= G 2 +G 2
√ i j (6)
The direction of gradient is given by
G
θ=arctan j
( )
Gi
(7)
Gi and Gj are the gradients in the i-th and j-dimensions respectively

3. Non-maximum suppression:
 Non maximum suppression is carried out to preserves all local maxima in the gradient
image, and deleting everything else this results in thin edges. For a pixel M (i, j):
 Firstly round the gradient direction Ѳ nearest45°, then compare the gradient
magnitude of the pixels in positive and negative gradient directions i.e. If gradient
direction is east then compare with gradient of the pixels in east and west
directions say E (i, j) and W (i, j) respectively.
 If the edge strength of pixel M (i, j) is largest than that of E (i, j) and W (i, j), then
preserve the value of gradient and mark M (i, j) as edge pixel, if not then suppress
or remove.
4. Double thresholding:
The output of non-maxima suppression still contains the local maxima created by noise.
Instead choosing a single threshold, for avoiding the problem of streaking two thresholds
thigh and tlow are used.
For a pixel M (i, j) having gradient magnitude G following conditions exists to detect
pixel as edge:
 If G < tlow than discard the edge.
 If G > thigh keep the edge.
 If tlow < G < thigh and any of its neighbors in a 3 ×3 region around it have gradient
magnitudes greater than thigh, keep the edge.
 If none of pixel (i, j)’s neighbors have high gradient magnitudes but at least one
falls between tlow and thigh search the 5 × 5 region to see if any of these pixels have
a magnitude greater than thigh. If so, keep the edge.
 Else, discard the edge.

C. K-Nearest Neighbor (KNN) Algorithm

The k Nearest Neighbour (kNN) is one of the most commonly used methods for pattern
recognition [6], and has been applied in a variety of cases [7][8][9]. Its simplicity and relatively
high convergence speed make it a popular choice. In pattern recognition, the k-Nearest
Neighbors algorithm (or k-NN for short) is a non-parametric method used for classification and
regression. In both cases, the input consists of the k closest training examples in the feature
space. The output depends on whether k-NN is used for classification or regression:
 In K-NN classification, the output is a class membership. An object is classified by a
majority vote of its neighbor, with the object being assigned to the class most common
among its k nearest neighbors. If k=1, then the object is simply assigned to the class of
that single nearest neighbor.
 In K-NN regression, the output is the property value for the object. This value is the
average of the values of its k nearest neighbors

The k-nearest-neighbor classifier is commonly based on the Euclidean distance between a test
sample and the specified training samples. Let xi be an input sample with p features (xi1,xi2,…,xip),
n be the total number of input samples (i=1,2,…,n) and p the total number of features (j=1,2,
…,p) . The Euclidean distance between sample xi and xl (l=1, 2…, n) is defined as
 d ( x i , x l )= √∑ ( x ij− xlj )2
(8)
The training phase of the algorithm consists only of storing the feature vectors and class labels of
the training samples. In the actual classification phase, the same features as before are computed
for the test sample (whose class is not known). Distances from the new vector to all stored
vectors are computed and k closest samples are selected. The new point is predicted to belong to
the most numerous class within the set.
The best choice of k depends upon the data; generally, larger values of k reduce the effect of
noise on the classification, but make boundaries between classes less distinct. A good k can be
selected by parameter optimization using, for example, cross-validation. The special case where
the class is predicted to be the class of the closest training sample (i.e. when k = 1) is called the
nearest neighbor algorithm.

The accuracy of the k-NN algorithm can be severely degraded by the presence of noisy or
irrelevant features, or if the features scales are not consistent with their relevance. A particularly
popular approach is the use of evolutionary algorithms to optimize feature scaling. Another
popular approach is to scale features by the mutual information of the training data with the
training classes. The algorithm is easy to implement, but it is computationally intensive,
especially when the size of the training set grows.

D. Linear Discriminant Analysis (LDA)

There are many possible techniques for classification of data. Linear Discriminant Analysis
(LDA) is commonly used techniques for data classification and dimensionality reduction. Linear
Discriminant Analysis easily handles the case where the within-class frequencies are unequal and
their performances have been examined on randomly generated test data.
The basic idea of LDA is to find a linear transformation that best discriminate among classes and
the classification is then performed in the transformed space based on some metric such as
Euclidean distance. Mathematically a typical LDA implementation is carried out via scatter
matrix analysis. [10]
Fisher first introduced LDA for two classes and its idea was to transform the multivariate
observations x to univariate observations y such that the y’s derived from the two classes were
separated as much as possible. For a set of m p-dimensional samples x1, x2, …, xm (where xi = (xi1,
. . . , xip)) belonging to two different classes, namely c1 and c2. For the two classes, the scatter
matrices are given as
¿ ¿
S i= ∑ ( x−x i )( x−x i )'
x ∈ ci
(9)
1
x i= ∑ x
Where mi x ∈ ci and m is the number of samples in c . Hence the total intra class
i i
scatter matrix is given by
¿ ¿
S w =S1 + S 2 =∑ ∑ ( x−x i )( x−x i )'
i x ∈ ci
(10)
The inter class scatter matrix is given by
 ¿ ¿ ¿ ¿
S B =( x 1−x 2 )( x 1−x 2 )' (11)
Fisher’s criterion suggests that the linear transformation Φ to maximize the so called Rayleigh
coefficient, i.e., the ratio of the determinant of the inter class scatter matrix of the projected
samples to the intra class scatter matrix of the projected samples:
|ΦT S B Φ|
ℑθ= T (12)
|Φ S w Φ|
If
Sw Φ
is non-singular, equation (12) can be solved as a conventional
S S
eigenvalue problem and Φ is given by the eigenvectors of matrix w−1 B

D.1. Multi Class LDA

If the number of classes are more than two, then a natural extension of Fisher Linear
discriminant exists using multiple discriminant analysis. As in two-class case, the projection is
from high dimensional space to a low dimensional space and the transformation suggested still
maximizes the ratio of intra-class scatter to the inter-class scatter. But unlike the two-class case,
the maximization should be done among several competing classes. Suppose that now there are n
classes.
For the n classes, the intra-class matrix is given by
n ¿ ¿
S w =S1 + ¿⋅¿+ S n=∑ ∑ ( x−x i )( x−x i )'
i=1 x ∈ c
i
(13)
The inter-class scatter matrix is given by
n ¿ ¿ ¿ ¿
S B =∑ mi ( xi −x )( x i−x )'
i=1
(14)
¿

Where
mi is the number of training samples for each class, x i is the mean for each class
¿
1 n ¿
¿¿
x= ∑ m x
and mi(xi−x) is total mean vector given by m i=1 i i

After obtaining B and S S

w the linear transformation ɸ can be obtained by solving the
generalized eigenvalue problem:
S B Φ=λS w Φ .
(15)

The upper bounds of the rank of S B and w S

are respectively m−n and n−1. Multiple
discriminant analysis provides an elegant way for classification using discriminant features.
D.2. Classification:
Once the transformation ɸ is given, the classification is then performed in the transformed space

based on some distance metric, such as Euclidean distance

d ( x , y )= √∑ i ( x i− y i )
2

and

d ( x , y )=1−
√∑ i ( x i− yi )
2

cosine measure
√∑ √∑ i x i2 i y2
i
. Then upon the arrival of the new instance z,
¿
arg min d ( zφ , x k φ ) ¿

it is classified to k , where x k is the centroid of kth class.

References:
1. James Clerk Maxwell,1868 DIGITAL IMAGE PROCESSING Mathematical and
Computational Methods.
2. R .Gonzalez and R. Woods, Digital Image Processing, ,Addison Wesley, 1992, pp 414 -
428.
3. S. Sridhar, Oxford university publication. , Digital Image Processing.
4. Shamik Tiwari , Danpat Rai & co.(P) LTD. “Digital Image processing”
5. J. F. Canny. “A computational approach to edge detection”. IEEE Trans. Pattern Anal.
MachineIntell., vol.PAMI-8, no. 6, pp. 679-697, 1986 Journal of Image Processing (IJIP),
Volume (3) : Issue
6. Moreno-Seco, F., Mico, L. and Oncina, J. A Modification of the LAESA Algorithm for
Approximated k-NN Classification. Pattern Recognition Letters 24 (2003), pp. 47–53
7. Abidin, T. and Perrizo, W. SMART-TV: A Fast and Scalable Nearest Neighbor Based
Classifier for Data Mining. Proceedings of ACM SAC-06, Dijon, France,April 23-27,
2006. ACM Press, New York, NY, pp. 536-540
8. Khan, M., Ding, Q. and Perrizo, W. K-Nearest Neighbors Classification of Spatial Data
Streams using P-trees. Proceedings of the PAKDD, 2002, pp. 517-528
9. Yu, K. and Ji, L. Karyotyping of Comparative Genomic Hybridization Human
Metaphases Using Kernel Nearest-Neighbor Algorithm, Cytometry, 48, 202–208, 2002.
10. S. Balakrishnama, A. Ganapathiraju, Linear Discriminant Analyais - A Brief Tutorial,
Institute for Signal and Information Processing, Department of Electrical and Computer
Engineering, Mississippi State University, page 2-3