Classification by

backpropagation, SVM,kNN
21CSE355T – Data Mining and Analytics
Unit 3 - Part-2
Classification by backpropagation
“What is backpropagation?” Backpropagation is a neural network learning algorithm.
• The backpropagation algorithm performs learning on a multilayer feed-forward neural network. It
iteratively learns a set of weights for prediction of the class label of tuples.
• A multilayer feed-forward neural network consists of an input layer, one or more hidden layers,
and an output layer.
• Each layer is made up of units. The inputs to the network correspond to the attributes measured
for each training tuple.
• The inputs are fed simultaneously into the units making up the input layer.
• These inputs pass through the input layer and are then weighted and fed simultaneously to a
second layer of “neuronlike” units, known as a hidden layer.
• The outputs of the hidden layer units can be input to another hidden layer, and so on. The
number of hidden layers is arbitrary, although in practice, usually only one is used.
• The weighted outputs of the last hidden layer are input to units making up the output layer,
which emits the network’s prediction for given tuples.
Network Topology
• Neural networks can be used for both classification (to predict the class label of a
given tuple) and numeric prediction (to predict a continuous-valued output).
• For classification, one output unit may be used to represent two classes (where
the value 1
• represents one class, and the value 0 represents the other).
• If there are more than two classes, then one output unit per class is used.
(strategies used in multiclass classification)
• There are no clear rules as to the “best” number of hidden layer units.
• Network design is a trial-and-error process and may affect the accuracy of the
resulting trained network.
• The initial values of the weights may also affect the resulting accuracy.
• Once a network has been trained and its accuracy is not considered acceptable, it
is common to repeat the training process with a different network topology or a
different set of initial weights.
How does backpropagation work
• Backpropagation learns by iteratively processing a data set of training tuples,
comparing the network’s prediction for each tuple with the actual known target
value.
• The target value may be the known class label of the training tuple (for
classification problems) or a continuous value (for numeric prediction).
• For each training tuple, the weights are modified so as to minimize the mean-
squared error between the network’s prediction and the actual target value.
• These modifications are made in the “backwards” direction (i.e., from the output
layer) through each hidden layer down to the first hidden layer (hence the name
backpropagation).
• Although it is not guaranteed, in general the weights will eventually converge,
and the learning process stops.
Backpropagation algorithm
Backpropagation
• Initialize the weights: The weights in the network are initialized to small random numbers (e.g., ranging
from 􀀀1.0 to 1.0, or 􀀀0.5 to 0.5).
• Each unit has a bias associated with it, as explained later. The biases are similarly initialized to small random
numbers.
• Each training tuple, X, is processed by the following steps.
• Propagate the inputs forward: First, the training tuple is fed to the network’s input layer. The inputs pass
through the input units, unchanged. That is, for an input unit, j, its output, Oj , is equal to its input value, Ij .
• Next, the net input and output of each unit in the hidden and output layers are computed. The net input to
a unit in the hidden or output layers is computed as a linear combination of its inputs.
• To help illustrate this point, a hidden layer or output layer unit shown in the following figure.
Backpropagation
• Each such unit has a number of inputs to it that are, in fact, the outputs of the
units connected to it in the previous layer.
• Each connection has a weight. To compute the net input to the unit, each input
connected to the unit is multiplied by its corresponding weight, and this is
summed.
• Given a unit, j in a hidden or output layer, the net input, Ij , to unit j is

• where wij is the weight of the connection from unit i in the previous layer to unit j;
• Oi is the output of unit i from the previous layer; and j is the bias of the unit.
• The bias acts as a threshold in that it serves to vary the activity of the unit.
Backpropagation
• Backpropagate the error: The error is propagated backward by updating the weights and biases
to reflect the error of the network’s prediction. For a unit j in the output layer, the error Errj is
computed by

• The weights and biases are updated to reflect the propagated errors.
• Terminating condition:
• Training stops when
• Change in the weight in the previous epoch are so small as to be below some specified threshold, or
• The percentage of tuples misclassified in the previous epoch is below some threshold, or
• A prespecified number of epochs has expired.
Sample calculations for learning by the backpropagation algorithm
This example shows the calculations for backpropagation, given the first training tuple, X.
Sample calculations for learning by the backpropagation algorithm
Backpropagation
Classify an unknown tuple using a trained network

• To classify an unknown tuple, X, the tuple is input to the

trained network, and the net input and output of each unit
are computed.
• (There is no need for computation and/or backpropagation
of the error.)
• If there is one output node per class, then the output node
with the highest value determines the predicted class label
for X.
• If there is only one output node, then output values greater
than or equal to 0.5 may be considered as belonging to the
positive class, while values less than 0.5 may be considered
negative.
•
What is support vector ?
• “Support Vector Machine” (SVM) is a supervised machine learning algorithm which can
be used for both classification or regression challenges.
• However, it is mostly used in classification problems.
• In this algorithm, we plot each data item as a point in n-dimensional space (where n is
number of features you have) with the value of each feature being the value of a
particular coordinate.
• Then, we perform classification by finding the hyperplane that differentiate the two
classes very well.
• Generally, Support Vector Machines is considered to be a classification approach, it but
can be employed in both types of classification and regression problems.
• It can easily handle multiple continuous and categorical variables.
• SVM constructs a hyperplane in multidimensional space to separate different classes.
SVM generates optimal hyperplane in an iterative manner, which is used to minimize an
error.
• The core idea of SVM is to find a maximum marginal hyperplane(MMH) that best
divides the dataset into classes.
Decision vectors
Definitions
• Support Vectors
• Support vectors are the data points, which are closest to the hyperplane. These points will define the
separating line better by calculating margins. These points are more relevant to the construction of the
classifier.
• Hyperplane
• A hyperplane is a decision plane which separates between a set of objects having different class
memberships.
• Margin
• A margin is a gap between the two lines on the closest class points.
• This is calculated as the perpendicular distance from the line to support vectors or closest points. – If
the margin is larger in between the classes, then it is considered a good margin, a smaller margin is a
bad margin.
How SVM works ?
How classification will work ?
How SVM works ?
• The main objective is to segregate the given dataset in the best possible
way.
• The distance between the either nearest points is known as the margin.
• The objective is to select a hyperplane with the maximum possible
margin between support vectors in the given dataset.
• SVM searches for the maximum marginal hyperplane in the following
steps:
• – Generate hyperplanes which segregates the classes in the best way.
• – Select the right hyperplane with the maximum segregation from the
either nearest data points.
How SVM works ?
Non-linear and inseparable planes
• Some problems can’t be solved using linear hyperplane.
• In such situation, SVM uses a kernel trick to transform the input space to a higher dimensional
space as shown on the right.
• The data points are plotted on the x-axis and zaxis (Z is the squared sum of both x and y:
z=x^2=y^2).
• Now you can easily segregate these points using linear separation.
kNN
• An idea that can be used for machine learning as does
another maxim involving poultry: "birds of a feather
flock together."
• In other words, things that are alike are likely to have
properties that are alike.
• We can use this principle to classify data by placing it
in the category with the most similar, or "nearest"
neighbors.
Nearest Neighbour Classification
• In a single sentence, nearest neighbor classifiers are defined by their
characteristic of classifying unlabeled examples by assigning them the
class of the most similar labeled examples.
• Despite the simplicity of this idea, nearest neighbor methods are
extremely powerful. They have been used successfully for:
• Computer vision applications, including optical character recognition and
facial recognition in both still images and video
• Predicting whether a person enjoys a movie which he/she has been
recommended (as in the Netflix challenge)
• Identifying patterns in genetic data, for use in detecting specific protein
or diseases
kNN algorithm
• The kNN algorithm begins with a training dataset made up of examples
that are classified into several categories, as labeled by a nominal
variable.
• Assume that we have a test dataset containing unlabeled examples that
otherwise have the same features as the training data.
• For each record in the test dataset, kNN identifies k records in the
training data that are the "nearest" in similarity, where k is an integer
specified in advance.
• The unlabeled test instance is assigned the class of the majority of the k
nearest neighbors
Example:
Example
Example
Classify the “tomato”
Find the distance between neighbours and tomato
Distance calculation
• Locating the tomato's nearest neighbors requires a distance
function, or a formula that measures the similarity between
two instances.
• There are many different ways to calculate distance.
• Traditionally, the kNN algorithm uses Euclidean distance,
which is the distance one would measure if you could use a
ruler to connect two points, illustrated in the previous figure
by the dotted lines connecting the tomato to its neighbors.
Distance calculation
Euclidean distance
• Euclidean distance is specified by the following formula, where p and q are the examples to be
compared, each having n features. The term p1 refers to the value of the first feature of example
p, while q1 refers to the value of the first feature of example q:

• The distance formula involves comparing the values of each feature. For example, to calculate the
distance between the tomato (sweetness = 6, crunchiness = 4), and the green bean (sweetness =
3, crunchiness = 7), we can use the formula as follows:
Distance
Closest neighbours
Choosing appropriate k value
• Deciding how many neighbors to use for kNN
determines how well the mode will generalize to
future data.
• The balance between overfitting and underfitting the
training data is a problem known as the biasvariance
tradeoff.
• Choosing a large k reduces the impact or variance
caused by noisy data, but can bias the learner such
that it runs the risk of ignoring small, but important
patterns.
Choosing appropriate k value
Choosing appropriate k value
• In practice, choosing k depends on the difficulty of the
concept to be learned and the number of records in the
training data.
• Typically, k is set somewhere between 3 and 10. One common
practice is to set k equal to the square root of the number of
training examples.
• In the classifier, we might set k = 4, because there were 15
example ingredients in the training data and the square root
of 15 is 3.87.
Min-Max Normalisation
• The traditional method of rescaling features for kNN is minmax normalization.
• This process transforms a feature such that all of its values fall in a range between 0
and 1. The formula for normalizing a feature is as follows. Essentially, the formula
subtracts the minimum of feature X from each value and divides by the range of X:

• Normalized feature values can be interpreted as indicating how far, from

0 percent to 100 percent, the original value fell along the range between
the original minimum and maximum.
The Lazy learning
• Using the strict definition of learning, a lazy learner is not
really learning anything.
• Instead, it merely stores the training data in it. This allows the
training phase to occur very rapidly, with a potential downside
being that the process of making predictions tends to be
relatively slow.
• Due to the heavy reliance on the training instances, lazy
learning is also known as instance-based learning or rote
learning.
Lazy Learning algorithms

• K Nearest Neighbors
• Local Regression
• Lazy Naive Bayes
kNN algorithm
END OF UNIT-III