UNIT – IV

Support Vector Machine: Optimal separation, Kernels, The Support Vector

Machine Algorithm.
Artificial Neural Networks: Introduction, Artificial Neural Networks (ANN),
biological neural networks, Basic building blocks of ANN, ANN terminologies,
McCulloch-Pitts Neuron Model, Hebb Net. Perceptron Networks: Single Layer
Perceptron, Multilayer Perceptron Networks, back propagation of error,
derivation and learning algorithm
 Support Vector Machine Algorithm

• Support Vector Machine or SVM is one of the most popular

Supervised Learning algorithms, which is used for Classification as
well as Regression problems. However, primarily, it is used for
Classification problems in Machine Learning.
• The goal of the SVM algorithm is to create the best line or decision
boundary that can segregate n-dimensional space into classes so
that we can easily put the new data point in the correct category in
the future. This best decision boundary is called a hyperplane.
• SVM chooses the extreme points/vectors that help in creating the
hyperplane. These extreme cases are called as support vectors, and
hence algorithm is termed as Support Vector Machine. Consider the
below diagram in which there are two different categories that are
classified using a decision boundary or hyperplane:
• Example: SVM can be understood with the example that
we have used in the KNN(K-Nearest Neighbor(KNN) Algorithm)
classifier. Suppose we see a strange cat that also has
some features of dogs, so if we want a model that can
accurately identify whether it is a cat or dog, so such a
model can be created by using the SVM algorithm. We
will first train our model with lots of images of cats and
dogs so that it can learn about different features of cats
and dogs, and then we test it with this strange creature.
So as support vector creates a decision boundary
between these two data (cat and dog) and choose
extreme cases (support vectors), it will see the extreme
case of cat and dog. On the basis of the support vectors,
it will classify it as a cat. Consider the below diagram:

• SVM algorithm can be used for Face

detection, image classification, text
categorization, etc.
 Types of SVM
SVM can be of two types:
• Linear SVM: Linear SVM is used for linearly separable
data, which means if a dataset can be classified into two
classes by using a single straight line, then such data is
termed as linearly separable data, and classifier is used
called as Linear SVM classifier.
• Non-linear SVM: Non-Linear SVM is used for non-linearly
separated data, which means if a dataset cannot be
classified by using a straight line, then such data is termed
as non-linear data and classifier used is called as Non-
linear SVM classifier.
Hyperplane and Support Vectors in the SVM
algorithm:

• Hyperplane: There can be multiple lines/decision boundaries

to segregate the classes in n-dimensional space, but we need
to find out the best decision boundary that helps to classify
the data points. This best boundary is known as the
hyperplane of SVM.
• The dimensions of the hyperplane depend on the features
present in the dataset, which means if there are 2 features (as
shown in image), then hyperplane will be a straight line. And if
there are 3 features, then hyperplane will be a 2-dimension
plane.
• We always create a hyperplane that has a maximum margin,
which means the maximum distance between the data points.
 Support Vectors:
• The data points or vectors that are the closest to the
hyperplane and which affect the position of the
hyperplane are termed as Support Vector. Since these
vectors support the hyperplane, hence called a Support
vector.
How does SVM works?

• Linear SVM:
• The working of the SVM
algorithm can be understood
by using an example. Suppose
we have a dataset that has
two tags (green and blue), and
the dataset has two features
x1 and x2. We want a classifier
that can classify the pair(x1,
x2) of coordinates in either
green or blue. Consider the
below image:
• So as it is 2-d space so by
just using a straight line,
we can easily separate
these two classes. But
there can be multiple lines
that can separate these
classes. Consider the
below image:
• Hence, the SVM algorithm helps to
find the best line or decision
boundary; this best boundary or
region is called as a hyperplane.
SVM algorithm finds the closest
point of the lines from both the
classes. These points are called
support vectors. The distance
between the vectors and the
hyperplane is called as margin.
And the goal of SVM is to maximize
this margin. The hyperplane with
maximum margin is called
the optimal hyperplane.
Non-Linear SVM:

• If data is linearly arranged,

then we can separate it by
using a straight line, but
for non-linear data, we
cannot draw a single
straight line. Consider the
below image:
• So to separate these data points, we
need to add one more dimension. For
linear data, we have used two
dimensions x and y, so for non-linear
data, we will add a third dimension z. It
can be calculated as:
z=x2 +y2
• By adding the third dimension, the
sample space will become as below
image:
• So now, SVM will divide
the datasets into classes
in the following way.
Consider the below image:
• Since we are in 3-d Space,
hence it is looking like a
plane parallel to the x-
axis. If we convert it in 2d
space with z=1, then it will
become as:
• Hence we get a
circumference of radius 1
in case of non-linear data.
 Artificial Neural
Networks
The term "Artificial neural network" refers to a
biologically inspired sub-field of artificial
intelligence modeled after the brain. An Artificial
neural network is usually a computational network
based on biological neural networks that construct
the structure of the human brain.
Similar to a human brain has neurons
interconnected to each other, artificial neural
networks also have neurons that are linked to each
other in various layers of the networks. These
neurons are known as nodes.
The artificial neural network tutorial covers all the
aspects related to the artificial neural network. In
this tutorial, we will discuss ANNs, Adaptive
resonance theory, Kohonen self-organizing map,
Building blocks, unsupervised learning, Genetic
algorithm, etc.
What is Artificial Neural Network?

• The term "Artificial Neural

Network" is derived from
Biological neural networks that
develop the structure of a
human brain. Similar to the
human brain that has neurons
interconnected to one another,
artificial neural networks also
have neurons that are
interconnected to one another
in various layers of the
networks. These neurons are
known as nodes.
• The given figure illustrates
the typical diagram of
Biological Neural Network.
• The typical Artificial Neural
Network looks something
like the given figure.
• Dendrites from Biological
Neural Networks represent
inputs in Artificial Neural
Networks, cell nucleus
represents Nodes, synapse
represents Weights, and
Axon represents Output.
Relationship between Biological
neural network and artificial neural
network:
Biological Neural Network Artificial Neural Network

Dendrites Inputs

Cell nucleus Nodes

Synapse Weights

Axon Output
• An Artificial Neural Network in the field of Artificial intelligence where it
attempts to mimic the network of neurons makes up a human brain so that
computers will have an option to understand things and make decisions in a
human-like manner. The artificial neural network is designed by programming
computers to behave simply like interconnected brain cells.
• There are around 1000 billion neurons in the human brain. Each neuron has
an association point somewhere in the range of 1,000 and 100,000. In the
human brain, data is stored in such a manner as to be distributed, and we can
extract more than one piece of this data, when necessary, from our memory
parallelly. We can say that the human brain is made up of incredibly amazing
parallel processors.
• We can understand the artificial neural network with an example, consider an
example of a digital logic gate that takes an input and gives an output. “OR”
gate, which takes two inputs. If one or both the inputs are “On,” then we get
“On” in the output. If both the inputs are “Off,” then we get “Off” in output.
Here the output depends upon the input. Our brain does not perform the
same task. The outputs-to-inputs relationship keeps changing because of the
neurons in our brain, which are "learning."
 Basic Building Blocks of ANN
ANN depends upon the following three building blocks −
• Network Topology
• Adjustments of Weights or Learning
• Activation Functions
 Network Topology
• A network topology is the arrangement of a network along
with its nodes and connecting lines. According to the
topology, ANN can be classified as the following kinds −
• Feedforward Network
• It is a non-recurrent network having processing units/nodes
in layers and all the nodes in a layer are connected with the
nodes of the previous layers. The connection has different
weights upon them. There is no feedback loop means the
signal can only flow in one direction, from input to output. It
may be divided into the following two types −
• Single layer feedforward network
• Multilayer feedforward network
• Single layer feedforward network − The concept is of
feedforward ANN having only one weighted layer. In other
words, we can say the input layer is fully connected to the
output layer.
• Single layer feedforward network − The concept is of
feedforward ANN having only one weighted layer. In other
words, we can say the input layer is fully connected to the
output layer.
• Multilayer feedforward network − The concept is of
feedforward ANN having more than one weighted layer. As
this network has one or more layers between the input and
the output layer, it is called hidden layers.
Feedback Network
• As the name suggests, a feedback network has feedback
paths, which means the signal can flow in both directions
using loops. This makes it a non-linear dynamic system,
which changes continuously until it reaches a state of
equilibrium. It may be divided into the following types −
• Recurrent networks
• Fully recurrent network
• Recurrent networks − They are feedback networks with
closed loops. Following are the two types of recurrent
networks.
• Fully recurrent network − It is the simplest neural
network architecture because all nodes are connected to
all other nodes and each node works as both input and
output.
Jordan network − It is a closed loop network in which the
output will go to the input again as feedback as shown in
the following diagram.
 Adjustments of Weights or Learning
• Learning, in artificial neural network, is the method of
modifying the weights of connections between the
neurons of a specified network. Learning in ANN can be
classified into three categories namely supervised
learning, unsupervised learning, and reinforcement
learning.
Supervised Learning
• As the name suggests, this type of learning
is done under the supervision of a teacher.
This learning process is dependent.
• During the training of ANN under
supervised learning, the input vector is
presented to the network, which will give
an output vector. This output vector is
compared with the desired output vector.
An error signal is generated, if there is a
difference between the actual output and
the desired output vector. On the basis of
this error signal, the weights are adjusted
until the actual output is matched with the
desired output.
Unsupervised Learning
• As the name suggests, this type of
learning is done without the supervision of
a teacher. This learning process is
independent.
• During the training of ANN under
unsupervised learning, the input vectors
of similar type are combined to form
clusters. When a new input pattern is
applied, then the neural network gives an
output response indicating the class to
which the input pattern belongs.
• There is no feedback from the
environment as to what should be the
desired output and if it is correct or
incorrect. Hence, in this type of learning,
the network itself must discover the
patterns and features from the input data,
and the relation for the input data over
the output.
Reinforcement Learning
• As the name suggests, this type of learning
is used to reinforce or strengthen the
network over some critic information. This
learning process is similar to supervised
learning, however we might have very less
information.
• During the training of network under
reinforcement learning, the network
receives some feedback from the
environment. This makes it somewhat
similar to supervised learning. However, the
feedback obtained here is evaluative not
instructive, which means there is no
teacher as in supervised learning. After
receiving the feedback, the network
performs adjustments of the weights to get
better critic information in future.
 Activation Functions
It may be defined as the extra force or effort applied over
the input to obtain an exact output. In ANN, we can also
apply activation functions over the input to get the exact
output. Followings are some activation functions of interest
−
Linear Activation Function
• It is also called the identity function as it performs no
input editing. It can be defined as −
F(x)=x
Sigmoid Activation Function
 Sigmoid Activation Function
• It is of two type as follows −
Binary sigmoidal function − This activation function
performs input editing between 0 and 1. It is positive in
nature. It is always bounded, which means its output
cannot be less than 0 and more than 1. It is also strictly
increasing in nature, which means more the input higher
would be the output. It can be defined as

Bipolar sigmoidal function − This activation function

performs input editing between -1 and 1. It can be positive
or negative in nature. It is always bounded, which means
its output cannot be less than -1 and more than 1. It is also
strictly increasing in nature like sigmoid function. It can be
defined as
Hebb Net
Perceptron Network