BAYESIAN MODULE 4

LEARNING
• Bayesian Learning is a learning method that
describes and represents knowledge in
an uncertain domain
• and provides a way to reason about this
knowledge using probability measure
• . It uses Bayes theorem to infer the unknown
parameters of a model.
• Bayesian inference is useful in many
applications which involve reasoning and
diagnosis such as game theory, medicine,
etc.
• Bayesian inference is much more powerful in
handling missing data and for
estimating any uncertainty in
predictions.
 INTRODUCTION TO PROBABILITY-
BASED LEARNING
• Probability-based learning combines prior
knowledge or prior probabilities with
observed data.
• Probabilistic learning uses the concept of
probability theory that describes how to model
randomness, uncertainty, and noise to predict
future events.
• It is a tool for modelling large datasets and uses
Bayes rule to infer unknown quantities,
predict and learn from data.
• In a probabilistic model, randomness plays a
major role which gives probability
distribution a solution, while in a
deterministic model there is no randomness
 INTRODUCTION TO PROBABILITY-
BASED LEARNING
• Bayesian learning differs from probabilistic
learning as it uses subjective probabilities
(i.e., probability that is based on an individual's
belief or interpretation about the outcome of an
event and it can change over time) to infer
parameters of a model.
• Two practical learning algorithms called Naïve
Bayes learning and Bayesian Belief Network
(BBN) form the major part of Bayesian learning.
These algorithms use prior probabilities and apply
Bayes rule to infer useful information.
FUNDAMENTALS OF BAYES THEOREM

• Prior Probability
• It is the initial probability that is believed before
any new information is collected.
• Likelihood Probability
• Likelihood probability is the relative probability
of the observation occurring for each class
or the sampling density for the evidence given the
hypothesis.
• It is stated as P (Evidence | Hypothesis),
• which denotes the likeliness of the occurrence of
the evidence given the parameters.
FUNDAMENTALS OF BAYES THEOREM

• Posterior Probability
• It is the updated or revised probability of an
event taking into account the observations from
the training data.
• P (Hypothesis | Evidence) is the posterior
distribution representing the belief about the
hypothesis, given the evidence from the training
data. Therefore,
FUNDAMENTALS OF BAYES THEOREM

• Posterior Probability
• It is the updated or revised probability of an
event taking into account the observations from
the training data.
• P (Hypothesis | Evidence) is the posterior
distribution representing the belief about the
hypothesis, given the evidence from the training
data. Therefore,
CLASSIFICATION USING BAYES MODEL
• Naïve Bayes Classification models work on the
principle of Bayes theorem.
• Bayes' rule is a mathe-matical formula used to
determine the posterior probability, given prior
probabilities of events.
• . It is based on the calculation of the posterior
probability and is stated as
CLASSIFICATION USING BAYES MODEL
CLASSIFICATION USING BAYES MODEL
CLASSIFICATION USING BAYES MODEL
ZERO PROBABILITY ERROR
 ZERO PROBABILITY ERROR
• Since the probability value is zero, the model
fails to predict, and this is called as
ZeroProbability error.
• This problem arises because there are no
instances
• This zero-probability error can be solved by
applying a smoothing technique called
Laplace correction
• which means given 1000 data instances in the
training dataset, if there are zero instances for a
particular value of a feature we can add 1
instance for each attribute value pair of that
feature which will not make much difference for
1000 data instances and the overall probability
does not become zero
ZERO PROBABILITY ERROR
ZERO PROBABILITY ERROR
BAYES OPTIMAL CLASSIFIER
BAYES OPTIMAL CLASSIFIER
 ARTIFICIAL NEURAL NETWORKS
• Artificial Neural Networks (ANNs) imitate human brain
behaviour and the way in which learning happens in a
human.
• The human brain constitutes a mass of neurons that are all
connected as a network, which is actually a directed
graph.
• These neurons are the processing units which receive
information, process it and then transmit this data to other
neurons that allows humans to learn almost any task.
• ANN is a learning mechanism that models a human
brain to solve any non-linear and complex problem.
• Each neuron is modelled as a computing unit, or
simply called as a node in ANN, that is capable of
doing complex calculations
 ARTIFICIAL NEURAL NETWORKS
• They are widely used in developing artificial
learning systems and have inspired
researchers and industry in Machine Learning
nowadays.
• Some typical applications of ANN in the field
of computer science are Natural Language
Processing (NLP), pattern recognition,
face recognition, speech recognition,
character recognition, text processing,
stock prediction, computer vision, etc.
• ANNs also have been considerably used in
other engineering fields such as Chemical
industry, Medicine, Robotics,
Communications, Banking, and Marketing
 INTRODUCTION
• The human nervous system has billions of
neurons that are the processing units which
make humans to perceive things, to hear, to see and
to smell.
• It is a learning system that consists of functional
units called nerve cells, typically called as
neurons.
• The human nervous system is divided into two
sections called the Central Nervous System (CNS)
and the Peripheral Nervous System (PNS).
• The brain and the spinal cord constitute the CNS and
the neurons inside and outside the CNS constitute
the PNS.
• The neurons are basically classified into three
types called sensory neurons, motor neurons
and interneurons.
 INTRODUCTION
• Sensory neurons get information from
different parts of the body and bring it into the
CNS,
• whereas motor neurons receive information
from other neurons and transmit commands to
the body parts.
• The CNS consists of only interneurons which
connect one neuron to another neuron by
receiving information from one neuron and
transmitting it to another.
• The basic functionality of a neuron is to
receive information, process it and then
transmit it to another neuron or to a body
part.
•
 BIOLOGICAL NEURONS
•
 BIOLOGICAL NEURONS
• A typical biological neuron has four parts called
• dendrites, soma, axon and synapse.
• The body of the neuron is called as soma.
• Dendrites accept the input information and process it in
the cell body called soma.
• A single neuron is connected by axons to around 10,000
neurons and through these axons the processed
information is passed from one neuron to another neuron.
• A neuron gets fired if the input information crosses a
threshold value and transmits signals to another neuron
through a synapse.
• A synapse gets fired with an electrical impulse called
spikes which are transmitted to another neuron.
• A single neuron can receive synaptic inputs from one
neuron or multiple neurons.
• These neurons form a network structure which processes
input information and gives out a response.
 ARTIFICIAL NEURONS
• Artificial neurons are like biological
neurons which are called as nodes.
• A node or a neuron can receive one or more
input information and process it.
• Artificial neurons or nodes are connected by
connection links to one another.
• Each connection link is associated with a
synaptic weight.
 SIMPLE MODEL OF AN ARTIFICIAL
NEURON
• The first mathematical model of a
biological neuron was designed by
McCulloch & Pitts in 1943. It includes
two steps:
• 1. It receives weighted inputs from other
neurons
• 2. It operates with a threshold function
or activation function
• The received inputs are computed as a
weighted sum which is given to the
activation function and if the sum
exceeds the threshold value the neuron
gets fired
 PERCEPTRON AND LEARNING THEORY
• The first neural network model 'Perceptron',
designed by Frank Rosenblatt in 1958, is a linear
binary classifier used for supervised learning.
• He modified the McCulloch & Pitts Neuron modelby
combining two concepts, McCulloch-Pitts model of an
artificial neuron and Hebbian learning rule of adjusting
weights.
• He introduced variable weight values and an extra input
that represents bias to this model.
• He proposed that artificial neurons could actually learn
weights and thresholds from data and came up with a
supervised learning algorithm that enabled the artificial
neurons to learn the correct weights from training data by
itself.
• The perceptron model consists of 4 steps: 1. Inputs
from other neurons 2. Weights and bias 3. Net sum
4. Activation function
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
PERCEPTRON AND LEARNING THEORY
 XOR PROBLEM

• A perceptron model can solve all Boolean functions which are

linearly separable.
• However, the XOR problem was identified in 1969 by Minsky and Papert.
• An XOR function returns a 1, if the two inputs are not equal and a 0 if they
are equal.
 XOR PROBLEM
• Since XOR is not linearly separable, the single layer
perceptron failed to classify and hence this problem led to
the evolution of a multi-layer perceptron.
• Initially, the Multi-Layer Perceptron (MLP) was not a success due to
the lack of an appropriate learning algorithm.
• In 1974, Werbos introduced a back propagation algorithm for
the three-layered perceptron network and in 1986, a general
back propagation algorithm for a multi-layered perceptron
was introduced by Rummelhart and Mclelland.
• Then, again ANN and Deep Neural Networks became a success,
solving many complex problems in the current era.
DELTA LEARNING RULE AND GRADIENT DESCENT
• Generally, learning in neural networks is performed by adjusting the
network weights in order to minimize the difference between the
desired and estimated outputs.
• This delta difference is measured as an error function or also
called as cost function.
• The cost function, being linear and continuous, is differentiable.
• This way of learning called as delta rule (also known as
Widrow-Hoff" rule or Adaline rule) is a type of back
propagation applied for training the network.
• The training error of a hypothesis is half the squared difference
between the desired target output and actual output and is given as
follows:
DELTA LEARNING RULE AND GRADIENT DESCENT

• The principle of gradient descent is an optimization approach

which is used to minimize the cost function by converging to
a local minimal point moving in the negative direction of the
gradient and each step size during movement is determined
by the learning rate and the slope of the gradient.
 TYPES OF ARTIFICIAL NEURAL NETWORKS
• Feed Forward Neural Network
• This is the simplest neural network that consists of neurons which are
arranged in layers and the information is propagated only in the
forward direction.
• This model may or may not contain a hidden layer and there is
no back propagation.
• Based on the number of hidden layers they are further classified into
single-layered and multi-layered feed forward networks.
• These ANNs are simple to design and easy to maintain. They are fast
but cannot be used for complex learning. They are used for simple
classification and simple image processing, etc.
 TYPES OF ARTIFICIAL NEURAL NETWORKS
• Fully Connected Neural Network
• Fully connected neural networks are the ones in which all the
neurons in a layer are connected to all other neurons in the next
layer.
 TYPES OF ARTIFICIAL NEURAL NETWORKS
Multi-Layer Perceptron (MLP)
• This ANN consists of multiple layers with one input layer, one output
layer and one or more hidden layers.
• Every neuron in a layer is connected to all neurons in the next layer
and thus they are fully connected.
• The information flows in both the directions. In the forward direction,
the inputs are multiplied by weights of neurons and forwarded to the
activation function of the neuron and output is passed to the next
layer.
 TYPES OF ARTIFICIAL NEURAL NETWORKS
Multi-Layer Perceptron (MLP)
• This ANN consists of multiple layers with one input layer, one output
layer and one or more hidden layers.
• Every neuron in a layer is connected to all neurons in the next layer
and thus they are fully connected.
• The information flows in both the directions. In the forward direction,
the inputs are multiplied by weights of neurons and forwarded to the
activation function of the neuron and output is passed to the next
layer.
 TYPES OF ARTIFICIAL NEURAL NETWORKS
Feedback Neural Network
Feedback neural networks have feedback connections between
neurons that allow information flow in both directions in the network.
The output signals can be sent back to the neurons in the same layer
or to the neurons in the preceding layers.
 POPULAR APPLICATIONS OF ARTIFICIAL NEURAL
NETWORKS
• ANN learning mechanisms are used in many complex applications that involve
modelling of non-linear processes.
• ANN is a useful model that can handle even noisy and incomplete data. They
are used to model complex patterns, recognize patterns and solve prediction
problems like
1. Real-time applications: Face recognition, emotion detection, self-driving
cars, navigation systems, routing systems, target tracking, vehicle scheduling,
etc.
2. Business applications: Stock trading, sales forecasting, customer behaviour
modelling, Market research and analysis, etc.
3. Banking and Finance: Credit and loan forecasting, fraud and risk evaluation,
currency price prediction, real-estate appraisal, etc.
4. Education: Adaptive learning software, student performance modelling, etc.
5. Healthcare: Medical diagnosis or mapping symptoms to a medical case,
image interpre-tation and pattern recognition, drug discovery, etc.
6. Other Engineering Applications: Robotics, aerospace, electronics,
 ADVANTAGES AND DISADVANTAGES OF ANN

• Advantages of ANN
• 1. ANN can solve complex problems involving non-linear
processes.
• 2. ANNs can learn and recognize complex patterns and solve
problems as humans solve a problem.
• 3. ANNs have a parallel processing capability and can predict in less
time.
• 4. They have an ability to work with inadequate knowledge. It can
even handle incomplete and noisy data.
• 5. They can scale well to larger data sets and outperforms other
learning mechanisms.
 ADVANTAGES AND DISADVANTAGES OF ANN

• Limitations of ANN
• 1. An ANN requires processors with parallel processing capability
to train the network running for many epochs. The function of each
node requires a CPU capability which is difficult for very large networks
with a large amount of data.
• 2. They work like a 'black box' and it is exceedingly difficult to
understand their working in inner layers. Moreover, it is hard to
understand the relationship between the representations learned at each
layer.
• 3. The modelling with ANN is also extremely complicated and the
development takes a much longer time.
• 4. Generally, neural networks require more data than traditional
machine learning algorithms, and they do not perform well on
small datasets.
• 5. They are also more computationally expensive than traditional
 CHALLENGES OF ARTIFICIAL NEURAL NETWORKS

• The major challenges while modelling a real-time application with ANNs

are:
• 1. Training a neural network is the most challenging part of using
this technique. Overfitting or underfitting issues may arise if datasets
used for training are not correct. It is also hard to generalize to the real-
world data when trained with some simulated data. Moreover, neural
network models normally need a lot of training data to be robust and are
usable for a real-time application.
• 2. Finding the weight and bias parameters for neural networks is
also hard and it is difficult to calculate an optimal model.