Deep Learning

Unit -2
 Back Propagation in deep learning
Back-propagation is the essence of neural net training. It is the method of
fine-tuning the weights of a neural net based on the error rate obtained in the
previous epoch (i.e., iteration). Proper tuning of the weights allows you to reduce
error rates and to make the model reliable by increasing its generalization.

Backpropagation is a short form for "backward propagation of errors." It is

a standard method of training artificial neural networks. This method helps to
calculate the gradient of a loss function with respects to all the weights in the
network.
Back Propagation in deep learning
Back Propagation in deep learning
Back Propagation in deep learning (epoch)
 For further assistance

Visit Stack Exchange

https://stats.stackexchange.com/questions/154879/a-list-of-
cost-functions-used-in-neural-networks-alongside-applicatio
ns
Back Propagation in deep learning

What is Bias
Back Propagation (reduce Cost
function)
(Batch Gradient Descent,
Stochastic Gradient Descent, Mini-
batch Gradient Descent)
 What is Bias
Bias is just like an intercept added in a linear equation. It is an additional
parameter in the Neural Network which is used to adjust the output along with
the weighted sum of the inputs to the neuron. Moreover, bias value allows you to
shift the activation function to either right or left.
output = sum (weights * inputs) + bias
The output is calculated by multiplying the inputs with their weights and then
passing it through an activation function like the Sigmoid function, etc. Here, bias
acts like a constant which helps the model to fit the given data. The steepness of
the Sigmoid depends on the weight of the inputs.
A simpler way to understand bias is through a constant c of a linear function
y =mx + c
What is Bias

W*x+b
 Different types of Neural Network
• Perceptron (Multilayer Perceptron) & ANN
• Feedforward Neural Network – Artificial Neuron
• Radial Basis Function Neural Network
• Convolutional Neural Network
• Recurrent Neural Network(RNN) –
Long Short Term Memory
 ANN
A perceptron is a network with two layers, one input and one output. ... Artificial neural network, which has
input layer, output layer, and two or more trainable weight layers (constisting of Perceptrons) is called
multilayer perceptron or MLP
 Feedforward Neural Network
It is one of the simplest types of artificial neural networks. In a feedforward neural
network, the data passes through different input nodes until it reaches the output node. In other
words, the data moves in only one direction from the first range until it reaches the output node.
It is also known as a front propagating wave which is usually obtained using a graded activation
function. Unlike more complex types of neural networks, backpropagation and data move in only
one direction. A feedforward neural network may consist of a single layer or may contain hidden
layers. In a feedful neural network, the products of the inputs and their weights are calculated.
This is then fed to the output.

whereas
Backpropagation is a training algorithm
consisting of 2 steps:
•Feedforward the values.
•Calculate the error and propagate it back to
the earlier layers.
 Radial Basis Function Neural Network
A radial basis function (RBF) is a function that assigns a real value to each input from its domain (it is a real-value
function), and the value produced by the RBF is always an absolute value; i.e. it is a measure of distance and cannot
be negative.
f(x) = f(||x||)

Euclidean distance, the straight-line distance between two points in Euclidean space, is typically used.
Radial basis functions are used to approximate functions, much as neural networks act as function approximators.
RBF network represents a radial basis function network. The radial basis functions act as activation functions.
The approximant f(x) is differentiable with respect to the weights W, which are learned using iterative updater
methods coming among neural networks.

Radial basis function neural networks are extensively applied in power restoration systems. In recent
decades, power systems have become larger and more complex.
This increases the risk of blackout. This neural network is used in power restoration systems to restore power in
the least amount of time.
 Convolutional Neural Network
Convolutional Neural Networks (CNN) is one of the variants of neural networks used heavily in
the field of Computer Vision. It derives its name from the type of hidden layers it consists of.
The hidden layers of a CNN typically consist of convolutional layers, pooling layers, fully
connected layers, and normalization layers. Here it simply means that instead of using the
normal activation functions defined above, convolution and pooling functions are used as
activation functions.
 Recurrent Neural Network
Recurrent Neural Network(RNN) are a type of Neural Network where the output from
previous step are fed as input to the current step. In traditional neural networks, all the inputs
and outputs are independent of each other, but in cases like when it is required to predict the
next word of a sentence, the previous words are required and hence there is a need to
remember the previous words. Thus RNN came into existence, which solved this issue with the
help of a Hidden Layer. The main and most important feature of RNN is Hidden state, which
remembers some information about a sequence.
principle
principle
 Step
STEP 1: Randomly initialize the weights to small numbers close to 0 (but not 0)

STEP 2: Input the first observation of your dataset in the input layer, each feature in one input node.

STEP 3: Forward-Propagation: from left to right, the neurons are activated in a way that the impact of each
neuron's activation is limited by the weights. Propagate the activations until getting the predicted result y.

STEP 4: Compare the predicted result to the actual result. Measure the generated error.

STEP 5: Back-Propagation: from right to left, the error is back-propagated.

Update the weights according to how much they are responsible for the error. The learning rate decides
by how much we update the weights.

STEP 6: Repeat Steps 1 to 5 and update the weights after each observation (Reinforcement Learning).
Or: Repeat Steps 1 to 5 but update the weights only after a batch of observations (Batch Learning).
 To Handle Overfitting Problem

• Bias and Variance Problem (tradeoff)

• irreducible Error
• Regularization (L1 Ridge Regression , L2 Lasso Regression
• Drop out Method (Regularization)
What is Bias and Variance
Underfitting and Overfitting (Bias and Variance)
So, What will be the best model

Low Bias, Low Variance

So, what about the irreducible Error, actually it is errors in data
collected at the source. Noisiness and outlies introduced at source
level from data collected!

Helpful link:
https://towardsdatascience.com/holy-grail-for-bias-variance-tradeoff-overfitting-underfitting-7fad64ab5d76
Regularization, Regularization is a process of
(L2) Ridge Regression introducing additional
(L1) Lasso Regression information in order to prevent
overfitting.
Drop out Method (Regularization)

Drop out ratio

0 <= p >= 1

Eg: = 0.2
Convolution neural network (CNN)
So, what is CNN
Convolution neural network (CNN)
So, what is CNN
 Convolution neural network (CNN)
So, what is CNN
A convolutional neural network (CNN) is a specific type of artificial neural network that uses perceptrons, a
machine learning unit algorithm, for supervised learning, to analyze data. CNNs apply to image processing,
natural language processing and other kinds of cognitive tasks.
Thank You