An Introduction To Mathematics Behind Neural Networks

An Introduction To Mathematics
Behind Neural Networks

December 23rd 2019
TWEET THIS
Today, with open source machine learning software

libraries such as TensorFlow, Keras or PyTorch we can create
neural network, even with a high structural complexity, with
just a few lines of code. Having said that, the Math behind
neural networks is still a mystery to some of us and having
the Math knowledge behind neural networks and deep
learning can help us understand what’s happening inside a
neural network. It is also helpful in architecture selection,
fine-tuning of Deep Learning models, hyperparameters tuning
and optimization.
Introduction
I ignored understanding the Math behind neural networks and

Deep Learning for a long time as I didn’t have good
knowledge of algebra or differential calculus. Few days ago, I
decided to to start from scratch and derive the methodology
and Math behind neural networks and Deep Learning, to
know how and why they work. I also decided to write this
article, which would be useful to people like me, who finds it
difficult to understand these concepts.
Perceptrons
Perceptrons — invented by Frank Rosenblatt in 1957, are the

simplest neural network that consist of n number of inputs,
only one neuron and one output, where n is the number of
features of our dataset. The process of passing the data
through the neural network is know as forward propagation
and the forward propagation carried out in a Perceptron is
explained in the following three steps.
Step 1 : For each input, multiply the input value xᵢ with
weights wᵢ and sum all the multiplied values. Weights —
represent the strength of the connection between neurons
and decides how much influence the given input will have on
the neuron’s output. If the weight w₁ has higher value than
the weight w₂, then the input x₁ will have higher influence on
the output than w₂.
The row vectors of the inputs and weights are x = [x₁, x₂, … ,
xₙ] and w =[w₁, w₂, … , wₙ] respectively and their dot
product is given by
Hence, the summation is equal to the dot product of the

vectors x and w
Step 2: Add bias b to the summation of multiplied values

and let’s call this z. Bias — also know as offset is necessary in
most of the cases, to move the entire activation function to
the left or right to generate the required output values .
Step 3 : Pass the value of z to a non-linear activation

function. Activation functions — are used to introduce non-
linearity into the output of the neurons, without which the
neural network will just be a linear function. Moreover, they
have a significant impact on the learning speed of the neural
network. Perceptrons have binary step function as their
activation function. However, we shall use Sigmoid — also
know as logistic function as our activation function.
where, σ denotes the Sigmoid activation function and the
output we get after the forward prorogation is know as
the predicted value y.
Learning Algorithm
The learning algorithm consist of two parts —

Backpropagation and Optimization.
Backpropagation : Backpropagation, short for backward
propagation of errors, refers to the algorithm for computing
the gradient of the loss function with respect to the weights.
However, the term is often used to refer to the entire learning
algorithm. The backpropagation carried out in a Perceptron is
explained in the following two steps.
Step 1 : To know an estimation of how far are we from our
desired solution a loss function is used. Generally, Mean
Squared Error is chosen as loss function for regression
problems and cross entropy for classification problems. Let’s
take a regression problem and its loss function be Mean
Squared Error, which squares the difference
between actual (yᵢ) and predicted value ( ŷᵢ ).
Loss function is calculated for the entire training dataset and

their average is called the Cost function C.
Step 2 : In order to find the best weights and bias for our
Perceptron, we need to know how the cost function changes
in relation to weights and bias. This is done with the help
the gradients (rate of change) — how one quantity changes
in relation to another quantity. In our case, we need to find
the gradient of the cost function with respect to the weights
and bias.
Let’s calculate the gradient of cost function C with respect to
the weight wᵢ using partial derivation. Since the cost function
is not directly related to the weight wᵢ, let’s use the chain
rule.
Now we need to find the following three gradients
Let’s start with the gradient of the Cost function (C) with
respect to the predicted value ( ŷ )
Let y = [y₁ , y₂ , … yₙ] and ŷ =[ ŷ₁ , ŷ₂ , … ŷₙ] be the row

vectors of actual and predicted values. Hence the above
equation is simplifies as
Now let’s find the the gradient of the predicted value with
respect to the z. This will be a bit lengthy.
The gradient of z with respect to the weight wᵢ is
Therefore we get,
What about Bias? — Bias is theoretically considered to have

an input of constant value 1. Hence,
Optimization : Optimization is the selection of a best
element from some set of available alternatives, which in our
case, is the selection of best weights and bias of the
perceptron. Let’s choose gradient descent as our optimization
algorithm, which changes the weights and bias, proportional
to the negative of the gradient of the Cost function with
respect to the corresponding weight or bias. Learning rate (α)
is a hyperparameter which is used to control how much the
weights and bias are changed.
The weights and bias are updated as follows and the
Backporpagation and gradient descent is repeated until
convergence.
Conclusion
I hope that you’ve found this article useful and understood

the maths behind the Neural Networks and Deep Learning. I
have explained the working of a single neuron in this article,
however the these basic concepts are applicable to all kinds
of Neural Networks with some modifications. If you have any
questions or if you found a mistake, please let me know in
the comment.

An Introduction To Mathematics Behind Neural Networks

Uploaded by

Copyright:

Available Formats

An Introduction To Mathematics Behind Neural Networks

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

An Introduction To Mathematics Behind Neural Networks

Uploaded by

Copyright:

Available Formats

What are perceptrons?

How does forward propagation work in perceptrons?

An Introduction To Mathematics

Behind Neural Networks

Today, with open source machine learning software

I ignored understanding the Math behind neural networks and

Perceptrons — invented by Frank Rosenblatt in 1957, are the

Hence, the summation is equal to the dot product of the

Step 2: Add bias b to the summation of multiplied values

Step 3 : Pass the value of z to a non-linear activation

The learning algorithm consist of two parts —

Loss function is calculated for the entire training dataset and

Now we need to find the following three gradients

Let y = [y₁ , y₂ , … yₙ] and ŷ =[ ŷ₁ , ŷ₂ , … ŷₙ] be the row

The gradient of z with respect to the weight wᵢ is

What about Bias? — Bias is theoretically considered to have

I hope that you’ve found this article useful and understood

You might also like