Unit 5

Basics of Neural networks: Understanding the Biological Neuron, Exploring Artificial

Neuron, Types of Activation Functions, McCulloch –Pitts model of a Neuron,
Architectures of Neural Network, Learning Process in ANN, Backpropagation, Deep
Learning.

Basics of Neural networks: Understanding the Biological Neuron

Biological Neural Network: Biological Neural Network (BNN) is a structure that consists
of Synapse, dendrites, cell body, and axon. In this neural network, the processing is carried
out by neurons. Dendrites receive signals from other neurons, Soma sums all the incoming
signals and axon transmits the signals to other cells.
Differences between ANN and BNN :
Biological Neural Networks (BNNs) and Artificial Neural Networks (ANNs) are both
composed of similar basic components, but there are some differences between them.
Neurons: In both BNNs and ANNs, neurons are the basic building blocks that process and
transmit information. However, BNN neurons are more complex and diverse than ANNs. In
BNNs, neurons have multiple dendrites that receive input from multiple sources, and
the axons transmit signals to other neurons, while in ANNs, neurons are simplified and
usually only have a single output.
Synapses: In both BNNs and ANNs, synapses are the points of connection between neurons,
where information is transmitted. However, in ANNs, the connections between neurons are
usually fixed, and the strength of the connections is determined by a set of weights, while in
BNNs, the connections between neurons are more flexible, and the strength of the
connections can be modified by a variety of factors, including learning and experience.
Neural Pathways: In both BNNs and ANNs, neural pathways are the connections between
neurons that allow information to be transmitted throughout the network. However, in BNNs,
neural pathways are highly complex and diverse, and the connections between neurons can
be modified by experience and learning. In ANNs, neural pathways are usually simpler and
predetermined by the architecture of the network.

Parameters ANN BNN

input dendrites
weight synapse
Structure
output axon
hidden layer cell body

very precise structures and

Learning they can tolerate ambiguity
formatted data

Processor complex simple

 high speed low speed
one or a few large number

separate from a processor integrated into processor

Memory localized distributed

non-content addressable content-addressable

centralized distributed

Computing sequential parallel

stored programs self-learning

Reliability very vulnerable robust

numerical and symbolic perceptual

Expertise
manipulations problems

well-defined poorly defined

Operating
Environment well-constrained un-constrained

the potential of fault performance degraded even on

Fault Tolerance
tolerance partial damage

The human nervous system has two main parts – the central nervous system (CNS) consisting
of the brain and spinal cord the peripheral nervous system consisting of nerves and ganglia
outside the brain and spinal cord. The CNS integrates all information, in the form of signals,
from the different parts of the body. The peripheral nervous system, on the other hand, connects
the CNS with the limbs and organs.
It has three main parts to carry out its primary functionality of receiving and transmitting
information:
1. Dendrites – to receive signals from neighbouring neurons.
2. Soma – main body of the neuron which accumulates the signals coming from the different
dendrites. It ‘fires’ when a sufficient amount of signal is accumulated.
3. Axon – last part of the neuron which receives signal from soma, once the neuron ‘fires’, and
 passes it on to the neighbouring neurons through the axon terminals (to the adjacent dendrite
of the neighbouring neurons).
There is a very small gap between the axon terminal of one neuron and the adjacent dendrite
of the neighbouring neuron. This small gap is known as synapse. The signals transmitted
through synapse may be excitatory or inhibitory.

EXPLORING ARTIFICIAL NEURON

The biological neural network has been modelled in the form of ANN with artificial neurons
simulating the function of biological neurons. As depicted in Figure input signal x (x , x , …,
x ) comes to an artificial neuron. Each neuron has three major components:
TYPES OF ACTIVATION FUNCTIONS
There are different types of activation functions. The most commonly used activation
functions are highlighted below.
➢ Identity function
➢ Threshold/step function
➢ ReLU (Rectified Linear Unit) function
➢ Sigmoid function
• Binary sigmoid function
• Bipolar sigmoid function
➢ Hyperbolic tangent function
The range of values of sigmoid functions can be varied depending on the application.
However, the range of (−1, +1) is most commonly adopted.
MCCULLOCH –PITTS MODEL OF A NEURON
• The McCulloch–Pitts neural model, which was the earliest ANN model, has only
two types of inputs – excitatory and inhibitory.
• The excitatory inputs have weights of positive magnitude and the inhibitory weights
have weights of negative magnitude.
• The inputs of the McCulloch– Pitts neuron could be either 0 or 1.
• It has a threshold function as activation function. So, the output signal y is 1 if the
input y is greater than or equal to a given threshold value, else 0.
To analyse the situations using the McCulloch–Pitts neural model, we can consider the
input signals as follows:
x 1→ Is it raining?
x 2→ Is it sunny?
So, the value of both x and x can be either 0 or 1. We can use the value of both weights x
and x as 1 and a threshold value of the activation function as 1. So, the neural model will
look as Figure 10.7a.
ARCHITECTURES OF NEURAL NETWORK
The connection between artificial neurons can transmit signal from one neuron to another.
There are multiple possibilities for connecting the neurons based on which architecture we
are going to adopt for a specific solution.
There may be just two layers of neuron in the network – the input and output layer. Other
than the input and output layers, there may be one or more intermediate ‘hidden’ layers of
neuron. The neurons may be connected with one or more of the neurons in the next layer.
The neurons may be connected with all neurons in the next layer. There may be single or
multiple output signals. If there are multiple output signals, they might be connected with
 each other. The output from one layer may become input to neurons in the same or
preceding layer.

There are many components to a neural network architecture. Each neural network has a few
components in common:

Input - Input is data that is put into the model for learning and training purposes.

Weight - Weight helps organize the variables by importance and impact of contribution.

Transfer function - Transfer function is when all the inputs are summarized and combined into
one output variable.

Activation function - The role of the activation function is to decide whether or not a specific
neuron should be activated. This decision is based on whether or not the neuron’s input will be
important to the prediction process.

Bias - Bias shifts the value given by the activation function.

Types of ANN
• Single Layer Feed Forward Network
• Multiple Layer Feed Forward Network
• Recurrent Network
• COMPETITIVE NETWORK

SINGLE-LAYER FEED FORWARD NETWORK

A feedforward neural network is an artificial neural network wherein connections between

the units do not form a cycle. The feed-forward model is the basic type of neural network
because the input is only processed in one direction. The data always flows in one direction
and never backwards/opposite.
The net signal input to the output neurons is given by
MULTI-LAYER FEED FORWARD ANNS
The multi-layer feed forward network is quite similar to the single-layer feed forward network,
except for the fact that there are one or more intermediate layers of neurons between the input
and the output layers.
A multilayer feedforward neural network is an interconnection of perceptrons in which data
and calculations flow in a single direction, from the input data to the outputs. The number of
layers in a neural network is the number of layers of perceptrons.
COMPETITIVE NETWORK
The competitive network is almost the same in structure as the single-layer feed forward
network. The only difference is that the output neurons are connected with each other.
RECURRENT NETWORK
In feed forward networks, signals always flow from the input layer towards the output layer
(through the hidden layers in the case of multi-layer feed forward networks), i.e. in one
direction. In the case of recurrent neural networks, there is a small deviation. There is a
feedback loop, as depicted in Figure 10.20, from the neurons in the output layer to the input
layer neurons. There may also be self-loops.
LEARNING PROCESS IN ANN
There are four major aspects which need to be decided:
1. The number of layers in the network
2. The direction of signal flow
3. The number of nodes in each layer
4. The value of weights attached with each interconnection between neurons

NUMBER OF LAYERS
In the case of a single layer, a set of neurons in the input layer receives signal, i.e. a single
feature per neuron, from the data set. The value of the feature is transformed by the activation
function of the input neuron. The signals processed by the neurons in the input layer are then
forwarded to the neurons in the output layer. The neurons in the output layer use their own
activation function to generate the final prediction. More complex networks may be designed
with multiple hidden layers between the input layer and the output layer. Most of the multi-
layer networks are fully connected.
DIRECTION OF SIGNAL FLOW
In certain networks, termed as feed forward networks, signal is always fed in one direction, i.e.
from the input layer towards the output layer through the hidden layers, if there is any.
However, certain networks, such as the recurrent network, also allow signals to travel from the
output layer to the input layer.
NUMBER OF NODES IN LAYERS
In the case of a multi-layer network, the number of nodes in each layer can be varied. However,
the number of nodes or neurons in the input layer is equal to the number of features of the input
data set. Similarly, the number of output nodes will depend on possible outcomes, e.g. number
of classes in the case of supervised learning. So, the number of nodes in each of the hidden
layers is to be chosen by the user. A larger number of nodes in the hidden layer help in
improving the performance. However, too many nodes may result in overfitting as well as an
increased computational expense.
WEIGHT OF INTERCONNECTION BETWEEN NEURONS
For solving a learning problem using ANN, we can start with a set of values for the synaptic
weights and keep doing changes to those values in multiple iterations. In the case of supervised
learning, the objective to be pursued is to reduce the number of misclassifications. Ideally, the
iterations for making changes in weight values should be continued till there is no
misclassification.
To summarize, learning process using ANN is a combination of multiple aspects – which
include deciding the number of hidden layers, number of nodes in each of the hidden layers,
direction of signal flow, and last but not the least, deciding the connection weight.

BACK PROPAGATION
Backpropagation is the essence of neural network training. It is the method of fine-tuning the
weights of a neural network based on the error rate obtained in the previous epoch (i.e.,
iteration). Proper tuning of the weights allows you to reduce error rates and make the model
reliable by increasing its generalization.
Backpropagation in neural network is a short form for “backward propagation of errors.” It is
a standard method of training artificial neural networks. This method helps calculate the
gradient of a loss function with respect to all the weights in the network.
How Backpropagation Algorithm Works
The Back propagation algorithm in neural network computes the gradient of the loss function
for a single weight by the chain rule. It efficiently computes one layer at a time, unlike a native
direct computation. It computes the gradient, but it does not define how the gradient is used. It
generalizes the computation in the delta rule.
Consider the following Back propagation neural network example diagram to understand:
How Backpropagation Algorithm Works
1. Inputs X, arrive through the preconnected path
2. Input is modeled using real weights W. The weights are usually randomly selected.
3. Calculate the output for every neuron from the input layer, to the hidden layers, to the
output layer.
4. Calculate the error in the outputs
ErrorB= Actual Output – Desired Output
5. Travel back from the output layer to the hidden layer to adjust the weights such that the
error is decreased.
Keep repeating the process until the desired output is achieved

One main part of the algorithm is adjusting the interconnection weights. This is done using a
technique termed as gradient descent. In simple terms, the algorithm calculates the partial
derivative of the activation function by each interconnection weight to identify the ‘gradient’
or extent of change of the weight required to minimize the cost function.

During the learning phase, the interconnection weights are adjusted on the basis of the errors
generated by the network, i.e. difference in the output signal of the network vis-à-vis the expected
value. These errors generated at the output layer are propagated back to the preceding layers. Because
of the backward propagation of errors which happens during the learning phase, these networks are
also called back-propagation networks.
In this network, X0 is the bias input to the hidden layer and Y0 is the bias input to the output layer.
As a part of the gradient descent algorithm, partial derivative of the cost function E has to be
done with respect to each of the interconnection weights w′01 w′02 w′03 Mathematically, it can
be represented as follows:
The weights and bias for the interconnection between the input and hidden layers need to be
updated as follows:
DEEP LEARNING
There are multiple choices of architectures for neural networks, multi-layer neural network
being one of the most adopted ones. However, in a multi-layer neural network, as we keep
increasing the number of hidden layers, the computation becomes very expensive. Going
beyond two to three layers becomes quite difficult computationally. The only way to handle
such intense computation is by using graphics processing unit (GPU) computing. Deep
learning is a more contemporary branding of deep neural networks, i.e. multilayer neural
networks having more than three layers.