Artificial Neural Networks and its Applications

Neurons or nerve cells that are the primary units of both the brain and the
nervous system. These neurons receive sensory input from the outside world
which they process and then provide the output which might act as the input
to the next neuron. Each of these neurons is connected to other neurons in
complex arrangements at synapses.

Artificial Neural Networks/Neural Nets

Artificial Neural Networks contain artificial neurons which are called units .
These units are arranged in a series of layers that together constitute the
whole Artificial Neural Network in a system. A layer can have only a dozen
units or millions of units as this depends on how the complex neural
networks will be required to learn the hidden patterns in the dataset.

Commonly, Artificial Neural Network has an input layer, an output layer as

well as hidden layers. The input layer receives data from the outside world
which the neural network needs to analyze or learn about.

Then this data passes through one or multiple hidden layers that transform
the input into data that is valuable for the output layer. Finally, the output
layer provides an output in the form of a response of the Artificial Neural
Networks to input data provided.

Neural Network Architecture

In the majority of neural networks, units are interconnected from one layer to
another. Each of these connections has weights that determine the influence
of one unit on another unit. As the data transfers from one unit to another,
the neural network learns more and more about the data which eventually
results in an output from the output layer.

The input layer of an artificial neural network is the first layer, and it
receives input from external sources and releases it to the hidden layer,
which is the second layer.

In the hidden layer, each neuron receives input from the previous layer
neurons, computes the weighted sum, and sends it to the neurons in the
next layer.

Connections between layers are "weighted," meaning each input from the
previous layer has a different level of importance. These weights are like
multipliers that adjust the impact of each input. During training, the network
learns how much each weight should be to make better predictions. In other
words, the network adjusts these weights over time to improve its
performance and make more accurate predictions.

Artificial neurons vs Biological neurons

 Synapses : Synapses are the links between biological neurons that

enable the transmission of impulses from dendrites to the cell body.
Synapses are the weights that join the one-layer nodes to the next-
layer nodes in artificial neurons. The strength of the links is determined
by the weight value.

 Learning : In biological neurons, learning happens in the cell body

nucleus or soma, which has a nucleus that helps to process the
 impulses. An action potential is produced and travels through the
axons if the impulses are powerful enough to reach the threshold.
This becomes possible by synaptic plasticity, which represents the
ability of synapses to become stronger or weaker over time in
reaction to changes in their activity. In artificial neural networks,
backpropagation is a technique used for learning, which adjusts the
weights between nodes according to the error or differences between
predicted and actual outcomes.

 Activation : In biological neurons, activation is the firing rate of the

neuron which happens when the impulses are strong enough to reach
the threshold. In artificial neural networks, A mathematical function
known as an activation function maps the input to the output, and
executes activations.

Backpropagation in Neural Network

Backpropagation is also known as "Backward Propagation of
Errors" and it is a method used to train neural network . Its goal
is to reduce the difference between the model’s predicted output
and the actual output by adjusting the weights and biases in the
network.

What is Backpropagation?
Backpropagation is a technique used in deep learning to train
artificial neural networks particularly feed-forward networks. It
works iteratively to adjust weights and bias to minimize the cost
function.
In each epoch the model adapts these parameters reducing loss
by following the error gradient. Backpropagation often uses
optimization algorithms like gradient descent or stochastic
gradient descent. The algorithm computes the gradient using
the chain rule from calculus allowing it to effectively navigate
complex layers in the neural network to minimize the cost
function.

Fig(a) A simple illustration of how the backpropagation works by adjustments

of weights

Backpropagation plays a critical role in how neural networks

improve over time. Here's why:
1. Efficient Weight Update: It computes the gradient of
the loss function with respect to each weight using the
chain rule making it possible to update weights efficiently.
 2. Scalability: The backpropagation algorithm scales well to
networks with multiple layers and complex architectures
making deep learning feasible.
3. Automated Learning: With backpropagation the
learning process becomes automated and the model can
adjust itself to optimize its performance.
Working of Backpropagation Algorithm
The Backpropagation algorithm involves two main steps:
the Forward Pass and the Backward Pass.
How Does Forward Pass Work?
In forward pass the input data is fed into the input layer. These
inputs combined with their respective weights are passed to
hidden layers. For example in a network with two hidden layers
(h1 and h2 as shown in Fig. (a)) the output from h1 serves as the
input to h2. Before applying an activation function, a bias is
added to the weighted inputs.
Each hidden layer applies an activation function like ReLU
(Rectified Linear Unit) which returns the input if it’s positive
and zero otherwise. This adds non-linearity allowing the model to
learn complex relationships in the data. Finally the outputs from
the last hidden layer are passed to the output layer where an
activation function such as softmax converts the weighted
outputs into probabilities for classification.
The forward pass using weights and biases

How Does the Backward Pass Work?

In the backward pass the error (the difference between the
predicted and actual output) is propagated back through the
network to adjust the weights and biases. One common method
for error calculation is the Mean Squared Error (MSE) given by:
MSE=(Predicted Output−Actual Output)2
Once the error is calculated the network adjusts weights
using gradients which are computed with the chain rule. These
gradients indicate how much each weight and bias should be
adjusted to minimize the error in the next iteration. The backward
pass continues layer by layer ensuring that the network learns
and improves its performance. The activation function through its
derivative plays a crucial role in computing these gradients during
backpropagation.

Example of Backpropagation in Machine

Learning
Let’s walk through an example of backpropagation in machine
learning. Assume the neurons use the sigmoid activation function
for the forward and backward pass. The target output is 0.5, and
the learning rate is 1.

Example (1) of backpropagation sum

Forward Propagation
 aj=∑(wi,j∗xi)aj = ∑(wi,j∗xi)
1. Initial Calculation
The weighted sum at each node is calculated using:

Where,
 aj is the weighted sum of all the inputs and weights at
each node
 wi,jwi,j represents the weights associated with the jth input
to the ith neuron
 xi represents the value of the jth1 input
2. Sigmoid Function
The sigmoid function returns a value between 0 and 1,
introducing non-linearity into the model.

To find the outputs of y3, y4 and y5

3. Computing Outputs
At h1 node
Once we calculated the a1 value, we can now proceed to find the

y3 value:
Similarly find the values of y4 at h2 and y5 at O3

Values of y3, y4 and y5

4. Error Calculation

Our actual output is 0.5 but we obtained 0.67. To calculate the

error we can use the below formula:
Using this error value we will be backpropagating.
Backpropagation
1. Calculating Gradients
The change in each weight is calculated as:

Where:
 δj is the error term for each unit,
 η is the learning rate.
2. Output Unit Error
For O3:

4. Hidden Unit Error

For h1:

For h2:

4. Weight Updates
For the weights from hidden to output layer:
New weight:

For weights from input to hidden layer:

New weight:

Similarly other weights are updated:

The updated weights are illustrated below

Through backward pass the weights are updated

After updating the weights the forward pass is repeated yielding:

Since y5=0.61 is still not the target output the process of

calculating the error and backpropagating continues until the
desired output is reached.
This process demonstrates how backpropagation iteratively
updates weights by minimizing errors until the network accurately
predicts the output.

This process is said to be continued until the actual output is

gained by the neural network.
Backpropagation Implementation in Python
for XOR Problem
This code demonstrates how backpropagation is used in a neural
network to solve the XOR problem. The neural network consists
of:
1. Defining Neural Network
 Input layer with 2 inputs
 Hidden layer with 4 neurons
 Output layer with 1 output neuron
 Using Sigmoid function as activation function
import numpy as np

class NeuralNetwork:
def __init__(self, input_size, hidden_size, output_size):
self.input_size = input_size
self.hidden_size = hidden_size
self.output_size = output_size

self.weights_input_hidden = np.random.randn(self.input_size,
self.hidden_size)
 self.weights_hidden_output = np.random.randn(self.hidden_size,
self.output_size)

self.bias_hidden = np.zeros((1, self.hidden_size))

self.bias_output = np.zeros((1, self.output_size))

def sigmoid(self, x):

return 1 / (1 + np.exp(-x))

def sigmoid_derivative(self, x):

return x * (1 - x)
 def __init__(self, input_size, hidden_size, output_size)::
constructor to initialize the neural network
 self.input_size = input_size: stores the size of the input
layer
 self.hidden_size = hidden_size: stores the size of the hidden
layer
 self.weights_input_hidden = np.random.randn(self.input_size,
self.hidden_size): initializes weights for input to hidden
layer
 self.weights_hidden_output = np.random.randn(self.hidden_size,
self.output_size): initializes weights for hidden to output
layer
 self.bias_hidden = np.zeros((1, self.hidden_size)): initializes
bias for hidden layer
 self.bias_output = np.zeros((1, self.output_size)): initializes
bias for output layer
2. Defining Feed Forward Network
In Forward pass inputs are passed through the network activating
the hidden and output layers using the sigmoid function.
def feedforward(self, X):
self.hidden_activation = np.dot(X, self.weights_input_hidden) +
self.bias_hidden
self.hidden_output = self.sigmoid(self.hidden_activation)

self.output_activation = np.dot(self.hidden_output,
self.weights_hidden_output) + self.bias_output
self.predicted_output = self.sigmoid(self.output_activation)

return self.predicted_output

 self.hidden_activation = np.dot(X, self.weights_input_hidden)

+ self.bias_hidden: calculates activation for hidden layer
  self.hidden_output = self.sigmoid(self.hidden_activation):
applies activation function to hidden layer
 self.output_activation = np.dot(self.hidden_output,
self.weights_hidden_output) + self.bias_output: calculates
activation for output layer
 self.predicted_output = self.sigmoid(self.output_activation):
applies activation function to output layer
3. Defining Backward Network
In Backward pass (Backpropagation) the errors between the
predicted and actual outputs are computed. The gradients are
calculated using the derivative of the sigmoid function and
weights and biases are updated accordingly.
def backward(self, X, y, learning_rate):
output_error = y - self.predicted_output
output_delta = output_error *
self.sigmoid_derivative(self.predicted_output)

hidden_error = np.dot(output_delta,
self.weights_hidden_output.T)
hidden_delta = hidden_error *
self.sigmoid_derivative(self.hidden_output)

self.weights_hidden_output += np.dot(self.hidden_output.T,
output_delta) * learning_rate
self.bias_output += np.sum(output_delta, axis=0, keepdims=True)
* learning_rate
self.weights_input_hidden += np.dot(X.T, hidden_delta) *
learning_rate
self.bias_hidden += np.sum(hidden_delta, axis=0, keepdims=True)
* learning_rate
 output_error = y - self.predicted_output: calculates the error
at the output layer
 output_delta = output_error *
self.sigmoid_derivative(self.predicted_output): calculates the
delta for the output layer
 hidden_error = np.dot(output_delta,
self.weights_hidden_output.T): calculates the error at the
hidden layer
 hidden_delta = hidden_error *
self.sigmoid_derivative(self.hidden_output): calculates the
delta for the hidden layer
 self.weights_hidden_output += np.dot(self.hidden_output.T,
output_delta) * learning_rate: updates weights between
hidden and output layers
  self.weights_input_hidden += np.dot(X.T, hidden_delta) *
learning_rate: updates weights between input and hidden
layers
4. Training Network
The network is trained over 10,000 epochs using the
backpropagation algorithm with a learning rate of 0.1
progressively reducing the error.
def train(self, X, y, epochs, learning_rate):
for epoch in range(epochs):
output = self.feedforward(X)
self.backward(X, y, learning_rate)
if epoch % 4000 == 0:
loss = np.mean(np.square(y - output))
print(f"Epoch {epoch}, Loss:{loss}")
 output = self.feedforward(X): computes the output for the
current inputs
 self.backward(X, y, learning_rate): updates weights and
biases using backpropagation
 loss = np.mean(np.square(y - output)): calculates the mean
squared error (MSE) loss
5. Testing Neural Network
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

nn = NeuralNetwork(input_size=2, hidden_size=4, output_size=1)

nn.train(X, y, epochs=10000, learning_rate=0.1)

output = nn.feedforward(X)
print("Predictions after training:")
print(output)
 X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]): defines the
input data
 y = np.array([[0], [1], [1], [0]]): defines the target values
 nn = NeuralNetwork(input_size=2, hidden_size=4, output_size=1):
initializes the neural network
 nn.train(X, y, epochs=10000, learning_rate=0.1): trains the
network
 output = nn.feedforward(X): gets the final predictions after
training
Output:
 Trained Model
The output shows the training progress of a neural network over
10,000 epochs. Initially the loss was high (0.2713) but it gradually
decreased as the network learned reaching a low value of 0.0066
by epoch 8000. The final predictions are close to the expected
XOR outputs: approximately 0 for [0, 0] and [1, 1] and
approximately 1 for [0, 1] and [1, 0] indicating that the network
successfully learned to approximate the XOR function.
Advantages of Backpropagation for Neural
Network Training
The key benefits of using the backpropagation algorithm are:
 Ease of Implementation: Backpropagation is beginner-
friendly requiring no prior neural network knowledge and
simplifies programming by adjusting weights with error
derivatives.
 Simplicity and Flexibility: Its straightforward design
suits a range of tasks from basic feedforward to complex
convolutional or recurrent networks.
 Efficiency: Backpropagation accelerates learning by
directly updating weights based on error especially in
deep networks.
 Generalization: It helps models generalize well to new
data improving prediction accuracy on unseen examples.
 Scalability: The algorithm scales efficiently with larger
datasets and more complex networks making it ideal for
large-scale tasks.
Challenges with Backpropagation
While backpropagation is powerful it does face some challenges:
1. Vanishing Gradient Problem: In deep networks the
gradients can become very small during backpropagation
making it difficult for the network to learn. This is common
when using activation functions like sigmoid or tanh.
 2. Exploding Gradients: The gradients can also become
excessively large causing the network to diverge during
training.
3. Overfitting: If the network is too complex it might
memorize the training data instead of learning general
patterns.
Backpropagation is a technique that makes neural network learn.
By propagating errors backward and adjusting the weights and
biases neural networks can gradually improve their predictions.
Though it has some limitations like vanishing gradients many
techniques like ReLU activation or optimizing learning rates have
been developed to address these issues.

How do Artificial Neural Networks learn?

Artificial neural networks are trained using a training set. For example,
suppose you want to teach an ANN to recognize a cat. Then it is shown
thousands of different images of cats so that the network can learn to
identify a cat. Once the neural network has been trained enough using
images of cats, then you need to check if it can identify cat images correctly.
This is done by making the ANN classify the images it is provided by deciding
whether they are cat images or not. The output obtained by the ANN is
corroborated by a human-provided description of whether the image is a cat
image or not. If the ANN identifies incorrectly then back-propagation is used
to adjust whatever it has learned during training. Backpropagation is done by
fine-tuning the weights of the connections in ANN units based on the error
rate obtained. This process continues until the artificial neural network can
correctly recognize a cat in an image with minimal possible error rates.

What are the types of Artificial Neural Networks?

 Feedforward Neural Network : The feedforward neural network is

one of the most basic artificial neural networks. In this ANN, the data or
the input provided travels in a single direction. It enters into the ANN
through the input layer and exits through the output layer while hidden
layers may or may not exist. So the feedforward neural network has a
front-propagated wave only and usually does not have
backpropagation.
 Convolutional Neural Network : A Convolutional neural network has
some similarities to the feed-forward neural network, where the
connections between units have weights that determine the influence
of one unit on another unit. But a CNN has one or more than one
convolutional layer that uses a convolution operation on the input and
then passes the result obtained in the form of output to the next layer.
CNN has applications in speech and image processing which is
particularly useful in computer vision.

 Modular Neural Network: A Modular Neural Network contains a

collection of different neural networks that work independently towards
obtaining the output with no interaction between them. Each of the
different neural networks performs a different sub-task by obtaining
unique inputs compared to other networks. The advantage of this
modular neural network is that it breaks down a large and complex
computational process into smaller components, thus decreasing its
complexity while still obtaining the required output.

 Radial basis function Neural Network: Radial basis functions are

those functions that consider the distance of a point concerning the
center. RBF functions have two layers. In the first layer, the input is
mapped into all the Radial basis functions in the hidden layer and then
the output layer computes the output in the next step. Radial basis
function nets are normally used to model the data that represents any
underlying trend or function.

 Recurrent Neural Network: The Recurrent Neural Network saves the

output of a layer and feeds this output back to the input to better
predict the outcome of the layer. The first layer in the RNN is quite
similar to the feed-forward neural network and the recurrent neural
network starts once the output of the first layer is computed. After this
layer, each unit will remember some information from the previous
step so that it can act as a memory cell in performing computations.

Applications of Artificial Neural Networks

1. Social Media: Artificial Neural Networks are used heavily in Social

Media. For example, let’s take the ‘People you may know’ feature on
Facebook that suggests people that you might know in real life so that
you can send them friend requests. Well, this magical effect is
achieved by using Artificial Neural Networks that analyze your profile,
your interests, your current friends, and also their friends and various
 other factors to calculate the people you might potentially know.
Another common application of Machine Learning in social media
is facial recognition . This is done by finding around 100 reference
points on the person’s face and then matching them with those already
available in the database using convolutional neural networks.

2. Marketing and Sales: When you log onto E-commerce sites like
Amazon and Flipkart, they will recommend your products to buy based
on your previous browsing history. Similarly, suppose you love Pasta,
then Zomato, Swiggy, etc. will show you restaurant recommendations
based on your tastes and previous order history. This is true across all
new-age marketing segments like Book sites, Movie services,
Hospitality sites, etc. and it is done by implementing personalized
marketing . This uses Artificial Neural Networks to identify the
customer likes, dislikes, previous shopping history, etc., and then tailor
the marketing campaigns accordingly.

3. Healthcare : Artificial Neural Networks are used in Oncology to train

algorithms that can identify cancerous tissue at the microscopic level
at the same accuracy as trained physicians. Various rare diseases may
manifest in physical characteristics and can be identified in their
premature stages by using Facial Analysis on the patient photos. So
the full-scale implementation of Artificial Neural Networks in the
healthcare environment can only enhance the diagnostic abilities of
medical experts and ultimately lead to the overall improvement in the
quality of medical care all over the world.

4. Personal Assistants: I am sure you all have heard of Siri, Alexa,

Cortana, etc., and also heard them based on the phones you have!!!
These are personal assistants and an example of speech recognition
that uses Natural Language Processing to interact with the users
and formulate a response accordingly. Natural Language Processing
uses artificial neural networks that are made to handle many tasks of
these personal assistants such as managing the language syntax,
semantics, correct speech, the conversation that is going on, etc.