Neural Networks

Artificial Neural Networks (ANNs) are computer systems inspired by the way the
human brain works. They are made up of simple building blocks called artificial
neurons or units, which are organized into layers. These layers work together to
process information and solve complex tasks, such as recognizing images,
predicting values, or understanding speech.
Structure of an Artificial Neural Network
An Artificial Neural Network is typically made up of three types of layers:
1. Input Layer
o The input layer is the starting point of the neural network.
o It receives raw data from the outside world.
o Each unit (or neuron) in the input layer represents one feature or
attribute of the data.
o Example: If we are building a network to recognize handwritten
digits, each pixel of the image would be an input unit.
2. Hidden Layers
o After the input layer, the data moves through one or more hidden
layers.
o These layers perform complex transformations and calculations to
detect patterns or relationships in the data.
o The hidden layers are called "hidden" because we do not directly see
their operations; they work behind the scenes.
o A simple network might have just one hidden layer, but modern deep
learning models often have many hidden layers.
3. Output Layer
o The final layer in the network is the output layer.
o It produces the result or prediction based on the processed
information.
o Example: In a digit recognition system, the output layer would tell
us whether the input image is a 0, 1, 2, etc.
Connections and Weights
  Units in one layer are connected to units in the next layer.
 Each connection has an associated weight.
 The weight determines how much influence one neuron has on another.
 Higher weights mean stronger influence, while lower weights mean
weaker influence.
Example:
If a neural network is learning to recognize animals, the feature "has four legs"
might have a strong weight when identifying a dog, but a weaker weight when
identifying a bird.

How Data Flows Through the Network

1. Forward Propagation:
o The input data is passed from the input layer through the hidden
layers to the output layer.
o Each neuron calculates a value based on the inputs it receives and
the weights of its connections.
o These calculations continue layer by layer until the final output is
produced.
2. Learning and Weight Adjustment:
o Initially, the network might give wrong answers because the
weights are random.
o Using a method called training, the network adjusts the weights
step-by-step to improve its accuracy.
o This is done using techniques like backpropagation and gradient
descent, where the network compares its output to the correct
answer and makes small corrections.
Simple Real-World Example
Imagine teaching a child to recognize fruits:
 Input: You show a picture of a fruit (say, an apple).
 Hidden Processing: The child notices feature - it's red, round, and has a
stem.
 Output: The child says, "It's an apple!"
At first, the child might confuse an apple with a tomato. But after seeing many
examples and learning the right features, the child gets better - just like a neural
network improves after training.
Perceptron:
The Perceptron is one of the simplest and most important types of artificial
neurons. It is the basic building block of a neural network and was invented by
Frank Rosenblatt in 1958.
The main idea behind a perceptron is to mimic how a human brain neuron works
by taking several inputs, processing them, and producing an output.
Structure of a Perceptron
A perceptron has the following key components:
1. Inputs
o These are the data or features fed into the perceptron.
o Example: If you want to decide whether an email is spam or not,
the inputs could be words like "offer", "win", or "free".
2. Weights
o Each input is associated with a weight.
o Weights determine how important each input is to the final
decision.
o Higher weight means the input is more important.
3. Summation Function
o The perceptron multiplies each input by its corresponding weight
and then adds them all together.
4. Activation Function
 o After summing, the perceptron passes the result through an
activation function.
o In the basic perceptron, the activation function is simple:
 If the result is greater than a threshold value, output = 1.
 Otherwise, output = 0.
5. Output
o The final decision made by the perceptron — either 0 or 1.
o This output can represent different decisions, like "yes or no",
"spam or not spam", etc.

Working of a Perceptron - Step-by-Step

1. Take the input values.
2. Multiply each input by its weight.
3. Add all the weighted inputs together.
4. Compare the sum to a threshold.
5. If the sum is greater than the threshold, output 1; otherwise, output 0.
Simple Example
Suppose you are building a perceptron to decide if a student passes or fails
based on:
 Hours of study (Input 1)
 Number of classes attended (Input 2)
 Input Weight
Hours studied = 5 0.6
Classes attended = 8 0.4
Now:
 Multiply inputs with weights:
o (5 × 0.6) + (8 × 0.4) = 3 + 3.2 = 6.2
 Suppose the threshold is 5.
 Since 6.2 > 5, the perceptron outputs 1 (Pass).
If the result were less than 5, the output would be 0 (Fail).
Learning in a Perceptron
 In the beginning, the weights are random.
 During training, the perceptron adjusts its weights to minimize errors.
 Each time it makes a wrong prediction; it updates the weights slightly.
 Over time, with more examples, the perceptron learns the correct
decisions.
Limitations of a Simple Perceptron
 A single perceptron can only solve linearly separable problems (where
two classes can be separated by a straight line).
 It cannot solve more complex problems (like recognizing patterns that are
not linearly separable - e.g., XOR problem).
 To solve complex problems, we use multi-layer perceptron (which have
hidden layers too).
Types of Perceptron
Perceptron come in different types based on their structure and complexity.
Each type is designed to handle different kinds of problems, from simple to
complex.
Here are the main types of perceptron:
1. Single-Layer Perceptron
 The simplest form of perceptron.
  It has only one layer of output nodes (neurons).
 The input layer just passes the data, and no computation happens at the
input layer.
 The processing is done in the output layer, where inputs are multiplied by
weights, summed up, and passed through an activation function.
Example:
 Classifying whether an email is spam or not spam based on a few
keywords.
Limitations:
 Cannot solve non-linear problems like XOR (Exclusive OR) - problems
where you cannot draw a straight line to separate classes.
2. Multi-Layer Perceptron (MLP)
 A more advanced and powerful version of perceptron.
 Has multiple layers:
o Input layer (to receive the data),
o One or more hidden layers (to process the data),
o Output layer (to give the result).
 Hidden layers allow the network to learn complex patterns and solve non-
linear problems.
 Each neuron in one layer is connected to every neuron in the next layer
(this is called a fully connected network).
How It Works:
 Data passes forward through the layers (forward propagation).
 Errors are sent backward to adjust weights (backpropagation) during
training.
Used for:
 Image recognition,
 Speech recognition,
 Language translation,
 Medical diagnosis, and many more.
Backpropagation:
Backpropagation is a very important concept in training neural networks,
especially multilayer perceptrons (MLPs). It is the method that allows a neural
network to learn from its mistakes by adjusting its internal weights.
Without backpropagation, a neural network would not be able to improve and
give accurate results.
What is Backpropagation?
 After a neural network makes a prediction, it checks how far the output
is from the correct answer (this difference is called the error).
 Backpropagation is the process of sending this error backward through
the network to update the weights, so the network performs better next
time.
In simple words: Backpropagation teaches the network by showing it how wrong
it was and how to fix it.
Steps of Backpropagation
The backpropagation process mainly happens in four steps:
1. Forward Propagation
 Inputs are fed into the network.
 Each neuron processes the data and passes it to the next layer.
 Finally, the network gives an output (prediction).
2. Calculate the Error
 The network compares its output to the actual (target) output.
 The error is calculated using a formula like Mean Squared Error (MSE)
or another loss function.
3. Backward Propagation of Error
 The error is sent backward from the output layer toward the input layer.
 Each neuron’s contribution to the total error is calculated.
 This involves applying the chain rule from calculus to compute how
much each weight contributed to the error.
4. Update the Weights
 Based on how much each weight contributed to the error, the weights are
adjusted.
  Weights are usually updated using an optimization technique called
Gradient Descent.
 The goal is to reduce the error by adjusting weights in small steps in the
right direction.
Practice Question:
Given:
 Input:
x= [1,0]
 Weights:
w1=0.4, w2=0.5
 Bias:
Assume bias = 0 for simplicity.
 Activation Function:
Use simple identity function (i.e., output = input).
 Target Output:
t=1
 Learning Rate:
α=0.1
Step 1: Forward Propagation
Calculate the weighted sum:
𝑧 = (𝑥1 × 𝑤1) + (𝑥2 × 𝑤2)
Substituting the values:
z= (1×0.4) + (0×0.5) = 0.4+0 = 0.4
Since we are using the identity function, the output o is simply:
o=z=0.4
Forward output is 0.4
Step 2: Calculate Error
Now, calculate the error:
Error=Target−Output
Error=1−0.4=0.6
Error is 0.6.
Step 3: Backward Propagation
Now, update the weights using the rule:
New weight=Old weight + (Learning rate × Input ×Error)
Let’s update each weight:
For w1:
w1=0.4+(0.1×1×0.6)
w1=0.4+0.06=0.46
New w1=0.46
For w2:
Since x2=0, no change in w2:
w2=0.5+(0.1×0×0.6)
w2=0.5+0=0.5
w2 remains 0.5
Final Updated Weights:
Weight New Value
w1w_1w1 0.46
w2w_2w2 0.5