Artificial Neural Network

Mr. Tapan Kumar Dey

Manipal University Jaipur

February 27, 2025

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 1 / 30
Outline

1 Artificial Neural Network

2 McCulloch Pitts Neuron

3 Perception

4 Multi-layerd perceptron

5 Feedfroward Neural Network

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 2 / 30
Introduction

An Artificial Neural Network (ANN) is a computational model inspired by

the way biological neural networks in the human brain process information.
It is a core technology in the field of machine learning and deep learning,
designed to solve complex problems by mimicking the structure and function
of neurons. Key Components:
Neurons (Nodes): Process inputs and generate outputs.
Layers:
Input Layer: Takes input data.
Hidden Layers: Perform computations.
Output Layer: Produces the final result.
Weights and Biases: Determine the importance of connections.
Activation Functions: Introduce non-linearity (e.g., Sigmoid, ReLU).

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 3 / 30
Artificial Neuron
The most fundamental unit of a deep neural network is called an
artificial neuron.
Why is it called a neuron ? Where does the inspiration come from ?
The inspiration comes from biology (more specifically, from the brain)
Biological neurons = neural cells = neural processing units.
We will first see what a biological neuron looks like.

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 4 / 30
Artificial Neuron

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 5 / 30
Biologcal Neuron

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 6 / 30
Biological Neuron

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 7 / 30
Biological Neuron

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 8 / 30
McCulloch Pitts Neuron

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 9 / 30
Properties of McCulloch-Pitts Neurons
Binary Model: Each neuron produces a binary output (0 or 1), depending
on whether its input surpasses a threshold.
Input and Weights: Inputs are binary (0 or 1), and each input is associated
with a weight that determines its significance.
Threshold Logic Unit: The neuron activates (outputs 1) if the weighted
sum of inputs exceeds a pre-defined threshold:
( P
1, if i wi xi ≥ Threshold
Output =
0, otherwise

Linearity: It is a linear model and cannot handle non-linear relationships.

Simple Logic Implementation: Capable of modeling basic logic gates
(AND, OR, NOT) by appropriately setting weights and thresholds.
Deterministic Model: Operates based on fixed rules; there is no
randomness.
Lack of Learning: Weights are static and do not adapt through training.
Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 10 / 30
Boolean Function Representation using McCulloch-Pitts
Neurons

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 11 / 30
Perception
Frank Rosenblatt, an American psychologist, proposed the classical
perception model (1958)
A more general computational model than McCulloch–Pitts neurons.
Main differences: Introduction of numerical weights for inputs and a
mechanism for learning these weights(weight adjustment (learning
algorithm).
Inputs are no longer limited to boolean values

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 12 / 30
Perception

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 13 / 30
Difference between McCulloch-Pitts Neurons and
Perceptron

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 14 / 30
Perceptron

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 15 / 30
A perceptron can only solve linearly separable problems
(e.g., AND, OR)

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 16 / 30
Perceptron Learning Algorithm

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 17 / 30
Limitation of perceptron

Non-Linearly Separable Problems:

Perceptron cannot solve problems where data is not linearly separable
(e.g., XOR problem).
Rigid Activation Function:
Uses a step function, which limits the ability to learn non-linear
relationships.
No Probabilistic Output:
Provides binary outputs (0 or 1) without confidence scores.
Single-Layer Constraint:
Cannot handle multi-class classification or hierarchical features without
extension to multi-layer perceptrons (MLPs).

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 18 / 30
Example
Suppose we want to predict whether a web user will click on an ad for a
refrigerator. given Four features:
Recently searched “refrigerator repair” (assume x1 =0).
Recently searched for ’refrigerator reviews’ (assume x2 = 1).
Recently bought a refrigerator (assume x3 =0).
Has clicked on any ad in the recent past. ((assume x3 =0)
These are all binary features (values can be either 0 or 1)

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 19 / 30
Introduction to MLP
A Multilayer Perceptron (MLP) is a type of artificial neural network that
consists of multiple layers of neurons, each fully connected to the neurons
in the subsequent layer. It is commonly used for supervised learning tasks
like classification and regression.
Example:

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 20 / 30
Comparision between Multilayered Perceptron and Feed
Forwarded Neural Network
Definition: A feed-forward neural network is one of the simplest types of artificial
neural networks devised. In this network, the information moves in only one direc-
tion: forward, from the input nodes, through the hidden nodes (if any), and to the
output nodes.
Feature Multilayer Feedforward
Perceptron (MLP) Neural Network
(FNN)
Definition A type of FNN with A broad class of
only fully connected neural networks
layers. with unidirectional
data flow (no
cycles).
Architecture Consists of fully Can include fully
connected layers connected,
only. convolutional,
pooling, and other
specialized layers.
Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 21 / 30
Comparision between Multilayered Perceptron and Feed
Forwarded Neural Network

Scope Subset of General category

Feedforward Neural encompassing
Networks. various architectures
(e.g., CNNs,
Transformers).
Data Input Works with Handles both
structured data structured and
(e.g., vectors, unstructured data
tabular data). (e.g., images, text,
audio).
Data Processing Requires data to be Can process
flattened into structured data
fixed-size vectors. directly (e.g., image
pixels, sequences).

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 22 / 30
Feedfroward Neural Network

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 23 / 30
Feedfroward Neural Network

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 24 / 30
Feedfroward Neural Network

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 25 / 30
Back propagation Algorithm

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 26 / 30
Frame Title
Network Architecture:
Input layer: 2 neurons (x1 , x2 ).
Hidden layer: 2 neurons (h1 , h2 ) with sigmoid activation.
Output layer: 1 neuron (y ) with sigmoid activation.
Given:
Weights:    
0.15 0.2 0.4
W1 = , W2 =
0.25 0.3 0.45

Biases:  
0.35
b1 = , b2 = 0.6
0.35

Input: x1 = 0.05, x2 = 0.1.

Target Output: ytarget = 0.01.
Learning Rate: η = 0.5.
Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 27 / 30
Example

Hidden Layer:

z1 = 0.15·0.05+0.2·0.1+0.35 = 0.3775, z2 = 0.25·0.05+0.3·0.1+0.35 = 0.3925

1
h1 = σ(z1 ) = ≈ 0.593, h2 = σ(z2 ) ≈ 0.596
1 + e −0.3775
Output Layer:

z3 = 0.4 · 0.593 + 0.45 · 0.596 + 0.6 = 1.10585

1
y = σ(z3 ) = ≈ 0.751
1+ e −1.10585
Compute Loss
1 1
L= (ytarget − y )2 = (0.01 − 0.751)2 ≈ 0.274
2 2

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 28 / 30
Exampel

Output Layer Error:

δ3 = (y − ytarget ) · σ ′ (z3 )

σ ′ (z3 ) = y · (1 − y ) =⇒ σ ′ (z3 ) ≈ 0.751 · (1 − 0.751) ≈ 0.187

δ3 = (0.751 − 0.01) · 0.187 ≈ 0.138
Hidden Layer Error:
δ1 = δ3 · W2,1 · σ ′ (z1 )
σ ′ (z1 ) = h1 · (1 − h1 ) =⇒ σ ′ (z1 ) ≈ 0.593 · (1 − 0.593) ≈ 0.242
δ1 = 0.138 · 0.4 · 0.242 ≈ 0.013
Similarly

δ2 = δ3 · W2,2 · σ ′ (z2 ) ≈ 0.138 · 0.45 · 0.241 ≈ 0.015

Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 29 / 30
Example
Gradients:

∇W2,1 = δ3 ·h1 = 0.138·0.593 ≈ 0.0818, ∇W2,2 = δ3 ·h2 = 0.138·0.596 ≈ 0.0822

∇W1,1 = δ1 ·x1 = 0.013·0.05 ≈ 0.00065, ∇W1,2 = δ2 ·x2 = 0.015·0.05 = 0.00075

Using gradient descent:

W2,1 ← W2,1 − η · ∇W2,1 =⇒ W2,1 ← 0.4 − 0.5 · 0.0818 ≈ 0.359

Similarly, W2,2 ≈ 0.409.

For the hidden layer:

W1,1 ← W1,1 − η · ∇W1,1 =⇒ W1,1 ← 0.15 − 0.5 · 0.00065 ≈ 0.1497

After one iteration:

Updated weights and biases are used for the next forward pass.
The loss reduces iteratively until convergence.
Mr. Tapan Kumar Dey (MUJ) Artificial Neural Network February 27, 2025 30 / 30