2Module 4 chapter 2

Artificial neural network

10.1 INTRODUCTION

Introduction to the Human Nervous System and Learning

1. The human nervous system helps us sense the world — we can see, hear, smell, and understand our
surroundings because of it.

2. It allows us to perceive who we are, what we are doing, and where we are.

3. It also helps us to remember, recognize, and connect things around us — this process is called learning.

4. The system is made up of functional units called nerve cells, commonly known as neurons.

Main Divisions of the Nervous System

5. The human nervous system is divided into two main parts:

o Central Nervous System (CNS)

o Peripheral Nervous System (PNS)

6. The Central Nervous System (CNS) consists of:

o The brain

o The spinal cord

7. The Peripheral Nervous System (PNS) includes:

o All the neurons inside and outside the CNS

o It connects the body parts to the CNS

Types of Neurons

8. There are three main types of neurons:

o Sensory Neurons
Collect information from different parts of the body (like skin, eyes, ears, etc.)
Send that information to the CNS

o Motor Neurons
Receive commands from the CNS or other neurons
Send signals to muscles or body parts to take action
 o Interneurons
Found only in the CNS
Connect one neuron to another neuron
Help in processing and relaying information within the CNS

Function of a Neuron

9. The basic function of a neuron is:

o To receive information

o To process that information

o To transmit it either to:

▪ Another neuron

▪ Or a body part (like a muscle or organ)

10.2 Basic Structure of a Biological Neuron

1. A biological neuron is the basic unit of the nervous system that processes and transmits information.

2. A typical neuron has four main parts:

o Dendrites

o Soma (cell body)

o Axon

o Synapse

Functions of Each Part

3. Soma (Cell Body)

o This is the main body of the neuron.

o It is also called the cell body.

o It processes the input signals received by the dendrites.

4. Dendrites

o These are branch-like structures attached to the soma.

o They receive input information from other neurons.

5. Axon

o It is a long, thin structure that connects a neuron to many other neurons.

 o A single neuron can be connected to around 10,000 neurons through its axon.

o The axon carries the processed signal from the soma to other neurons.

6. Synapse

o This is the connection point between two neurons.

o It passes the signal from one neuron to the next.

How a Neuron Fires and Passes Signals

7. A neuron gets fired when the total input it receives crosses a certain threshold value.

8. Once fired, it transmits signals to the next neuron through the synapse.

9. A synapse is activated by an electrical impulse called a spike.

10. These spikes carry the message from one neuron to another.

Neuron Connections and Networks

11. A single neuron can receive input from one or many other neurons.

12. These connections between neurons form a network.

13. The neural network processes the input information and gives an output response — this is how learning and
decisions happen in the brain.

10.3 Introduction to Artificial Neurons

1. Artificial neurons are inspired by biological neurons.

2. In artificial systems, these are often called nodes or simply neurons.

 Functionality of Artificial Neurons

3. An artificial neuron (or node) can:

o Receive one or more inputs

o Process the input information

o Pass the result to the next neuron or layer

Connections Between Artificial Neurons

4. Artificial neurons are connected to each other through connection links.

5. Each connection link between neurons is associated with a synaptic weight:

o This weight decides how strong or weak the input is.

o It plays a major role in the learning process of the model.

Structure of a Single Artificial Neuron

6. The structure of one artificial neuron includes:

o Inputs

o Weights for each input

o A processing function (usually a sum or activation function)

o An output to pass on to the next layer

10.3.1 Simple Model of an Artificial Neuron

Introduction to the First Artificial Neuron Model

1. The first mathematical model of a biological neuron was proposed by McCulloch & Pitts in 1943.

2. This model mimics how a biological neuron works using a simple two-step process.
Two Key Steps in the Model

3. Step 1: Receiving Weighted Inputs

o The neuron receives inputs from other neurons.

o Each input has an associated weight (importance level).

4. Step 2: Applying a Threshold/Activation Function

o The neuron calculates a weighted sum of all inputs.

o This sum is passed through an activation function.

o If the sum exceeds a certain threshold, the neuron gets fired (produces output).

Mathematical Function of a Neuron

5. The inputs are represented as:

o x1,x2,...,xn

6. Each input has an associated weight:

o w1,w2,...,wn

7. The weighted sum is calculated using the formula:

o Net-sum=∑xi⋅wi

Role of the Neuron

8. The neuron acts as a basic processing unit that:

o Takes input signals,

o Computes a net weighted sum,

o Decides to fire or not based on the activation function.

Boolean Function Representation by McCulloch & Pitts Model

1.AND Function:

o The neuron fires (outputs 1) only when all inputs are 1.

o Example:

▪ Input: 1, 1 → Output: 1

▪ Input: 1, 0 → Output: 0

2.OR Function:

o The neuron fires (outputs 1) if at least one input is 1.

o Example:

▪ Input: 1, 0 → Output: 1

▪ Input: 0, 0 → Output: 0

Limitations of McCulloch & Pitts Model

3.The weights and threshold values in this model are fixed and not learnable.

4.Because of this limitation:

o The model cannot adapt or learn from data.

o It cannot represent complex functions like XOR, which require learning or more layers.

10.3.2 Artificial Neural Network Structure

What is an Artificial Neural Network (ANN)?

1. An Artificial Neural Network (ANN) is a model that imitates the human brain.

2. It is designed to simulate human intelligence, especially the learning and decision-making process.

Structure of ANN
3. An ANN is represented as a directed graph:

o It has nodes (representing neurons)

o And edges (representing the connections between neurons)

4. The connections (edges) are called connection links and carry signals between neurons.

Layers in an ANN

5. Neurons in an ANN are arranged in a layered structure:

o Input Layer – Takes the input values (e.g., x1,x2,...,xn)

o Hidden Layer(s) – Performs processing and computations

o Output Layer – Gives the final result or prediction

6. Each connection between neurons in different layers has a synaptic weight (also called connection weight).

How Information is Processed

7. The input layer receives the input signals and passes them to the hidden layer neurons.

8. Hidden layer neurons:

o Compute the weighted sum of inputs: ∑(xi⋅wi)+bias\sum (x_i \cdot w_i) + \text{bias}∑(xi⋅wi)+bias

o Use this result in an activation function

9. If the weighted sum exceeds the threshold, the neuron fires and passes the output forward.

Activation Function Role

10. The activation function:

• Transforms the weighted sum into an output

• Often gives a non-linear output

• Helps the network learn complex patterns

Final Output

11. The output layer neuron gives the final result of the network based on all the processing done in previous
layers.
 What an activation function is
An activation function is a mathematical rule that each neuron in a neural network uses to convert its total input signal
into an output signal.

Why it’s needed

It decides whether a neuron should “fire” (pass its signal forward) or stay inactive based on the strength of its input.

How it normalizes outputs

By mapping raw input values into a fixed range—commonly between 0 and 1, or between –1 and +1—it keeps neuron
outputs stable and comparable.

Two main categories

Activation functions can be either linear or non-linear, depending on the shape of their input–output mapping.

Linear activation functions

• They produce an output directly proportional to the input (i.e., 𝑓(𝑥)=𝑘·𝑥).

• Their output range is unbounded unless explicitly clipped.

• They are mainly used in simple binary perceptrons for straightforward “yes/no” decisions.

Limitations of purely linear networks

A network built only from linear activations is effectively equivalent to a single-layer model, no matter how many layers
you stack, so it cannot learn complex patterns.

Non-linear activation functions

• They introduce curves (non-linearity) into the mapping, enabling the network to model intricate relationships.

• They typically squash inputs into bounded ranges like (0, 1) or (–1, 1).

• They are essential when working with high-dimensional or complex data such as audio signals, video frames,
and images.

Common non-linear examples

• Sigmoid (Logistic) Function: 𝑓(𝑥)=1/(1+e^(–x)), maps inputs smoothly into (0, 1).

• Tanh Function: 𝑓(𝑥)=tanh(x), maps inputs into (–1, 1) and is zero-centered.

• ReLU (Rectified Linear Unit): 𝑓(𝑥)=max(0, x), combines simplicity with fast training in deep networks.
 Perceptron and Learning Theory (10.12) – Easy Explanation

• The Perceptron was the first neural network model, designed by Frank Rosenblatt in 1958.

• It is a linear binary classifier used in supervised learning, meaning it learns from labeled data.

• Rosenblatt modified the McCulloch & Pitts Neuron model by combining it with the Hebbian learning rule.

• He introduced two key changes:

o Variable weight values (instead of fixed values).

o An extra input called bias to shift the activation threshold.

• He proposed that artificial neurons can learn the correct weights and thresholds from data on their own.

• He developed a learning algorithm that helps the model automatically adjust its weights during training.

Structure of the Perceptron Model

1. Inputs from other neurons – The neuron receives a set of input values.

2. Weights and bias – Each input is multiplied by a weight, and a bias is added.

3. Net sum – The model calculates the weighted sum of all inputs + bias.

4. Activation function – It processes the net sum to produce the final output (usually 0 or 1).

• So, the model receives inputs x1,x2,...,xn with corresponding weights w1,w2,..., wn,
and a bias term is added.

• The Net-sum function computes the total as:

Net sum=(w1⋅x1)+(w2⋅x2)+…+(wn⋅xn)+bias

• This total is then passed through an activation function to determine the output.
10.4.1 XOR Problem

• A Perceptron model can solve only those Boolean functions that are linearly separable.

• In 1969, Minsky and Papert pointed out a limitation in perceptrons using the XOR problem as an example.

Understanding XOR

• The XOR (Exclusive OR) function gives:

o 1 if the two input values are different

o 0 if the two input values are the same

XOR Truth Table:

Input A Input B XOR Output

0 0 0

0 1 1

1 0 1

1 1 0

• This function cannot be separated by a straight line, which means it is not linearly separable.

• As a result, the single-layer perceptron fails to solve the XOR problem.

Birth of Multi-Layer Perceptron (MLP)

• The failure of the single-layer perceptron with XOR led to the development of the Multi-Layer Perceptron
(MLP).

• Initially, MLPs were not successful due to the absence of a proper learning algorithm.

Introduction of Backpropagation

• In 1974, Werbos introduced the backpropagation algorithm for a 3-layered perceptron.

• In 1986, Rumelhart and McClelland developed a general backpropagation algorithm for multi-layered
perceptrons.

• This algorithm allowed neural networks to learn from errors by adjusting weights layer by layer.

Modern Impact

• With backpropagation, Artificial Neural Networks (ANNs) and Deep Neural Networks (DNNs) started solving
complex, non-linear problems.

• The MLP model thus became a powerful tool, capable of handling non-linearly separable problems like XOR
and many others.

10.4.2 Delta Learning Rule and Gradient Descent

Basic Idea of Learning in Neural Networks

• Neural networks learn by adjusting their weights to reduce the error between:

o The desired (target) output

o The actual (estimated) output

 Delta Learning Rule

• Also called the Widrow-Hoff rule or Adaline rule.

• It is a supervised learning method where weights are updated based on the error (delta).

• The goal is to minimize the cost function, which measures how far the network's output is from the desired
output.

Cost Function / Error Function

• The cost (or error) function shows how well the model is performing.

• It is continuous and differentiable, so we can apply calculus-based optimization.

• The training error is calculated as:

Training Error=1/2∑d∈T(Od−O^d)2

Where:

• T = Training dataset

• Od = Desired output for training instance d

• O^d= Estimated (actual) output from the network

Gradient Descent Principle

• It is an optimization technique to minimize the cost function.

• Works by moving in the negative direction of the gradient (slope) of the cost function.

• The amount we move in each step depends on:

o The learning rate (how fast we move)

o The slope of the gradient (how steep the curve is)

Connection to Backpropagation

• Gradient Descent is the core principle behind the Backpropagation Algorithm.

• Backpropagation uses delta learning + gradient descent to train Multi-Layer Perceptrons (MLPs).

10.5 Types of Artificial Neural Networks (ANNs)

• Artificial Neural Networks (ANNs) are made up of many neurons arranged in layers.

• Different types of ANNs exist based on:

o Network structure
 o Activation function used

o Learning rules followed

• Every ANN has three types of layers:

Input Layer – Receives the input data.

Hidden Layer(s) – Processes the data (can be one or more).

Output Layer – Produces the final result.

• A typical ANN contains:

o One input layer

o One output layer

o Zero or more hidden layers

10.5.1 Feed Forward Neural Network

• This is the simplest type of neural network.

• Neurons are arranged in layers, and data flows only in the forward direction (no loops).

• It may or may not have a hidden layer.

• No backpropagation is involved in this model.

Types:based on hidden layers

• Single-layered (no hidden layer)

• Multi-layered (one or more hidden layers)

Key Features:

• Simple to design

• Easy to maintain

• Fast in execution

• Not suitable for complex learning tasks

Applications:

• Simple classification tasks

• Basic image processing

10.5.2 Fully Connected Neural Network

Fully connected neural networks are the ones in which all the neurons in a layer are connected to all other neurons in
the next layer.

10.5.3 Multi-Layer Perceptron (MLP)

• MLP is a type of ANN with:

o One input layer

o One output layer

o One or more hidden layers

Key Features:

• Fully connected: Every neuron in one layer connects to all neurons in the next layer.

• Information flows in both directions:

o Forward direction: Inputs are multiplied by weights → sent to activation function → output goes to
next layer.

o Backward direction: If output is wrong, error is backpropagated to adjust weights and biases.

Learning Process:

• The network learns from training data by adjusting weights and biases using the backpropagation algorithm.

Applications:

• Used in deep learning

• Useful for:

o Complex classification

o Speech recognition
 o Medical diagnosis

o Forecasting

Limitations:

• More complex than feedforward networks

• Slower in processing due to backpropagation

10.5.4 Feedback Neural Network – Simplified

• These networks have feedback connections between neurons.

• Information flows in both directions:

The output signals can be sent back to the neurons in the same layer or to the neurons in the preceding layers.

o Forward (like normal flow)

o Backward (feedback) to the same layer or previous layers

Key Feature:

• Because of feedback, the network becomes more dynamic during training.

10.10 ADVANTAGES AND DISADVANTAGES OF ANN

Advantages of ANN

1. Solves Complex Problems

o Can handle non-linear processes effectively.

2. Learns Like Humans

o Can learn and recognize complex patterns just like how humans solve problems.

3. Faster Predictions with Parallel Processing

o Has parallel processing ability, which helps in making predictions quickly.

4. Handles Inadequate or Noisy Data

o Can still work even when some data is missing or noisy.

5. Scalable and Efficient

o Scales well with large datasets and often performs better than other learning models.

Limitations of ANN

1. Requires High Computational Power

o Needs processors with parallel processing capability, especially when training for many epochs.

o Each node and layer needs good CPU/GPU performance, which is tough for very large networks.

2. Acts Like a Black Box

o It is very hard to understand how the network makes decisions.

o Inner workings and learned representations are not easily interpretable.

3. Complicated to Develop

o Designing and modeling an ANN is complex and time-consuming.

4. Needs Large Datasets

o Performs poorly on small datasets.

o Requires more data compared to traditional machine learning algorithms.

5. Computationally Expensive

o Needs more memory, processing power, and time than traditional learning techniques.

10.11 Challenges of Artificial Neural Networks (ANNs)

1. Training is Difficult

o Training a neural network is one of the most challenging tasks in ANN development.

2. Overfitting or Underfitting
 o If the training dataset is not proper, the model may:

▪ Overfit (learn too much from training data and fail on new data)

▪ Underfit (fail to learn patterns properly)

3. Poor Generalization to Real-world Data

o Models trained on simulated or limited data may not work well on real-world applications.

4. Large Training Data Required

o ANNs need a lot of data to become robust and reliable, especially for real-time applications.

5. Difficult to Optimize Weights and Biases

o Finding the right weights and biases is hard.

o It’s tough to compute an optimal model.