UNIT – 4

Artificial Neural Networks

Introduction to Artificial Neural Networks (ANNs)
Definition:
Artificial Neural Networks (ANNs) are computational models inspired by
the biological neural networks in the human brain. They are used in
machine learning to recognize patterns, make predictions, and solve
complex problems.
Why ANNs?
 ANNs can model complex, non-linear relationships in data.
 They are capable of learning from data without being explicitly
programmed.
 They generalize well to unseen data when trained properly.
Example:
 Image Recognition: ANNs are used in facial recognition systems
(e.g., Facebook's photo tagging). The network learns to identify
faces by analyzing thousands of labeled images.
 Speech Recognition: Virtual assistants like Siri and Alexa use
ANNs to convert spoken language into text.
What is a Neural Network?
Neural networks are machine learning models that mimic the complex
functions of the human brain. These models consist of interconnected
nodes or neurons that process data, learn patterns, and enable tasks such
as pattern recognition and decision-making.
In this article, we will explore the fundamentals of neural networks, their
architecture, how they work, and their applications in various fields.
Understanding neural networks is essential for anyone interested in the
advancements of artificial intelligence.
Understanding Neural Networks in Deep Learning
Neural networks are capable of learning and identifying patterns directly
from data without pre-defined rules. These networks are built from several
key components:
1. Neurons: The basic units that receive inputs, each neuron is
governed by a threshold and an activation function.
 2. Connections: Links between neurons that carry information,
regulated by weights and biases.
3. Weights and Biases: These parameters determine the strength
and influence of connections.
4. Propagation Functions: Mechanisms that help process and
transfer data across layers of neurons.
5. Learning Rule: The method that adjusts weights and biases over
time to improve accuracy.
Learning in neural networks follows a structured, three-stage
process:
1. Input Computation: Data is fed into the network.
2. Output Generation: Based on the current parameters, the network
generates an output.
3. Iterative Refinement: The network refines its output by adjusting
weights and biases, gradually improving its performance on diverse
tasks.
In an adaptive learning environment:
 The neural network is exposed to a simulated scenario or dataset.
 Parameters such as weights and biases are updated in response to
new data or conditions.
 With each adjustment, the network’s response evolves, allowing it to
adapt effectively to different tasks or environments
The image illustrates the analogy between a biological neuron and an
artificial neuron, showing how inputs are received and processed to
produce outputs in both systems.

Importance of Neural Networks

Neural networks are pivotal in identifying complex patterns, solving
intricate challenges, and adapting to dynamic environments. Their ability
to learn from vast amounts of data is transformative, impacting
technologies like natural language processing, self-driving vehicles,
and automated decision-making.
Neural networks streamline processes, increase efficiency, and support
decision-making across various industries. As a backbone of artificial
intelligence, they continue to drive innovation, shaping the future of
technology.
Layers in Neural Network Architecture
1. Input Layer: This is where the network receives its input data. Each
input neuron in the layer corresponds to a feature in the input data.
2. Hidden Layers: These layers perform most of the computational
heavy lifting. A neural network can have one or multiple hidden
layers. Each layer consists of units (neurons) that transform the
inputs into something that the output layer can use.
3. Output Layer: The final layer produces the output of the model.
The format of these outputs varies depending on the specific task
(e.g., classification, regression).
How Neurons Process Data in a Neural Network
In a neural network, input data is passed through multiple layers,
including one or more hidden layers. Each neuron in these hidden layers
performs several operations, transforming the input into a usable output.
1. Input Layer: The input layer contains 3 nodes that indicates the
presence of each keyword.
2. Hidden Layer
 The input data is passed through one or more hidden layers.
 Each neuron in the hidden layer performs the following operations

Weighted Sum: Each input is multiplied by a corresponding weight

assigned to the connection. For example, if the weights from the
input layer to the hidden layer neurons are as follows:
o Weights for Neuron H1: [0.5, -0.2, 0.3]
o Weights for Neuron H2: [0.4, 0.1, -0.5]

Calculate Weighted Input:

o For Neuron H1:
o Calculation=(1×0.5)+(0×−0.2)+(1×0.3)=0.5+0+
0.3=0.8Calculation=(1×0.5)+(0×−0.2)+(1×0.3)=
0.5+0+0.3=0.8
o For Neuron H2:
 o Calculation=(1×0.4)+(0×0.1)+(1×−0.5)=0.4+0−
0.5=−0.1Calculation=(1×0.4)+(0×0.1)+(1×−0.5)
=0.4+0−0.5=−0.1
o Activation Function: The result is passed through an
activation function (e.g., ReLU or sigmoid) to introduce
non-linearity.
o For H1, applying
ReLU: ReLU(0.8)=0.8ReLU(0.8)=0.8
o For H2, applying
ReLU: ReLU(−0.1)=0ReLU(−0.1)=0
3. Output Layer
 The activated outputs from the hidden layer are passed to the
output neuron.
 The output neuron receives the values from the hidden layer
neurons and computes the final prediction using weights:
o Suppose the output weights from hidden layer to output
neuron are [0.7, 0.2].
o Calculation:
o Input=(0.8×0.7)+(0×0.2)=0.56+0=0.56Input=(0.8×0.7
)+(0×0.2)=0.56+0=0.56
o Final Activation: The output is passed through a sigmoid
activation function to obtain a probability:
o σ(0.56)≈0.636σ(0.56)≈0.636
4. Final Classification
 The output value of approximately 0.636 indicates the probability of
the email being spam.
 Since this value is greater than 0.5, the neural network classifies the
email as spam (1).
Appropriate Problems for Neural Network Learning
Characteristics of Problems Suitable for ANNs:
1. Non-Linearity: Problems where the relationship between input and
output is complex and non-linear.
2. Large Datasets: ANNs require large amounts of data to learn
effectively.
3. Pattern Recognition: Problems involving pattern recognition, such as
image, speech, or text analysis.
4. Real-Valued Inputs: ANNs work well with real-valued inputs (e.g.,
sensor data, pixel values).
Real-Time Examples:
1. Image Classification:
o Problem: Classify images into categories (e.g., cats vs. dogs).
o Why ANN?: ANNs can learn complex features (e.g., edges,
shapes) from raw pixel data.
o Example: Google Photos uses ANNs to categorize photos.
2. Speech-to-Text Conversion:
o Problem: Convert spoken language into written text.
o Why ANN?: ANNs can model temporal dependencies in audio
data.
o Example: Google Translate uses ANNs for real-time speech
translation.
3. Autonomous Driving:
 o Problem: Make driving decisions based on sensor data (e.g.,
camera, LIDAR).
o Why ANN?: ANNs can process high-dimensional data and
learn to detect objects, lanes, and obstacles.
o Example: Tesla's Autopilot system uses ANNs for object
detection and path planning.
4. Fraud Detection:
o Problem: Detect fraudulent transactions in real-time.
o Why ANN?: ANNs can learn patterns in transaction data and
flag anomalies.
o Example: Banks use ANNs to detect credit card fraud.

5. Perceptrons: A perceptron is one of the simplest types of

artificial neural networks and serves as the building block for more
complex neural networks. It is a binary classifier that can decide
whether an input belongs to one class or another. The perceptron
algorithm was introduced by Frank Rosenblatt in 1957 and is the
foundation of modern deep learning.

A perceptron is a single-layer neural network that consists of:

1. Input Layer: Receives input features.

2. Weights and Bias: Each input feature is associated with a weight,
and a bias term is added to the weighted sum.
3. Activation Function: A step function that determines the output
based on the weighted sum.
4. Output Layer: Produces a binary output (0 or 1).

Mathematical Representation
 The perceptron computes the weighted sum of the inputs and applies
an activation function to produce the output.

How Does a Perceptron Learn?

The perceptron learns by adjusting its weights and bias to minimize

classification errors. The learning process involves the following steps:
Example: Spam Email Classification

Problem: Classify emails as spam (1) or not spam (0) based on features
like the presence of specific keywords.

Features:

 x1: Number of times the word "free" appears.

 x2: Number of times the word "money" appears
Multilayer networks and the back propagation algorithm

Multilayer networks, also known as Multi-Layer Perceptrons (MLPs), are a

class of feedforward artificial neural networks that consist of multiple
layers of nodes (neurons). These networks are capable of learning
complex, non-linear patterns and are widely used in tasks like
classification, regression, and pattern recognition. The backpropagation
algorithm is the key method used to train these networks by minimizing
the error between predicted and actual outputs.
Structure of Multilayer Networks

A multilayer network consists of:

1. Input Layer: Receives the input features.

2. Hidden Layers: One or more layers of neurons that process the
input data.
3. Output Layer: Produces the final output (e.g., class labels or
continuous values).

Each neuron in a layer is connected to every neuron in the next layer via
weighted connections. The network learns by adjusting these weights
during training.

Backpropagation Algorithm
Backpropagation is the process of propagating the error backward through
the network to update the weights and biases. It involves the following
steps:

Remarks on the back propagation algorithm

The backpropagation algorithm is a cornerstone of training artificial neural

networks, particularly Multi-Layer Perceptrons (MLPs). It is a supervised
learning technique that adjusts the weights and biases of a neural
network to minimize the error between the predicted output and the
actual output. Below are detailed remarks on the backpropagation
algorithm, including its strengths, limitations, and practical considerations.

1. How Backpropagation Works

Backpropagation consists of two main phases:

1. Forward Propagation:
o Input data is passed through the network to compute the
output.
o The loss (error) is calculated by comparing the predicted
output with the actual output.
2. Backward Propagation:
 o The error is propagated backward through the network.
o The gradients of the loss with respect to the weights and
biases are computed.
o The weights and biases are updated using gradient descent.

2. Strengths of Backpropagation

1. Efficiency:
o Backpropagation efficiently computes gradients using the
chain rule of calculus, making it suitable for training large
neural networks.
2. Versatility:
o It can be applied to various network architectures, including
feedforward networks, convolutional neural networks (CNNs),
and recurrent neural networks (RNNs).
3. Scalability:
o Backpropagation scales well with the size of the dataset and
the complexity of the model.
4. Automatic Feature Learning:
o It enables neural networks to automatically learn hierarchical
features from raw data, reducing the need for manual feature
engineering.
5. Wide Applicability:
o It is used in a wide range of applications, including image
recognition, natural language processing, and speech
recognition.

3. Limitations of Backpropagation

1. Vanishing and Exploding Gradients:

o In deep networks, gradients can become very small
(vanishing) or very large (exploding), making training difficult.
o Solutions: Use activation functions like ReLU, weight
initialization techniques, and normalization methods (e.g.,
batch normalization).
2. Local Minima:
o The loss landscape may have many local minima, and
backpropagation can get stuck in suboptimal solutions.
o Solutions: Use optimization techniques like momentum, Adam,
or simulated annealing.
3. Computational Cost:
o Backpropagation requires significant computational resources,
especially for large datasets and deep networks.
o Solutions: Use GPUs or TPUs for acceleration.
4. Overfitting:
o Neural networks trained with backpropagation can overfit the
training data, especially if the dataset is small.
o Solutions: Use regularization techniques like dropout, weight
decay, or data augmentation.
 5. Sensitivity to Hyperparameters:
o The performance of backpropagation depends heavily on
hyperparameters like learning rate, batch size, and network
architecture.
o Solutions: Use grid search, random search, or Bayesian
optimization for hyperparameter tuning.

4. Practical Considerations

1. Learning Rate:
o The learning rate determines the size of the weight updates.
o Too high: The algorithm may diverge.
o Too low: The algorithm may converge slowly.
o Solution: Use learning rate schedules or adaptive learning rate
methods (e.g., Adam).
2. Batch Size:
o The batch size affects the stability and speed of training.
o Smaller batches: Noisier gradients but faster convergence.
o Larger batches: Smoother gradients but slower convergence.
o Solution: Experiment with different batch sizes.
3. Weight Initialization:
o Proper weight initialization is crucial for effective training.
o Poor initialization can lead to vanishing or exploding gradients.
o Solution: Use techniques like Xavier or He initialization.
4. Regularization:
o Regularization techniques prevent overfitting and improve
generalization.
o Common methods: L2 regularization (weight decay), dropout,
and early stopping.
5. Monitoring Training:
o Monitor training and validation loss to detect overfitting or
underfitting.
o Use techniques like learning curves and validation checks.

Scenario: A company wants to implement a face recognition

system for employee attendance.

Steps:

1. Data Collection:
o Collect facial images of all employees.
2. Data Preprocessing:
o Resize images to 64x64 pixels and convert to
grayscale.
o Normalize pixel values to [0, 1].
3. Model Training:
o Train the ANN model on the preprocessed dataset.
4. Deployment:
o Deploy the trained model to a real-time system.
 o Use a camera to capture facial images and predict the
employee's identity.
5. Monitoring:
o Continuously monitor the system's performance and retrain
the model as needed.

Face recognition using ANNs is a powerful application of machine

learning. By following the steps outlined above, you can build a
robust face recognition system that can be deployed in real-time
scenarios. The key to success lies in careful data preparation,
thoughtful model design, and thorough evaluation.

Advanced topics in Artificial Neural Networks

Advanced topics in artificial neural networks (ANNs) in machine learning

encompass concepts like deep learning architectures (CNNs, RNNs),
reinforcement learning, and more, which build upon the foundations of
basic ANNs to handle complex data and tasks.