Chapter four

neural network
• Understanding the brain

Outline
• The Perceptron
• Artificial neural network
• Multilayer Perceptron
• Back propagation algorithm
• Training procedures
• Tuning the network size

2
 Understanding the brain

• The brain is a complex organ that processes

information and controls various functions.
• Neural networks aim to replicate the brain's
structure and function to build intelligent systems.
• By studying the brain, we can gain insights into
how neural networks can effectively process
information
2
Neural networks for parallel processing
• Neural networks are designed to perform parallel processing, similar to the
brain.
• Parallel processing allows for efficient computation and the ability to handle
multiple tasks simultaneously.
• Neural networks leverage interconnected nodes to distribute computational
tasks across the network.

3
 The Perceptron

• The Perceptron is a basic building block of artificial neural networks.

• It simulates a single neuron, receiving multiple inputs and producing an output.
• The Perceptron applies weights to the inputs and uses a threshold function to
make binary decisions.
• Training a Perceptron involves iteratively adjusting the weights to improve its
accuracy.
• During training, input patterns are presented to the Perceptron, and the output is
compared to the expected output.
• The difference between the predicted and expected outputs is used to update
the weights.
• This iterative process continues until the Perceptron achieves the desired level
of accuracy.
4
 Artificial Neural Network
• An artificial neural network consists of interconnected nodes, or perceptrons.
• These nodes are organized into layers, including an input layer, hidden layers,
and an output layer.
• The network processes input signals through the layers to generate output
predictions.
• By adjusting the weights and biases of the connections, the network can learn
and improve its predictions.

5
 Multilayer Perceptron
• The Multilayer Perceptron (MLP) is an artificial neural network with one or more
hidden layers.
• MLPs can learn complex patterns and solve nonlinear problems more
effectively.
• The hidden layers allow for the extraction of higher-level features and enhance
the network's modeling capability.

6
 Backpropagation algorithm
• Backpropagation is a widely used algorithm for training MLPs.
• It involves a forward pass, where input data is propagated through the network
to generate predictions.
• The error between the predictions and the expected outputs is then propagated
backward through the network.
• During the backward pass, the weights and biases are updated based on the
calculated error gradient using gradient descent optimization.

7
Nonlinear regression
Nonlinear Regression
• Nonlinear regression is a statistical method used to model relationships between a
dependent variable and one or more independent variables when the relationship is not
linear.
• Unlike linear regression, which assumes a straight-line relationship, nonlinear
regression can capture more complex, curved patterns in data.
• Unlike linear regression, it models data using a nonlinear function, such as
exponential, logarithmic, polynomial, or power functions.
Key Features:
1.Model Form: The relationship is expressed as a nonlinear function, such as
exponential, logarithmic, polynomial, or sigmoidal.
Example: y=Y=aebx+c
2.Parameters: Nonlinear models often involve parameters that are not linear
coefficients, requiring iterative optimization techniques (e.g., gradient descent)
for estimation.
3.Applications: Widely used in fields like biology (growth curves), economics
(demand curves), and engineering (system modeling).
Backpropagation algorithm …Cont’d
Two-Class Discrimination
• Backpropagation can also handle two-class discrimination problems, where the
network learns to classify inputs into two distinct classes.
• Through training, the network adjusts its parameters to optimize the decision
boundary between the classes.

10
Backpropagation algorithm …Cont’d
Multiclass Discrimination
• Backpropagation can be extended to solve multiclass discrimination problems,
where inputs belong to multiple classes.
• The network is trained to produce the correct class label for each input by
adjusting its weights and biases.
Multiple Hidden Layers
• Backpropagation can effectively train MLPs with multiple hidden layers.
• Multiple hidden layers allow for more complex representations and capture
intricate relationships in the data.

11
 Training procedures
• Training procedures are vital for optimizing the performance of neural networks.
• These procedures involve a series of steps that contribute to the network's
ability to learn from data.
Step 1: Data Preparation
• Collect relevant training data.
• Preprocess the data by cleaning and normalizing it.
• Split the data into training and validation sets.
Step 2: Network Architecture Design
• Design the structure of the neural network.
• Determine the number of layers and types of neurons.
• Establish connections between the layers.
12
 Training procedures …
Step 3: Initialization
• Initialize the network's parameters (weights and biases).
• Random initialization or using pre-trained values.
• Proper initialization aids in faster convergence.
Step 4: Forward Propagation
• Feed the training data through the network.
• Neurons calculate weighted sums of inputs.
• Apply activation functions to produce output.
Step 5: Loss Calculation
• Assess the difference between predicted and desired output.
• Use appropriate loss function (e.g., mean squared error, cross-entropy).
• Quantify network performance on the training data.
13
 Training procedures …
Step 6: Backpropagation
• Compute gradients of the loss function.
• Propagate error gradients from output to input layers.
• Adjust weights and biases based on gradient information.
Step 7: Gradient Descent
• Update network parameters using gradient descent algorithm.
• Adjust weights and biases in the opposite direction of gradients.
• Minimize the loss function with an appropriate learning rate.
Step 8: Repeat
• Repeat steps 4-7 for multiple epochs.
• Each epoch involves passing the entire training data through the network.
• Iteration allows the network to learn and refine its parameters.
14
 Training procedures …
Step 9: Validation
• Evaluate network performance on a separate validation set.
• Monitor generalization and avoid overfitting.
• Adjust hyperparameters (e.g., learning rate) based on validation results.
Step 10: Testing
• Assess the trained network's performance on an independent test set.
• Provides an unbiased evaluation of network performance.
• Indicates how well the network will perform in real-world scenarios.

15
 Improving convergence

• Overtraining, or overfitting, occurs when the network becomes too specialized

in the training data and fails to generalize well to new data.
• Techniques like early stopping, dropout, or regularization can help prevent
overtraining and improve generalization.
• Various techniques can improve the convergence of training, such as choosing
appropriate activation functions and weight initialization methods.
• Regularization techniques, like L1or L2 regularization, can also prevent
overfitting and enhance convergence.

16
 Structuring the network
• Structuring the network involves determining the number of layers and the
number of neurons in each layer.
• Network structure affects the network's capacity to learn and its ability to
generalize to new data.
Tuning the network size
• Tuning the network size involves finding the right balance between complexity
and performance.
• Increasing the network size can improve performance, but it may also increase
computational complexity and the risk of overfitting.
• A smaller network size may be more efficient but could limit the network's
capacity to learn complex patterns. 16
 Case Study - Image Recognition

• Neural Network Case Study: Image recognition in autonomous vehicles.

• Neural networks used for object detection in autonomous driving systems.
• NVIDIA DriveNet network trained on labeled image dataset.
• Network processes camera input, identifies objects like pedestrians and traffic
signs.
• Enables real-time decision-making and control for safer autonomous driving.
• Demonstrates power of deep learning in complex, real-world tasks.

18
Introduction to ML
“You can have data without information, but you
cannot have information without data.”
~ Daniel Keys Moran

19