Syllabus:

Supervised Learning: Perceptron learning, Single layer/multilayer, Adaline,

Madaline, Back propagation network, RBFN, Application of Neural network in
forecasting, data compression and image compression.

 Supervised Learning : Ans in first unit

 Perceptron Learning

 Perceptron learning is a supervised learning algorithm used for

binary classification tasks. It is based on the perceptron model,
which is a simple artificial neuron that makes decisions based on
input data.
 The primary objective of the perceptron learning algorithm is to
minimize prediction errors. This is achieved by adjusting the weights
with each training example. The perceptron learns to classify the
data correctly by updating its weights until it can accurately separate
the data into two classes.
In the perceptron, x1,x2,…,xnx_1, x_2, \dots, x_nx1,x2,…,xn are the
features, and w1,w2,…,wnw_1, w_2, \dots, w_nw1,w2,…,wn are the
weights. w0w_0w0 is the bias term. The formula to calculate the weighted
sum is:

z = w0x0 + w1x1 + w2x2 ……+ wnxn

The activation function is then used to transform this weighted sum zzz
into a value within a specific range. The activation function can be any
function, such as a step function (which gives an output of -1 or 1), a
sigmoid function (which gives the probability of each class), or even mean
squared error.

Since the perceptron is a binary classifier, the output will always be 0 or 1

(or -1 or 1, depending on the activation function).

Perceptron Learning Algorithm

 Types of Perceptron

Single-Layer Perceptron:
 This is the basic form of the perceptron, consisting of one layer of
neurons (or units).
 It has an input layer, a set of weights, an activation function, and an
output layer. It is used for simple binary classification tasks where data is
linearly separable.
 Input Layer: Receives raw data (features) and passes it to the next layer.
 Hidden Layer: Processes input data using weights and activation
functions, capturing patterns or relationships in the data.
 Output Layer: Produces the final result (classification or prediction)
based on the processed data.


Multi-Layer Perceptron (MLP):

 This is a more advanced version that includes multiple layers: an input
layer, one or more hidden layers, and an output layer.
  MLP can handle complex classification tasks, including non-linearly
separable data. The multiple layers allow it to learn more complex
patterns.
 Input Layer: Receives raw data (features) and passes it to the next layer.
 Hidden Layer: Processes input data using weights and activation
functions, capturing patterns or relationships in the data.
 Output Layer: Produces the final result (classification or prediction)
based on the processed data.

Applications of Perceptron:

1. Image Recognition: Classifies images into categories (e.g., identifying

objects in photos).
2. Speech Recognition: Converts spoken words into text.
3. Text Classification: Categorizes text (e.g., spam detection).
4. Medical Diagnosis: Helps in disease prediction based on medical data.
5. Finance: Used for stock market prediction and fraud detection.

Advantages:

1. Simple to Understand: Easy to implement and comprehend.

2. Fast Computation: Works efficiently for small datasets.
3. Binary Classification: Effective for clear, binary classification tasks.
4. Adaptable: Can learn from errors and improve over time.
5. Linear Separation: Excellent for linearly separable data.

Disadvantages:

1. Limited to Linear Problems: Struggles with non-linear data.

2. Sensitive to Input Data: Performance heavily depends on feature
scaling.
3. Slow Convergence: May take time to converge with complex data.
4. No Memory: Can't remember past patterns outside training data.
5. Overfitting Risk: May overfit if the model is too complex for the data.

 Adaptive Linear Neuron (Adaline)

Adaptive Linear Neuron (Adaline) is a type of neural network similar to the

perceptron but with a key difference in its learning algorithm. It uses a linear
activation function instead of a step function, which allows it to handle more
complex learning tasks.

Key Features of Adaline:

1. Linear Activation Function: Unlike the perceptron, which uses a binary

step function, Adaline uses a continuous linear activation function. This
means the output is a real number instead of just 0 or 1.
2. Error Calculation: Adaline uses the Mean Squared Error (MSE)
between the predicted and actual outputs as its error function. The goal is
to minimize this error across all training samples.
3. Learning Rule: The weights are updated using the Delta Rule (also
called the Least Mean Squares, or LMS rule). The weight update is:

w i =w i +η⋅(y− y ^ )⋅x i

where:

o η is the learning rate.

o y is the target output.
o y^ is the predicted output.
o Xi is the input feature.
4. Convergence: Adaline is guaranteed to converge in situations where the
data is linearly separable, but it can also handle some non-linearly
separable cases better than the perceptron.

Architecture of Adaline

First, calculate the net input to your Adaline network then apply the activation
function to its output then compare it with the original output if both the equal,
then give the output else send an error back to the network and update the
weight according to the error which is calculated by the delta learning rule. i.e

w_{i}(new)=w_{i}(old) + α(t-y_{in})x_{i} {gfg}

where w_{i} ,y_{in} and t are the weight, predicted output, and true value
respectively.

Adaline Algorithm:
Applications of Adaline:

1. Pattern Recognition: Used for binary classification tasks like identifying

handwritten digits.
2. Signal Processing: Helps in noise reduction by adapting to input data.
3. Control Systems: Applied in adaptive control systems for real-time
decision making.
4. Medical Diagnosis: Used for predicting disease outcomes based on
clinical data.
5. Financial Forecasting: Helps predict stock prices or trends based on
historical data.

Advantages of Adaline:

1. Continuous Output: Provides real-valued predictions, useful for

regression tasks.
2. Efficient Learning: Converges faster due to smooth error surface.
3. Simple to Implement: Easy to understand and implement.
4. Stable Training: Uses gradient descent, providing stable weight updates.
5. Versatile: Can be used for both classification and regression tasks.

Disadvantages of Adaline:

1. Linear Limitation: Struggles with non-linear problems.

2. Sensitive to Input Scaling: Performance can be affected if features are
not scaled properly.
3. Slow Convergence for Complex Data: May take longer for complex
tasks with many features.
4. No Memory of Past Data: Can't remember previous examples once
weights are updated.
5. Prone to Overfitting: May overfit if data is too noisy or if model
complexity is too high.
  Multiple Adaptive Linear Neurons (Madaline)

Multiple Adaptive Linear Neurons (Madaline) is an extension of the Adaline

model. While Adaline uses a single neuron, Madaline uses multiple Adaline
neurons to handle more complex classification tasks, especially for non-linearly
separable data.

Key Features of Madaline:

1. Multiple Neurons: Unlike Adaline, which uses only one neuron,

Madaline employs multiple Adaline units (neurons) to form a network.
2. Layered Structure: It typically consists of an input layer, a hidden layer
with multiple Adaline neurons, and an output layer. The neurons in the
hidden layer work together to capture more complex relationships in the
data.
3. Activation Functions: Like Adaline, Madaline uses linear activation
functions for the neurons. However, each unit in the hidden layer can be
seen as processing the data with its weights and producing outputs that
help in final decision-making.
4. Learning Algorithm: Madaline uses a combination of:
o Adaline's Delta Rule for weight updates within individual
neurons.
o A unique Madaline rule for the connections between neurons,
allowing for more efficient learning when dealing with non-linear
separability.
5. Handling Non-linear Data: By using multiple neurons in the hidden
layer, Madaline can handle data that is not linearly separable, which is a
major improvement over Adaline.

Architecture of Madaline
 There are three types of a layer present in Madaline First input layer contains
all the input neurons, the Second hidden layer consists of an adaline layer, and
weights between the input and hidden layers are adjustable and the third layer is
the output layer the weights between hidden and output layer is fixed they are
not adjustable.

Madaline Algorithm:

1. Initialization: Start by randomly initializing the weights of the neurons.

2. Input Data: Prepare your training dataset with input features and target
outputs.

3. Feedforward Process:

- For each input sample, calculate the output of each neuron by taking the
weighted sum of inputs and applying an activation function.

- Pass the output to the next layer until you reach the final output layer.
4. Error Calculation: Compare the predicted output with the actual target output.

5. Weight Adjustment: If there is an error, update the weights using:

- New weight = Old weight + Learning rate * (Target output - Predicted

output) * Input

6. Iteration: Repeat the feedforward and weight adjustment steps for several
epochs until the weights stabilize or you reach a desired accuracy.

7. Final Prediction: After training, use the network to predict outputs for new
data by following the feedforward process.

This algorithm helps the Madaline network learn patterns in the data for
classification tasks

Advantages of Madaline:

1. Handles Non-linearly Separable Data: Can solve more complex

problems than a single Adaline neuron.
2. Multiple Neurons for Complex Tasks: More neurons mean it can learn
and adapt to more complex patterns.
3. Better Generalization: With multiple neurons, it can generalize better to
new, unseen data.
4. Flexible Structure: More layers and neurons offer flexibility in adapting
to a wider variety of problems.
5. Improved Convergence: Multiple neurons can improve learning speed
and stability in more complex environments.

Disadvantages of Madaline:

1. Computationally Intensive: More neurons and layers require more

computation and memory.
2. Not Completely Non-linear: Though it works better than Adaline for
complex data, it still faces challenges with highly non-linear problems.
 3. Sensitive to Initialization: Like other neural networks, it can be sensitive
to initial weights and requires careful tuning.
4. Prone to Overfitting: If not properly regularized, it may overfit to the
training data, especially with many neurons.
5. More Complex to Train: The training process is more complex and
might require more sophisticated optimization techniques compared to
Adaline.

Applications of Madaline:

1. Pattern Recognition: Used in image or speech recognition tasks where

data is complex and not linearly separable.
2. Signal Processing: Helps in detecting patterns or features in noisy
signals.
3. Medical Diagnosis: For classifying medical data, like identifying
diseases from patient records.
4. Time Series Prediction: Can be used to predict future values in time-
dependent data.
5. Control Systems: For adaptive control systems, where the model learns
to adapt over time based on input data.

 Back Propagation Network:

Backpropagation is a supervised learning algorithm used to train artificial

neural networks. It works by calculating the error between the predicted
output and the actual output, then adjusting the weights of the network in
the opposite direction of the error to minimize it. This process is repeated
for multiple iterations to improve the model's accuracy.

Architecture:
 Key Features of Backpropagation:

1. Supervised Learning: Backpropagation is a supervised learning

algorithm, meaning it requires labeled data for training.
2. Error Minimization: It aims to minimize the error between the predicted
output and the actual target output by adjusting the weights of the
network.
3. Multi-layer Network: Typically used in networks with multiple layers
(input, hidden, and output layers), making it suitable for more complex
patterns and non-linear problems.

How Backpropagation Works:

1. Forward Propagation:
o Input data is passed through the network, layer by layer, until the
final output is produced.
o Each neuron in the network performs a weighted sum of inputs,
applies an activation function, and passes the output to the next
layer.
2. Calculate Error:
o The error is calculated as the difference between the predicted
output and the actual target output.
3. Backward Propagation:
o The error is propagated backward from the output layer to the
hidden layers, adjusting weights to reduce the error.
o This is done using the gradient descent optimization algorithm to
minimize the error.
4. Weight Update:
o Weights are updated in the direction that reduces the error, based
on the gradient of the error with respect to the weights.
5. Repeat:
o The process is repeated for multiple iterations (epochs) until the
error is minimized or converges.

Backpropagation Algorithm Steps:

1. Initialize weights and biases randomly.

2. For each training example:
o Perform forward propagation: Calculate output at each layer.
o Calculate error: Compute the difference between predicted and
actual output.
Compute the error: actual output - predicted output.
o Backward propagation: Calculate the gradient of the error and
propagate it backward to update weights.
3. Repeat until the error converges to a minimum value or a stopping
criterion is met.

Applications of Backpropagation:

1. Image Recognition: Used in convolutional neural networks (CNNs) for

tasks like face or object recognition.
2. Speech Recognition: Employed in models that convert spoken language
into text.
3. Natural Language Processing: Used in recurrent neural networks
(RNNs) for tasks like machine translation and sentiment analysis.
4. Medical Diagnosis: Helps in predicting diseases based on patient data.
5. Financial Predictions: Applied in stock price prediction and algorithmic
trading.

Advantages:

1. Efficient Training: Allows for the efficient training of multi-layer

networks.
2. Flexibility: Can be applied to a wide range of problems, including
classification and regression.
3. Scalability: Works well with large datasets.

Disadvantages:
1. Slow Convergence: It may take a long time to train, especially with deep
networks.
2. Prone to Overfitting: Can overfit the training data, especially if not
regularized properly.
3. Requires Large Datasets: Needs a substantial amount of labeled data for
effective training.

 Radial Basis Function (RBF) Network:

 A Radial Basis Function Network (RBF Network) is a type of

artificial neural network that uses radial basis functions as
activation functions. It is primarily used for classification,
regression, and function approximation tasks.
 The key concept behind RBF networks is that the output of a
neuron is based on the distance between the input and a central
point, which allows the network to perform well on tasks that
require distinguishing between data points that are similar or close
to each other.
 Key Components:

1. Input Layer: Accepts the input features.

2. Hidden Layer: Contains neurons that use radial basis functions (e.g.,
Gaussian functions) to compute the output based on the distance between
input and center.
3. Output Layer: Produces the final output using a linear combination of
the hidden layer outputs.

How RBF Works:

1. Distance Measure: Each hidden neuron computes a distance between the

input and a center (reference point) and applies a radial basis function
(e.g., Gaussian function) to this distance.
2. Activation: The output of each hidden unit is determined by how close
the input is to its center. The closer the input, the higher the output value.
3. Output Calculation: The output layer combines the outputs of the
hidden units, typically with weights, to generate the final prediction.

Training Algorithm:
1. Initialize: Set the number of hidden neurons and randomly select centers
from training data.
2. Calculate Spread: Define the spread (sigma) for the RBFs.
3. Compute Hidden Layer Output: For each input, calculate the output of
RBF neurons using:
- Output = exp(-((Input - Center)²) / (2 * sigma²))
4. Output Layer Calculation: Combine hidden layer outputs with weights
to get the final output.
5. Error Calculation: Compare predicted output with actual target.
6. Adjust Weights: Update weights based on error.
7. Repeat: Iterate until convergence.
8. Predict: Use the trained network for new data.
This allows the RBF network to learn and make predictions.
Applications of RBF Networks:
1. Function Approximation: Approximating complex functions based on
available data.
2. Classification: Used for tasks like handwritten digit recognition or
categorizing data points into different classes.
3. Time Series Prediction: Predicting future values based on historical data
patterns.
4. Signal Processing: Filtering and analyzing signals in various applications
like audio and image processing.
5. Control Systems: In robotics and autonomous systems for real-time
control.
Advantages of RBF Networks:
1. Fast Learning: The network can learn quickly because the hidden layer
weights are determined by the proximity of the input to the centers.
2. Effective for Nonlinear Problems: RBF networks are well-suited for
handling complex, nonlinear decision boundaries.
3. Good Generalization: RBF networks often generalize well to unseen
data, especially for function approximation.
Disadvantages of RBF Networks:
1. Sensitive to Centers Selection: The choice of centers can affect
performance, and selecting them optimally can be challenging.
2. Need for Clustering: Finding the optimal centers (e.g., through K-
means) adds complexity to the training process.
3. Computationally Expensive for Large Datasets: RBF networks may
require significant computational resources for large datasets due to the
need to compute pairwise distances.
 Applications of Neural Networks in Forecasting, Data Compression,
and Image Compression

1. Neural Networks in Forecasting:

Neural networks are powerful tools for predicting future values or trends
based on historical data. Here’s how they are applied in different
forecasting domains:
 Time Series Forecasting: Neural networks can model time-dependent
patterns and trends, making them ideal for predicting future values in
time series data (e.g., stock prices, weather conditions, sales, or demand
forecasting).
o Example: Predicting stock market trends using Recurrent Neural
Networks (RNNs) or Long Short-Term Memory (LSTM)
networks, which can handle sequential data and capture temporal
dependencies.

 Financial Forecasting: In finance, ANNs help predict stock prices, forex

rates, and market movements.
o Example: Forecasting cryptocurrency prices using historical
market data.
 Weather Forecasting: Neural networks can predict weather conditions
by analyzing past climate data.
o Example: Predicting temperature, precipitation, or storm patterns
using data from sensors and satellites.

 Energy Load Forecasting: Predicting the future electricity demand helps

in better resource planning and energy distribution.
o Example: Forecasting power consumption in cities for efficient
grid management.

 Healthcare Forecasting: Predicting the spread of diseases or patient

outcomes using historical data and medical factors.
 o Example: Predicting the future course of an illness (e.g., cancer
progression) based on patient data.

2. Neural Networks in Data Compression:

Data compression is the process of reducing the size of data to save space
or transmission time. Neural networks are increasingly used in this
domain for both lossy and lossless compression.

 Dimensionality Reduction: Neural networks can reduce the number of

features in the data, effectively compressing the information.
o Example: Using Autoencoders (a type of neural network) for
compressing high-dimensional data, such as text or speech, by
encoding it into a smaller representation and decoding it back to
the original form.

 Text Data Compression: Neural networks, particularly Recurrent

Neural Networks (RNNs), are used for compressing sequences of text
data by predicting the next word in a sequence and reducing redundant
information.
o Example: Text compression for efficient storage or transmission in
communication systems.

 Speech Compression: Neural networks can compress speech signals by

removing redundant patterns and encoding the information efficiently.
o Example: Reducing bandwidth in voice over IP (VoIP) systems.

3. Neural Networks in Image Compression:

Image compression involves reducing the file size of an image while

maintaining its quality. Neural networks are used to achieve lossless and
lossy image compression by learning efficient representations of images.

 Autoencoders for Image Compression: Autoencoders are widely used

for image compression. The encoder compresses the image into a lower-
dimensional space, and the decoder reconstructs it back to the original.
 The compressed representation can be transmitted or stored more
efficiently.
o Example: Convolutional Neural Networks (CNNs) in image
compression, where the network learns to compress images while
maintaining key features, used in applications like streaming
services.

 Lossy Compression Using Deep Learning: In lossy image compression,

neural networks are trained to reduce image size by eliminating less
important details.
o Example: JPEG-like compression techniques using neural
networks that reduce file sizes while keeping the perceptual quality
high.

 Generative Models for Image Compression: Generative Adversarial

Networks (GANs) can be used for compressing images by learning to
generate high-quality reconstructions from compressed data.
o Example: Using GANs to create efficient image compression
methods that maintain image quality even at lower bitrates.

 Super-Resolution Compression: Using deep learning techniques to

compress and then reconstruct high-resolution images from lower-
resolution versions.
o Example: Compressing images in low resolution and then
applying neural networks for upscaling the image to its original or
higher quality.
Summary:
 In Forecasting: Neural networks predict future trends or values in
various domains like finance, weather, energy, healthcare, and more, by
learning from historical data patterns.
 In Data Compression: Neural networks help in reducing the size of data
by encoding it efficiently, such as through autoencoders or recurrent
neural networks, for tasks like speech, text, and general data compression.
 In Image Compression: Neural networks (e.g., autoencoders, GANs)
compress images by learning to reduce data size while maintaining visual
quality, making them ideal for image storage and transmission
applications.