MODULE 1

1) Widrow Hoff Learning rule

The Widrow-Hoff learning rule, also known as the Delta rule or the Least Mean Squares
(LMS) algorithm, is a method used for training artificial neural networks in supervised
learning scenarios. It's named after its creators, Bernard Widrow and Ted Hoff.

Here's a breakdown of how it works:

1. Objective: The main goal of the Widrow-Hoff learning rule is to adjust the weights of
connections between neurons in a neural network so that the network produces
outputs that are as close as possible to the desired outputs for a given set of input
data.

2. Error Calculation: The algorithm starts by computing the error, which is the
difference between the actual output produced by the network and the desired
output for a specific input. This error quantifies how far off the network's prediction is
from what it should be.

3. Weight Update: The weights of the connections between neurons are adjusted
iteratively based on the error calculated. The adjustment is made in the direction that
minimizes the error, effectively updating the network's parameters to improve its
performance.

4. Gradient Descent: The Widrow-Hoff learning rule can be seen as a form of gradient
descent, a popular optimization technique in machine learning. It aims to minimize
the mean squared error between the network's output and the desired output by
iteratively adjusting the weights.

5. Learning Rate: A crucial parameter in the Widrow-Hoff learning rule is the learning
rate (η), which determines the size of the steps taken during the weight updates. A
larger learning rate means larger steps, which can lead to faster convergence but may
also risk overshooting the optimal solution. Conversely, a smaller learning rate results
in smaller steps, which can lead to slower convergence but may provide more stable
learning.

6. Iterative Process: The learning process continues iteratively, with the weights being
updated for each training example in the dataset. This iterative process gradually
improves the network's ability to approximate the desired outputs for the given
inputs.

7. Convergence: With sufficient training data and appropriate parameter settings, the
Widrow-Hoff learning rule aims to converge to a set of weights that minimize the
error across the entire dataset, thus producing a well-trained neural network.
MODULE 1

2) Compare 3 modes of learning in ANN

Supervised Learning:

Definition: Supervised learning involves training the neural network on a

dataset consisting of input-output pairs, where the desired output is provided
for each input. The network learns to map inputs to outputs by adjusting its
weights to minimize the difference between predicted and actual outputs.
Characteristics:
i. Requires labeled data for training.
ii. Network learns to generalize patterns from the training data to make
predictions on unseen data.
iii. Commonly used for tasks such as classification and regression.
Examples: Image classification, speech recognition, predicting house prices.

Unsupervised Learning:

Definition: Unsupervised learning involves training the neural network on

input data without explicit output labels. The network learns to find patterns
or structure in the data without guidance, typically by clustering similar data
points or reducing the dimensionality of the input space.
Characteristics:
iv. Doesn't require labeled data; the network learns from the inherent
structure of the data.
v. Useful for tasks such as clustering, anomaly detection, and feature
learning.
vi. Can uncover hidden patterns or relationships in data.
Examples: Clustering customer segments, detecting outliers in data,
dimensionality reduction.
MODULE 1

Reinforcement Learning:

Definition: Reinforcement learning (RL) involves training the neural network to

take actions in an environment to maximize cumulative rewards. The network
learns through trial and error, receiving feedback in the form of rewards or
penalties based on its actions.
Characteristics:
vii. Learns through interactions with an environment.
viii. Uses a reward signal to guide the learning process.
ix. Balances exploration (trying new actions) and exploitation (leveraging
known actions) to maximize long-term rewards.
Examples: Training autonomous agents in games, robotics control,
optimizing resource allocation in dynamic environments.

3) Compare biological neuron and artificial neuron

Biological neurons and artificial neurons have some similarities and differences in their
structure and function. Here are some of the main points of comparison:
Structure: Biological neurons have a complex and organic structure, consisting of dendrites,
soma, axon, and synapses. Artificial neurons have a simple and mathematical structure,
consisting of inputs, weights, bias, and activation function.
Function: Biological neurons process and transmit electrical and chemical signals, using
action potentials and neurotransmitters. Artificial neurons process and transmit numerical
values, using weighted sums and activation functions.
Learning: Biological neurons learn and adapt through synaptic plasticity, changing the
strength and number of synapses based on experience and stimuli. Artificial neurons learn
and adapt through weight adjustment, changing the value and number of weights based on
error and feedback.
Efficiency: Biological neurons are highly efficient and parallel, processing and transmitting
signals at high speed and low energy consumption. Artificial neurons are less efficient and
sequential, requiring more time and power to perform computations and communications.

Artificial Neurons at Work

Now that we know what artificial neurons are, let’s see how they work together to form a
neural network. Artificial neurons are usually organized into layers, forming a neural network.
The first layer receives the input data, the last layer produces the output, and the
intermediate layers are called hidden layers. Each layer performs a specific transformation on
the data, passing it to the next layer. The more layers and neurons a neural network has, the
more complex functions it can learn.
The input layer has three neurons, corresponding to three features of the data. The hidden
layer has four neurons, performing some computation on the input data. The output layer
has one neuron, producing the final prediction or decision.
MODULE 1

How do they work together?

The way neural networks work is by adjusting the weights of the connections between the
neurons based on the error of the network predictions compared to the actual data. This is
called the training process, where the network learns from the data and improves its
performance. The training process can be done using various algorithms, such as gradient
descent, backpropagation, stochastic gradient descent, etc.
The goal of the training process is to minimize the error or loss function, which measures
how well the network fits the data. The lower the error or loss, the better the network
performs. The training process can be repeated until the network reaches a satisfactory level
of accuracy or meets some predefined criteria.

4) Discuss the role of mean square error in delta learning

rule . Explain the impact of continuous activation
function in it
In the context of the Delta learning rule, which is often used in supervised learning scenarios
for updating the weights of connections in artificial neural networks, mean square error
(MSE) plays a crucial role as the objective function that the algorithm aims to minimize.
Here's a breakdown of its role and the impact of continuous activation functions:

1. Role of Mean Square Error (MSE):

 Objective Function: Mean square error quantifies the difference between the
actual output produced by the neural network and the desired output for a
given input. It represents the discrepancy or error in the network's predictions.
 Minimization Objective: The goal of the Delta learning rule is to minimize
the mean square error across all training examples. By iteratively adjusting the
weights of the network to reduce the MSE, the network learns to make better
predictions and approximate the desired outputs more accurately.
 Gradient Descent: MSE serves as the basis for calculating the gradient of the
error with respect to the network's weights. This gradient guides the weight
MODULE 1

updates in the direction that minimizes the error, following the principles of
gradient descent optimization.

2. Impact of Continuous Activation Functions:

 Derivatives for Gradient Descent: Continuous activation functions are

crucial for the Delta learning rule because they enable the calculation of
derivatives, which are needed for gradient-based optimization methods like
backpropagation.
 Smooth Error Surface: Continuous activation functions lead to smooth error
surfaces, which make it easier to navigate during the weight update process.
This smoothness helps prevent convergence issues and ensures more stable
learning.
 Non-linear Transformations: Continuous activation functions introduce non-
linear transformations to the network's output, allowing it to capture complex
relationships and patterns in the data. This is essential for learning tasks that
involve non-linear mappings between inputs and outputs.
 Differentiation: Continuous activation functions enable the computation of
derivatives throughout the network, facilitating efficient backpropagation of
errors and weight updates.
 Common Choices: Common continuous activation functions include sigmoid,
hyperbolic tangent (tanh), and rectified linear unit (ReLU), each offering
different properties and benefits for learning tasks

5) Which are the different activation functions

MODULE 1

6) Mc cullock pits model

The McCulloch-Pitts (MCP) neuron model, proposed by Warren McCulloch and Walter Pitts
in 1943, is one of the earliest formalizations of artificial neurons. It serves as a fundamental
building block for understanding the functioning of artificial neural networks. Here's an
explanation of the MCP neuron model:

1. Basic Structure:

 The MCP neuron model is a simplified abstraction of a biological neuron's

functionality.
 It consists of a set of binary inputs 𝑥1,𝑥2,...,𝑥𝑛x1,x2,...,xn, each with a value
of either 0 or 1, representing the presence or absence of a signal or feature.

 Each input is associated with a weight 𝑤1,𝑤2,...,𝑤𝑛w1,w2,...,wn, which

represents the strength of the connection between the input and the neuron.

2. Activation Function:

 The neuron computes a weighted sum of its inputs, ∑𝑖=1𝑛𝑤𝑖⋅𝑥𝑖∑i=1nwi⋅xi,

where 𝑤𝑖wi is the weight associated with input 𝑥𝑖xi.

 If the weighted sum exceeds a certain threshold 𝜃θ, the neuron produces an
output signal of 1; otherwise, it produces an output signal of 0.

 Mathematically, the output 𝑦y of the neuron is computed as:

𝑦={1,if ∑𝑖=1𝑛𝑤𝑖⋅𝑥𝑖≥𝜃0,otherwisey={1,0,if ∑i=1nwi⋅xi≥θotherwise

3. Thresholding:

 The threshold 𝜃θ acts as a decision boundary. If the weighted sum of inputs

exceeds this threshold, the neuron "fires" and produces an output of 1;
otherwise, it remains inactive with an output of 0.
 The threshold parameter allows the MCP neuron model to implement logical
operations such as AND, OR, and NOT.
MODULE 1

4. Functionality:

 The MCP neuron model can be used to represent basic logical operations and
compute simple decision boundaries.
 It forms the basis of more complex artificial neural networks by serving as the
building block for interconnected layers of neurons with more sophisticated
activation functions and learning mechanisms.

5. Limitations:

 The MCP neuron model is limited in its ability to represent complex patterns
or perform nonlinear transformations, as it operates with binary inputs and
produces binary outputs.
 It does not incorporate mechanisms for learning or adaptation; the weights
and threshold are typically set manually rather than learned from data.

7) Elaborate winner take all learning rule

The Winner-Take-All (WTA) learning rule is a simple yet effective algorithm used in artificial
neural networks (ANNs) for competitive learning. It is based on the principle of competition
among neurons, where only one neuron or a small subset of neurons are allowed to "win" or
become active at a time. Here's an elaboration on the Winner-Take-All learning rule:

1. Competitive Learning:

 In competitive learning paradigms like Winner-Take-All, neurons compete

among themselves to become activated based on the input stimuli.
 The goal is to select the most relevant or representative neuron(s) for a given
input pattern while suppressing the activity of other neurons.

2. Neural Architecture:

 The Winner-Take-All learning rule typically applies to a layer of neurons in an

artificial neural network, often the output layer or an intermediate layer.
MODULE 1

 Each neuron in the layer receives inputs from the preceding layer or external
sources.

3. Activation Competition:

 When presented with an input pattern, each neuron computes its activation
based on its inputs and current weights.
 The neuron with the highest activation, or the one that best matches the input
pattern, is declared the "winner."
 In some variants of the WTA rule, multiple neurons may win, forming a subset
of active neurons. However, the key principle remains that only a limited
number of neurons are allowed to activate.

4. Winner Determination:

 The winner neuron(s) are typically identified through a comparison of their

activations. The neuron(s) with the highest activation value(s) surpassing a
certain threshold are selected as the winners.

5. Weight Update:

 After determining the winner neuron(s), the weights of the connections

leading to these neurons are updated to enhance their responsiveness to
similar input patterns in the future.
 The weight update process reinforces the winning neuron(s) to become more
selective and specialized in recognizing specific patterns.

6. Competition Dynamics:

 The competition among neurons drives them to specialize in recognizing

different patterns or features of the input space.
 Over time, the network's neurons become tuned to respond to distinct input
patterns, facilitating pattern recognition or classification tasks.

7. Applications:

 Winner-Take-All learning is commonly used in various applications, including

self-organizing maps, vector quantization, feature extraction, and clustering.
 It is particularly useful in scenarios where input patterns need to be classified
into discrete categories or where only the most relevant features need to be
extracted from the input data.
MODULE 1

8) Limitations of perceptron model

Linear Separability:

The perceptron model can only learn linearly separable patterns. It is limited
to tasks where the input space can be divided into two classes by a
hyperplane.
For non-linearly separable data, such as XOR or more complex patterns, a
single-layer perceptron cannot converge to a solution.

Binary Inputs and Outputs:

The perceptron model typically operates with binary inputs (0 or 1) and

produces binary outputs (0 or 1).
Binary representations may not adequately capture the complexity of real-
world data, limiting the model's expressive power.

Lack of Hidden Layers:

The original perceptron model consists of a single layer of neurons directly

connected to the input layer.
Without hidden layers, perceptrons are unable to learn complex mappings
between inputs and outputs or capture hierarchical representations of data.

Limited Expressiveness:

Perceptrons are limited in their ability to represent complex functions or learn

intricate patterns from data.
They lack the capacity to generalize well to unseen data or to model
relationships beyond simple linear separations.
MODULE 1

No Weight Updates for Misclassifications:

In the original perceptron learning algorithm, weight updates are only applied
when misclassifications occur.
This limitation can lead to slow convergence or failure to converge, especially for
data that is not perfectly separable by a hyperplane.

Sensitivity to Scaling and Bias:

The performance of perceptrons can be sensitive to the scaling of input

features and the choice of bias terms.
Small changes in input values or bias parameters can lead to significant changes
in the model's output, affecting its stability and robustness.

No Probabilistic Outputs:

Perceptrons do not naturally provide probabilistic outputs or uncertainty

estimates, which are essential in many modern machine learning tasks.
Probabilistic outputs allow for more nuanced decision-making and model
interpretation.
MODULE 1
MODULE 1
MODULE 1
MODULE 1

9) Implement AND and XOR using perceptron model ( from note)