Name: Sub: Artificial Neural Network (ANN) CS-

Dharmendra 306C
kumar maurya Branch & Sem: CSE 6th Sem.
Roll: 20CS41CL Assignment 5 Unit 5

Short Answered Type

Questions Q1. Define learning at controls the step size of the weight update during gradient
rate.
The learning rate is a hyperparameter t
descent. A high learning rate can lead to large weight updates, which can cause the network to overshoot the
minimum of the loss function and potentially diverge. On the other hand, a low learning rate can lead to slow
convergence and require more epochs to reach the minimum. Therefore, choosing an appropriate learning rate is
crucial for achieving fast and stable convergence. One common approach is to use a learning rate schedule that
reduces the learning rate over time, such as by dividing it by a constant or using an adaptive method such as Adam.

Q2. What is supervised learning?

This type of learning is done under the supervision of a teacher. This learning process s dependent. During
the training of ANN under supervised learning, the input vector is presented to the network, which will give an
output vector. This output vector is compared with the desired output vector. An error signal is generated, if there is
a difference between the actual output and the desired output vector. Based on this error signal, the weights are
adjusted until the actual output is matched with the desired output.

Q3. What is self-organizing Map?

A Self-Organizing Map (SOM) is an artificial neural network model that is used for unsupervised learning and data
visualization. It is a type of competitive learning algorithm that organizes input data into a low- dimensional grid of
neurons, capturing the underlying structure and patterns in the data.
The SOM consists of a grid of neurons, where each neuron represents a prototype or codebook vector that
characterizes a region of the input space. During training, the SOM learns to map input patterns to the neurons by
adjusting their weights based on the similarity between the input patterns and the prototypes. The main idea is to
encourage nearby neurons to respond similarly to similar input patterns, thus creating a topological mapping of the
input space.

Q4. What is unsupervised learning?

This type of learning is done without the supervision of a teacher. This learning process is independent. During
the training of ANN under unsupervised learning, the input vectors of similar type are combined to form clusters.
When a new input pattern is applied, then the neural network gives an output response indicating the class to
which the input pattern belongs.
There is no feedback from the environment as to what should be the desired output and if it is correct or incorrect.
Hence, in this type of learning, the network itself must discover the patterns and features from the input data, and the
relation for the input data over the output.
Q5. Define weights.
Weight: It represents the dimension or strength of the connection between units. If the
weight to node 1 to node 2 has a higher quantity, then neuron 1 has a more considerable influence on the neuron.
weights are the parameters that connect the neurons in the network. Each connection between two neurons has a
weight associated with it, which determines the strength of the connection. These weights are essentially the
numerical values that are learned during the training process of the ANN. It carries information denoted by w1, w2,
…wn. It may be fixed or can take a random value.

Q6. Define Clusters.

Clustering is the process of dividing a dataset into meaningful and distinct clusters based on the inherent similarities
or dissimilarities between the data points. The goal of clustering is to identify natural groupings or patterns in the
data, where data points within the same cluster are more similar to each other compared to data points in different
clusters. Clustering is an unsupervised learning technique, meaning that it does not rely on predefined class labels or
target values. Instead, it focuses on discovering the underlying structure of the data based solely on the similarities
between data points. Clustering algorithms assign data points to clusters based on certain criteria, such as
minimizing the intra-cluster distance (distance between data points within the same cluster) and maximizing the
inter-cluster distance (distance between data points in different clusters).

Long Answered Type Questions

Q1. Explain winner-take all
learning.
Winner take all (WTA) learning is a type of learning mechanism used in artificial neural networks (ANNs) and other
machine learning models. It is commonly associated with competitive learning, where multiple neurons or units in
the network compete with each other to become the "winner" and activate more strongly than their competitors.
In winner take all learning, the competition is based on the relative strengths of the inputs to the neurons. The neuron
or unit with the highest input strength is declared the winner, and its output is set to a high value (e.g., 1), while the
outputs of the other neurons are set to a low value (e.g., 0). This way, the winner neuron "takes all" the activation,
hence the name. Winner take all learning can be used for various purposes in neural networks. One common
application is feature selection or dimensionality reduction, where the network is trained to identify the most relevant
features from a set of inputs. By applying winner take all learning, the network can effectively select the most salient
features by activating the corresponding neurons.
Another application is in clustering or categorization tasks, where the network is trained to assign inputs to specific
classes or categories. Winner take all learning can help identify the best matching category for an input by activating
the corresponding neuron associated with that category. It's worth noting that winner take all learning can be
implemented in different ways. One common approach is to use a hard competition, where only one winner is
selected, and all other neurons are completely suppressed. However, there are also variations that allow for more
flexibility, such as soft competition, where multiple winners can be selected with different activation levels.
Winner take all learning provides a simple and effective mechanism to promote competition among neurons in a
neural network, enabling the network to learn important features and make categorical decisions based on the
strongest inputs. Winner take all learning can be implemented using various algorithms. Here are a few commonly
used algorithms for winner take all learning:
ALGO
1. Max Selection: In this algorithm, the neuron with the highest input value is selected as the winner. All
other neurons are suppressed, meaning their outputs are set to zero. This algorithm is simple and widely used
in various applications.
2. Softmax Function: The softmax function is commonly used in winner take all learning to implement a
more flexible and probabilistic approach. The inputs to the neurons are transformed using the softmax function,
which converts them into probabilities. The neuron with the highest probability becomes the
winner, and its output is set to 1, while the outputs of other neurons are set to 0. The softmax function allows for
multiple winners with varying activation levels.
3. Kohonen Self-Organizing Maps (SOM): SOM is a popular unsupervised learning algorithm that
incorporates winner take all learning. It is commonly used for clustering and dimensionality reduction tasks. In
SOM, a grid of neurons is organized in such a way that nearby neurons have similar weights. During training, the
input vectors are compared to the weights of the neurons, and the neuron with the closest weight vector to the input
becomes the winner. The winning neuron's weights are then adjusted to better represent the input.

Q2. Explain the concept of Initialisation of weights.

The initialization of weights in a neural network is an important step that can significantly affect the
learning and performance of the network. The weights determine the strength of connections between
neurons and play a crucial role in information processing and decision-making within the network.
Here are a few common approaches for weight initialization in neural networks:
1. Zero Initialization: In this approach, all the weights in the network are initialized to zero. While this
method is simple, it is generally not recommended because it leads to symmetric weight updates during
training, causing all neurons to learn the same features and limiting the network's capacity to learn diverse
representations.
2. Random Initialization: A common practice is to initialize the weights with random values. The random
values are typically drawn from a Gaussian distribution with zero mean and a small standard deviation. This
helps break the symmetry and allows neurons to learn different features and adapt independently during
training.
3. Xavier/Glorot Initialization: Xavier initialization is a popular method for weight initialization,
particularly in shallow feedforward neural networks. It sets the initial weights from a Gaussian
distribution with zero mean and a variance that depends on the number of input and output neurons of
the weight layer. This initialization aims to ensure that the signals have a reasonable magnitude
throughout the network, facilitating learning.
4. He Initialization: He initialization is similar to Xavier initialization but is specifically designed for
networks that use rectified linear units (ReLU) as activation functions. It scales the initial weights based
on the number of input neurons to prevent the neurons from saturating in the nonlinear region of the
ReLU activation function.
5. Uniform Initialization: In uniform initialization, weights are initialized from a uniform distribution within
a specified range. This method can be useful in certain scenarios but requires careful tuning of the range to
avoid weight values that are too large or too small, which could hinder learning. These are some commonly
used weight initialization techniques in neural networks. It's worth noting that the choice of weight
initialization method can depend on the specific architecture, activation functions, and problem domain.
Additionally, advanced techniques such as variance scaling, orthogonal initialization, or using pretrained
weights from pretraining or transfer learning can also
be employed in specific cases.

Q3. What are the seperability limitations?

The separability limitation refers to the challenge of learning when the classes or patterns to be classified in
a machine learning problem are not easily separable or distinguishable. In other words, if the classes or
patterns overlap or share similar characteristics, it becomes difficult for a learning algorithm to accurately
separate them.
The separability limitation can arise in various types of machine learning tasks, including binary
classification, multi-class classification, and clustering. When faced with this limitation, the performance of
the learning algorithm may suffer, resulting in lower accuracy or misclassifications. There are several
factors that can contribute to the separability limitation:
1. Overlapping feature distributions: If the feature distributions of different classes overlap significantly, it
becomes challenging for a classifier to draw clear decision boundaries and accurately separate the classes.
2. Inherent noise or variability: When the data contains inherent noise or variability, it can further blur
the boundaries between classes, making them harder to separate.
3. Class imbalance: In situations where the class distribution is imbalanced, with significantly more
samples from one class compared to others, it can be more difficult to accurately classify the minority
class due to the scarcity of representative samples.
4. Non-linear relationships: If the underlying relationships between features and classes are nonlinear, linear
classifiers may struggle to find appropriate decision boundaries to separate the classes effectively.
To overcome the separability limitation, several strategies can be employed:
1. Feature engineering: Careful feature engineering can help enhance the separability between classes. By
selecting or creating informative features that capture the relevant characteristics of the classes, the learning
algorithm may have an easier time distinguishing between them.
2. Non-linear classifiers: Non-linear classifiers, such as support vector machines with kernel functions or
neural networks with non-linear activation functions, can be more effective in capturing complex decision
boundaries and handling non-linear relationships between features and classes.
3. Ensemble methods: Ensemble methods, such as random forests or boosting algorithms, combine the
predictions of multiple classifiers to improve overall performance. They can help compensate for the
separability limitation by leveraging the diversity of individual classifiers.
4. Data augmentation: Generating additional synthetic data or perturbing existing data to introduce
variations and increase the separability between classes can be beneficial, especially when dealing with
limited training data.
5. Collecting more labeled data: Increasing the size and diversity of the labeled training dataset can
improve the chances of finding discriminative patterns and separating the classes effectively.
It's important to note that the separability limitation is not always a hard constraint, and it depends on the
specific problem and data characteristics. By employing appropriate techniques and understanding the
nuances of the data, it's often possible to mitigate the separability limitation and achieve satisfactory
performance in challenging learning tasks.

Q4. Draw SOM Network Architecture.

Self Organizing Map (or Kohonen Map or SOM) is a type of Artificial Neural Network which is also inspired by
biological models of neural systems from the 1970s. It follows an unsupervised learning approach and trained its
network through a competitive learning algorithm. SOM is used for clustering and mapping (or dimensionality
reduction) techniques to map multidimensional data onto lower-dimensional which allows people to reduce complex
problems for easy interpretation. SOM has two layers, one is the Input layer and the other one is the Output layer.
The architecture of the Self Organizing Map with two clusters and n input features of any sample is given below:

How do SOM works?

Let’s say an input data of size (m, n) where m is the number of training examples and n is the number of features
in each example. First, it initializes the weights of size (n, C) where C is the number of clusters. Then iterating
over the input data, for each training example, it updates the winning vector (weight vector
wij = wij(old) + alpha(t) * (xik - wij(old))
where alpha is a learning rate at time t, j denotes the winning vector, i denotes the ith f ature of training
example and k denotes the kth training example from the input data. After training the SOM network, trained
weights are used for clustering new examples. A new example falls in the cluster of winning vectors.

Q5. Discuss any two application of SOM.

Self-Organizing Maps (SOMs) have a wide range of applications in various fields. Here are two notable applications
of SOMs:

1. Data Visualization: SOMs are often used for data visualization tasks, particularly for high-dimensional data.
By mapping the input data onto a low-dimensional grid of neurons, SOMs can reveal the underlying structure and
relationships in the data in a visually intuitive manner. Each neuron represents a cluster or a prototype, and the
spatial arrangement of neurons on the grid reflects the similarities and dissimilarities between the input patterns.
This visualization capability is useful for exploring and understanding complex datasets, identifying clusters and
patterns, and gaining insights into the data. It can be applied in fields such as market research, customer
segmentation, image analysis, and exploratory data analysis.
2. Clustering and Pattern Recognition: SOMs are widely used for clustering analysis, where they group
similar data points together based on their input similarities. The SOM's topological mapping of input patterns
allows for efficient clustering, as neighboring neurons tend to respond similarly to similar input patterns. Each
neuron represents a cluster or a prototype, and new data points can be assigned to the cluster represented by the
closest neuron.
SOMs can be used for pattern recognition tasks. Once trained on a specific dataset, the SOM can classify new input
patterns by assigning them to the cluster represented by the closest neuron. This makes SOMs useful for tasks such
as image recognition, speech recognition, text categorization, and anomaly detection. SOMs provide a powerful tool
for visualizing and analyzing complex data, uncovering underlying patterns and relationships, and performing
clustering and pattern recognition tasks. Their ability to organize and map high-dimensional data onto a low-
dimensional grid enables intuitive interpretation and efficient analysis of the data, making them valuable in various
domains.

Q6. Explain the learning algorithms in ANN.

Learning in ANN can be classified into three categories namely supervised lear ing, unsupervised
learning, and reinforcement learning.
Supervised Learning
This type of learning is done under the supervision of a teacher. This learning process is dependent. During
the training of ANN under supervised learning, the input vector is presented to the network, which will give
an output vector. This output vector is compared with the desired
e if there is a difference between the actual output and
output vector. An error signal is gen rated,
the desired output vector. Based on this error signal, the weights are adjusted until the actual output is
matched with the desired output.

Backpropagation: This is the most widely used learning algorithm for supervised learni g in ANNs. It
involves calculating the gradient of the network's error with respect to its weights and biases and adjusting
Unsupervised Learning
This type of learning is done without the supervision of a teacher. This learning process is
independent. During the training of ANN under unsupervised learning, the input vectors of similar
type are combined to form clusters. When a new input pattern is applied, then t e neuralh network
gives an output response indicating the class to which the input pattern belongs.
There is no feedback from the envir nmento as to what should be the desired output and if it is
correct or incorrect. Hence, in this type of learning, the network itself must discover the patterns and
features from the input data, and the relation for the input data over the output.

● Self-Organizing Maps (SOM): SOMs are a type of unsupervised learning algorithm that organizes input data
into a low-dimensional grid of neurons. The neurons compete to respond to input patterns
and adjust their weights to capture the underlying structure of the data.
● Hebbian Learning: This learning algorithm is based on the Hebbian principle, which states that
"neurons that fire together, wire together." It strengthens the connections between neurons that are active
simultaneously, thus promoting the learning of correlated input patterns and reinforcing relevant
connections.

Reinforcement Learning
This type of learning is used to reinf o rce or strengthen the network over some critic information. ed
This learning process is like supervi s learning; however, we might have very less information.
During the training of network under reinforcement learning, the network receiv s somee feedback
from the environment. This makes it somewhat like supervised learning. However, the feedback
u which means there is no teacher as in supervised
obtained here is evaluative not instr ctive,
learning. After receiving the feedback, the network performs adjustments of the weights to get better critic
information in future.

● Q-Learning: Q-Learning is a reinforcement learning algorithm that uses a value function (Q-function) to
estimate the expected rewards for different actions in a given state. The algorithm learns by
iteratively updating the Q-function based on the rewards received during exploration of the environment.
● Policy Gradient: This algorithm learns by directly optimizing the policy of the network, which
specifies the actions to take in different states. It uses gradient ascent to find the policy that maximizes the
expected cumulative reward.