• Dimensionality Reduction (Lower Dimension):

• Purpose: Dimensionality reduction techniques like

PCA are used to reduce the number of features
(dimensions) in your data while preserving as much
information as possible.
• Use Cases:
• Data Visualization: Lower-dimensional
representations can be helpful for visualizing high-
dimensional data in a more manageable form

• Noise Reduction: By reducing dimensionality, you

can remove noise or redundant information from the
data, which can improve the performance of some
machine learning algorithms.
• Simplification: In cases where high dimensionality
leads to computational or model complexity issues,
dimensionality reduction can make the problem more
tractable.
• The key aspect of dimensionality reduction in SOMs is that the
grid of neurons typically has a lower dimensionality than the
original data. For example, if you have high-dimensional data, you
might use a 2D grid of neurons to visualize it. The SOM arranges
the neurons in such a way that similar data points in the high-
dimensional space are mapped to nearby neurons on the grid.
This results in a lower-dimensional representation of the data
that captures its underlying structure.

• In summary, SOMs perform dimensionality reduction through a

process of competitive learning and topological mapping, where
they arrange neurons on a grid to capture the structure of high-
dimensional data in a lower-dimensional space.
 Self-Organizing Map
• Introduced by Prof. Teuvo Kohonen in 1982
• Also known as Kohonen feature map
• Unsupervised neural network
• Clustering tool of
high-dimensional
and complex data

4
Self-Organizing Maps (SOMs)

• Self-Organizing Maps (SOMs), also known as Kohonen

maps, are a type of artificial neural network that can be used
for dimensionality reduction and data visualization. Unlike
techniques like Principal Component Analysis (PCA) that
explicitly project data into a lower-dimensional space,
SOMs perform dimensionality reduction in an implicit
manner through a process called topological mapping.
Here's how SOMs achieve dimensionality reduction:
• Create a Grid of Neurons: SOMs consist of a
grid of neurons or nodes organized in a two-
dimensional lattice. Each neuron in the grid is
associated with a weight vector of the same
dimension as the input data.

• Initialization: Initially, the weight vectors of the

neurons are initialized with random values or
values sampled from the data.
• Competitive Learning: During training, an input data point is presented
to the SOM. The SOM performs a competitive learning process to
determine which neuron's weight vector is most similar to the input data
point. The neuron with the closest weight vector is called the "winning"
neuron.

• Updating Neuron Weights: The winning neuron, as well as its

neighboring neurons (usually defined within a certain neighborhood on
the grid), adjust their weight vectors. They do this by moving their weight
vectors closer to the input pattern. The magnitude of the adjustment
typically decreases with distance from the winning neuron. This process
allows neurons to become more similar to the input pattern.
(Neighbor Relationships: Neurons on the grid are not isolated but are
connected to neighboring neurons. This connectivity allows information to
propagate across the grid during training. Neighboring neurons adjust
their weight vectors in response to the winning neuron's adaptation,
helping to capture the data's topological relationships)
• Repeat: Steps 3 and 4 are repeated for many iterations, gradually fine-tuning
the neuron weights and causing them to organize themselves in a way that
reflects the underlying structure of the data.
• Visualization: After training, the SOM's neurons have organized
themselves in such a way that similar input patterns are grouped
together. The grid positions of the neurons represent clusters or regions
of similarity in the input data. This organization can be used for
clustering, visualization, and exploratory data analysis.
• Once the SOM is trained, it can be used to visualize the data. The
positions of the neurons on the grid represent a lower-dimensional
representation of the data. Neurons that are close to each other on the
grid have weight vectors that are similar, indicating that they represent
similar features or patterns in the data.

• The grid structure is a key feature of SOMs that enables them to

perform topological mapping and dimensionality reduction. It helps
organize and visualize high-dimensional data in a way that simplifies the
analysis and reveals patterns and relationships within the data.
Let's take a simple example of color classification using a Self-
Organizing Map (SOM) to classify colors into three categories:
Red, Green, and Blue. In this example, we'll use a 2x2 grid SOM for
simplicity.
• We start with a 2x2 grid of neurons, each representing a potential
category.
• Each neuron has a weight vector of three values (R, G, B), initially set to
random values
• Neuron 1 (R, G, B) = (0.2, 0.5, 0.8)
• Neuron 2 (R, G, B) = (0.7, 0.3, 0.6)
• Neuron 3 (R, G, B) = (0.4, 0.9, 0.1)
• Neuron 4 (R, G, B) = (0.8, 0.2, 0.4)
• Training:
• We provide a dataset of colors for training, where each
color is represented as an (R, G, B) vector.
• During training iterations, we randomly pick a color from
the dataset.
• We find the Best Matching Unit (BMU), which is the
neuron whose weight vector is closest to the input color.
This is usually done using Euclidean distance.
• We adjust the BMU's weight vector and the weight
vectors of its neighbors to be closer to the input color.
The adjustment is controlled by the learning rate and
the neighborhood radius.
• We repeat this process for a number of training
iterations.
• After training, the SOM has organized itself such that similar colors
are grouped together.
• For our example, we might have something like this:

• Neuron 1 (R, G, B) = (0.9, 0.1, 0.1) #Representative of Red

• Neuron 2 (R, G, B) = (0.2, 0.9, 0.2) # Representative of Green
• Neuron 3 (R, G, B) = (0.1, 0.2, 0.9) # Representative of Blue
• Neuron 4 (R, G, B) = (0.7, 0.8, 0.2) # Mixed colors
1.Classification:

1.To classify a new color, we present it to the SOM.

2.The SOM finds the BMU for the new color.
3.Based on the BMU's category (e.g., Red, Green, Blue), we
classify the new color accordingly.
• Suppose we present a new color with the RGB values (0.85, 0.15,
0.1) to the trained SOM. The BMU is Neuron 1, which is the
representative of Red. Therefore, we classify the new color as
"Red.“

1. Input Vector Dimensionality: The input vectors represent the data that you want to classify or cluster. Each
element in the input vector represents a feature or attribute of the data point. In the context of color classification,
each input vector is typically a 3D vector representing the Red, Green, and Blue (RGB) values of a color. Hence,
the input vector dimensionality is 3.
2. Neuron Weight Vector Dimensionality: Each neuron in the SOM has a weight vector associated with it. This
weight vector also has the same dimensionality as the input vectors. In the case of color classification, each
neuron's weight vector is also a 3D vector representing an RGB color.
an example of pattern identification using a Self-Organizing Map (SOM) in a
step-by-step manner. In this example, we'll consider the task of classifying
flowers into different species based on their petal length and width. We'll use a
simple 2x2 SOM grid for illustration.

• Example: Flower Species Classification

• Imagine you have a dataset of flowers with two features: petal
length and petal width. You want to use a SOM to identify clusters
of flowers based on these features without specifying the flower
species in advance.
• Step 1: Initialization
• You start with a 2x2 grid of neurons. Each neuron in the grid has a
weight vector with two values (petal length and petal width).
• The weight vectors are initialized randomly.
• Neuron 1 (Petal Length, Petal Width) = (0.2, 0.5)
• Neuron 2 (Petal Length, Petal Width) = (0.7, 0.3)
• Neuron 3 (Petal Length, Petal Width) = (0.4, 0.9)
• Neuron 4 (Petal Length, Petal Width) = (0.8, 0.2)
• Step 2: Training
• You use your dataset of flower measurements to train the SOM.
• During training, the SOM calculates the similarity between flower
measurements (petal length and petal width) and the weight vectors of
neurons.
• The Best Matching Unit (BMU) is identified for each flower based on
the closest neuron (lowest Euclidean distance).
• The BMU's neuron and its neighboring neurons have their weight
vectors adjusted to become more like the flower measurements. The
adjustment is controlled by learning rate and neighborhood radius.
For simplicity, let's use a dataset of 5 flowers with their petal length
and petal width:
• Flower 1: (1.0, 0.6)
• Flower 2: (0.3, 0.8)
• Flower 3: (0.7, 0.5)
• Flower 4: (0.5, 0.3)
• Flower 5: (0.9, 0.7)
• Flower 1 (1.0, 0.6):Compute Euclidean distances:
• Distance to Neuron 1: 0.421
• Distance to Neuron 2: 0.616
• Distance to Neuron 3: 0.329
• Distance to Neuron 4: 0.640
• Identify BMU: Neuron 3 (closest distance)
• Update weights of Neuron 3 and its neighbors:
Updated Neuron Weights (After Training):
Neuron 1 (Cluster- A) (Petal Length, Petal Width): (0.25, 0.65)
Neuron 2 (Cluster- B) (Petal Length, Petal Width): (0.6, 0.3)
Neuron 3 (Cluster- C) (Petal Length, Petal Width): (0.87, 0.6)
Neuron 4 (Cluster- D) (Petal Length, Petal Width): (0.9, 0.4)
Example Input Pattern:
Input Pattern: (0.6, 0.6) (Petal Length, Petal Width)
To classify the input pattern, we'll calculate the Euclidean distance
between the input pattern and each neuron's weight. The closest neuron
(smallest Euclidean distance) will classify the input pattern.
Euclidean distance to Neuron 1: sqrt((0.6 - 0.25)^2 + (0.6 - 0.65)^2) ≈
0.375
Euclidean distance to Neuron 2: sqrt((0.6 - 0.6)^2 + (0.6 - 0.3)^2) = 0.3
Euclidean distance to Neuron 3: sqrt((0.6 - 0.87)^2 + (0.6 - 0.6)^2) ≈ 0.27
Euclidean distance to Neuron 4: sqrt((0.6 - 0.9)^2 + (0.6 - 0.4)^2) ≈ 0.316
The input pattern (0.6, 0.6) is closest to Neuron 3 (Cluster- C) (with the smallest
Euclidean distance of approximately 0.27), so we classify it as associated with
Neuron 3.
•After training, each neuron in the SOM represents
a cluster of similar flowers based on petal length
and width. Neurons with similar weight vectors
(similar flower measurements) are located close to
each other on the map.

•In this way, the SOM has identified patterns in the

flower measurements and can classify new flowers
into segments based on those patterns, without

needing predefined labels or species information .

 Self-Organizing Map

• Maintains the topology of the dataset

• Training occurs via competition between the neurons
• Impossible to assign network nodes to specific input
classes in advance
• Can be used for detecting similarity and degrees of
similarity
• It is assumed that input pattern fall into sufficiently
large distinct groupings
• Random weight vector initialization

18
 Terminology used
• Clustering
• Unsupervised learning
• Euclidean Distance
p  (p1 , p 2 ,..., p n )
q  (q1 , q 2 ,..., q n )
n
ED   (p
i 1
i  qi ) 2

19
 Self-Organizing Maps (SOMs)
• Biologically inspired
by human brain and
nervous system
• Clustering algorithm
that maps input to
best matching unit
• Maps n-dimensional
input space onto a
two dimensional map
• Preserves
topological
relationships in the
data

08 October 2023
21
22
23
SOM algorithm – Finding winning neuron
Find neuron whose weights closest match the inputs

Yj  arg min j ( || X W ||)

i ij

winner

08 October 2023
25
SOM algorithm – weight adaptation

Wij(t  1)  Wij(t )  hij( X (t )  Wij(t ))

•Wij(t+1) is the updated weight.
•Wij(t) ​ is the current weight of the neuron.
•α is the learning rate, controlling the step size of the weight update.
•hij is the neighborhood function.
•X(t) is the input pattern.

/ 2 )
2 2
h ij  exp( dij
The neighborhood function hij is typically a decreasing function of the distance between the winning neuron and
the neuron being updated.
27
A larger σ will result in a broader neighborhood, meaning that more
neurons will be influenced by the BMU during training. Conversely, a
smaller σ will result in a narrower neighborhood, where only nearby
neurons will be significantly influenced by the BMU.

Typically, σ is initialized with a relatively large value at the beginning of

the training process and is gradually reduced over epochs (iterations)
to narrow down the neighborhood as the SOM learns and the weights
stabilize. This process is often referred to as the "learning rate decay"
or "neighborhood radius decay.“

Adjusting σ effectively controls the learning dynamics of the SOM,

influencing how fast or slow neurons adapt to the input data and how
far-reaching their influence is during the learning process.

28
29
In a Self-Organizing Map (SOM), each cluster is represented by a "cluster
center," which is typically the best-matching unit (BMU) or winner neuron for
that cluster. A smaller cluster size means that the SOM is organizing the input
data into finer and more distinct groupings.
Smaller Cluster Size:
•Results in more focused and specific representations of patterns or features in the
input data.
•Each neuron, as a cluster center, represents a specific and localized region of the input
space.
1.Larger Cluster Size:
1. Represents broader, more general patterns in the input data.

1. Each neuron, as a cluster center, represents a larger region of the input space.

By adjusting the SOM's parameters, such as the neighborhood size (σ), during training,
we can control the cluster size and, consequently, the specificity of the representation.
This flexibility allows you to tailor the SOM's organization to match the complexity and
detail of the input data.

30
31
Kohonen Learning Algorithm

1. Initialise weights (random values) and set topological

neighbourhood and learning rate parameters
2. While stopping condition is false, do steps 3-8
3. For each input vector X, do steps 4-6
4. For each neuron j, compute Euclidean distance:
Yj  arg min j (|| Xi  Wij ||)
5. Find index j such that YJ is a minimum
6. For all units j within a specified neighbourhood of j,
and for all i: Wij(t  1)  Wij(t )  hij( X (t )  Wij(t ))
7. Update learning rate ()
8. Reduce topological neighbourhood at specified times
9. Test stopping condition.
 Suppose we have a set of 2D data points as
follows:
•Data Point 1: (3, 5)
•Data Point 2: (8, 9)
•Data Point 3: (2, 2)
•Data Point 4: (9, 7)
•Data Point 5: (6, 4)

Neuron Initial Weight (Random)

1 (0.2, 0.5)
2 (0.7, 0.3)
3 (0.4, 0.9)
4 (0.8, 0.2)
•Data Point 1: (3, 5) •Data Point 3: (2, 2)
BMU: Neuron 1 (closest to (3, 5)) BMU: Neuron 1 (closest to (2, 2))
Update weights using a learning rate (e.g., 0.1): Update weights using a learning rate (e.g., 0.1):
•Neuron 1: (0.2, 0.5) + 0.1 * [(3, 5) - (0.2, 0.5)] = (0.47, •Neuron 1: (0.47, 0.75) + 0.1 * [(2, 2) - (0.47, 0.75)] =
0.75) (0.63, 0.8)
•Neuron 2: No update (not in the neighborhood) •Neuron 2: No update (not in the neighborhood)
•Neuron 3: No update (not in the neighborhood) •Neuron 3: No update (not in the neighborhood)
•Neuron 4: No update (not in the neighborhood) •Neuron 4: No update (not in the neighborhood)

•Data Point 2: (8, 9) •Data Point 4: (9, 7)

BMU: Neuron 4 (closest to (8, 9)) BMU: Neuron 4 (closest to (9, 7))
Update weights using a learning rate (e.g., 0.1): Update weights using a learning rate (e.g., 0.1):
•Neuron 1: No update (not in the neighborhood) •Neuron 1: No update (not in the neighborhood)
•Neuron 2: No update (not in the neighborhood) •Neuron 2: No update (not in the neighborhood)
•Neuron 3: No update (not in the neighborhood) •Neuron 3: No update (not in the neighborhood)
•Neuron 4: (0.8, 0.2) + 0.1 * [(8, 9) - (0.8, 0.2)] = (1.02, •Neuron 4: (1.02, 1.1) + 0.1 * [(9, 7) - (1.02, 1.1)] = (1.5,
1.1) 1.4)
•Data Point 5: (6, 4) New Input Pattern: (0.6, 0.6)
BMU: Neuron 1 (closest to (6, 4)) We'll find the Best-Matching Unit (BMU) for this
Update weights using a learning rate (e.g., 0.1): input and visualize its position on the SOM.
•Neuron 1: (0.63, 0.8) + 0.1 * [(6, 4) - (0.63, 0.8)] = Finding BMU:
(0.87, 0.92) We'll calculate the Euclidean distance between
•Neuron 2: No update (not in the neighborhood) the new input pattern (0.6, 0.6) and each
•Neuron 3: No update (not in the neighborhood) neuron's weight to find the BMU.
•Neuron 4: No update (not in the neighborhood) •Distance to Neuron 1:
(0.6−0.87)^2+(0.6−0.92)^2=0.32
•Updated Neuron Weights: •Distance to Neuron 2: (0.6−0.7)^2+(0.6−0.3)^2=0.3
•Neuron Updated Weight •Distance to Neuron 3: (0.6−0.4)^2+(0.6−0.9)^2=
•1 (0.87, 0.92) 0.36
•2 (0.7, 0.3) Distance to Neuron 4: (0.6−1.5)^2+(0.6−1.4)^2 =0.9
•3 (0.4, 0.9)
•4 (1.5, 1.4) The BMU is Neuron 2, as it has the
smallest distance to the new input
pattern (0.6, 0.6).
Let's mark the position of the new input pattern (0.6, 0.6) as 'x' and
indicate the BMU.

Neuron 1 (0.87, 0.92)

o

Neuron 2 (0.7, 0.3)

o x

Neuron 3 (0.4, 0.9)

o

Neuron 4 (1.5, 1.4)

o

In this visualization, 'o' represents the positions of

the neurons, 'x' represents the position of the new
input pattern (0.6, 0.6), and the 'o' with 'x' represents
the BMU.
 Suppose we have a 2D SOM with 5 neurons
arranged in a line:

Neuron Initial Weight

1 (0.2, 0.5)
2 (0.7, 0.3)
3 (0.4, 0.9)
4 (0.8, 0.2)
5 (0.6, 0.7)

Input Pattern: (0.3, 0.6)

We'll calculate the Euclidean distance between the input pattern (0.3, 0.6) and
each neuron's weight. Then, we'll use the Gaussian neighborhood function to
calculate the neighborhood weights based on the distance and a specified
neighborhood radius (σ).
•Input Pattern: (0.3, 0.6)
Calculate Distances and Neighborhood Weights
•Distance to Neuron 1: dist(X,W1)=(0.3−0.2)^2+(0.6−0.5)^2≈0.1414

•Distance to Neuron 2: dist(X,W2)=(0.3−0.7)^2+(0.6−0.3)^2≈0.509

•Distance to Neuron dist(X,W3​)=(0.3−0.4)2+(0.6−0.9)2​≈0.3162

•Distance to Neuron 4: dist(X,W4​)=(0.3−0.8)2+(0.6−0.2)2​≈0.6403

•Distance to Neuron 5: dist(X,W5​)=(0.3−0.6)2+(0.6−0.7)2​≈0.3162

Wij(t  1)  Wij(t )  hij( X (t )  Wij(t ))
• H1​=1 (BMU)
• H2​=exp(−dist(1,2)^2/((2×0.52​)^2)
• H3​=exp(−dist(1,3)^2/((2×0.52​)^2)
• H4​=exp(−dist(1,4)^2​/((2×0.52)^2)
• H5​=exp(−dist(1,5)^2​/((2×0.52)^2)
• Neuron 1 (BMU): New Weight1=(0.2,0.5)+0.1×1×[(0.3,0.6)−(0.2,0.5)]
•Neuron 2 (Right Neighbor):
New Weight2=(0.7,0.3)+0.1×exp(−dist(1,2)^2/((2×0.52​)^2)
×[(0.3,0.6)−(0.7,0.3)]

•Neuron 3 (Top Neighbor): New Weight3=(0.4,0.9)+0.1×exp(−dist(1,3)^2/((2×0.52​)^2)

×[(0.3,0.6)−(0.4,0.9)]

•Neuron 4 (Left Neighbor): New Weight4=(0.8,0.2)+0.1×exp(−dist(1,4)^2​/((2×0.52)^2)

×[(0.3,0.6)−(0.8,0.2)]

•Neuron 5 (Bottom Neighbor):

New Weight5=(0.6,0.7)+0.1×exp(−dist(1,5)^2​/((2×0.52)^2)
)×[(0.3,0.6)−(0.6,0.7)]
41
42
43
44
45
46