APEX INSTITUTE OF TECHNOLOGY

COMPUTER SCIENCE &

ENGINEERING
Bachelor of Engineering (Computer Science)
Types of Learning Algorithm
20CSF-286
Prof. (Dr.) Paras Chawla (E5653)

Unit 1 : Machine Learning DISCOVER . LEARN . EMPOWER

Unsupervised Learning
• Unsupervised Learning is that does not involve direct control of the developer. If the main point of
supervised machine learning is that you know the results and need to sort out the data, then in the case of
unsupervised machine learning algorithms the desired results are unknown and yet to be defined. Ex:
Recommender Systems, buying habits.
• Another big difference between the two is that supervised learning uses labeled data exclusively, while
unsupervised learning feeds on unlabeled data.

The unsupervised machine learning algorithm is used for:

• exploring the structure of the information;
• extracting valuable insights;
• detecting patterns;

In other words, unsupervised machine learning

describes information by sifting through it and
making sense of it.
The most widely used algorithms are:
• k-means clustering
• t-SNE (t-Distributed Stochastic Neighbor Embedding)
• PCA (Principal Component Analysis)
• Association rule
Unsupervised Learning

Unsupervised learning algorithms apply the following techniques to describe the data:
• Clustering: it is an exploration of data used to segment it into meaningful groups (i.e., clusters)
based on their internal patterns without prior knowledge of group credentials.
• The credentials are defined by the similarity of individual data objects and also aspects of their
dissimilarity from the rest (which can also be used to detect anomalies).
• Dimensionality reduction: there is a lot of noise in the incoming data. Machine learning algorithms
use dimensionality reduction to remove this noise while distilling the relevant information.
 Clustering
• Clustering is a technique for finding similarity groups in data, called clusters. i.e., groups data instances
that are similar to (near) each other in one cluster and data instances that are very different (far away) from
each other into different clusters.
• Clustering is often called an unsupervised learning task, as no class values denoting an a priori grouping
of the data instances are given, which is the case in supervised learning.
• It is a main task of exploratory data mining , and a common technique for statistical data analysis . Used in
many fields, including machine learning, pattern recognition, image analysis, etc.
• For example, a taxi agent might gradually develop a concept of “good traffic days” and “bad traffic days”
without ever being given labeled examples of each by a teacher.

4
Unsupervised Learning Process Flow
The data has no labels. The machine just looks for whatever patterns it can find.

Unsupervised Learning Model

Feature
Training
Text,
Documents
Vectors
, Images,
etc.

Machine
Learning
Algorithm

Feature
Likelihood
New Text, Vectors or Cluster ID
Document, Predictive or Better
Images, etc. Model Representatio
n
 Unsupervised Learning vs. Supervised Learning
The only difference is the labels in the training data

Feature Feature
Training Text, Training Text,
Vectors Documents, Vectors
Documents,
Images, etc. Images, etc.

Machine
Machine
Learning Labels Learning
Algorithm
Algorithm

Feature
Likelihood Feature
New Text, or Cluster ID
Vectors Predictive New Text,
Document, or Better Expected
Model Vectors Predictive
Images, etc. Representation Document,
Model Label
Images, etc.
Unsupervised Learning: Example
Clustering like-looking birds/animals based on their features

Unsupervised
Learning
 Application of Unsupervised Learning
Unsupervised learning can be used for anomaly detection as well as
clustering
Anomaly
0.9 detection

7 0.8
6
0.7
5 + xxx
x xx
0.6 + +++ x
+ x x x x x xx
4 +++++ x
xxx 0.0033
+++ 0.0251
3 0.5 + 0.008
2 0.0119
0.4
1
0.3
0
0.2
-1
-2 0.1
0 2 4 6 8 10 0 0.5 0.6 0.7 0.8 0.9 0.1
0.1 0.2 0.3 0.4
Identifying
similarities in
groups
(Clustering)
 Clustering
Grouping objects based on the information found in data that describes the
objects or their relationship

The goal is to see that

similar objects are
grouped into one
cluster and different
from objects in
another cluster

Cluster 0
Cluster 1
Cluster 2
Cluster 3
Cluster 4
Need of Clustering
To determine the intrinsic grouping in a set of unlabeled data

To organize data into clusters showing internal structure of the data

To partition the data points

To understand and extract value from large sets of structured and unstructured data
 Types of Clustering
Clustering

Hierarchical Partitional
clustering
clustering

Agglomerative Divisive K-means Fuzzy C-means

 Hierarchical Clustering
Outputs a hierarchy, a structure that is more informative than the unstructured set of clusters returned by flat clustering

B C B C B C B C
A A A A

D D D D
F E F E F E F E
Dissimilarity

Dissimilarity

Dissimilarity

Dissimilarity
A
A B C D E B C D E F A B C D E F
A B C D E
F
Combine A and B based on similarity Combination of A and B is combined Final tree contains all clusters
Combine D and E based on with C Combined into a single cluster
Combination of D and E is combined
similarity
2 with F 3
1 4
 Working: Hierarchical Clustering

Step 1 Step 2 Step 3 Step 4

Assign each item to Find the closest (most Compute distances Repeat steps 2 and 3
its own cluster, such similar) pair of (similarities) between until all items are
that if you have N clusters and merge the new cluster and clustered into a single
number of items, you them into a single every old cluster cluster of size N
now have N number cluster. Now you have
of clusters one less cluster
Distance Measures
Complete - Linkage clustering
• Find the maximum possible distance between points belonging to two different
clusters

Single - Linkage Clustering

• Find the minimum possible distance between points belonging to
two different
clusters

Mean - Linkage Clustering

• Find all possible pair-wise distances for points belonging to
two different clusters and then
calculate the average

Centroid - Linkage Clustering

• Find the centroids of each cluster and calculate
the distance between them
 The Dendrogram
Dendrogram ((in Greek, dendro means tree and gramma means drawing) is a tree diagram
frequently used to illustrate the arrangement of the clusters produced by hierarchical
clustering.

Agglomerative

Divisive
Hierarchical Clustering: Example
A hierarchical clustering of distances between cities in kilometers

MI
TO

FI

RM

NA BA
RM
BA NA RM FI TO MI
Hierarchical Clustering: Step 1
Create distance matrix of data

BA FI MI NA RM TO

BA 0 662 877 255 412 996

FI 662 0 295 468 268 400

MI 877 295 0 754 564 138

Distance between TO and MI
NA 255 468 754 0 219 869

RM 412 268 564 219 0 669

TO 996 400 138 869 669 0

Distance Matrix
Hierarchical Clustering: Step 2
From the distance matrix, you can see that MI has least distance with TO and they form a cluster together

BA FI MI NA RM TO
BA FI MI/TO NA RM
BA 0 662 877 255 412 996
BA 0 662 877 255 412
FI 662 0 295 468 268 400
FI 662 0 295 468 268
MI 877 295 0 754 564 138
MI/TO 877 295 0 754 564
NA 255 468 754 0 219 869
NA 255 468 754 0 219
RM 412 268 564 219 0 669
RM 412 268 564 219 0
TO 996 400 138 869 669 0

TO MI

As the MI column has lower values than TO column,

MI/TO consists of MI column values
Hierarchical Clustering: Step 3
Repeat clustering until a single cluster is obtained with all the members in it

BA FI MI/TO NA RM
BA 0 662 877 255 412
FI 662 0 295 468 268
MI/TO 877 295 0 754 564 NA RM TO MI
NA 255 468 754 0 219
RM 412 268 564 219 0
BA FI MI/TO NA/RM
BA 0 662 877 255
FI 662 0 295 268
MI/TO 877 295 0 564
NA/RM 255 468 564 0
Hierarchical Clustering: Step 3 (Contd.)

BA FI MI/TO NA/RM
BA 0 662 877 255
FI 662 0 295 268
MI/TO 877 295 0 564 NA RM TO MI
NA/RM 255 268 564 0

BA/(NA/RM) FI MI/TO
BA/(NA/RM) 0 268 564

FI 268 0 295
MI/TO 564 295 0
Hierarchical Clustering: Step 3 (Contd.)

BA/(NA/RM) FI MI/TO
BA/(NA/RM) 0 268 564

FI 268 0 295 BA
MI/TO 564 295 0 NA RM FI MI
TO

BA/(NA/RM)/FI (MI/TO)
BA/(NA/RM)/FI 0 295

(MI/TO) 295 0
Hierarchical Clustering: Step 4
Derive the final dendrogram

BA/(NA/RM)/FI (MI/TO)
BA/(NA/RM)/FI 0 295
BA
(MI/TO) 295 0 NA RM FI TO
MI
 K-means Algorithm: Steps

1 Randomly chooses k datapoints as initial centroids

2 Assigns each datapoint closest to the centroid

3 Calculates new cluster centroids

4 Checks if the convergence criterion is met

K-means: Example
Consider the below datapoints
K-means: Example (Contd.)
Initialize centers randomly
K-means: Example (Contd.)
Assign points to the nearest center
K-means: Example (Contd.)
Readjust centers
K-means: Example (Contd.)
Assign points to the nearest center
K-means: Example (Contd.)
Re-adjust centres
K-means: Example (Contd.)
Assign points to the nearest center
K-means: Example (Contd.)
Readjust centers
K-means: Example (Contd.)
Assign points to the nearest center
Optimal Number of Clusters

Objective Function Value

If you plot k against the SSE, you will see that the error
decreases as k increases

i.e., Distortion
This is because their size decreases and hence distortion is
also smaller"

The goal of elbow method is to choose k where SSE

decreases abruptly Elbow Plot
Knowledge Check
Knowledge
Check Can decision trees be used for performing clustering?
1

a. Tru
e
b. False
 Knowledge
Check Can decision trees be used for performing clustering?
1

a. Tru
e
b. False

The correct answer is a. True

Decision trees can also be used to for clusters in the data, but it often generates natural clusters and is not dependent on any
objective function.
Knowledge Which of the following can act as possible termination condition in K-Means?
Check 1. Fixed number of iterations.
2. Assigning observations to clusters such that they don’t change between iterations, except for cases with a bad local minimum.
2 3. Stationary centroids appear between successive iterations.
4. When RSS falls below a threshold.

a. 1,3, and 4

b. 1, 2, and 3

c. 1, 2, and 4

d. All the above

 Which of the following can act as possible termination condition in K-Means?
Knowledge
Check 1. Fixed number of iterations.
2. Assigning observations to clusters such that they don’t change between iterations, except for cases with a bad local minimum.
2 3. Stationary centroids appear between successive iterations.
4. When RSS falls below a threshold.

a. 1,3, and 4

b. 1, 2, and 3

c. 1, 2, and 4

d. All the above

The correct answer is d. All the above

All the above options are true.
References

Web Sources:

1. https://machinelearningmastery.com/linear-regression-for-machine-learning/

Video Source:

1. https://www.youtube.com/watch?v=zPG4NjIkCjc
THANK
YOU
For Queries,
Write at : [email protected]