INTRODUCTION TO ARTIFICIAL INTELLIGENCE & MACHINE LEARNING

UNIT -IV
Basic Methods in Supervised Learning: Distance-based methods, Nearest-Neighbors, Decision
Trees, Support Vector Machines, Nonlinearity and Kernel Methods.
Unsupervised Learning: Clustering, K-means, Dimensionality Reduction, PCA and kernel.
=================================================================================

Supervised learning
Supervised learning is a type of machine learning where a model is trained on labelled data, meaning
the input data is paired with the correct output or label. The goal is for the model to learn the
mapping from inputs to outputs, enabling it to make accurate predictions on new, unseen data.
Common algorithms include linear regression, decision trees, and neural networks. Supervised
learning is used for tasks like classification (predicting categories) and regression (predicting
continuous values). The effectiveness of the model depends on the quality and quantity of the
labelled data used during training.

Distance-based methods
Distance-based methods in supervised learning are techniques that classify data based on the
similarity or distance between data points. These methods use distance metrics, such as Euclidean or
Manhattan distance, to measure how close an unknown data point is to known labelled instances.
Popular algorithms like k-Nearest Neighbours (k-NN) and Support Vector Machines (SVM) rely on
these distances to make predictions. In k-NN, the class of the majority of the nearest neighbours
determines the class of the new data point. These methods are simple, intuitive, and effective for
problems where spatial proximity is a strong indicator of classification.
 Distance-based methods in machine learning rely on measuring the distance between data
points to make predictions.
 These methods are used both in classification and regression problems.

Common Distance Metrics:

 Euclidean Distance: The straight-line distance between two points in Euclidean space. This is the
most common distance metric and is used in algorithms like k-Nearest Neighbors (k-NN).
 Manhattan Distance: The sum of the absolute differences of their coordinates. It’s also known
as L1 norm.
 Minkowski Distance: A generalization of both Euclidean and Manhattan distances. It can be
adjusted with a parameter to resemble other types of distances.
 Cosine Similarity: A measure of similarity between two vectors, often used in text classification
or document clustering.
 Hamming Distance: The number of positions at which the corresponding symbols are different
in two strings or vectors. This is commonly used for binary data.
 Mahalanobis Distance: Takes into account the correlations of the data set and is useful for
identifying outliers.
 Minkowski distance or Minkowski metric is a metric in a normed vector space which can be
considered as a generalization of both the Euclidean distance and the Manhattan distance.
 It is named after the Polish mathematician Hermann Minkowski.

 Cosine similarity is a metric used to measure how similar two vectors (or data points) are, based
on their orientation, regardless of their magnitude.
 It is commonly used in text analysis, information retrieval, and machine learning, especially
when comparing documents or words in high-dimensional spaces.
 Hamming distance between two strings or vectors of equal length is the number of positions at
which the corresponding symbols are different.
 In other words, it measures the minimum number of substitutions required to change one string
into the other, or equivalently, the minimum number of errors that could have transformed one
string into the other.
 In a more general context, the Hamming distance is one of several string metrics for measuring
the edit distance between two sequences.
 It is named after the American mathematician Richard Hamming.
 Mahalonobis distance is the distance between a point and a distribution. and not between two
distinct points.
 It is effectively a multivariate equivalent of the Euclidean distance.
 It was introduced by Prof. P. C. Mahalanobis in 1936 and has been used in various statistical
applications ever since.
 However, it’s not so well known or used in the machine learning practice.
 How is Mahalanobis distance different from Euclidean distance?
• It transforms the columns into uncorrelated variables
• Scale the columns to make their variance equal to 1
• Finally, it calculates the Euclidean distance.

where,
• D^2 is the square of the Mahalanobis
distance.
• x is the vector of the observation (row in a
dataset),
• m is the vector of mean values of
independent variables (mean of each
column),
• C^(-1) is the inverse covariance matrix of
independent variables.

Nearest Neighbours
Nearest neighbors is a concept used in machine learning, statistics, and computer science, referring
to the idea of finding data points that are most similar to a given point within a dataset. It's often
used for classification, regression, and clustering problems. The key ideas around nearest neighbors:
1. K-Nearest Neighbors (KNN):
o A popular algorithm based on the nearest neighbors concept, where K is a parameter
that defines how many neighbors to consider when making a prediction for a new data
point.
o Classification: The algorithm assigns the class most common among the K nearest data
points to the new data point.
o Regression: The algorithm averages the target values of the K nearest neighbors to
predict a value for the new point.
2. Distance Metrics:
o The similarity between data points is typically measured using a distance metric, such as
Euclidean distance, Manhattan distance, or Minkowski distance, depending on the
problem and the nature of the data.
3. Curse of Dimensionality:
o As the number of features increases, the concept of "nearness" can become less
meaningful. In high-dimensional spaces, all points might seem equally distant from one
another, leading to reduced accuracy in the nearest neighbors' predictions.
4. Choice of K:
o The value of K affects the model's performance. A small K can make the model sensitive
to noise, while a large K can smooth out predictions but potentially overlook small
patterns or details.
5. Efficiency:
o For large datasets, finding the nearest neighbors can be computationally expensive,
especially in high-dimensional spaces. Techniques like KD-trees, Ball Trees, or
Approximate Nearest Neighbors (ANN) are used to optimize search times.
6. Applications:
o Nearest neighbors are widely used in recommendation systems, image recognition,
anomaly detection, and other areas where similar data points provide meaningful
insights or predictions.
Nearest neighbors is a foundational method in machine learning, particularly useful in scenarios
where similarity between data points can help make decisions or predictions.

k-Nearest Neighbors (k-NN)

 k-NN is a simple, instance-based learning algorithm that makes predictions based on the k
nearest neighbors of a point.
 It does not involve explicit model training but memorizes the training data.
How it Works:
• Choose a value for k (e.g., 3, 5, etc.).
• For a new input, calculate the distance between the input and all training data points.
• Select the k nearest neighbors.
• Classification: Take a majority vote from the k neighbors' class labels.
• Regression: Average the outputs of the k neighbors.
Decision Trees
Decision trees are a supervised learning method and are popular machine learning algorithm used
for both classification and regression tasks. They model data by splitting it into smaller subsets based
on certain criteria, ultimately leading to predictions. The concept of decision trees are presented
here:

1. Structure of a Decision Tree:

 A decision tree is a tree-like structure where each internal node represents a decision based
on a feature, and each leaf node represents an outcome (class or value).
 The tree splits the data into branches based on feature values to make predictions at the
leaves.
2. How it Works:
 Root Node: The starting point of the tree, representing the entire dataset.
 Splitting: The data is split at each node based on the feature that best separates the data.
Splitting continues until certain conditions are met (e.g., maximum depth, purity of nodes).
 Leaf Nodes: Final predictions made at the leaf nodes, either as a class label (classification) or
a continuous value (regression).
3. Choosing Splits:
 Decision trees use impurity measures to decide the best feature to split on:
o Gini Impurity (used for classification): Measures the impurity or disorder of the
node.
o Entropy (used for classification): Measures the amount of information disorder in
the dataset.
o Mean Squared Error (MSE) (used for regression): Measures the variance in the
target variable.
 The feature that results in the best separation (lowest impurity) is chosen for the split.
4. Overfitting and Pruning:
 Decision trees are prone to overfitting, especially when they grow too deep, memorizing the
training data instead of generalizing.
 Pruning is the process of removing branches that add little predictive power to prevent
overfitting.
 Max Depth, Min Samples Split, and other hyperparameters can be set to limit the tree's
growth and prevent overfitting.
5. Advantages:
 Interpretability: Decision trees are easy to understand and visualize, making them a good
choice for explainable AI.
  Non-Linear Relationships: They can model complex, non-linear relationships between
features and the target variable.
 Handles Both Numerical and Categorical Data: Can work with both types of data without
the need for scaling or encoding.
6. Disadvantages:
 Overfitting: If not controlled, trees can become too complex and overfit the data.
 Instability: Small changes in the data can lead to a completely different tree structure.
 Bias Toward Features with More Categories: Decision trees may prefer features with more
categories for splits, even if they aren’t the most informative.
7. Applications:
 Decision trees are widely used in areas like customer segmentation, medical diagnoses,
financial forecasting, and any other task where the data can be easily split based on
features.
8. Ensemble Methods:
 Random Forest: A collection of decision trees that improves accuracy by averaging the
predictions of multiple trees to reduce overfitting.
 Gradient Boosting Trees: Another ensemble method that builds trees sequentially to correct
errors made by previous trees, improving predictive accuracy.
Decision trees are a powerful, interpretable method in machine learning that works well for both
classification and regression tasks, though they require careful tuning to avoid overfitting. Examples
for decision trees are given below:
Support Vector Machines (SVM)
Support Vector Machines (SVM) are powerful and versatile supervised machine learning algorithms
used for classification and regression tasks. It finds a hyperplane (decision boundary) that best
separates the data points of different classes and is shown in the following figure:

1. Basic Concept:
 Classification: SVMs are primarily used for classification tasks. They aim to find the optimal
hyperplane (a decision boundary) that best separates different classes in the feature space.
 Regression: In regression (SVR), SVMs try to predict a continuous value by fitting the best
hyperplane while allowing for some margin of error.
2. Key Components:
 Support Vectors: The data points that are closest to the hyperplane and play a critical role in
defining the optimal boundary. These points are used to create the decision boundary.
 Hyperplane: A hyperplane is a boundary that divides the data into two classes. In 2D, this is
a line; in higher dimensions, it's a plane or hyperplane. SVM aims to maximize the margin
between classes while ensuring the data points are correctly classified.
3. Margin Maximization:
 The goal of SVM is to maximize the margin between the hyperplane and the support vectors.
A larger margin is associated with better generalization, meaning the model is less likely to
overfit.
4. Linear vs. Nonlinear SVM:
 Linear SVM: When the data is linearly separable (i.e., a straight line or hyperplane can
separate the classes), a linear SVM can be used.
 Nonlinear SVM: If the data isn't linearly separable, SVM uses a technique called the kernel
trick to map the data into a higher-dimensional space where a linear hyperplane can
separate the classes. Common kernels include:
o Polynomial kernel
o Radial Basis Function (RBF) kernel
o Sigmoid kernel
5. Kernel Trick:
 The kernel trick enables SVM to operate in higher-dimensional spaces without explicitly
computing the transformation, making it efficient even for complex, non-linear
relationships.
6. Cost Parameter (C):
 The parameter C controls the trade-off between maximizing the margin and minimizing
classification errors. A high C value puts more emphasis on minimizing errors (which can lead
to overfitting), while a lower C allows for a wider margin but allows some misclassification
(which can improve generalization).
7. Advantages:
 Effective in high-dimensional spaces: SVM is effective in situations where there are many
features (high-dimensional data).
 Robust to overfitting: Especially when using the correct kernel and regularization
parameters.
 Works well for both linear and non-linear data: Thanks to the kernel trick, SVM can handle
complex data distributions.
 Unique solution: The optimization problem in SVM has a unique global solution.
8. Disadvantages:
 Computationally expensive: SVM can be slow to train, especially with large datasets or high-
dimensional data, as it involves solving a complex quadratic optimization problem.
 Sensitive to parameter tuning: Choosing the right kernel, the parameter C, and the kernel-
specific parameters (like the RBF gamma) can significantly affect performance.
 Not ideal for large datasets: Because of its computational complexity, SVM may not scale
well with very large datasets.
9. Applications:
 SVM is used in various applications, including:
o Image classification: Recognizing objects in images.
o Text classification: Spam filtering, sentiment analysis.
o Bioinformatics: Gene classification, disease prediction.
o Handwriting recognition and other pattern recognition tasks.
Support Vector Machines are a powerful classification and regression tool that works by finding the
optimal hyperplane to separate classes with the largest possible margin. The use of kernels allows
SVM to handle both linear and non-linear problems, making it versatile, though it can be
computationally intensive and sensitive to parameter selection.
Nonlinearity and Kernel Methods
Nonlinearity and kernel methods are key concepts in machine learning, especially when dealing with
complex data that cannot be separated or modelled well using linear methods. The concept can be
seen in the following figures:

1. Nonlinearity in Machine Learning:

 Nonlinear Problems: Many real-world problems cannot be modelled using a straight line or
simple hyperplane. Data may have intricate patterns or relationships between features that
are not linear, meaning the classes or target variables cannot be separated by a linear
boundary.
 Linear Methods: Algorithms like Linear Regression, Logistic Regression, and Linear Support
Vector Machines (SVM) work well when the data has a linear relationship.
 Need for Nonlinearity: When the data is nonlinear, such models are limited. For example, in
image recognition, speech processing, or complex pattern recognition, the relationship
between features can be highly nonlinear.
2. Kernel Methods:
 Introduction to Kernel Methods: Kernel methods are a powerful technique used to handle
nonlinearity. The idea is to map the original data into a higher-dimensional feature space
where the data becomes linearly separable (or easier to model) without explicitly computing
this transformation.
 Kernel Trick: The kernel trick is the heart of kernel methods. Instead of performing the
expensive computation of transforming the data into a higher-dimensional space, kernel
methods compute the inner product between data points in this higher-dimensional space
directly, using a kernel function. This allows the model to perform complex transformations
efficiently.
3. Key Kernel Functions
4. How Kernel Methods Work:
 Transforming Data: The idea behind kernel methods is to map the original data to a higher-
dimensional space, where a linear model can be applied to solve the problem. In this space,
even complex nonlinear relationships may become linear.
 Inner Product in Feature Space: The kernel function computes the inner product between
transformed data points, effectively performing the mapping without explicitly calculating it.
This saves computational resources and makes the method more efficient.
5. Applications of Kernel Methods:
 Support Vector Machines (SVM): One of the most popular uses of kernel methods is in
SVMs, where the kernel allows for the separation of data in higher dimensions, thus
enabling the classification of nonlinear data.
 Kernel Principal Component Analysis (PCA): Kernel PCA is a method for nonlinear
dimensionality reduction, extending traditional PCA to kernel-induced spaces.
 Kernel Ridge Regression: A variant of ridge regression that uses kernels to handle nonlinear
regression problems.
 Clustering: Kernel methods are also used in clustering techniques like Kernel K-means,
where the kernel trick is applied to allow for more complex cluster shapes.
6. Advantages of Kernel Methods:
 Handle Nonlinearity: Kernel methods allow for the handling of highly nonlinear data without
needing to explicitly transform the data, making them very powerful in complex real-world
problems.
 Flexibility: By choosing different kernel functions, one can model various types of
nonlinearities in the data.
 Efficiency: With the kernel trick, we avoid the computational cost of explicitly computing
high-dimensional feature mappings.
7. Challenges and Considerations:
 Choice of Kernel: The choice of kernel and its parameters (like γ for the RBF kernel)
significantly affects the model’s performance. Choosing the wrong kernel or poor
parameters can lead to underfitting or overfitting.
 Computational Complexity: Kernel methods can be computationally expensive, especially
for large datasets, as they often require computing the kernel matrix, which can be of size
O(n2), where n is the number of data points.
  Interpretability: Models using kernel methods (such as SVM with an RBF kernel) are less
interpretable than linear models, as the decision boundary is complex and not easy to
visualize.
Nonlinearity and kernel methods are essential for tackling machine learning problems where data
cannot be easily separated by linear models. Kernel methods, especially through techniques like the
kernel trick, allow algorithms like Support Vector Machines to efficiently learn from data with
complex, nonlinear patterns by mapping it into higher-dimensional spaces. By choosing appropriate
kernels, machine learning models can handle various types of nonlinearities while maintaining
computational efficiency.

Each of these supervised learning methods has its strengths and weaknesses, and the choice of
which to use depends on the dataset and problem at hand. Here's a quick recap of their applications:
 Distance-based Methods (e.g., k-NN): Good for small datasets and classification/regression
based on closeness of data points.
 Decision Trees: Easy to interpret, but can overfit and be unstable.
 Support Vector Machines: Effective for both linear and non-linear problems, but
computationally expensive.
 Kernel Methods: Used to deal with non-linear data by transforming the feature space, but may
be tricky to tune.

 ML algorithms vary in complexity and suitability based on the structure of the data.
 Each method provides different strengths in handling tasks like classification and regression.
 Important part of learning, is understanding when to apply each method based on the problem
you’re trying to solve.

Unsupervised Learning
 Unsupervised learning is a type of machine learning where the model is trained on unlabelled
data, meaning the input data does not come with corresponding target labels or outcomes. The
goal of unsupervised learning is to identify hidden patterns, structures, or relationships in the
data.
 Unlike supervised learning, where the algorithm learns from labelled data to make predictions,
unsupervised learning focuses on discovering inherent patterns within the data itself.
 Common tasks in unsupervised learning include clustering, where the algorithm groups similar
data points together (e.g., customer segmentation in marketing), and dimensionality reduction,
which aims to reduce the number of features in the data while retaining important information
(e.g., Principal Component Analysis, or PCA).
 Other techniques in unsupervised learning include anomaly detection, which identifies unusual
or outlier data points, and association rule learning, which finds interesting relationships
between variables (e.g., market basket analysis).
 One of the main challenges in unsupervised learning is the absence of labelled data, making it
harder to evaluate the performance of the model and interpret the results.
 Popular algorithms for unsupervised learning include K-means clustering, Hierarchical
clustering, Gaussian Mixture Models (GMM), and Self-Organizing Maps (SOM) for clustering,
and PCA and t-SNE for dimensionality reduction.
 Unsupervised learning is widely used in areas like customer behavior analysis, image and speech
recognition, anomaly detection in security, and natural language processing.
 The flexibility and ability to work with unlabelled data make unsupervised learning especially
valuable in real-world scenarios where obtaining labelled data is costly, time-consuming, or
impractical.
Clustering
 Clustering is an unsupervised machine learning technique that involves grouping similar data
points together based on certain characteristics or features, without the need for labeled data.
 The goal of clustering is to organize a dataset into subsets, or clusters, where data points within
each cluster are more similar to each other than to those in other clusters.
 Clustering is widely used in various fields, including customer segmentation, image recognition,
anomaly detection, and market research.
 There are several popular clustering algorithms, such as K-means, which divides data into a
predefined number of clusters by minimizing the variance within each cluster, and Hierarchical
clustering, which builds a tree-like structure of nested clusters, often used for hierarchical
relationships.
 DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is another popular
method that groups together points that are close to each other based on a distance metric, and
it is especially effective at handling noise and outliers.
 Gaussian Mixture Models (GMM) assume that data points are generated from a mixture of
several Gaussian distributions and can capture more complex, non-spherical cluster shapes.
 The choice of algorithm depends on the data type, the distribution of the data, and the desired
outcome.
 Key challenges in clustering include determining the optimal number of clusters and selecting
appropriate distance metrics for measuring similarity.
 Evaluation of clustering performance can be difficult due to the lack of ground truth labels, but
methods like silhouette scores and Davies-Bouldin index can provide some insights into the
quality of the clusters.
 Clustering is a powerful tool for uncovering hidden patterns in data, making it valuable for
exploratory data analysis and uncovering natural groupings within complex datasets.
Example:
 In customer segmentation, a business might want to group customers based on their purchasing
behavior.
 Clustering algorithms can automatically identify groups like high spenders, low spenders,
frequent buyers, etc., without needing predefined labels for each customer.
Key Algorithms:
• K-Means
• Hierarchical Clustering
• DBSCAN
K-Means Clustering
 K-means clustering is a widely used unsupervised machine learning algorithm that groups data
points into a predefined number of clusters based on their similarity.
 The goal of K-means is to partition the data into K clusters, where each cluster contains points
that are more similar to each other than to those in other clusters.
The algorithm works through an iterative process:
1. Initialization: K initial centroids (one for each cluster) are chosen, either randomly or using
methods like K-means++ to improve convergence.
2. Assignment Step: Each data point is assigned to the nearest centroid based on a distance
metric, usually Euclidean distance.
3. Update Step: After the points are assigned to clusters, the centroids are recalculated as the
mean of all data points in each cluster.
4. Repeat: The assignment and update steps are repeated until the centroids no longer change
significantly, indicating convergence.
 K-means is computationally efficient and works well when clusters are spherical and roughly
equal in size.
 However, it has some limitations: it requires the number of clusters (K) to be predefined, it can
be sensitive to the initial placement of centroids (which may lead to different results on different
runs), and it may struggle with clusters of varying shapes or densities.
 Additionally, K-means is sensitive to outliers, as they can distort the centroid calculation.
 Despite these challenges, K-means is popular in tasks like customer segmentation, image
compression, and anomaly detection due to its simplicity and effectiveness in many scenarios.
 It is important to evaluate the results using metrics like the silhouette score or elbow method to
determine the optimal number of clusters.

Dimensionality reduction
 Dimensionality reduction is a technique used in machine learning and data analysis to reduce
the number of features (variables) in a dataset while preserving its essential structure and
information.
 High-dimensional datasets can be computationally expensive to process, and visualizing or
interpreting them becomes challenging.
 Dimensionality reduction helps by transforming the data into a lower-dimensional space, making
it easier to analyze and visualize.
 Two common methods for dimensionality reduction are Principal Component Analysis (PCA)
and t-Distributed Stochastic Neighbor Embedding (t-SNE).
 PCA identifies the directions (principal components) in which the data varies the most and
projects the data onto those directions, reducing the number of features while retaining the
most important variance.
 t-SNE, on the other hand, is often used for visualizing high-dimensional data in two or three
dimensions, preserving local structures in the data.
 Dimensionality reduction is beneficial for improving computational efficiency, reducing noise,
and preventing overfitting, especially in high-dimensional spaces where models may struggle to
generalize.
 However, it comes with trade-offs, as reducing dimensions can sometimes lead to a loss of
important information.
 It is widely used in applications such as image compression, feature extraction, and pre-
processing for machine learning models.
Example:
 In an image dataset, each image might have thousands of pixels, making analysis slow.
 Dimensionality reduction techniques can reduce the number of dimensions while still preserving
important patterns.

Principal Component Analysis (PCA)

 Principal Component Analysis (PCA) is a popular dimensionality reduction technique used in
machine learning and statistics to simplify complex datasets by transforming them into a smaller
set of uncorrelated variables called principal components.
 The primary goal of PCA is to reduce the number of features in a dataset while retaining as much
variance (information) as possible.
 PCA works by identifying the directions (principal components) in which the data varies the
most, and then projecting the data onto these new axes.
 The first principal component captures the highest variance, the second captures the second-
highest variance, and so on.
 This process results in a set of new features (principal components) that are linear combinations
of the original features.
 PCA is particularly useful in situations where data has many correlated variables, as it helps
eliminate redundancy and makes the dataset easier to analyze.
 It is widely used for data compression, noise reduction, and feature extraction, and it can
improve the performance of machine learning models by reducing overfitting.
 However, PCA assumes that the principal components are linear combinations and is sensitive to
the scaling of the data, so it is important to standardize the data before applying PCA in many
cases.
Steps:
 Standardize the data (mean = 0, variance = 1).
 Compute the covariance matrix to find relationships between features.
 Calculate the eigenvectors and eigenvalues of the covariance matrix.
 Select the top 'K' eigenvectors (principal components) to form a reduced-dimensional
representation of the data.
Example:
 In face recognition systems, PCA can be used to reduce the dimensions of facial features (such as
eye distance, nose size) while retaining enough information to distinguish between different
individuals.

Kernel Methods
 Kernel methods in unsupervised learning are powerful techniques that allow algorithms to
handle nonlinearity by mapping data into higher-dimensional spaces where complex patterns
can become linear and easier to analyze.
 The core idea is to use a kernel function to compute the inner product between data points in
this higher-dimensional feature space without explicitly transforming the data, a technique
known as the kernel trick.
 Kernel Trick: Instead of directly mapping data points to higher-dimensional spaces, the kernel
function computes the inner product in that space using the original data points.
 This makes kernel methods computationally efficient, especially when dealing with complex data
structures.
 In unsupervised learning, kernel methods are commonly applied in tasks such as clustering and
dimensionality reduction.
 For example, in Kernel Principal Component Analysis (Kernel PCA), the kernel trick is used to
perform dimensionality reduction in a nonlinear feature space, capturing more complex
structures in the data than traditional PCA.
 In Kernel K-means clustering, the kernel allows for the identification of more complex, non-
linear clusters compared to traditional K-means, making it effective in cases where clusters are
not easily separable by a simple linear boundary.
 Common kernels include the Radial Basis Function (RBF) kernel, polynomial kernel, and sigmoid
kernel, each suited for different types of data.
 Overall, kernel methods provide flexibility and power in unsupervised learning tasks, enabling
the modelling of intricate relationships in high-dimensional data.
 Kernel methods are techniques used to enable algorithms to operate in higher-dimensional
spaces without explicitly computing the coordinates in that space.
 This is particularly useful for non-linear data that cannot be separated in lower dimensions.

Example: Support Vector Machines (SVM) use kernel methods to find decision boundaries that can
separate non-linearly separable data by mapping them to higher-dimensional space.
Common Kernels:
• Linear Kernel
• Polynomial Kernel
• Radial Basis Function (RBF) Kernel
Example:
 For a classification task where you want to distinguish between two classes (e.g., cats and
dogs), a linear separator might not work well.
 Using a kernel function, the data can be transformed into a higher-dimensional space where
the classes become separable by a linear boundary.

Summary of Key Concepts:

 Clustering: Grouping similar data points together (e.g., customer segmentation).
 K-Means: A clustering algorithm that groups data into 'K' clusters by iterating between assigning
points to centroids and updating centroids.
 Dimensionality Reduction: Reducing the number of features while retaining important
information (e.g., reducing the dimensions of images).
 PCA: A technique for dimensionality reduction by projecting data into a new set of orthogonal
axes that capture the maximum variance.
 Kernel Methods: Techniques for transforming non-linear problems into linear ones by mapping
data into higher dimensions using kernel functions.
These concepts form the foundation of unsupervised learning techniques and are critical for data
analysis, pattern recognition, and machine learning applications.
 Assignment 4
(Submit within a week)

1) Describe Nearest-Neighbor Classification in detail.

2) Explain in detail about Decision Tree with an example.
3) Describe SVM algorithm with example.
4) Discuss in detail about Soft Margin SVM. How to identify soft margin?
5) Discuss in detail about Distance Based Clustering. Write its importance in machine learning.
6) Explain the distance-based methods in Machine Learning.
7) Explain in detail about Dimensionality Reduction. Compare various methods used for it.
8) Discuss various Kernel methods in machine learning.
9) What kind of data is suitable for SVM? How does SVM avoid over-fiting?
10) Give the merits and demerits of K–means algorithm? How to overcome the demerits of it?
11) Explain about K-means algorithm with an example. Describe its convergence.
12) Explain about Principal Component Analysis in detail. How will assist in dimensionality
reduction?