Visualizing Support Vector Machines (SVM) using Python

Support Vector Machines (SVM) are powerful machine learning algorithms used for classification tasks. They work by finding the best hyperplane that separates different classes in the feature space. SVM is particularly useful in both linear and non-linear classification problems. We’ll demonstrate how SVM works with simple datasets and show how the decision boundary changes with different kernels and parameters

Support Vector Machine (SVMs)

Support Vector Machines are supervised learning algorithms that operate by determining the best hyperplane that maximally separates various classes in an existing dataset. The primary goal of SVM is to push the margin between classes to its maximum value, which is the distance from the hyperplane to the nearest data points representing each class, referred to as the support vectors.

• Linear SVM: The data can be linearly separated by a straight line (or a hyperplane in higher dimensions).
• Non-Linear SVM: SVM can also handle data that is not linearly separable by using different kernel functions like:
  • Linear
  • Polynomial
  • Radial Basis Function (RBF)
  • Sigmoid

The flexibility in choosing the kernel allows SVMs to tackle complex classification problems, making them suitable for a wide range of applications.

Visualizing Linear SVMs

Let's start by visualizing a simple linear SVM using Iris dataset. We will create the data and train the SVM model with Scikit-Learn. Then, we will plot the decision boundary and support vectors to see how the model distinguishes between classes.

Step 1: Importing Necessary Libraries and load the Dataset

We will use scikit-learn to load the Iris dataset and Matplotlib for plotting the visualization

import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.svm import SVC

# Load the Iris dataset
iris = datasets.load_iris()
X = iris.data[:, :2]  # Only first two features for visualization
y = iris.target

Step 2 :Training SVM with Linear Kernel

An SVM model with a linear kernel is trained on the Iris dataset. A linear kernel is suitable for linearly separable data, aiming to find the best hyperplane that separates different classes.

clf_linear = SVC(kernel='linear')
clf_linear.fit(X, y)

Step 3: Visualizing the Decision Boundary

To visualize the decision boundary, we create a meshgrid over the feature space. The decision boundary is determined by the SVM's classification of each point in this grid.

# Create a mesh to plot decision boundaries
h = 0.02  # step size in the mesh
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
                     
# Plot decision boundary of Linear SVM
Z_linear = clf_linear.predict(np.c_[xx.ravel(), yy.ravel()])
Z_linear = Z_linear.reshape(xx.shape)
plt.contourf(xx, yy, Z_linear, cmap=plt.cm.Paired, alpha=0.8)
plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired)
plt.title('Linear SVM')
plt.xlabel('Sepal Length')
plt.ylabel('Sepal Width')
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.show()

Output

In the above visualization, linear SVM has classified the data points in a linear way. Even though the accuracy is not that great, we can clearly see that the red section has a ton of misclassified datapoints, but that's where non-linear svm will come into picture.

Visualizing Non-linear SVMs

SVMs are able to deal with non-linear decision boundaries through the aid of kernels. For non-linear classification, the Radial Basis Function (RBF) kernel will be applied, which happens to be a very popularly used kernel with non-linearly separable data

Step 1: Understanding the Impact of Gamma in RBF Kernels

The gamma parameter significantly impacts the RBF kernel's behavior in SVMs. It essentially determines the influence of individual data points on the decision boundary.

• A lower gamma value results in a wider influence of each data point, leading to a smoother decision boundary.
• Conversely, a higher gamma value narrows the influence of data points, creating a more complex and potentially overfitted decision boundary.

Step 2: Defining Gamma Values

A list named gamma_values is defined, containing different values of the gamma hyperparameters, controlling the influence of a single training example.

import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.svm import SVC

iris = datasets.load_iris()
X = iris.data[:, :2]  # Only first two features for visualization
y = iris.target

gamma_values = [0.1, 1, 10, 50, 100, 200]

Step 3: Plotting Decision Boundaries for Each Gamma Value

• Figure: A 20x10 inch figure is created with subplots for each gamma value.
• Gamma Iteration: SVM with an RBF kernel is trained for each gamma.
• Meshgrid: A meshgrid is generated to visualize the decision boundary.
• Plotting: Decision boundaries are visualized with contour plots, and data points are overlaid with class labels.

# Plot decision boundaries for each gamma value
plt.figure(figsize=(20, 10))
for i, gamma in enumerate(gamma_values, 1):
    # Train SVM with RBF kernel
    clf_rbf = SVC(kernel='rbf', gamma=gamma)
    clf_rbf.fit(X, y)

    # Create a mesh to plot decision boundaries
    h = 0.02  # step size in the mesh
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),np.arange(y_min, y_max, h))

    # Plot decision boundary
    plt.subplot(2, 3, i)
    Z_rbf = clf_rbf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z_rbf = Z_rbf.reshape(xx.shape)
    plt.contourf(xx, yy, Z_rbf, cmap=plt.cm.Paired, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Paired)
    plt.title(f'RBF SVM (Gamma={gamma})')
    plt.xlabel('Sepal Length')
    plt.ylabel('Sepal Width')
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())

plt.tight_layout()
plt.show()

Output

In the above visualization, clearly gamma values impacts a lot on the accuracy and complexity of the model.

• We chose 6 different gamma values and due to that we have 6 different visualization, higher the gamma value, higher the accuracy, that's why in the visualization where the gamma value is 200, our model has classified the data points almost perfectly.
• But when it comes to the model where we have chosen gamma value as 50, its alsmot very similar to the result of linear SVM and the accuracy is not that high too.