Implementing SVM from Scratch in Python

Support Vector Machines (SVMs) is a supervised machine learning algorithms used for classification and regression tasks. They work by finding the optimal hyperplane that separates data points of different classes with the maximum margin. We can use Scikit library of python to implement SVM but in this article we will implement SVM from scratch as it enhances our knowledge of this algorithm and have better clarity of how it works.

Prerequisites: Support Vector Machines

1. Importing libraries and Data Visualization

We will be using Iris dataset that is available in Scikit library library.

import numpy as np
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt

data = load_iris()
X = data.data[:, :2]
y = data.target
y = np.where(y == 0, -1, 1)

plt.scatter(X[:, 0], X[:, 1], c=y, cmap='bwr')
plt.xlabel('Sepal Length')
plt.ylabel('Sepal Width')
plt.title('Iris Dataset (Setosa vs. Non-Setosa)')
plt.show()

Output:

2. Defining SVM Class

We'll define an SVM class with methods for training and predicting.

• __init__: Initializes learning rate, regularization, iterations, weight vector and bias.
• fit: Transforms labels, updates weights and bias using gradient descent with regularization and margin checks.
• predict: Computes decision function and returns predicted class based on the sign.

class SVM:
    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iters=1000):
        self.lr = learning_rate
        self.lambda_param = lambda_param
        self.n_iters = n_iters
        self.w = None
        self.b = None

    def fit(self, X, y):
        n_samples, n_features = X.shape
        y_ = np.where(y <= 0, -1, 1)

        self.w = np.zeros(n_features)
        self.b = 0

        for _ in range(self.n_iters):
            for idx, x_i in enumerate(X):
                condition = y_[idx] * (np.dot(x_i, self.w) - self.b) >= 1
                if condition:
                    self.w -= self.lr * (2 * self.lambda_param * self.w)
                else:
                    self.w -= self.lr * (2 * self.lambda_param * self.w - np.dot(x_i, y_[idx]))
                    self.b -= self.lr * y_[idx]

    def predict(self, X):
        approx = np.dot(X, self.w) - self.b
        return np.sign(approx)

3. Training the Model

Now, we can train our SVM model on the dataset. We ceates a mesh grid of points across the feature space, use the model to predict on the mesh grid and reshapes the result to match the grid. After that we draw the decision boundary using contour plots ( ax.contourf ) visualizing the regions classified as -1 or 1.

• svm = SVM(...): Initializes the SVM model with specified learning rate, regularization parameter and number of iterations.
• svm.fit(X, y): Trains the SVM model on the data (X, y).
• plot_decision_boundary(): It plots data points (X[:, 0], X[:, 1]) colored by their labels (y).

svm = SVM(learning_rate=0.001, lambda_param=0.01, n_iters=1000)
svm.fit(X, y)

def plot_decision_boundary(X, y, model):
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap='bwr')
    ax = plt.gca()
    xlim = ax.get_xlim()
    ylim = ax.get_ylim()

    xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 50), np.linspace(ylim[0], ylim[1], 50))
    xy = np.vstack([xx.ravel(), yy.ravel()]).T
    Z = model.predict(xy).reshape(xx.shape)

    ax.contourf(xx, yy, Z, alpha=0.3, cmap='bwr')
    plt.show()

plot_decision_boundary(X, y, svm)

Output:

4. Prediction on Model

To predict the class of new samples we use the predict method of our SVM class.

new_samples = np.array([[0, 0], [4, 4]])
predictions = svm.predict(new_samples)
print(predictions)

Output:

[-1. -1.]

The output [-1. -1.] means that both of the new data points ([0, 0] and [4, 4]) were classified as -1 by the SVM model.

5. Testing the Implementation

To validate our implementation we can compare it with the SVM implementation from popular libraries Scikit-learn.

• clf = SVC(kernel='linear'): Initializes an SVM model using Scikit-learn's SVC class with a linear kernel.
• clf.fit(X, y): Trains the SVM model using the data (X, y).
• print(clf.predict(new_samples)): Uses the trained SVM model to predict the class labels for the new data points and prints the predictions.

from sklearn.svm import SVC
=
clf = SVC(kernel='linear')
clf.fit(X, y)

print(clf.predict(new_samples))

Output:

[-1 -1]

With this we can see it validates the results of our SVM model made in python from scratch. This implementation can be extended further to handle more complex scenarios such as multi-class classification and non-linear data.

Related Posts:

• Supervised Learning
• Scikit-learn