Implementing SVM and Kernel SVM with Python's Scikit-Learn

In this article we will implement a classification model using Scikit learn implementation for SVM model in Python. Then we will try to understand what is a kernel and how it can helps us to achieve better performance by learning non-linear boundaries in the dataset.

What is a SVM algorithm?

Support vector machines (SVMs) are a type of supervised learning algorithm that can be used for classification or regression tasks. In simple terms, an SVM constructs a hyperplane or set of hyperplanes in a high-dimensional space, which can be used to separate different classes or to predict continuous variables.

The main idea behind SVMs is to find the hyperplane in the high-dimensional space that has the maximum margin, or the maximum distance between the data points of the different classes. This hyperplane is called the maximum-margin hyperplane, and the data points that are closest to the hyperplane are called support vectors. The position and orientation of the maximum-margin hyperplane can be determined using mathematical optimization techniques.

SVMs have been widely used in many applications, including image and text classification, protein classification, and regression problems. They have several advantages, such as the ability to handle high-dimensional data, the ability to perform non-linear classification using the kernel trick, and the ability to provide probability estimates for classification tasks. However, they can be computationally expensive and may not be suitable for very large datasets.

To show the usage of the kernel SVM let's import the necessary libraries and the iris dataset.

# Import necessary libraries
from sklearn import svm
from sklearn import datasets

# Load the Iris dataset
iris = datasets.load_iris()
# We only take the first two
# features for simplicity
X = iris.data[:, :2]
y = iris.target

Now we will use SupportVectorClassifier as currently we are dealing with a classification problem.

# Fit the SVM model
model = svm.SVC(kernel='linear')
model.fit(X, y)

# Predict using the SVM model
predictions = model.predict(X)

# Evaluate the predictions
accuracy = model.score(X, y)
print("Accuracy of SVM:", accuracy)

Output : 

Accuracy of SVM: 0.82

The above code is an example of using a support vector machine (SVM) model to make predictions on the Iris dataset. The Iris dataset is a well-known dataset in machine learning that contains measurements of various characteristics of iris flowers, such as sepal length and width, and the species of the flower.

The code first imports the necessary modules and libraries, including the SVM module from Scikit-learn and the Iris dataset from Scikit-learn's datasets module. Then, it loads the Iris dataset and extracts the first two features from each example (sepal length and width), as well as the target labels (the species of the flower).

Next, the code creates an SVM model using the SVC class from Scikit-learn, and specifies that it should use a linear kernel. Then, it trains the model on the data using the fit() method, and makes predictions on the same data using the predict() method. Finally, it evaluates the predictions by computing the accuracy of the model (the fraction of examples that were predicted correctly) using the score() method. The accuracy is then printed to the console.

What is Kernel SVM? 

Kernel support vector machines (SVMs) are a variant of support vector machines (SVMs) that use kernel functions to find the maximum-margin hyperplane in non-linear classification or regression problems. In simple terms, a kernel function transforms the original data into a higher-dimensional space, where it becomes linearly separable. The maximum-margin hyperplane is then found in this higher-dimensional space using an SVM.

Kernel SVMs have several advantages over regular SVMs. They can handle non-linear classification or regression tasks without having to explicitly perform the data transformation, which can be computationally expensive. They also allow the use of different kernel functions, which can provide different types of non-linear transformations and can be chosen based on the specific characteristics of the data.

The most commonly used kernel functions in kernel SVMs are the linear, polynomial, and radial basis function (RBF) kernels. The linear kernel is used for linear classification or regression tasks, the polynomial kernel can handle non-linear problems, and the RBF kernel is often used in classification tasks with a large number of features.

Kernel SVMs have been widely used in many applications, such as image and text classification, protein classification, and regression problems. However, they can be computationally expensive and may not be suitable for very large datasets. Additionally, choosing the right kernel function and the corresponding hyperparameters can be challenging and requires some domain knowledge and experimentation.

Here is an example of how to implement Support Vector Machines (SVM) and Kernel SVM with Python's Scikit-learn library:

from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

# Dummy function to simulate loading the data from a file

def load_data():
    X_train = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
    X_test = [[10, 11, 12], [13, 14, 15], [16, 17, 18]]
    y_train = [1, 0, 1]
    y_test = [0, 1, 0]
    return X_train, X_test, y_train, y_test

# Load the data and split it into training and test sets
X_train, X_test,\
    y_train, y_test = load_data()

Now let's normalize the dataset using the StandardScaler function from the sklearn library this will helps us achieve stable and faster training process of the model.

# Import the necessary modules and libraries
# Scale the features using standardization
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Next, we create a KSVM model using the radial basis function (RBF) kernel, with the gamma and C hyperparameters set to 0.1 and 10.0, respectively.

# Create a kernel support vector machine model
ksvm = svm.SVC(kernel='rbf',
               gamma=0.1,
               C=10.0)

# Train the model on the training data
ksvm.fit(X_train, y_train)

# Evaluate the model on the test data
accuracy = ksvm.score(X_test, y_test)
print('Accuracy:', accuracy)

Output : 

Accuracy: 0.3333333333333333

Difference between the SVM and kernel SVM: