Scribd
Upload
15 views

Support-Vector-Regression

Uploaded by

sathyanarayana.thirunagari
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
15 views

Support-Vector-Regression

Uploaded by

sathyanarayana.thirunagari
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 5

Support Vector Regression (SVR) is an extension of Support Vector Machines

(SVM), primarily designed for regression tasks rather than classification. Instead
of categorizing data points into classes, SVR aims to predict a continuous
variable, like a price or temperature, while also maximizing generalization. SVR
is particularly effective for datasets with complex relationships due to its ability
to handle non-linear data with kernel functions.
Key Concepts in Support Vector Regression

1. Support Vectors: In SVR, support vectors are data points that lie closest to
the predicted hyperplane. These points influence the shape and position of
the regression hyperplane. The objective is to find the hyperplane that is at
most a certain margin away from these support vectors while minimizing
error.
2. Hyperplane: Just like in SVM, the SVR model seeks a hyperplane that
best fits the data points. In a two-dimensional space, this would be a line,
but in higher-dimensional spaces, it can be considered a flat, n-dimensional
plane.
3. Margin of Tolerance (Epsilon, £): The margin of tolerance is a critical
aspect of SVR. It represents the acceptable range within which predictions
can deviate from the actual values. The width of the epsilon margin (often
just called "epsilon") is specified as a parameter. Any point lying within
this epsilon-tube (margin) is not penalized, while points outside it incur a
penalty in the objective function.
4. Error Penalty (C): Another critical parameter is CCC, which defines the
trade-off between the model’s complexity (allowing more data points
outside the margin) and the prediction accuracy. A higher CCC value
allows more data points outside the margin, making the model more
sensitive to errors. A lower CCC value leads to a simpler, smoother model
but may result in underfitting.
Mathematical Formulation of SVR

The SVR problem can be formulated as follows:

Given data points yi), where X{ is the feature vector and yj is the target

variable, SVR tries to find a function f(x) = w • x + b such that the

predictions are as close as possible to the actual target values within the

margin e.
The objective of SVR is:

1. To minimize the magnitude of the weights IIw|| which controls the


flatness of the hyperplane, and
2. To ensure that most of the data points fall within the epsilon margin.
The optimization problem is:

Minimize ^|M|2 +
C+ 4*)
t=l

subject to:

Vi - ( w ■ X i + b) <e+

4 (w ■ Xi + b) - yi <

e + 4*

4,4* >o

Where:

• ^i\xi_i^i and <^i*\xi_iA*^i* are slack variables for cases where points
fall outside the epsilon margin.
• CCC is the regularization parameter controlling the trade-off between
the flatness of the hyperplane and the amount of allowed deviation.
Kernel Functions in SVR

One of SVR’s most powerful features is its use of kernel functions.


Kernels allow SVR to handle non-linear data by implicitly mapping
input features into a higherdimensional space where linear regression is
possible. SVR supports several kernel functions:
1. Linear Kernel: Used for linearly separable data or when the relationship
between the features and the target variable is approximately linear.
2. Polynomial Kernel: Adds polynomial features of the input data, which
can be helpful when there are polynomial relationships in the data.
3. Radial Basis Function (RBF) Kernel: The most commonly used kernel,
RBF maps data points into an infinite-dimensional space and can capture
complex relationships.
4. Sigmoid Kernel: Often used in neural networks, the sigmoid kernel
introduces non-linearity and is suitable for specific types of data.
How SVR Works Step-by-Step

1. Data Transformation: If necessary, the input data is transformed using a


kernel function, mapping it to a higher-dimensional space where a linear
separation is possible.
2. Finding the Optimal Hyperplane: SVR tries to fit a hyperplane that
maximizes the margin between the data points and the decision boundary.
Instead of seeking zero error, SVR minimizes the prediction error by
keeping most predictions within a given margin (epsilon).
3. Support Vectors and Epsilon-Tube: Points within the epsilon-tube are not
considered in calculating the error. Only points outside the epsilon margin
contribute to the error and influence the position of the hyperplane.
4. Prediction: Once the model is trained, it uses the learned hyperplane to
make predictions on new data. The predictions are based on how well the
points lie within the acceptable epsilon tolerance around the hyperplane.
Visual Representation of SVR

Consider a 2D example where we have position levels on the x-axis and


salaries on the y-axis. SVR would:
• Try to fit a line (or hyperplane) within an epsilon margin.
• Only data points that fall outside this margin would affect the model's
training.
Implementing SVR in Python
python Copy code
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.svm import SVR
from sklearn.preprocessing import StandardScaler

# Load dataset
dataset = pd.read csv(’Position Salaries.csv’)
X = dataset.iloc[:, 1:2].values y =
dataset.iloc[:, 2].values
# Feature Scaling
scX = StandardScaler()
scy = StandardScaler()
X = sc X.fit
transform(X)
y = sc y.fit
transform(y.reshape(-
1, 1)).flatten()

# Initialize SVR with an RBF kernel


regressor = SVR(kernel='rbf')
regressor.fit(X, y)
# Predicting for a specific
level level = 6.5
scaled level = sc
X.transform([[level]])
predicted salary scaled = regressor.predict(scaled level) predicted
salary = sc y.inverse transform([predicted salary scaled])
print(f”Predicted Salary for level {level}: {predicted salary[0]}”)

# Visualize the SVR results


X grid = np.arange(min(X), max(X), 0.01).reshape(-1,
1) plt.scatter(X, y, color='red')
plt.plot(X grid, regressor.predict(X grid),
color='blue')
plt.title(’SVR Fit’)
plt.xlabel(’Position Level’)
plt.ylabel(’Salary’)
plt.show()

Tuning Hyperparameters in SVR


Tuning hyperparameters can significantly impact SVR’s
performance:
1. C: Controls the penalty for errors outside the epsilon margin. Larger
values of C mean less tolerance for errors.
2. Epsilon (£): Defines the margin within which no penalty is given.
Summary of Advantages
Increasing epsilon and
can Disadvantages
simplify the model by increasing tolerance.
3. Kernel Choice: Choose the kernel based on the data characteristics.
Advantages:
• Flexible and Robust: Can handle non-linear data with kernels.
• Support Vector Based: Only uses support vectors, making it less sensitive
to large datasets.
• Regularization: The C parameter provides regularization, helping control
overfitting.
Disadvantages:

• Scaling Requirement: SVR is sensitive to feature scaling.


• Computationally Intensive: SVR with non-linear kernels can be slow for
large datasets.
• Hyperparameter Sensitivity: Requires careful tuning of C, epsilon, and
kernel parameters.

Applications of SVR

SVR is widely used in areas where continuous predictions are essential,

such as:

• Finance: Stock price prediction, financial forecasting.


• Engineering: Predicting mechanical properties or temperature.
• Healthcare: Modeling disease progression or predicting patient
outcomes.
• Energy: Load forecasting or energy consumption predictions.
Conclusion

Support Vector Regression is a powerful method for regression tasks,


especially when paired with kernel functions to handle non-linear data. By
optimizing the margin of tolerance and minimizing errors, SVR provides a
flexible and robust solution for predicting continuous outcomes. With
careful tuning, SVR can deliver high-performance predictions across
various applications.

You might also like