Gaussian Process

Hao Yan

June 2, 2024

Contents
1 Gaussian Process 2
1.1 Gaussian Process Prior: . . . . . . . . . . . . . . . . . . . . . 2
1.2 Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Posterior Distribution: . . . . . . . . . . . . . . . . . . . . . . 2

2 Tutorial on GPytorch.models.exact_gp 2
2.1 Pytorch Implementation - ExactGP Class . . . . . . . . . . . 2
2.2 Pytorch Implementation - ExactGP . . . . . . . . . . . . . . . 3
2.3 Pytorch Implementation - Predictionstrategy . . . . . . . . . . 3

3 Examlpe in Gpytorch 4
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3.2 Data Preparation - Create Synthetic Data . . . . . . . . . . . 4
3.3 Data Preparation - Visualize Synthetic Data . . . . . . . . . . 4
3.4 Model Definition - Define the GP Model . . . . . . . . . . . . 5
3.5 Model Training - Train the GP Model . . . . . . . . . . . . . 5
3.6 Model Training - Training Results . . . . . . . . . . . . . . . 6
3.7 Model Parameters . . . . . . . . . . . . . . . . . . . . . . . . 7
3.8 Likelihood Parameters . . . . . . . . . . . . . . . . . . . . . . 7
3.9 Predictions - Make Predictions with the GP Model . . . . . . 7
3.10 Predictions - Visualize Predictions . . . . . . . . . . . . . . . 8
3.11 Summary - Key Points . . . . . . . . . . . . . . . . . . . . . . 8
3.12 Summary - Further Reading . . . . . . . . . . . . . . . . . . . 9

Area

Keywords

1
1 Gaussian Process
1.1 Gaussian Process Prior:
f (x) ∼ GP m(x), k x, x′

where m(x) is the mean function and k (x, x′ ) is the covariance function.

1.2 Likelihood
Given training data X = {xi }ni=1 and corresponding targets y = {yi }ni=1 , we
assume:
y = f (X) + ϵ, ϵ ∼ N 0, σn2 I

Here, σn2 represents the noise variance.

1.3 Posterior Distribution:

The posterior distribution over function values f∗ at test inputs X∗ is given
by:
p (f∗ | X∗ , X, y) = N (f∗ | µ∗ , Σ∗ )

−1
µ∗ = K∗f Kf f + σn2 I y
where Here:
2 −1


Σ∗ = K∗∗ − K∗f Kf f + σn I Kf ∗
• Kf f is the covariance matrix between training points.
• K∗f is the covariance matrix between test and training points.
• K∗∗ is the covariance matrix between test points.

2 Tutorial on GPytorch.models.exact_gp
2.1 Pytorch Implementation - ExactGP Class
• File: gpytorch/models/exact_gp.py
• __init__
def __init__(self, train_inputs, train_targets, likelihood):
...
self.train_inputs = tuple(tri.unsqueeze(-1) if tri.ndimension() == 1 else t
self.train_targets = train_targets
self.likelihood = likelihood
self.prediction_strategy = None

2
 • forward: Defines the mean and covariance functions for the GP prior.

def forward(self, x):

mean = self.mean_module(x)
covar = self.covar_module(x)
return gpytorch.distributions.MultivariateNormal(mean, covar)

2.2 Pytorch Implementation - ExactGP

• Call existing prediction_strategy.exact_prediction to compute
the posterior distribution

def __call__(self, *args, **kwargs):

...
# Posterior mode
...
if self.prediction_strategy is None:
train_output = super().__call__(*train_inputs, **kwargs)
self.prediction_strategy = prediction_strategy(
train_inputs=train_inputs,
train_prior_dist=train_output,
train_labels=self.train_targets,
likelihood=self.likelihood,
)
...
# Get the joint distribution for training/test data
full_output = super(ExactGP, self).__call__(*full_inputs, **kwargs)
...
predictive_mean, predictive_covar = self.prediction_strategy.exact_predicti
return full_output.__class__(predictive_mean, predictive_covar)

2.3 Pytorch Implementation - Predictionstrategy

• gpytorch/models/exact_prediction_strategies.py implement the
prediction strategy

−1
• exact_predictive_mean computes µ∗ = K∗f Kf f + σ 2 I y

−1
• exact_predictive_covar computes Σ∗ = K∗∗ −K∗f Kf f + σ 2 I Kf ∗

3
3 Examlpe in Gpytorch
3.1 Introduction
• Gaussian Process Regression (GPR) is a non-parametric method.

• Useful for predicting the value of a function at unseen points.

• Provides uncertainty estimates along with predictions.

3.2 Data Preparation - Create Synthetic Data

import torch
import gpytorch
from matplotlib import pyplot as plt

# Training data
train_x = torch.linspace(0, 1, 100)
train_y = torch.sin(train_x * (2 * torch.pi)) + torch.randn(train_x.size()) * 0.2

# Testing data
test_x = torch.linspace(0, 1, 51)
test_y = torch.sin(test_x * (2 * torch.pi))

3.3 Data Preparation - Visualize Synthetic Data

plt.figure(figsize=(10, 5))
plt.plot(train_x.numpy(), train_y.numpy(), 'k*')
plt.plot(test_x.numpy(), test_y.numpy(), 'b')
plt.title("Training Data and True Function")
plt.xlabel("x")
plt.ylabel("y")
plt.legend(['Observed Data', 'True Function'])
plt.show()

4
3.4 Model Definition - Define the GP Model
• Subclass ‘gpytorch.models.ExactGP‘ to define our model.

• Use ‘ConstantMean‘ and ‘RBFKernel‘.

class GPRegressionModel(gpytorch.models.ExactGP):
def __init__(self, train_x, train_y, likelihood):
super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)
self.mean_module = gpytorch.means.ConstantMean()
self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKe

def forward(self, x):

mean_x = self.mean_module(x)
covar_x = self.covar_module(x)
return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

# Define likelihood and model

likelihood = gpytorch.likelihoods.GaussianLikelihood()
model = GPRegressionModel(train_x, train_y, likelihood)

3.5 Model Training - Train the GP Model

• Use the Adam optimizer.

• Maximize the marginal log likelihood.

5
model.train()
likelihood.train()

# Use the Adam optimizer

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

# "Loss" for GPs - the marginal log likelihood

mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)

3.6 Model Training - Training Results

• Observe the loss decreasing over iterations.

• Model parameters are optimized to fit the data.

# Plot training loss

loss_values = []
training_iterations = 600
for i in range(training_iterations):
optimizer.zero_grad()
output = model(train_x)
loss = -mll(output, train_y)
loss_values.append(loss.item())
loss.backward()
optimizer.step()

plt.plot(loss_values)
plt.title("Training Loss Over Iterations")
plt.xlabel("Iteration")
plt.ylabel("Loss")
plt.show()

6
3.7 Model Parameters
3.8 Likelihood Parameters
3.9 Predictions - Make Predictions with the GP Model
• Use the trained model to make predictions on test data.

• Compute the posterior mean and covariance.

model.eval()
likelihood.eval()

# Make predictions
with torch.no_grad(), gpytorch.settings.fast_pred_var():
observed_pred = likelihood(model(test_x))

# Get mean and lower/upper confidence bounds

mean = observed_pred.mean
lower, upper = observed_pred.confidence_region()

7
3.10 Predictions - Visualize Predictions
• Plot the training data, true function, and predictions with uncertainty.

# Plot training data as black stars

plt.figure(figsize=(12, 6))
plt.plot(train_x.numpy(), train_y.numpy(), 'k*')
# Plot true function
plt.plot(test_x.numpy(), test_y.numpy(), 'b')
# Plot predictive means as blue line
plt.plot(test_x.numpy(), mean.numpy(), 'r')
# Shade between the lower and upper confidence bounds
plt.fill_between(test_x.numpy(), lower.numpy(), upper.numpy(), alpha=0.5)
plt.legend(['Observed Data', 'True Function', 'Mean', 'Confidence'])
plt.title("Predictions with Uncertainty")
plt.xlabel("x")
plt.ylabel("y")
plt.show()

3.11 Summary - Key Points

• ExactGP model in GPyTorch allows efficient Gaussian Process re-
gression.

• Training involves optimizing the marginal log likelihood.

• Predictions provide both mean estimates and uncertainty bounds.

8
3.12 Summary - Further Reading
• GPyTorch Documentation

• GPyTorch Paper: Advances in Scalable Gaussian Processes