PyTorch Tutorial

05. Linear Regression with PyTorch

Lecturer : Hongpu Liu Lecture 5-1 PyTorch Tutorial @ SLAM Research Group
Revision

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 𝑦

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠

Lecturer : Hongpu Liu Lecture 5-2 PyTorch Tutorial @ SLAM Research Group
Revision

print("predict (before training)", 4, forward(4).item())

for epoch in range(100):

for x, y in zip(x_data, y_data):
l = loss(x, y)
l.backward()
print('\tgrad:', x, y, w.grad.item())
w.data = w.data - 0.01 * w.grad.data

w.grad.data.zero_()

print("progress:", epoch, l.item())

print("predict (after training)", 4, forward(4).item())

Lecturer : Hongpu Liu Lecture 5-3 PyTorch Tutorial @ SLAM Research Group
PyTorch Fashion

Prepare dataset Design model using Class

1 2
we shall talk about this later inherit from nn.Module

Construct loss and optimizer Training cycle

3 4
using PyTorch API forward, backward, update

Lecturer : Hongpu Liu Lecture 5-4 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 1. Prepare dataset

In PyTorch, the computational graph is in mini-batch fashion, so X and Y are 3 × 1 Tensors.

(1)
𝑦𝑝𝑟𝑒𝑑
𝑥 (1)
(2)
𝑦𝑝𝑟𝑒𝑑 = 𝜔 ∙ 𝑥 (2) + 𝑏
(3)
𝑦𝑝𝑟𝑒𝑑 𝑥 (3)

import torch

x_data = torch.Tensor([[1.0], [2.0], [3.0]])

y_data = torch.Tensor([[2.0], [4.0], [6.0]])

Lecturer : Hongpu Liu Lecture 5-5 PyTorch Tutorial @ SLAM Research Group
Revision: Gradient Descent Algorithm
Derivative Gradient
𝑁
𝜕𝑐𝑜𝑠𝑡(𝜔) 𝜕 1 𝜕𝑐𝑜𝑠𝑡
= ෍ 𝑥𝑛 ∙ 𝜔 − 𝑦𝑛 2
𝜕𝜔 𝜕𝜔 𝑁 𝜕𝜔
𝑛=1
𝑁
1 𝜕 Update
= ෍ 𝑥𝑛 ∙ 𝜔 − 𝑦𝑛 2
𝑁 𝜕𝜔 𝜕𝑐𝑜𝑠𝑡
𝑛=1 𝜔 =𝜔−𝛼
𝑁 𝜕𝜔
1 𝜕(𝑥𝑛 ∙ 𝜔 − 𝑦𝑛 )
= ෍ 2 ∙ (𝑥𝑛 ∙ 𝜔 − 𝑦𝑛 )
𝑁 𝜕𝜔 Update
𝑛=1
𝑁 𝑁
1 1
= ෍ 2 ∙ 𝑥𝑛 ∙ (𝑥𝑛 ∙ 𝜔 − 𝑦𝑛 ) 𝜔 = 𝜔 − 𝛼 ෍ 2 ∙ 𝑥𝑛 ∙ (𝑥𝑛 ∙ 𝜔 − 𝑦𝑛 )
𝑁
𝑁 𝑛=1
𝑛=1

Lecturer : Hongpu Liu Lecture 5-6 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

Affine Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 + 𝑏 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑦
Linear Unit

𝑥 ∗ + 𝑦ො 𝑙𝑜𝑠𝑠(𝑦,
ො 𝑦) 𝑙𝑜𝑠𝑠

𝜔 𝑏

Lecturer : Hongpu Liu Lecture 5-7 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

Our model class should be inherit

from nn.Module, which is Base
class LinearModel(torch.nn.Module):
def __init__(self): class for all neural network
super(LinearModel, self).__init__()
self.linear = torch.nn.Linear(1, 1) modules.
def forward(self, x):
y_pred = self.linear(x)
return y_pred

model = LinearModel()

Lecturer : Hongpu Liu Lecture 5-8 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

class LinearModel(torch.nn.Module):
def __init__(self):
super(LinearModel, self).__init__() Member methods __init__() and
self.linear = torch.nn.Linear(1, 1)
forward() have to be implemented.
def forward(self, x):
y_pred = self.linear(x)
return y_pred

model = LinearModel()

Lecturer : Hongpu Liu Lecture 5-9 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

class LinearModel(torch.nn.Module):
def __init__(self):
super(LinearModel, self).__init__() Just do it. : )
self.linear = torch.nn.Linear(1, 1)

def forward(self, x):

y_pred = self.linear(x)
return y_pred

model = LinearModel()

Lecturer : Hongpu Liu Lecture 5-10 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

Class nn.Linear contain two

class LinearModel(torch.nn.Module):
def __init__(self): member Tensors: weight and bias.
super(LinearModel, self).__init__()
self.linear = torch.nn.Linear(1, 1)

def forward(self, x): Linear Unit

y_pred = self.linear(x)
return y_pred
𝑥 ∗ + 𝑦ො
model = LinearModel()

𝜔 𝑏

Lecturer : Hongpu Liu Lecture 5-11 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

Lecturer : Hongpu Liu Lecture 5-12 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

(1)
𝑦𝑝𝑟𝑒𝑑
𝑥 (1)
(2)
Output 𝑦𝑝𝑟𝑒𝑑 = 𝜔 ∙ 𝑥 (2) + 𝑏
(3)
𝑦𝑝𝑟𝑒𝑑 𝑥 (3) Input

Lecturer : Hongpu Liu Lecture 5-13 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

class LinearModel(torch.nn.Module):
def __init__(self): Class nn.Linear has implemented the
super(LinearModel, self).__init__()
self.linear = torch.nn.Linear(1, 1) magic method __call__(), which enable
def forward(self, x): the instance of the class can be called
y_pred = self.linear(x)
return y_pred just like a function. Normally the
model = LinearModel() forward() will be called.
Pythonic!!!

Lecturer : Hongpu Liu Lecture 5-14 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 2. Design Model

class LinearModel(torch.nn.Module):
def __init__(self):
super(LinearModel, self).__init__()
self.linear = torch.nn.Linear(1, 1)

def forward(self, x):

y_pred = self.linear(x)
return y_pred
Create a instance of class
model = LinearModel()
LinearModel.

Lecturer : Hongpu Liu Lecture 5-15 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 3. Construct Loss and Optimizer

criterion = torch.nn.MSELoss(size_average=False)

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Also inherit from nn.Module.

Lecturer : Hongpu Liu Lecture 5-16 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 3. Construct Loss and Optimizer

criterion = torch.nn.MSELoss(size_average=False)

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Lecturer : Hongpu Liu Lecture 5-17 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 3. Construct Loss and Optimizer

criterion = torch.nn.MSELoss(size_average=False)

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Lecturer : Hongpu Liu Lecture 5-18 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 4. Training Cycle

for epoch in range(100):

y_pred = model(x_data) Forward: Predict
loss = criterion(y_pred, y_data)
print(epoch, loss)

optimizer.zero_grad()
loss.backward()
optimizer.step()

Lecturer : Hongpu Liu Lecture 5-19 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 4. Training Cycle

for epoch in range(100):

y_pred = model(x_data)
loss = criterion(y_pred, y_data) Forward: Loss
print(epoch, loss)

optimizer.zero_grad()
loss.backward()
optimizer.step()

Lecturer : Hongpu Liu Lecture 5-20 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 4. Training Cycle

for epoch in range(100):

y_pred = model(x_data) NOTICE:
loss = criterion(y_pred, y_data)
print(epoch, loss) The grad computed by .backward()
optimizer.zero_grad() will be accumulated.
loss.backward()
optimizer.step() So before backward, remember set
the grad to ZERO!!!

Lecturer : Hongpu Liu Lecture 5-21 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 4. Training Cycle

for epoch in range(100):

y_pred = model(x_data)
loss = criterion(y_pred, y_data)
print(epoch, loss)

optimizer.zero_grad()
loss.backward() Backward: Autograd
optimizer.step()

Lecturer : Hongpu Liu Lecture 5-22 PyTorch Tutorial @ SLAM Research Group
Linear Regression – 4. Training Cycle

for epoch in range(100): for x, y in zip(x_data, y_data):

y_pred = model(x_data) ……
loss = criterion(y_pred, y_data) w.data = w.data - 0.01 * w.grad.data
print(epoch, loss)

optimizer.zero_grad()
loss.backward()
optimizer.step() Update

Lecturer : Hongpu Liu Lecture 5-23 PyTorch Tutorial @ SLAM Research Group
Linear Regression – Test Model

# Output weight and bias

print('w = ', model.linear.weight.item())
print('b = ', model.linear.bias.item())

# Test Model
x_test = torch.Tensor([[4.0]])
y_test = model(x_test)
print('y_pred = ', y_test.data)

100 Iterations 1000 Iterations

Lecturer : Hongpu Liu Lecture 5-24 PyTorch Tutorial @ SLAM Research Group
Linear Regression
import torch

Prepare dataset
1
x_data = torch.Tensor([[1.0], [2.0], [3.0]])
y_data = torch.Tensor([[2.0], [4.0], [6.0]])

class LinearModel(torch.nn.Module): we shall talk about this later

def __init__(self):
super(LinearModel, self).__init__()
self.linear = torch.nn.Linear(1, 1)

def forward(self, x):

Design model using Class
2
y_pred = self.linear(x)
return y_pred
model = LinearModel() inherit from nn.Module
criterion = torch.nn.MSELoss(size_average=False)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for epoch in range(1000):

y_pred = model(x_data)
Construct loss and optimizer
loss = criterion(y_pred, y_data)
print(epoch, loss.item()) 3
using PyTorch API
optimizer.zero_grad()
loss.backward()
optimizer.step()

print('w = ', model.linear.weight.item())

Training cycle
4
print('b = ', model.linear.bias.item())

x_test = torch.Tensor([[4.0]])
y_test = model(x_test) forward, backward, update
print('y_pred = ', y_test.data)

Lecturer : Hongpu Liu Lecture 5-25 PyTorch Tutorial @ SLAM Research Group
Exercise 5-1: Try Different Optimizer in Linear Regression

• torch.optim.Adagrad
• torch.optim.Adam
• torch.optim.Adamax
• torch.optim.ASGD
• torch.optim.LBFGS
• torch.optim.RMSprop
• torch.optim.Rprop
• torch.optim.SGD

Lecturer : Hongpu Liu Lecture 5-26 PyTorch Tutorial @ SLAM Research Group
Exercise 5-2: Read more example from official tutorial

https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

Lecturer : Hongpu Liu Lecture 5-27 PyTorch Tutorial @ SLAM Research Group
 PyTorch Tutorial
05. Linear Regression with PyTorch

Lecturer : Hongpu Liu Lecture 5-28 PyTorch Tutorial @ SLAM Research Group