PyTorch Tutorial

04. Back Propagation

Lecturer : Hongpu Liu Lecture 4-1 PyTorch Tutorial @ SLAM Research Group
Compute gradient in simple network

Neuron
Linear Model 𝒙
∗ ෝ
𝒚 𝒙 ∗ ෝ
𝒚
𝑦ො = 𝑥 ∗ 𝜔 𝝎
𝝎

Stochastic Gradient Descent Derivative of Loss Function

𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠𝑛
𝜔 =𝜔−𝛼 𝜕𝜔
= 2 ∙ 𝑥𝑛 ∙ (𝑥𝑛 ∙ 𝜔 − 𝑦𝑛 )
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-2 PyTorch Tutorial @ SLAM Research Group
What about the complicated network?

Gradient

𝜕𝑙𝑜𝑠𝑠
=?
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-3 PyTorch Tutorial @ SLAM Research Group
Computational Graph
𝑋 Input

Weight 𝑊1 𝑀𝑀 Matrix multiplication

A two layer neural network

𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2

Lecturer : Hongpu Liu Lecture 4-4 PyTorch Tutorial @ SLAM Research Group
Computational Graph
𝑋 Input

Weight 𝑊1 𝑀𝑀 Matrix multiplication

A two layer neural network Bias 𝑏1 𝐴𝐷𝐷 Vector addition

𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2

Lecturer : Hongpu Liu Lecture 4-5 PyTorch Tutorial @ SLAM Research Group
Computational Graph
𝑋

First Layer
𝑊1 𝑀𝑀

A two layer neural network 𝑏1 𝐴𝐷𝐷

𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2

Lecturer : Hongpu Liu Lecture 4-6 PyTorch Tutorial @ SLAM Research Group
Computational Graph
𝑋

First Layer
𝑊1 𝑀𝑀

A two layer neural network 𝑏1 𝐴𝐷𝐷

𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2
𝑊2 𝑀𝑀

Lecturer : Hongpu Liu Lecture 4-7 PyTorch Tutorial @ SLAM Research Group
Computational Graph
𝑋

First Layer
𝑊1 𝑀𝑀

A two layer neural network 𝑏1 𝐴𝐷𝐷

𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2
𝑊2 𝑀𝑀

𝑏2 𝐴𝐷𝐷

𝑦ො

Lecturer : Hongpu Liu Lecture 4-8 PyTorch Tutorial @ SLAM Research Group
Computational Graph
𝑋

First Layer
𝑊1 𝑀𝑀

A two layer neural network 𝑏1 𝐴𝐷𝐷

𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2
𝑊2 𝑀𝑀
Second Layer

𝑏2 𝐴𝐷𝐷

𝑦ො

Lecturer : Hongpu Liu Lecture 4-9 PyTorch Tutorial @ SLAM Research Group
What problem about this two layer neural network?
𝑋
A two layer neural network

𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2 𝑊1 𝑀𝑀

𝑋
= 𝑊2 ∙ 𝑊1 ∙ 𝑋 + (𝑊2 𝑏1 +𝑏2 )
𝑏1 𝐴𝐷𝐷 𝑊 𝑀𝑀
=𝑊∙𝑋+𝑏

𝑊2 𝑀𝑀 𝑏 𝐴𝐷𝐷

𝑦ො
𝑏2 𝐴𝐷𝐷

𝑦ො

Lecturer : Hongpu Liu Lecture 4-10 PyTorch Tutorial @ SLAM Research Group
What problem about this two layer neural network?
𝑋
A two layer neural network
First Layer
𝑦ො = 𝑊2 𝑊1 ∙ 𝑋 + 𝑏1 + 𝑏2 𝑊1 𝑀𝑀

= 𝑊2 ∙ 𝑊1 ∙ 𝑋 + (𝑊2 𝑏1 +𝑏2 )
Nonlinear
𝑏1 𝐴𝐷𝐷 𝜎
=𝑊∙𝑋+𝑏 Function

Second Layer
A nonlinear function is required 𝑊2 𝑀𝑀
by each layer.

We shall talk about this later. Nonlinear

𝑏2 𝐴𝐷𝐷 𝜎
Function

𝑦ො

Lecturer : Hongpu Liu Lecture 4-11 PyTorch Tutorial @ SLAM Research Group
The composition of functions and Chain Rule

Lecturer : Hongpu Liu Lecture 4-12 PyTorch Tutorial @ SLAM Research Group
Chain Rule – 1. Create Computational Graph (Forward)

𝑥
𝑥
𝑓(𝑥, 𝜔)

𝑓 𝑧 Loss

𝜔
𝜔

Lecturer : Hongpu Liu Lecture 4-13 PyTorch Tutorial @ SLAM Research Group
Chain Rule – 2. Local Gradient

𝑥
𝜕𝑧
𝑥 𝜕𝑥
𝑓(𝑥, 𝜔)

𝑓 𝑧 Loss

𝜔
𝜕𝑧
𝜔 𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-14 PyTorch Tutorial @ SLAM Research Group
Chain Rule – 3. Given gradient from successive node

𝑥
𝜕𝑧
𝑥 𝜕𝑥
𝑓(𝑥, 𝜔)

𝑓 𝑧 Loss
𝜕𝐿
𝜔 𝜕𝑧
𝜕𝑧
𝜔 𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-15 PyTorch Tutorial @ SLAM Research Group
Chain Rule – 4. Use chain rule to compute the gradient (Backward)

𝑥
𝜕𝑧
𝑥 𝜕𝑥
𝜕𝐿 𝑓(𝑥, 𝜔)
𝜕𝑥
𝑓 𝑧 Loss
𝜕𝐿
𝜔 𝜕𝑧
𝜕𝑧
𝜔 𝜕𝜔
𝜕𝐿
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-16 PyTorch Tutorial @ SLAM Research Group
Chain Rule – 4. Use chain rule to compute the gradient (Backward)

𝑥
𝜕𝑧
𝑥 𝜕𝑥
𝜕𝐿 𝜕𝐿 𝜕𝑧 𝑓(𝑥, 𝜔)
= ∙
𝜕𝑥 𝜕𝑧 𝜕𝑥
𝑓 𝑧 Loss
𝜕𝐿
𝜔 𝜕𝑧
𝜕𝑧
𝜔 𝜕𝜔
𝜕𝐿 𝜕𝐿 𝜕𝑧
= ∙
𝜕𝜔 𝜕𝑧 𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-17 PyTorch Tutorial @ SLAM Research Group
Example: 𝑓 = 𝑥 ∙ 𝜔

𝑥
𝜕𝑧
𝑥 𝜕𝑥
=ω
𝑧

𝑓 =𝑥∙𝜔 𝑧 Loss

𝜔
𝜕𝑧
𝜔 𝜕𝜔
=𝑥

Lecturer : Hongpu Liu Lecture 4-18 PyTorch Tutorial @ SLAM Research Group
Example: 𝑓 = 𝑥 ∙ 𝜔, 𝑥 = 2, 𝜔 = 3

Forward

𝑥=2
𝜕𝑧
𝑥 𝜕𝑥
=ω
𝑧=6

𝑓 =𝑥∙𝜔 𝑧 Loss

𝜔=3
𝜕𝑧
𝜔 𝜕𝜔
=𝑥

Lecturer : Hongpu Liu Lecture 4-19 PyTorch Tutorial @ SLAM Research Group
Example: Backward

Backward

𝑥=2
𝜕𝑧
𝑥 𝜕𝑥
=ω
𝑧=6

𝑓 =𝑥∙𝜔 𝑧 Loss
𝜕𝐿
𝜔=3 =5
𝜕𝑧
𝜕𝑧
𝜔 𝜕𝜔
=𝑥 Given by successive node

Lecturer : Hongpu Liu Lecture 4-20 PyTorch Tutorial @ SLAM Research Group
Example: Backward

Backward

𝑥=2
𝜕𝑧
𝑥 𝜕𝑥
=ω
𝜕𝐿 𝜕𝐿 𝜕𝑧 𝑧=6
= ∙
𝜕𝑥 𝜕𝑧 𝜕𝑥
𝑓 =𝑥∙𝜔 𝑧 Loss
𝜕𝐿
𝜔=3 =5
𝜕𝑧
𝜕𝑧
𝜔 𝜕𝜔
=𝑥
𝜕𝐿 𝜕𝐿 𝜕𝑧
= ∙
𝜕𝜔 𝜕𝑧 𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-21 PyTorch Tutorial @ SLAM Research Group
Example: Backward

Backward

𝑥=2
𝜕𝑧
𝑥 𝜕𝑥
=ω
𝜕𝐿 𝜕𝐿 𝜕𝑧 𝑧=6
= ∙
𝜕𝑥 𝜕𝑧 𝜕𝑥
= 5 ∙ 𝜔 = 15 𝑓 =𝑥∙𝜔 𝑧 Loss
𝜕𝐿
𝜔=3 =5
𝜕𝑧
𝜕𝑧
𝜔 𝜕𝜔
=𝑥
𝜕𝐿 𝜕𝐿 𝜕𝑧
= ∙
𝜕𝜔 𝜕𝑧 𝜕𝜔
= 5 ∙ 𝑥 = 10

Lecturer : Hongpu Liu Lecture 4-22 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model

𝑦ො = 𝑥 ∗ 𝜔

𝜔 ∗ 𝑦ො

Lecturer : Hongpu Liu Lecture 4-23 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model

𝑦ො = 𝑥 ∗ 𝜔

𝜔 ∗ 𝑦ො
𝜕𝑥𝜔
=𝑥
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-24 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model

𝑦ො = 𝑥 ∗ 𝜔

𝑥 =1

𝜔 ∗ 𝑦ො
𝜕𝑥𝜔
=𝑥
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-25 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟
𝜕𝑥𝜔
=𝑥
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-26 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟
𝜕𝑥𝜔 𝜕𝑦ො − 𝑦
=𝑥 =1
𝜕𝜔 𝜕𝑦ො

Lecturer : Hongpu Liu Lecture 4-27 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1
𝜕𝑥𝜔 𝜕𝑦ො − 𝑦
=𝑥 =1
𝜕𝜔 𝜕𝑦ො

Lecturer : Hongpu Liu Lecture 4-28 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1
𝜕𝑥𝜔 𝜕𝑦ො − 𝑦 𝜕𝑟 2
=𝑥 =1 = 2𝑟
𝜕𝜔 𝜕𝑦ො 𝜕𝑟

Lecturer : Hongpu Liu Lecture 4-29 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1
𝜕𝑥𝜔 𝜕𝑦ො − 𝑦 𝜕𝑟 2
=𝑥 =1 = 2𝑟
𝜕𝜔 𝜕𝑦ො 𝜕𝑟

𝜕𝑙𝑜𝑠𝑠
= 2𝑟 = −2
𝜕𝑟

Lecturer : Hongpu Liu Lecture 4-30 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1
𝜕𝑥𝜔 𝜕𝑦ො − 𝑦 𝜕𝑟 2
=𝑥 =1 = 2𝑟
𝜕𝜔 𝜕𝑦ො 𝜕𝑟

𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠 𝜕𝑟
= ∙ 𝜕𝑙𝑜𝑠𝑠
𝜕𝑦ො 𝜕𝑟 𝜕𝑦ො = 2𝑟 = −2
= −2 ∙ 1 = −2 𝜕𝑟

Lecturer : Hongpu Liu Lecture 4-31 PyTorch Tutorial @ SLAM Research Group
Computational Graph of Linear Model

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1
𝜕𝑥𝜔 𝜕𝑦ො − 𝑦 𝜕𝑟 2
=𝑥 =1 = 2𝑟
𝜕𝜔 𝜕𝑦ො 𝜕𝑟

𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠 𝜕 𝑦ො
= ∙ 𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠 𝜕𝑟
𝜕𝜔 𝜕 𝑦ො 𝜕𝜔 = ∙ 𝜕𝑙𝑜𝑠𝑠
𝜕𝑦ො 𝜕𝑟 𝜕𝑦ො = 2𝑟 = −2
= −2 ∙ 𝑥 𝜕𝑟
= −2 ∙ 1 = −2
= −2 ∙ 1 = −2

Lecturer : Hongpu Liu Lecture 4-32 PyTorch Tutorial @ SLAM Research Group
Exercise 4-1: Compute the gradient with Computational Graph

Linear Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =2 𝑦 =4

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠

𝜕𝑙𝑜𝑠𝑠
=?
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-33 PyTorch Tutorial @ SLAM Research Group
Exercise 4-2: Compute gradient of Affine model

Affine Model Loss Function

𝑦ො = 𝑥 ∗ 𝜔 + 𝑏 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝑥 =1 𝑦 =2

𝜔 ∗ + 𝑦ො 𝑙𝑜𝑠𝑠(𝑦,
ො 𝑦) 𝑙𝑜𝑠𝑠

𝜕𝑙𝑜𝑠𝑠
𝜕𝜔
=? 𝑏 =2
𝜕𝑙𝑜𝑠𝑠
=?
𝜕𝑏

Lecturer : Hongpu Liu Lecture 4-34 PyTorch Tutorial @ SLAM Research Group
Tensor in PyTorch

In PyTorch, Tensor is the important

component in constructing dynamic Tensor 𝝎
computational graph.
Data Grad
𝝏𝒍𝒐𝒔𝒔
It contains data and grad, which storage 𝝎 𝝏𝝎
the value of node and gradient w.r.t loss
respectively.

Lecturer : Hongpu Liu Lecture 4-35 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

If autograd mechanics are required, the

import torch element variable requires_grad of
Tensor has to be set to True.
x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = torch.Tensor([1.0])
w.requires_grad = True

Lecturer : Hongpu Liu Lecture 4-36 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

Define the linear model:

Linear Model
def forward(x):
return x * w 𝑦ො = 𝑥 ∗ 𝜔

def loss(x, y):

y_pred = forward(x)
return (y_pred - y) ** 2

Lecturer : Hongpu Liu Lecture 4-37 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

Define the loss function:

def forward(x): Loss Function

return x * w 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

def loss(x, y):

y_pred = forward(x)
return (y_pred - y) ** 2

Lecturer : Hongpu Liu Lecture 4-38 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

print("predict (before training)", 4, forward(4).item())

for epoch in range(100):

for x, y in zip(x_data, y_data):
l = loss(x, y) Forward, compute the loss.
l.backward()
print('\tgrad:', x, y, w.grad.item())
w.data = w.data - 0.01 * w.grad.data

w.grad.data.zero_()

print("progress:", epoch, l.item())

print("predict (after training)", 4, forward(4).item())

Lecturer : Hongpu Liu Lecture 4-39 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

print("predict (before training)", 4, forward(4).item())

for epoch in range(100):

for x, y in zip(x_data, y_data): Backward, compute grad for
l = loss(x, y)
l.backward() Tensor whose requires_grad
print('\tgrad:', x, y, w.grad.item())
w.data = w.data - 0.01 * w.grad.data set to True
w.grad.data.zero_()

print("progress:", epoch, l.item())

print("predict (after training)", 4, forward(4).item())

Lecturer : Hongpu Liu Lecture 4-40 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

print("predict (before training)", 4, forward(4).item())

for epoch in range(100):

for x, y in zip(x_data, y_data):
l = loss(x, y)
l.backward()
print('\tgrad:', x, y, w.grad.item()) The grad is utilized to update
w.data = w.data - 0.01 * w.grad.data
weight.
w.grad.data.zero_()

print("progress:", epoch, l.item())

print("predict (after training)", 4, forward(4).item())

Lecturer : Hongpu Liu Lecture 4-41 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

print("predict (before training)", 4, forward(4).item())

for epoch in range(100):

for x, y in zip(x_data, y_data):
l = loss(x, y)
l.backward() NOTICE:
print('\tgrad:', x, y, w.grad.item())
w.data = w.data - 0.01 * w.grad.data The grad computed by .backward()
w.grad.data.zero_() will be accumulated.
print("progress:", epoch, l.item()) So after update, remember set the
grad
print("predict (after training)", 4, forward(4).item()) to ZERO!!!

Lecturer : Hongpu Liu Lecture 4-42 PyTorch Tutorial @ SLAM Research Group
Implementation of linear model with PyTorch

print("predict (before training)", 4, forward(4).item())

for epoch in range(100):

for x, y in zip(x_data, y_data):
l = loss(x, y)
l.backward()
print('\tgrad:', x, y, w.grad.item())
w.data = w.data - 0.01 * w.grad.data

w.grad.data.zero_()

print("progress:", epoch, l.item())

print("predict (after training)", 4, forward(4).item())

Lecturer : Hongpu Liu Lecture 4-43 PyTorch Tutorial @ SLAM Research Group
Forward/Backward in PyTorch

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠

Lecturer : Hongpu Liu Lecture 4-44 PyTorch Tutorial @ SLAM Research Group
Forward in PyTorch

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1

w = torch.Tensor([1.0])
w.requires_grad = True

l = loss(x, y)

Lecturer : Hongpu Liu Lecture 4-45 PyTorch Tutorial @ SLAM Research Group
Backward in PyTorch

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1

𝜕𝑙𝑜𝑠𝑠
l.backward() = 𝑤. 𝑔𝑟𝑎𝑑
𝜕𝜔

Lecturer : Hongpu Liu Lecture 4-46 PyTorch Tutorial @ SLAM Research Group
Update weight in PyTorch

𝑥 =1 𝑦 =2

𝜔 ∗ 𝑦ො − 𝑟 ^2 𝑙𝑜𝑠𝑠 = 1

𝜕𝑙𝑜𝑠𝑠
l.backward() = 𝑤. 𝑔𝑟𝑎𝑑
𝜕𝜔

w.data = w.data - 0.01 * w.grad.data

Lecturer : Hongpu Liu Lecture 4-47 PyTorch Tutorial @ SLAM Research Group
Exercise 4-3: Compute gradients using computational graph

Quadratic Model Loss Function

𝑦ො = 𝜔1 𝑥 2 + 𝜔2 𝑥 + 𝑏 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠

=? =? =?
𝜕𝜔1 𝜕𝜔2 𝜕𝑏

Lecturer : Hongpu Liu Lecture 4-48 PyTorch Tutorial @ SLAM Research Group
Exercise 4-4: Compute gradients using PyTorch

Quadratic Model Loss Function

𝑦ො = 𝜔1 𝑥 2 + 𝜔2 𝑥 + 𝑏 𝑙𝑜𝑠𝑠 = (𝑦ො − 𝑦)2 = (𝑥 ∙ 𝜔 − 𝑦)2

𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠 𝜕𝑙𝑜𝑠𝑠

=? =? =?
𝜕𝜔1 𝜕𝜔2 𝜕𝑏

Lecturer : Hongpu Liu Lecture 4-49 PyTorch Tutorial @ SLAM Research Group
 PyTorch Tutorial
04. Back Propagation

Lecturer : Hongpu Liu Lecture 4-50 PyTorch Tutorial @ SLAM Research Group