PyTorch Tutorial

12. Basic RNN

Lecturer : Hongpu Liu Lecture 12-1 PyTorch Tutorial @ SLAM Research Group
Revision: DNN

𝑥1 Linear Layer

Sigmoid Layer
𝑥2 𝑧1 𝑜1

𝑥3 𝑧2 𝑜2 𝑧1 𝑜1

𝑥4 𝑧3 𝑜3 𝑧2 𝑜2
𝑧1 𝑜1 𝑦ො
𝑥5 𝑧4 𝑜4 𝑧3 𝑜3

𝑥6 𝑧5 𝑜5 𝑧4 𝑜4

𝑥7 𝑧6 𝑜6

𝑥8

Lecturer : Hongpu Liu Lecture 12-2 PyTorch Tutorial @ SLAM Research Group
Revision: CNN

𝑪𝟏 𝑺𝟏 𝑪𝟐 𝑺𝟐 𝒏𝟏 𝒏𝟐
Input Feature maps Feature maps Feature maps Feature maps Output
1 × 28 × 28 4 × 24 × 24 4 × 12 × 12 8×8×8 8×4×4

0
1

8
9
5×5 2×2 5×5 2×2 Fully Fully
Convolution Subsampling Convolution Subsampling Connected Connected

Feature Extraction Classification

Lecturer : Hongpu Liu Lecture 12-3 PyTorch Tutorial @ SLAM Research Group
What is RNNs?

ℎ𝑡 ℎ1 ℎ2 ℎ3 ℎ4

RNN Cell ℎ0 RNN Cell RNN Cell RNN Cell RNN Cell

𝑥𝑡 𝑥1 𝑥2 𝑥3 𝑥4

Lecturer : Hongpu Liu Lecture 12-4 PyTorch Tutorial @ SLAM Research Group
What is RNN?

𝒉𝒕 = 𝐭𝐚𝐧𝐡 𝑾𝒊𝒉 𝒙𝒕 + 𝒃𝒊𝒉 + 𝑾𝒉𝒉 𝒉𝒕−𝟏 + 𝒃𝒉𝒉

RNN Cell

𝒉𝒕−𝟏 ∈ ℝ𝒉𝒊𝒅𝒅𝒆𝒏_𝒔𝒊𝒛𝒆 𝑾𝒉𝒉𝒉𝒕−𝟏 + 𝒃𝒉𝒉 + 𝒕𝒂𝒏𝒉 𝒉𝒕 ∈ ℝ𝒉𝒊𝒅𝒅𝒆𝒏_𝒔𝒊𝒛𝒆

𝑾𝒊𝒉 𝒙𝒕 + 𝒃𝒊𝒉

𝒙𝒕 ∈ ℝ𝒊𝒏𝒑𝒖𝒕_𝒔𝒊𝒛𝒆

Lecturer : Hongpu Liu Lecture 12-5 PyTorch Tutorial @ SLAM Research Group
What is RNNs?

ℎ𝑡 ℎ1 ℎ2 ℎ3 ℎ4

RNN Cell ℎ0 RNN Cell RNN Cell RNN Cell RNN Cell

𝑥𝑡 𝑥1 𝑥2 𝑥3 𝑥4

Lecturer : Hongpu Liu Lecture 12-6 PyTorch Tutorial @ SLAM Research Group
RNN Cell in PyTorch

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

RNN Cell

𝒉𝒕−𝟏 ∈ ℝ𝒉𝒊𝒅𝒅𝒆𝒏_𝒔𝒊𝒛𝒆 𝑾𝒉𝒉𝒉𝒕−𝟏 + 𝒃𝒉𝒉 + 𝒕𝒂𝒏𝒉 𝒉𝒕 ∈ ℝ𝒉𝒊𝒅𝒅𝒆𝒏_𝒔𝒊𝒛𝒆

𝑾𝒊𝒉 𝒙𝒕 + 𝒃𝒊𝒉

𝒙𝒕 ∈ ℝ𝒊𝒏𝒑𝒖𝒕_𝒔𝒊𝒛𝒆

Lecturer : Hongpu Liu Lecture 12-7 PyTorch Tutorial @ SLAM Research Group
RNN Cell in PyTorch

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

RNN Cell

𝒉𝒕−𝟏 ∈ ℝ𝒉𝒊𝒅𝒅𝒆𝒏_𝒔𝒊𝒛𝒆 𝑾𝒉𝒉𝒉𝒕−𝟏 + 𝒃𝒉𝒉 + 𝒕𝒂𝒏𝒉 𝒉𝒕 ∈ ℝ𝒉𝒊𝒅𝒅𝒆𝒏_𝒔𝒊𝒛𝒆

𝑾𝒊𝒉 𝒙𝒕 + 𝒃𝒊𝒉

𝒙𝒕 ∈ ℝ𝒊𝒏𝒑𝒖𝒕_𝒔𝒊𝒛𝒆

Lecturer : Hongpu Liu Lecture 12-8 PyTorch Tutorial @ SLAM Research Group
RNN Cell in PyTorch

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

hidden = cell(input, hidden)

input output

input of shape 𝑏𝑎𝑡𝑐ℎ, 𝑖𝑛𝑝𝑢𝑡_𝑠𝑖𝑧𝑒 hidden of shape 𝑏𝑎𝑡𝑐ℎ, ℎ𝑖𝑑𝑑𝑒𝑛_𝑠𝑖𝑧𝑒

hidden of shape 𝑏𝑎𝑡𝑐ℎ, ℎ𝑖𝑑𝑑𝑒𝑛_𝑠𝑖𝑧𝑒

Lecturer : Hongpu Liu Lecture 12-9 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell

• Suppose we have sequence with below properties:

• 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒 = 1
• 𝑠𝑒𝑞𝐿𝑒𝑛 = 3
• 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒 = 4
• ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒 = 2

Lecturer : Hongpu Liu Lecture 12-10 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell

• Suppose we have sequence with below properties:

• 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆 = 1
• 𝑠𝑒𝑞𝐿𝑒𝑛 = 3
• 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆 = 4
• ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒 = 2
• So the shape of inputs and outputs of RNNCell:
• 𝒊𝒏𝒑𝒖𝒕. 𝑠ℎ𝑎𝑝𝑒 = 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆
• 𝑜𝑢𝑡𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒

Lecturer : Hongpu Liu Lecture 12-11 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell

• Suppose we have sequence with below properties:

• 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆 = 1
• 𝑠𝑒𝑞𝐿𝑒𝑛 = 3
• 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒 = 4
• 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆 = 2
• So the shape of inputs and outputs of RNNCell:
• 𝑖𝑛𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
• 𝒐𝒖𝒕𝒑𝒖𝒕. 𝑠ℎ𝑎𝑝𝑒 = 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆

Lecturer : Hongpu Liu Lecture 12-12 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell

• Suppose we have sequence with below properties:

• 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆 = 1
• 𝒔𝒆𝒒𝑳𝒆𝒏 = 3
• 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆 = 4
• ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒 = 2
• So the shape of inputs and outputs of RNNCell:
• 𝑖𝑛𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
• 𝑜𝑢𝑡𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒
• The sequence can be warped in one Tensor with shape:
• 𝑑𝑎𝑡𝑎𝑠𝑒𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆

Lecturer : Hongpu Liu Lecture 12-13 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell
import torch

batch_size = 1
seq_len = 3
Parameters
input_size = 4
hidden_size = 2

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

# (seq, batch, features)

dataset = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(batch_size, hidden_size)

for idx, input in enumerate(dataset):

print('=' * 20, idx, '=' * 20)
print('Input size: ', input.shape)

hidden = cell(input, hidden)

print('outputs size: ', hidden.shape)

print(hidden)

Lecturer : Hongpu Liu Lecture 12-14 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size) Construction of RNNCell

# (seq, batch, features)

dataset = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(batch_size, hidden_size)

for idx, input in enumerate(dataset):

print('=' * 20, idx, '=' * 20)
print('Input size: ', input.shape)

hidden = cell(input, hidden)

print('outputs size: ', hidden.shape)

print(hidden)

Lecturer : Hongpu Liu Lecture 12-15 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

# (seq, batch, features) Wrapping the sequence into:

dataset = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(batch_size, hidden_size) 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
for idx, input in enumerate(dataset):
print('=' * 20, idx, '=' * 20)
print('Input size: ', input.shape)

hidden = cell(input, hidden)

print('outputs size: ', hidden.shape)

print(hidden)

Lecturer : Hongpu Liu Lecture 12-16 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

# (seq, batch, features)

dataset = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(batch_size, hidden_size) Initializing the hidden to zero
for idx, input in enumerate(dataset):
print('=' * 20, idx, '=' * 20)
print('Input size: ', input.shape)

hidden = cell(input, hidden)

print('outputs size: ', hidden.shape)

print(hidden)

Lecturer : Hongpu Liu Lecture 12-17 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

# (seq, batch, features)

dataset = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(batch_size, hidden_size)

for idx, input in enumerate(dataset): The shape of input is:

print('=' * 20, idx, '=' * 20)
print('Input size: ', input.shape) 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆

hidden = cell(input, hidden)

Input size: torch.Size([1, 4])
print('outputs size: ', hidden.shape)
print(hidden)

Lecturer : Hongpu Liu Lecture 12-18 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2

cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size)

# (seq, batch, features)

dataset = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(batch_size, hidden_size)

for idx, input in enumerate(dataset): The shape of hidden is:

print('=' * 20, idx, '=' * 20)
print('Input size: ', input.shape) 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
hidden = cell(input, hidden)
hidden size: torch.Size([1, 2])
print('hidden size: ', hidden.shape)
print(hidden)

Lecturer : Hongpu Liu Lecture 12-19 PyTorch Tutorial @ SLAM Research Group
How to use RNNCell
import torch

batch_size = 1
seq_len = 3
input_size = 4 ========== 0 ==========
hidden_size = 2 Input size: torch.Size([1, 4])
outputs size: torch.Size([1, 2])
cell = torch.nn.RNNCell(input_size=input_size, hidden_size=hidden_size) tensor([[-0.1579, 0.5140]])
========== 1 ==========
# (seq, batch, features) Input size: torch.Size([1, 4])
dataset = torch.randn(seq_len, batch_size, input_size)
outputs size: torch.Size([1, 2])
hidden = torch.zeros(batch_size, hidden_size)
tensor([[-0.9577, 0.6502]])
for idx, input in enumerate(dataset): ========== 2 ==========
print('=' * 20, idx, '=' * 20) Input size: torch.Size([1, 4])
print('Input size: ', input.shape) outputs size: torch.Size([1, 2])
tensor([[-0.7661, -0.9128]])
hidden = cell(input, hidden)

print('outputs size: ', hidden.shape)

print(hidden)

Lecturer : Hongpu Liu Lecture 12-20 PyTorch Tutorial @ SLAM Research Group
How to use RNN

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

ℎ1 ℎ2 ℎ3 ℎ4 ℎ𝑁

ℎ0 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell ℎ𝑁

𝑥1 𝑥2 𝑥3 𝑥4 𝑥𝑁

Lecturer : Hongpu Liu Lecture 12-21 PyTorch Tutorial @ SLAM Research Group
How to use RNN

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

out, hidden = cell(inputs, hidden)

ℎ1 ℎ2 ℎ3 ℎ4 ℎ𝑁

ℎ0 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell ℎ𝑁

𝑥1 𝑥2 𝑥3 𝑥4 𝑥𝑁

Lecturer : Hongpu Liu Lecture 12-22 PyTorch Tutorial @ SLAM Research Group
How to use RNN

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

out, hidden = cell(inputs, hidden) input

input of shape 𝑠𝑒𝑞𝑆𝑖𝑧𝑒, 𝑏𝑎𝑡𝑐ℎ, 𝑖𝑛𝑝𝑢𝑡_𝑠𝑖𝑧𝑒

hidden of shape 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠, 𝑏𝑎𝑡𝑐ℎ, ℎ𝑖𝑑𝑑𝑒𝑛_𝑠𝑖𝑧𝑒

output

Output of shape 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑏𝑎𝑡𝑐ℎ, ℎ𝑖𝑑𝑑𝑒𝑛_𝑠𝑖𝑧𝑒

hidden of shape 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠, 𝑏𝑎𝑡𝑐ℎ, ℎ𝑖𝑑𝑑𝑒𝑛_𝑠𝑖𝑧𝑒

Lecturer : Hongpu Liu Lecture 12-23 PyTorch Tutorial @ SLAM Research Group
How to use RNN

• Suppose we have sequence with below properties:

• 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒
• 𝑠𝑒𝑞𝐿𝑒𝑛
• 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒,
• 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠

Lecturer : Hongpu Liu Lecture 12-24 PyTorch Tutorial @ SLAM Research Group
How to use RNN

• Suppose we have sequence with below properties:

• 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆
• 𝒔𝒆𝒒𝑳𝒆𝒏
• 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒,
• 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠
• The shape of input and h_0 of RNN:
• 𝑖𝑛𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆
• ℎ_0. 𝑠ℎ𝑎𝑝𝑒 = 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒

Lecturer : Hongpu Liu Lecture 12-25 PyTorch Tutorial @ SLAM Research Group
How to use RNN

• Suppose we have sequence with below properties:

• 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆
• 𝑠𝑒𝑞𝐿𝑒𝑛
• 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆,
• 𝒏𝒖𝒎𝑳𝒂𝒚𝒆𝒓𝒔
• The shape of input and h_0 of RNN:
• 𝑖𝑛𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
• ℎ_0. 𝑠ℎ𝑎𝑝𝑒 = 𝒏𝒖𝒎𝑳𝒂𝒚𝒆𝒓𝒔, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆

Lecturer : Hongpu Liu Lecture 12-26 PyTorch Tutorial @ SLAM Research Group
How to use RNN

• Suppose we have sequence with below properties:

• 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆
• 𝒔𝒆𝒒𝑳𝒆𝒏
• 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆,
• 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠
• The shape of input and h_0 of RNN:
• 𝑖𝑛𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
• ℎ_0. 𝑠ℎ𝑎𝑝𝑒 = 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒
• The shape of output and h_n of RNN:
• 𝑜𝑢𝑡𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
• ℎ_𝑛. 𝑠ℎ𝑎𝑝𝑒 = 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒

Lecturer : Hongpu Liu Lecture 12-27 PyTorch Tutorial @ SLAM Research Group
How to use RNN

• Suppose we have sequence with below properties:

• 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆
• 𝑠𝑒𝑞𝐿𝑒𝑛
• 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆,
• 𝒏𝒖𝒎𝑳𝒂𝒚𝒆𝒓𝒔
• The shape of input and h_0 of RNN:
• 𝑖𝑛𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
• ℎ_0. 𝑠ℎ𝑎𝑝𝑒 = 𝑛𝑢𝑚𝐿𝑎𝑦𝑒𝑟𝑠, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒
• The shape of output and h_n of RNN:
• 𝑜𝑢𝑡𝑝𝑢𝑡. 𝑠ℎ𝑎𝑝𝑒 = 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, ℎ𝑖𝑑𝑑𝑒𝑛𝑆𝑖𝑧𝑒
• ℎ_𝑛. 𝑠ℎ𝑎𝑝𝑒 = 𝒏𝒖𝒎𝑳𝒂𝒚𝒆𝒓𝒔, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆

Lecturer : Hongpu Liu Lecture 12-28 PyTorch Tutorial @ SLAM Research Group
How to use RNN - numLayers

output ℎ1 ℎ2 ℎ3 ℎ4 ℎ𝑁

ℎ03 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell 3
ℎ𝑁

ℎ02 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell 2
ℎ𝑁

ℎ10 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell ℎ1𝑁

hidden

𝑥1 𝑥2 𝑥3 𝑥4 𝑥𝑁

Lecturer : Hongpu Liu Lecture 12-29 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4 Parameters
hidden_size = 2
num_layers = 1

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

# (seqLen, batchSize, inputSize)

inputs = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(num_layers, batch_size, hidden_size)

out, hidden = cell(inputs, hidden)

print('Output size:', out.shape)

print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-30 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

Construction of RNN
num_layers=num_layers)

# (seqLen, batchSize, inputSize)

inputs = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(num_layers, batch_size, hidden_size)

out, hidden = cell(inputs, hidden)

print('Output size:', out.shape)

print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-31 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

# (seqLen, batchSize, inputSize) Wrapping the sequence into:

inputs = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(num_layers, batch_size, hidden_size) 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
out, hidden = cell(inputs, hidden)

print('Output size:', out.shape)

print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-32 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

# (seqLen, batchSize, inputSize)

inputs = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(num_layers, batch_size, hidden_size)
Initializing the hidden to zero
out, hidden = cell(inputs, hidden)

print('Output size:', out.shape)

print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-33 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

# (seqLen, batchSize, inputSize)

inputs = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(num_layers, batch_size, hidden_size)
The shape of output is:
out, hidden = cell(inputs, hidden)
𝒔𝒆𝒒𝑺𝒊𝒛𝒆, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
print('Output size:', out.shape)
print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-34 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers)

# (seqLen, batchSize, inputSize)

inputs = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(num_layers, batch_size, hidden_size)
The shape of hidden is:
out, hidden = cell(inputs, hidden)
𝒏𝒖𝒎𝑳𝒂𝒚𝒆𝒓𝒔, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
print('Output size:', out.shape)
print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-35 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1
Output size: torch.Size([3, 1, 2])
cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size, Output: tensor([
num_layers=num_layers) [[ 0.2360, -0.5550]],
[[ 0.2685, 0.0106]],
# (seqLen, batchSize, inputSize) [[ 0.1172, -0.2262]]])
inputs = torch.randn(seq_len, batch_size, input_size) Hidden size: torch.Size([1, 1, 2])
hidden = torch.zeros(num_layers, batch_size, hidden_size) Hidden: tensor([[[ 0.1172, -0.2262]]])
out, hidden = cell(inputs, hidden)

print('Output size:', out.shape)

print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-36 PyTorch Tutorial @ SLAM Research Group
How to use RNN

• batch_first: if True, the input and output tensors are provided as:
• 𝑏𝑎𝑡𝑐ℎ𝑆𝑖𝑧𝑒, 𝑠𝑒𝑞𝐿𝑒𝑛, 𝑖𝑛𝑝𝑢𝑡_𝑠𝑖𝑧𝑒

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,

num_layers=num_layers, batch_first=True)

inputs = torch.randn(seq_len, batch_size, input_size)

inputs = torch.randn(batch_size, seq_len, input_size)

Lecturer : Hongpu Liu Lecture 12-37 PyTorch Tutorial @ SLAM Research Group
How to use RNN
import torch

batch_size = 1
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1
Output size: torch.Size([1, 3, 2])
cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size, Output: tensor([[
num_layers=num_layers, batch_first=True) [ 0.6582, -0.7281],
[ 0.5021, -0.8324],
# (seqLen, batchSize, inputSize) [-0.5225, -0.5621]]])
inputs = torch.randn(batch_size, seq_len, input_size) Hidden size: torch.Size([1, 1, 2])
hidden = torch.zeros(num_layers, batch_size, hidden_size) Hidden: tensor([[[-0.5225, -0.5621]]])
out, hidden = cell(inputs, hidden)

print('Output size:', out.shape)

print('Output:', out)
print('Hidden size: ', hidden.shape)
print('Hidden: ', hidden)

Lecturer : Hongpu Liu Lecture 12-38 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

• Train a model to learn:

• “hello”-> “ohlol”

𝑜 ℎ 𝑙 𝑜 𝑙

ℎ0 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell ℎ𝑁

ℎ 𝑒 𝑙 𝑙 𝑜

Lecturer : Hongpu Liu Lecture 12-39 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

• The inputs of RNN Cell should be vectors of numbers.

h 1 0 1 0 0
character index
e e 0 0 1 0 0 0
l h 1 2 0 0 1 0
l l 2 2 0 0 1 0
o 3
o 3 0 0 0 1

Input Dictionary Indices One-Hot Vectors

Lecturer : Hongpu Liu Lecture 12-40 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

• The inputs of RNN Cell should be vectors of numbers.

h 1 0 1 0 0
character index
e e 0 0 1 0 0 0
l h 1 2 0 0 1 0
l l 2 2 0 0 1 0
o 3
o 3 0 0 0 1

Input Dictionary Indices One-Hot Vectors

𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆 =?

Lecturer : Hongpu Liu Lecture 12-41 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

• The inputs of RNN Cell should be vectors of numbers.

h 1 0 1 0 0
character index
e e 0 0 1 0 0 0
l h 1 2 0 0 1 0
l l 2 2 0 0 1 0
o 3
o 3 0 0 0 1

Input Dictionary Indices One-Hot Vectors

𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆 = 𝟒

Lecturer : Hongpu Liu Lecture 12-42 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

𝑜 ℎ 𝑙 𝑜 𝑙

ℎ0 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell ℎ𝑁

ℎ 𝑒 𝑙 𝑙 𝑜

Lecturer : Hongpu Liu Lecture 12-43 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

• The outputs of RNN Cell should be vectors of prediction.

𝑃(𝑦 = 𝑒)

One-Hot
𝑃(𝑦 = 𝑒)
𝑥𝑡 RNN Cell Softmax NLLLoss 𝑦𝑡
𝑃(𝑦 = 𝑙)
𝑃(𝑦 = 𝑜)

ℎ𝑡−1 𝑙𝑜𝑠𝑠

𝒐𝒖𝒕𝒑𝒖𝒕𝑺𝒊𝒛𝒆 = ?

Lecturer : Hongpu Liu Lecture 12-44 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

• The outputs of RNN Cell should be vectors of prediction.

𝑃(𝑦 = 𝑒)

One-Hot
𝑃(𝑦 = 𝑒)
𝑥𝑡 RNN Cell Softmax NLLLoss 𝑦𝑡
𝑃(𝑦 = 𝑙)
𝑃(𝑦 = 𝑜)

ℎ𝑡−1 𝑙𝑜𝑠𝑠

𝒐𝒖𝒕𝒑𝒖𝒕𝑺𝒊𝒛𝒆 = 4

Lecturer : Hongpu Liu Lecture 12-45 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Using RNNCell

• The outputs of RNN Cell should be vectors of prediction.

CrossEntropyLoss

𝑃(𝑦 = 𝑒)

One-Hot
𝑃(𝑦 = 𝑒)
𝑥𝑡 RNN Cell Softmax NLLLoss 𝑦𝑡
𝑃(𝑦 = 𝑙)
𝑃(𝑦 = 𝑜)

ℎ𝑡−1 𝑙𝑜𝑠𝑠

𝒐𝒖𝒕𝒑𝒖𝒕𝑺𝒊𝒛𝒆 = 4

Lecturer : Hongpu Liu Lecture 12-46 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Parameters

import torch

input_size = 4
hidden_size = 4
batch_size = 1

Lecturer : Hongpu Liu Lecture 12-47 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Prepare Data

idx2char = ['e', 'h', 'l', 'o']

x_data = [1, 0, 2, 2, 3]
The dictionary
y_data = [3, 1, 2, 3, 2]

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data]

inputs = torch.Tensor(x_one_hot).view(-1, batch_size, input_size)

labels = torch.LongTensor(y_data).view(-1, 1)

Lecturer : Hongpu Liu Lecture 12-48 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Prepare Data

idx2char = ['e', 'h', 'l', 'o']

x_data = [1, 0, 2, 2, 3]
y_data = [3, 1, 2, 3, 2] The input sequence is ‘hello’

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data]

inputs = torch.Tensor(x_one_hot).view(-1, batch_size, input_size)

labels = torch.LongTensor(y_data).view(-1, 1)

Lecturer : Hongpu Liu Lecture 12-49 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Prepare Data

idx2char = ['e', 'h', 'l', 'o']

x_data = [1, 0, 2, 2, 3]
y_data = [3, 1, 2, 3, 2]
The output sequence is ‘ohlol’

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data]

inputs = torch.Tensor(x_one_hot).view(-1, batch_size, input_size)

labels = torch.LongTensor(y_data).view(-1, 1)

Lecturer : Hongpu Liu Lecture 12-50 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Prepare Data

idx2char = ['e', 'h', 'l', 'o']

x_data = [1, 0, 2, 2, 3]
y_data = [3, 1, 2, 3, 2]

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]] Convert indices into one-hot
x_one_hot = [one_hot_lookup[x] for x in x_data]
vector
inputs = torch.Tensor(x_one_hot).view(-1, batch_size, input_size)
labels = torch.LongTensor(y_data).view(-1, 1)

Lecturer : Hongpu Liu Lecture 12-51 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Prepare Data

idx2char = ['e', 'h', 'l', 'o']

x_data = [1, 0, 2, 2, 3]
y_data = [3, 1, 2, 3, 2]

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data]
Reshape the inputs to
inputs = torch.Tensor(x_one_hot).view(-1, batch_size, input_size)
𝒔𝒆𝒒𝑳𝒆𝒏, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆
labels = torch.LongTensor(y_data).view(-1, 1)

Lecturer : Hongpu Liu Lecture 12-52 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Prepare Data

idx2char = ['e', 'h', 'l', 'o']

x_data = [1, 0, 2, 2, 3]
y_data = [3, 1, 2, 3, 2]

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0],
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data] Reshape the labels to

inputs = torch.Tensor(x_one_hot).view(-1, batch_size, input_size) 𝒔𝒆𝒒𝑳𝒆𝒏, 𝟏

labels = torch.LongTensor(y_data).view(-1, 1)

Lecturer : Hongpu Liu Lecture 12-53 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Design Model

class Model(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size):
super(Model, self).__init
self.batch_size = batch_size
Initial the parameters
self.input_size = input_size
self.hidden_size = hidden_size
self.rnncell = torch.nn.RNNCell(input_size=self.input_size,
hidden_size=self.hidden_size)

def forward(self, input, hidden):

hidden = self.rnncell(input, hidden)
return hidden

def init_hidden(self):
return torch.zeros(self.batch_size, self.hidden_size)

net = Model(input_size, hidden_size, batch_size)

Lecturer : Hongpu Liu Lecture 12-54 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Design Model

class Model(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size):
super(Model, self).__init__()
# self.num_layers = num_layers
self.batch_size = batch_size
self.input_size = input_size Shape of inputs :
self.hidden_size = hidden_size
self.rnncell = torch.nn.RNNCell(input_size=self.input_size, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆
hidden_size=self.hidden_size) Shape of hidden:

def forward(self, input, hidden): 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆

hidden = self.rnncell(input, hidden)
return hidden

def init_hidden(self):
return torch.zeros(self.batch_size, self.hidden_size)

net = Model(input_size, hidden_size, batch_size)

Lecturer : Hongpu Liu Lecture 12-55 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Design Model

class Model(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size):
super(Model, self).__init__()
# self.num_layers = num_layers
self.batch_size = batch_size
self.input_size = input_size
self.hidden_size = hidden_size
self.rnncell = torch.nn.RNNCell(input_size=self.input_size,
hidden_size=self.hidden_size)

def forward(self, input, hidden):

hidden = self.rnncell(input, hidden)
return hidden

def init_hidden(self):
return torch.zeros(self.batch_size, self.hidden_size) Provide initial hidden

net = Model(input_size, hidden_size, batch_size)

Lecturer : Hongpu Liu Lecture 12-56 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Loss and Optimizer

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=0.1)

Lecturer : Hongpu Liu Lecture 12-57 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Training Cycle

for epoch in range(15):

loss = 0
optimizer.zero_grad() Training steps
hidden = net.init_hidden()
print('Predicted string: ', end='')
for input, label in zip(inputs, labels):
hidden = net(input, hidden)
loss += criterion(hidden, label)
_, idx = hidden.max(dim=1)
print(idx2char[idx.item()], end='')
loss.backward()
optimizer.step()
print(', Epoch [%d/15] loss=%.4f' % (epoch+1, loss.item()))

Lecturer : Hongpu Liu Lecture 12-58 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Training Cycle

for epoch in range(15):

loss = 0
optimizer.zero_grad()
hidden = net.init_hidden() Shape of inputs :
print('Predicted string: ', end='')
for input, label in zip(inputs, labels): 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆
hidden = net(input, hidden) Shape of input:
loss += criterion(hidden, label)
𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
_, idx = hidden.max(dim=1)
print(idx2char[idx.item()], end='')
loss.backward()
optimizer.step()
print(', Epoch [%d/15] loss=%.4f' % (epoch+1, loss.item()))

Lecturer : Hongpu Liu Lecture 12-59 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Training Cycle

for epoch in range(15):

loss = 0
optimizer.zero_grad()
hidden = net.init_hidden() Shape of labels :
print('Predicted string: ', end='') 𝒔𝒆𝒒𝑺𝒊𝒛𝒆, 𝟏
for input, label in zip(inputs, labels):
hidden = net(input, hidden) Shape of label:
loss += criterion(hidden, label) 𝟏
_, idx = hidden.max(dim=1)
print(idx2char[idx.item()], end='')
loss.backward()
optimizer.step()
print(', Epoch [%d/15] loss=%.4f' % (epoch+1, loss.item()))

Lecturer : Hongpu Liu Lecture 12-60 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Training Cycle

for epoch in range(15):

loss = 0
optimizer.zero_grad()
hidden = net.init_hidden()
print('Predicted string: ', end='')
for input, label in zip(inputs, labels):
hidden = net(input, hidden) RNN Cell
loss += criterion(hidden, label)
_, idx = hidden.max(dim=1)
print(idx2char[idx.item()], end='')
loss.backward()
optimizer.step()
print(', Epoch [%d/15] loss=%.4f' % (epoch+1, loss.item()))

Lecturer : Hongpu Liu Lecture 12-61 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Training Cycle

for epoch in range(15):

loss = 0
optimizer.zero_grad()
hidden = net.init_hidden()
print('Predicted string: ', end='')
for input, label in zip(inputs, labels):
hidden = net(input, hidden)
loss += criterion(hidden, label)
_, idx = hidden.max(dim=1) Output prediction
print(idx2char[idx.item()], end='')
loss.backward()
optimizer.step()
print(', Epoch [%d/15] loss=%.4f' % (epoch+1, loss.item()))

Lecturer : Hongpu Liu Lecture 12-62 PyTorch Tutorial @ SLAM Research Group
Example 12-1: Code – Result

Predicted string: loeeh, Epoch [1/15] loss=8.0117

Predicted string: oooeh, Epoch [2/15] loss=7.2082
Predicted string: ooooh, Epoch [3/15] loss=6.6208
Predicted string: ooooh, Epoch [4/15] loss=6.1802
Predicted string: ooooh, Epoch [5/15] loss=5.8060
Predicted string: ooooh, Epoch [6/15] loss=5.4739
Predicted string: ooooh, Epoch [7/15] loss=5.1593
Predicted string: ooool, Epoch [8/15] loss=4.8593
Predicted string: ooool, Epoch [9/15] loss=4.5819
Predicted string: ohool, Epoch [10/15] loss=4.3287
Predicted string: ohlol, Epoch [11/15] loss=4.0909
Predicted string: ohlol, Epoch [12/15] loss=3.8573
Predicted string: ohlol, Epoch [13/15] loss=3.6246
Predicted string: ohlol, Epoch [14/15] loss=3.4007
Predicted string: ohlol, Epoch [15/15] loss=3.2005

Lecturer : Hongpu Liu Lecture 12-63 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Using RNN Module

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=0.05)

for epoch in range(15):

optimizer.zero_grad()
outputs = net(inputs)
loss = criterion(outputs, labels) Training steps
loss.backward()
optimizer.step()

_, idx = outputs.max(dim=1)
idx = idx.data.numpy()
print('Predicted: ', ''.join([idx2char[x] for x in idx]), end='')
print(', Epoch [%d/15] loss = %.3f' % (epoch + 1, loss.item()))

Lecturer : Hongpu Liu Lecture 12-64 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Change Model

class Model(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size, num_layers=1):
super(Model, self).__init__()
self.num_layers = num_layers input_size = 4
self.batch_size = batch_size hidden_size = 4
self.input_size = input_size num_layers = 1
self.hidden_size = hidden_size batch_size = 1
self.rnn = torch.nn.RNN(input_size=self.input_size, seq_len = 5
hidden_size=self.hidden_size,
num_layers=num_layers)

def forward(self, input):

hidden = torch.zeros(self.num_layers,
self.batch_size,
self.hidden_size)
out, _ = self.rnn(input, hidden)
return out.view(-1, self.hidden_size)

net = Model(input_size, hidden_size, batch_size, num_layers)

Lecturer : Hongpu Liu Lecture 12-65 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Change Model

class Model(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size, num_layers=1):
super(Model, self).__init__()
self.num_layers = num_layers
self.batch_size = batch_size
self.input_size = input_size
self.hidden_size = hidden_size
self.rnn = torch.nn.RNN(input_size=self.input_size, input_size = 4
hidden_size=self.hidden_size, hidden_size = 4
num_layers=num_layers) num_layers = 1
batch_size = 1
def forward(self, input): seq_len = 5
hidden = torch.zeros(self.num_layers,
self.batch_size,
self.hidden_size)
out, _ = self.rnn(input, hidden)
return out.view(-1, self.hidden_size)

net = Model(input_size, hidden_size, batch_size, num_layers)

Lecturer : Hongpu Liu Lecture 12-66 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Change Model

class Model(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size, num_layers=1):
super(Model, self).__init__()
self.num_layers = num_layers
self.batch_size = batch_size
self.input_size = input_size
input_size = 4
self.hidden_size = hidden_size hidden_size = 4
self.rnn = torch.nn.RNN(input_size=self.input_size, num_layers = 1
hidden_size=self.hidden_size, batch_size = 1
num_layers=num_layers) seq_len = 5
def forward(self, input):
hidden = torch.zeros(self.num_layers,
Shape of hidden :
self.batch_size,
self.hidden_size) 𝒏𝒖𝒎𝑳𝒂𝒚𝒆𝒓𝒔, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
out, _ = self.rnn(input, hidden)
return out.view(-1, self.hidden_size)

net = Model(input_size, hidden_size, batch_size, num_layers)

Lecturer : Hongpu Liu Lecture 12-67 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Change Model

class Model(torch.nn.Module):
def __init__(self, input_size, hidden_size, batch_size, num_layers=1):
super(Model, self).__init__()
self.num_layers = num_layers
self.batch_size = batch_size
self.input_size = input_size
self.hidden_size = hidden_size
self.rnn = torch.nn.RNN(input_size=self.input_size,
hidden_size=self.hidden_size,
num_layers=num_layers)

def forward(self, input):

hidden = torch.zeros(self.num_layers,
self.batch_size,
self.hidden_size) Reshape out to:
out, _ = self.rnn(input, hidden) 𝒔𝒆𝒒𝑳𝒆𝒏 × 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
return out.view(-1, self.hidden_size)

net = Model(input_size, hidden_size, batch_size, num_layers)

Lecturer : Hongpu Liu Lecture 12-68 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Change Data

input_size = 4
hidden_size = 4
idx2char = ['e', 'h', 'l', 'o'] num_layers = 1
x_data = [1, 0, 2, 2, 3] batch_size = 1
y_data = [3, 1, 2, 3, 2] seq_len = 5

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0], Shape of inputs :
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data] 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆

inputs = torch.Tensor(x_one_hot).view(seq_len, batch_size, input_size)

labels = torch.LongTensor(y_data)

Lecturer : Hongpu Liu Lecture 12-69 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Change Data

input_size = 4
hidden_size = 4
idx2char = ['e', 'h', 'l', 'o'] num_layers = 1
x_data = [1, 0, 2, 2, 3] batch_size = 1
y_data = [3, 1, 2, 3, 2] seq_len = 5

one_hot_lookup = [[1, 0, 0, 0],

[0, 1, 0, 0],
[0, 0, 1, 0], Shape of labels :
[0, 0, 0, 1]]
x_one_hot = [one_hot_lookup[x] for x in x_data] 𝒔𝒆𝒒𝑳𝒆𝒏 × 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝟏

inputs = torch.Tensor(x_one_hot).view(seq_len, batch_size, input_size)

labels = torch.LongTensor(y_data)

Lecturer : Hongpu Liu Lecture 12-70 PyTorch Tutorial @ SLAM Research Group
Example 12-2 Result

Predicted: lhhhh, Epoch [1/15] loss = 1.325

Predicted: lohhh, Epoch [2/15] loss = 1.190
Predicted: lolol, Epoch [3/15] loss = 1.090
Predicted: lolol, Epoch [4/15] loss = 1.017
Predicted: oolol, Epoch [5/15] loss = 0.958
Predicted: oolol, Epoch [6/15] loss = 0.904
Predicted: oolol, Epoch [7/15] loss = 0.852
Predicted: oolol, Epoch [8/15] loss = 0.804
Predicted: oolol, Epoch [9/15] loss = 0.759
Predicted: ohlol, Epoch [10/15] loss = 0.716
Predicted: ohlol, Epoch [11/15] loss = 0.678
Predicted: ohlol, Epoch [12/15] loss = 0.643
Predicted: ohlol, Epoch [13/15] loss = 0.613
Predicted: ohlol, Epoch [14/15] loss = 0.588
Predicted: ohlol, Epoch [15/15] loss = 0.568

Lecturer : Hongpu Liu Lecture 12-71 PyTorch Tutorial @ SLAM Research Group
Associate a vector with a word/character

• One-hot encoding of words and characters

• The one-hot vectors are high-dimension.
• The one-hot vectors are sparse.
• The one-hot vectors are hardcoded.

Lecturer : Hongpu Liu Lecture 12-72 PyTorch Tutorial @ SLAM Research Group
Associate a vector with a word/character

• One-hot encoding of words and characters

• The one-hot vectors are high-dimension.
• The one-hot vectors are sparse.
• The one-hot vectors are hardcoded.
• Do we have a way to associate a vector with a word/character
with following specification:
• Lower-dimension
• Dense
• Learned from data
• A popular and powerful way is called EMBEDDING.

Lecturer : Hongpu Liu Lecture 12-73 PyTorch Tutorial @ SLAM Research Group
Associate a vector with a word/character

• One-hot encoding of words and characters

• The one-hot vectors are high-dimension.
• The one-hot vectors are sparse.
• The one-hot vectors are hardcoded.
• Do we have a way to associate a vector with a word/character
with following specification:
• Lower-dimension
• Dense
• Learned from data
• A popular and powerful way is called EMBEDDING.

Lecturer : Hongpu Liu Lecture 12-74 PyTorch Tutorial @ SLAM Research Group
One-hot vs Embedding

One-hot vectors: Embedding vectors:

• High-dimension • Lower-dimension
• Sparse • Dense
• Hardcoded • Learned from data

Lecturer : Hongpu Liu Lecture 12-75 PyTorch Tutorial @ SLAM Research Group
Embedding in PyTorch

𝑒𝑚𝑏𝑒𝑑𝑑𝑖𝑛𝑔𝑆𝑖𝑧𝑒

-1.1195 -0.3138 -1.4441 -0.4976 1.1591

0.9164 1.2496 0.2717 -0.8645 1.0408
𝑖𝑛𝑝𝑢𝑡𝑆𝑖𝑧𝑒
0.3896 1.6515 -0.9640 1.2836 1.2430
2.3852 1.6643 -1.0360 -1.3234 -1.3370

Lecturer : Hongpu Liu Lecture 12-76 PyTorch Tutorial @ SLAM Research Group
Embedding in PyTorch

-1.1195 -0.3138 -1.4441 -0.4976 1.1591

Lookup 0.9164 1.2496 0.2717 -0.8645 1.0408
2
0.3896 1.6515 -0.9640 1.2836 1.2430
2.3852 1.6643 -1.0360 -1.3234 -1.3370

Lecturer : Hongpu Liu Lecture 12-77 PyTorch Tutorial @ SLAM Research Group
Embedding in PyTorch

Embedding Layer

-1.1195 -0.3138 -1.4441 -0.4976 1.1591

Lookup 0.9164 1.2496 0.2717 -0.8645 1.0408
2
0.3896 1.6515 -0.9640 1.2836 1.2430
2.3852 1.6643 -1.0360 -1.3234 -1.3370

Lecturer : Hongpu Liu Lecture 12-78 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

𝑠𝑒𝑞𝐿𝑒𝑛, 4 𝑜1 𝑜2 𝑜3 𝑜4 𝑜5

Linear Linear Linear Linear Linear

Layer Layer Layer Layer Layer

ℎ0 RNN Cell RNN Cell RNN Cell RNN Cell RNN Cell ℎ5

Embed Embed Embed Embed Embed

LongTensor 𝑥1 𝑥2 𝑥3 𝑥4 𝑥5

Lecturer : Hongpu Liu Lecture 12-79 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

Lecturer : Hongpu Liu Lecture 12-80 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

Lecturer : Hongpu Liu Lecture 12-81 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

Lecturer : Hongpu Liu Lecture 12-82 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__() Lookup matrix of Embedding:
self.emb = torch.nn.Embedding(input_size, embedding_size)
self.rnn = torch.nn.RNN(input_size=embedding_size, 𝒊𝒏𝒑𝒖𝒕𝑺𝒊𝒛𝒆, 𝒆𝒎𝒃𝒆𝒅𝒅𝒊𝒏𝒈𝑺𝒊𝒛𝒆
hidden_size=hidden_size,
num_layers=num_layers,
batch_first=True)
self.fc = torch.nn.Linear(hidden_size, num_class)

def forward(self, x):

hidden = torch.zeros(num_layers, x.size(0), hidden_size)
x = self.emb(x) # (batch, seqLen, embeddingSize)
x, _ = self.rnn(x, hidden)
x = self.fc(x)
return x.view(-1, num_class)

Lecturer : Hongpu Liu Lecture 12-83 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__()
self.emb = torch.nn.Embedding(input_size, embedding_size)
self.rnn = torch.nn.RNN(input_size=embedding_size,
hidden_size=hidden_size,
num_layers=num_layers,
batch_first=True)
Input should be LongTensor:
self.fc = torch.nn.Linear(hidden_size, num_class)
𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒔𝒆𝒒𝑳𝒆𝒏
def forward(self, x):
hidden = torch.zeros(num_layers, x.size(0), hidden_size) Output with shape:
x = self.emb(x) # (batch, seqLen, embeddingSize)
x, _ = self.rnn(x, hidden)
𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒆𝒎𝒃𝒆𝒅𝒅𝒊𝒏𝒈𝑺𝒊𝒛𝒆
x = self.fc(x) Notice: batch FIRST
return x.view(-1, num_class)

Lecturer : Hongpu Liu Lecture 12-84 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__()
self.emb = torch.nn.Embedding(input_size, embedding_size) Input of RNN:
self.rnn = torch.nn.RNN(input_size=embedding_size,
hidden_size=hidden_size, 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒆𝒎𝒃𝒆𝒅𝒅𝒊𝒏𝒈𝑺𝒊𝒛𝒆
num_layers=num_layers,
batch_first=True) Output of RNN:
self.fc = torch.nn.Linear(hidden_size, num_class)
𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆
def forward(self, x):
hidden = torch.zeros(num_layers, x.size(0), hidden_size)
x = self.emb(x) # (batch, seqLen, embeddingSize)
x, _ = self.rnn(x, hidden)
x = self.fc(x)
return x.view(-1, num_class)

Lecturer : Hongpu Liu Lecture 12-85 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__()
self.emb = torch.nn.Embedding(input_size, embedding_size)
self.rnn = torch.nn.RNN(input_size=embedding_size,
hidden_size=hidden_size,
num_layers=num_layers,
batch_first=True)
self.fc = torch.nn.Linear(hidden_size, num_class) Input of FC Layer:

def forward(self, x): 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒉𝒊𝒅𝒅𝒆𝒏𝑺𝒊𝒛𝒆

hidden = torch.zeros(num_layers, x.size(0), hidden_size)
Output of FC Layer:
x = self.emb(x) # (batch, seqLen, embeddingSize)
x, _ = self.rnn(x, hidden) 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒏𝒖𝒎𝑪𝒍𝒂𝒔𝒔
x = self.fc(x)
return x.view(-1, num_class)

Lecturer : Hongpu Liu Lecture 12-86 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

class Model(torch.nn.Module):
def __init__(self):
super(Model, self).__init__()
self.emb = torch.nn.Embedding(input_size, embedding_size)
self.rnn = torch.nn.RNN(input_size=embedding_size,
hidden_size=hidden_size,
num_layers=num_layers,
batch_first=True)
self.fc = torch.nn.Linear(hidden_size, num_class)

def forward(self, x):

hidden = torch.zeros(num_layers, x.size(0), hidden_size) Reshape result to use Cross Entropy
x = self.emb(x) # (batch, seqLen, embeddingSize)
x, _ = self.rnn(x, hidden) Loss:
x = self.fc(x)
return x.view(-1, num_class) 𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆 × 𝒔𝒆𝒒𝑳𝒆𝒏, 𝒏𝒖𝒎𝑪𝒍𝒂𝒔𝒔

Lecturer : Hongpu Liu Lecture 12-87 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

# parameters
num_class = 4
input_size = 4
hidden_size = 8
embedding_size = 10
num_layers = 2
batch_size = 1
seq_len = 5

Lecturer : Hongpu Liu Lecture 12-88 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

idx2char = ['e', 'h', 'l', 'o'] # parameters

x_data = [[1, 0, 2, 2, 3]] # (batch, seq_len) num_class = 4
y_data = [3, 1, 2, 3, 2] # (batch * seq_len) input_size = 4
hidden_size = 8
inputs = torch.LongTensor(x_data) embedding_size = 10
labels = torch.LongTensor(y_data) num_layers = 2
batch_size = 1
seq_len = 5
Input should be LongTensor:
𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆, 𝒔𝒆𝒒𝑳𝒆𝒏
Target should be LongTensor:
𝒃𝒂𝒕𝒄𝒉𝑺𝒊𝒛𝒆 × 𝒔𝒆𝒒𝑳𝒆𝒏

Lecturer : Hongpu Liu Lecture 12-89 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

net = Model()

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=0.05)

Lecturer : Hongpu Liu Lecture 12-90 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

for epoch in range(15):

optimizer.zero_grad()
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()

_, idx = outputs.max(dim=1)
idx = idx.data.numpy()
print('Predicted: ', ''.join([idx2char[x] for x in idx]), end='')
print(', Epoch [%d/15] loss = %.3f' % (epoch + 1, loss.item()))

Lecturer : Hongpu Liu Lecture 12-91 PyTorch Tutorial @ SLAM Research Group
Example 12-3 Using embedding and linear layer

Predicted: lllll, Epoch [1/15] loss = 1.412

Predicted: lllll, Epoch [2/15] loss = 1.038
Predicted: lhlll, Epoch [3/15] loss = 0.799
Predicted: ohlol, Epoch [4/15] loss = 0.652
Predicted: ohloo, Epoch [5/15] loss = 0.548
Predicted: ohlol, Epoch [6/15] loss = 0.459
Predicted: ohlol, Epoch [7/15] loss = 0.381
Predicted: ohlol, Epoch [8/15] loss = 0.313
Predicted: ohlol, Epoch [9/15] loss = 0.248
Predicted: ohlol, Epoch [10/15] loss = 0.188
Predicted: ohlol, Epoch [11/15] loss = 0.136
Predicted: ohlol, Epoch [12/15] loss = 0.098
Predicted: ohlol, Epoch [13/15] loss = 0.072
Predicted: ohlol, Epoch [14/15] loss = 0.055
Predicted: ohlol, Epoch [15/15] loss = 0.043

Lecturer : Hongpu Liu Lecture 12-92 PyTorch Tutorial @ SLAM Research Group
Exercise 12 – 1 Using LSTM

Lecturer : Hongpu Liu Lecture 12-93 PyTorch Tutorial @ SLAM Research Group
Exercise 12 – 1 Using LSTM

https://pytorch.org/docs/stable/nn.html#lstm

Lecturer : Hongpu Liu Lecture 12-94 PyTorch Tutorial @ SLAM Research Group
Exercise 12 – 1 Using LSTM

𝒄𝒕+𝟏

𝒉𝒕+𝟏

Lecturer : Hongpu Liu Lecture 12-95 PyTorch Tutorial @ SLAM Research Group
Exercise 12 – 1 Using LSTM

https://pytorch.org/docs/stable/nn.html#lstm

Lecturer : Hongpu Liu Lecture 12-96 PyTorch Tutorial @ SLAM Research Group
Exercise 12 – 2 Using GRU

Lecturer : Hongpu Liu Lecture 12-97 PyTorch Tutorial @ SLAM Research Group
Exercise 12 – 2 Using GRU

https://pytorch.org/docs/stable/nn.html#gru

Lecturer : Hongpu Liu Lecture 12-98 PyTorch Tutorial @ SLAM Research Group
Exercise 12 – 2 Using GRU

https://pytorch.org/docs/stable/nn.html#gru

Lecturer : Hongpu Liu Lecture 12-99 PyTorch Tutorial @ SLAM Research Group
 PyTorch Tutorial
12. Basic RNN

Lecturer : Hongpu Liu Lecture 12-100 PyTorch Tutorial @ SLAM Research Group