Machine Learning
Pytorch Tutorial
TA : 曾元（Yuan Tseng）
2022.02.18
Outline
● Background: Prerequisites & What is Pytorch?
● Training & Testing Neural Networks in Pytorch
● Dataset & Dataloader
● Tensors
● torch.nn: Models, Loss Functions
● torch.optim: Optimization
● Save/load models
Prerequisites
● We assume you are already familiar with…
1. Python3
■ if-else, loop, function, file IO, class, ...
■ refs: link1, link2, link3
2. Deep Learning Basics
■ Prof. Lee’s 1st & 2nd lecture videos from last year
■ ref: link1, link2
Some knowledge of NumPy will also be useful!
What is PyTorch?
● An machine learning framework in Python.
● Two main features:
○ N-dimensional Tensor computation (like NumPy) on GPUs
○ Automatic differentiation for training deep neural networks
Training Neural Networks
Define Neural Optimization
Loss Function
Network Algorithm
Training
More info about the training process in last year's lecture video.
Training & Testing Neural Networks
Training Validation Testing
Guide for training/validation/testing can be found here.
Training & Testing Neural Networks - in Pytorch
Step 1.
torch.utils.data.Dataset &
Load Data torch.utils.data.DataLoader
Training Validation Testing
Dataset & Dataloader
● Dataset: stores data samples and expected values
● Dataloader: groups data in batches, enables multiprocessing
● dataset = MyDataset(file)
● dataloader = DataLoader(dataset, batch_size, shuffle=True)
Training: True
Testing: False
More info about batches and shuffling here.
Dataset & Dataloader
from torch.utils.data import Dataset, DataLoader
class MyDataset(Dataset):
def __init__(self, file):
self.data = ... Read data & preprocess
def __getitem__(self, index):
return self.data[index] Returns one sample at a time
def __len__(self):
return len(self.data) Returns the size of the dataset
Dataset & Dataloader
dataset = MyDataset(file)
dataloader = DataLoader(dataset, batch_size=5, shuffle=False)
DataLoader
__getitem__(0) 0
__getitem__(1) 1
Dataset __getitem__(2) 2 batch_size
__getitem__(3) 3
__getitem__(4) 4
mini-batch
Tensors
● High-dimensional matrices (arrays)
1-D tensor 2-D tensor 3-D tensor
e.g. audio e.g. black&white e.g. RGB images
images
Tensors – Shape of Tensors
● Check with .shape
4
3
5
3
5 5
(5, ) (3, 5) (4, 5, 3)
dim 0 dim 0 dim 1 dim 0 dim 1 dim 2
Note: dim in PyTorch == axis in NumPy
Tensors – Creating Tensors
● Directly from data (list or numpy.ndarray) tensor([[1., -1.],
x = torch.tensor([[1, -1], [-1, 1]]) [-1., 1.]])
x = torch.from_numpy(np.array([[1, -1], [-1, 1]]))
● Tensor of constant zeros & ones tensor([[0., 0.],
[0., 0.]])
x = torch.zeros([2, 2])
x = torch.ones([1, 2, 5]) tensor([[[1., 1., 1., 1., 1.],
shape [1., 1., 1., 1., 1.]]])
Tensors – Common Operations
Common arithmetic functions are supported, such as:
● Addition ● Summation
z = x + y y = x.sum()
● Subtraction ● Mean
z = x - y y = x.mean()
● Power
y = x.pow(2)
Tensors – Common Operations
● Transpose: transpose two specified dimensions
>>> x = torch.zeros([2, 3])
2
>>> x.shape
3
torch.Size([2, 3])
>>> x = x.transpose(0, 1)
>>> x.shape 3
torch.Size([3, 2])
2
Tensors – Common Operations
● Squeeze: remove the specified dimension with length = 1
>>> x = torch.zeros([1, 2, 3])
>>> x.shape 1
3
2
torch.Size([1, 2, 3])
>>> x = x.squeeze(0)
(dim = 0)
>>> x.shape 2
torch.Size([2, 3]) 3
Tensors – Common Operations
● Unsqueeze: expand a new dimension
>>> x = torch.zeros([2, 3]) 2
>>> x.shape
3
torch.Size([2, 3])
>>> x = x.unsqueeze(1) (dim = 1)
>>> x.shape 2
torch.Size([2, 1, 3]) 3
1
Tensors – Common Operations
x 2
3
1
● Cat: concatenate multiple tensors
y 2
>>> x = torch.zeros([2, 1, 3])
3
3
>>> y = torch.zeros([2, 3, 3])
>>> z = torch.zeros([2, 2, 3]) z
2
>>> w = torch.cat([x, y, z], dim=1) 3
2
>>> w.shape
w
torch.Size([2, 6, 3]) 2
3
6
more operators: https://pytorch.org/docs/stable/tensors.html
Tensors – Data Type
● Using different data types for model and data will cause errors.
Data type dtype tensor
32-bit floating point torch.float torch.FloatTensor
64-bit integer (signed) torch.long torch.LongTensor
see official documentation for more information on data types.
Tensors – PyTorch v.s. NumPy
● Similar attributes
PyTorch NumPy
x.shape x.shape
x.dtype x.dtype
see official documentation for more information on data types.
ref: https://github.com/wkentaro/pytorch-for-numpy-users
Tensors – PyTorch v.s. NumPy
● Many functions have the same names as well
PyTorch NumPy
x.reshape / x.view x.reshape
x.squeeze() x.squeeze()
x.unsqueeze(1) np.expand_dims(x, 1)
ref: https://github.com/wkentaro/pytorch-for-numpy-users
Tensors – Device
● Tensors & modules will be computed with CPU by default
Use .to() to move tensors to appropriate devices.
● CPU
x = x.to(‘cpu’)
● GPU
x = x.to(‘cuda’)
Tensors – Device (GPU)
● Check if your computer has NVIDIA GPU
torch.cuda.is_available()
● Multiple GPUs: specify ‘cuda:0’, ‘cuda:1’, ‘cuda:2’, ...
● Why use GPUs?
○ Parallel computing with more cores for arithmetic calculations
○ See What is a GPU and do you need one in deep learning?
Tensors – Gradient Calculation
1 >>> x = torch.tensor([[1., 0.], [-1., 1.]], requires_grad=True)
2 >>> z = x.pow(2).sum()
3 >>> z.backward()
4 >>> x.grad
1 2
tensor([[ 2., 0.],
[-2., 2.]])
3 4
See here to learn about gradient calculation.
Training & Testing Neural Networks – in Pytorch
Step 2.
torch.nn.Module
Load Data
Define Neural
Network
Loss Function Training Validation Testing
Optimization
Algorithm
torch.nn – Network Layers
● Linear Layer (Fully-connected Layer)
nn.Linear(in_features, out_features)
Input Tensor Output Tensor
nn.Linear(32, 64)
* x 32 * x 64
can be any shape (but last dimension must be 32)
e.g. (10, 32), (10, 5, 32), (1, 1, 3, 32), ...
torch.nn – Network Layers
● Linear Layer (Fully-connected Layer)
ref: last year's lecture video
torch.nn – Neural Network Layers
● Linear Layer (Fully-connected Layer)
y1
x1
y2
x2
32 y3 64 W x x + b = y
x3 (64x32)
...
...
x32
y64
torch.nn – Network Parameters
● Linear Layer (Fully-connected Layer)
>>> layer = torch.nn.Linear(32, 64)
>>> layer.weight.shape
torch.Size([64, 32]) W x x + b = y
(64x32)
>>> layer.bias.shape
torch.Size([64])
torch.nn – Non-Linear Activation Functions
● Sigmoid Activation
nn.Sigmoid()
● ReLU Activation
nn.ReLU()
See here to learn about why we need activation functions.
torch.nn – Build your own neural network
import torch.nn as nn
class MyModel(nn.Module):
def __init__(self):
super(MyModel, self).__init__()
self.net = nn.Sequential(
nn.Linear(10, 32), Initialize your model & define layers
nn.Sigmoid(),
nn.Linear(32, 1)
)
def forward(self, x):
Compute output of your NN
return self.net(x)
torch.nn – Build your own neural network
import torch.nn as nn import torch.nn as nn
class MyModel(nn.Module): class MyModel(nn.Module):
def __init__(self): def __init__(self):
super(MyModel, self).__init__() super(MyModel, self).__init__()
self.net = nn.Sequential( self.layer1 = nn.Linear(10, 32)
nn.Linear(10, 32), self.layer2 = nn.Sigmoid(),
nn.Sigmoid(), = self.layer3 = nn.Linear(32,1)
nn.Linear(32, 1)
) def forward(self, x):
out = self.layer1(x)
def forward(self, x): out = self.layer2(out)
return self.net(x) out = self.layer3(out)
return out
Training & Testing Neural Networks – in Pytorch
Step 3.
torch.nn.MSELoss
torch.nn.CrossEntropyLoss etc.
Load Data
Define Neural
Network
Loss Function Training Validation Testing
Optimization
Algorithm
torch.nn – Loss Functions
● Mean Squared Error (for regression tasks)
criterion = nn.MSELoss()
● Cross Entropy (for classification tasks)
criterion = nn.CrossEntropyLoss()
● loss = criterion(model_output, expected_value)
Training & Testing Neural Networks – in Pytorch
Step 4.
torch.optim
Load Data
Define Neural
Network
Loss Function Training Validation Testing
Optimization
Algorithm
torch.optim
● Gradient-based optimization algorithms that adjust network
parameters to reduce error. (See Adaptive Learning Rate lecture video)
● E.g. Stochastic Gradient Descent (SGD)
torch.optim.SGD(model.parameters(), lr, momentum = 0)
torch.optim
optimizer = torch.optim.SGD(model.parameters(), lr, momentum = 0)
● For every batch of data:
1. Call optimizer.zero_grad() to reset gradients of model parameters.
2. Call loss.backward() to backpropagate gradients of prediction loss.
3. Call optimizer.step() to adjust model parameters.
See official documentation for more optimization algorithms.
Training & Testing Neural Networks – in Pytorch
Load Data
Define Neural
Network
Loss Function Training Validation Testing
Optimization
Algorithm Step 5.
Entire Procedure
Neural Network Training Setup
dataset = MyDataset(file) read data via MyDataset
tr_set = DataLoader(dataset, 16, shuffle=True) put dataset into Dataloader
model = MyModel().to(device) construct model and move to device (cpu/cuda)
criterion = nn.MSELoss() set loss function
optimizer = torch.optim.SGD(model.parameters(), 0.1) set optimizer
Neural Network Training Loop
for epoch in range(n_epochs): iterate n_epochs
model.train() set model to train mode
for x, y in tr_set: iterate through the dataloader
optimizer.zero_grad() set gradient to zero
x, y = x.to(device), y.to(device) move data to device (cpu/cuda)
pred = model(x) forward pass (compute output)
loss = criterion(pred, y) compute loss
loss.backward() compute gradient (backpropagation)
optimizer.step() update model with optimizer
Neural Network Validation Loop
model.eval() set model to evaluation mode
total_loss = 0
for x, y in dv_set: iterate through the dataloader
x, y = x.to(device), y.to(device) move data to device (cpu/cuda)
with torch.no_grad(): disable gradient calculation
pred = model(x) forward pass (compute output)
loss = criterion(pred, y) compute loss
total_loss += loss.cpu().item() * len(x) accumulate loss
avg_loss = total_loss / len(dv_set.dataset) compute averaged loss
Neural Network Testing Loop
model.eval() set model to evaluation mode
preds = []
for x in tt_set: iterate through the dataloader
x = x.to(device) move data to device (cpu/cuda)
with torch.no_grad(): disable gradient calculation
pred = model(x) forward pass (compute output)
preds.append(pred.cpu()) collect prediction
Notice - model.eval(), torch.no_grad()
● model.eval()
Changes behaviour of some model layers, such as dropout and batch
normalization.
● with torch.no_grad()
Prevents calculations from being added into gradient computation
graph. Usually used to prevent accidental training on validation/testing
data.
Save/Load Trained Models
● Save
torch.save(model.state_dict(), path)
● Load
ckpt = torch.load(path)
model.load_state_dict(ckpt)
More About PyTorch
● torchaudio
○ speech/audio processing
● torchtext
○ natural language processing
● torchvision
○ computer vision
● skorch
○ scikit-learn + pyTorch
More About PyTorch
● Useful github repositories using PyTorch
○ Huggingface Transformers (transformer models: BERT, GPT, ...)
○ Fairseq (sequence modeling for NLP & speech)
○ ESPnet (speech recognition, translation, synthesis, ...)
○ Most implementations of recent deep learning papers
○ ...
References
● Machine Learning 2021 Spring Pytorch Tutorial
● Official Pytorch Tutorials
● https://numpy.org/
Any questions?
