PyTorch

Aurimas Aleksandras Nausėdas

2023
PyTorch

Agenda

1. Introduction
2. PyTorch Basics
3. Train A Simple Neural Net
4. PyTorch in Action
5. Data Loading, Loss & Optimizers
6. Visualisation
7. Save & Load
8. Conclusion

1
PyTorch

Terms
- PyTorch
- Tensor
- FloatTensor
- Autograd
- Variable
- Compuational Graph
- Torch.nn
- Torch.nn.functional
- GPU
- GRU
- Dataloader
- Batch size
- Iterations
- Epoch
- Loss Function
- Optimizer
- Visdom
- TensorBoard

2
Introduction
PyTorch

What is PyTorch?
- It’s a Python-based scientific computing package targeted
at two sets of audiences1
- A replacement for NumPy to use the power of GPUs

- A deep learning research platform that provides maximum

flexibility and speed

1 – https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html
3
PyTorch

What is PyTorch?

CPU GPU
Complicate & sequential processing Simple but fast & parallel computing

We would like to use GPU to accelerate the training of our neural nets

4
PyTorch

What is PyTorch?

Data Your code

Abundant NumPy arrays Training procedure
GPU-accelerated libraries

Parallel, fast training

GPU

However, CUDA is a low-level language and not available on all machines..

5
PyTorch

What is PyTorch?
Deep learning framework

Data Your code

Abundant NumPy arrays Training procedure

GPU-accelerated libraries

Parallel, fast training

GPU
6
PyTorch

Why PyTorch?

Life is short, you need

7
PyTorch

Why PyTorch?
- It is pythonic - concise, close to Python conventions
- Strong GPU support
- Autograd - automatic differentiation
- Many algorithms and components are already
implemented
- Similar to NumPy

8
PyTorch

Why PyTorch?

9
PyTorch

Three Levels of Abstraction

- Tensor: Imperative ndarray
- Variable: Node in a computational graph (data, grad)
- Module: A neural network layer

10
PyTorch

Major Components of PyTorch

Package Description
torch a Tensor library like NumPy, with strong GPU support

torch.autograd a tape based automatic differentiation library that supports all

differentiable Tensor operations in torch

torch.nn a neural networks library deeply integrated with autograd designed for
maximum flexibility

torch.optim an optimization package to be used with torch.nn with standard

optimization methods such as SGD, RMSProp, LBFGS, Adam etc.

torch.utils DataLoader, Dataset and other utility functions for convenience

11
PyTorch Basics
PyTorch

PyTorch Tensors

- Like numpy arrays, but they can run on GPU.

- No built-in notion of computatiomnal graph,
or gradients, or deep learning.
- Here you see a fit of a two-layer net using
PyTorch Tensors.2
- The biggest difference between a numpy array
and a PyTorch Tensor is that a PyTorch Tensor
can run on either CPU or GPU.2

2 - Stanford cs231n & https://pytorch.org/tutorials/beginner/examples_tensor/two_layer_net_tensor.html

12
PyTorch

PyTorch Tensors
To run on GPU just cast
tensors to CUDA type

Stanford cs231n 13
PyTorch

PyTorch Tensors
Create random tensors
for data and weights

Stanford cs231n 14
PyTorch

PyTorch Tensors

Forward prop: compute

predictions and loss

Stanford cs231n 15
PyTorch

PyTorch Tensors

Backward prop:
Manually compute
gradients

Stanford cs231n 16
PyTorch

PyTorch Tensors

Gradient descent
step on weights
Stanford cs231n 17
PyTorch

Initializing a Tensor

18
PyTorch

Attributes of a Tensor

19
PyTorch

Operations on Tensors

20
PyTorch

Operations on Tensors

21
PyTorch

Operations on Tensors

22
PyTorch

PyTorch: Autograd
A PyTorch Variable is a node in a
computational graph

x.Data is a Tensor

x.Grad is a Variable of gradients

(same shape as x.data)

x.grad.data is a Tensor of gradients

Stanford cs231n 23
PyTorch

Variable
The autograd package provides automatic differentiation for all
operations on Tensors.

“autograd. Variable is the central class of the package. It wraps a

Tensor, and supports nearly all of operations defined on it.

Once you finish your computation you can call .backward() and
have all the gradients computed automatically. “
Pytorch Tutorial. www.pytorch.org 24
PyTorch

Computational Graphs

25
PyTorch

Computational Graphs

Define Variables to
start building a
computational graph

26
PyTorch

Computational Graphs

Forward prop
looks just like
numpy

27
PyTorch

Computational Graphs

Calling c.backward()
computes all
gradients
28
PyTorch

Computational Graphs

Run on GPU by
calling .cuda()

29
PyTorch

Module

Other layers:
Dropout, Linear,
Normalization Layer

30
PyTorch

Module – torch.nn

Containers Module, Sequential,ModuleList,ParameterList

Convolution Layers Conv1d,Conv2d,Conv3d…
Recurrent Layers RNN,LSTM,GRU,RNNCell…
Linear Layers Linear, Bilinear
Non-linear Activations ReLU,Sigmoid,Tanh,LeakyReLU…
Loss Functions NLLLoss,BCELoss,CrossEntropyLoss…

31
PyTorch

Module – torch.nn.funtional

Torch.nn

Torch.nn.functional

32
Train a Simple Neural Net
PyTorch

Train a simple Neural Net

1. Forward: compute output of each layer

2. Backward: compute gradient
3. Update: update the parameters with computed gradient
33
PyTorch

PyTorch: nn

Higher-level wrapper for

working with Neural Nets

Similar to Keras and friends …

But only one and it’s good

34
PyTorch

PyTorch: nn

Define our model as a

sequence of layers

nn also define common

loss functions

35
PyTorch

PyTorch: nn

Forward prop: feed data

to model, and prediction
to loss function

36
PyTorch

PyTorch: nn

Backward prop:
compute all gradients
37
PyTorch

PyTorch: nn

Make gradient step on

each model parameter 38
PyTorch

PyTorch: optim

Use an optimzer for

different rules

39
PyTorch

PyTorch: optim

Update all parameters

after computing gradients
40
PyTorch in Action
PyTorch

MNIST Dataset

41
PyTorch

MNIST Example

42
PyTorch

MLP for MNIST (0-d features)

10

64

28*28

43
PyTorch

MLP for MNIST

Last Test Acc: 95.8% 44

PyTorch

RNN for MNIST (1-d features)

GRU GRU … GRU

28

…
28

45
PyTorch

RNN for MNIST

Last Test Acc: 97.7% 46

PyTorch

CNN for MNIST (2-d features)

47
PyTorch

CNN for MNIST

Last Test Acc: 99.2% 48

Data Loading, Loss &
Optimizers
PyTorch

Data Loading

Pytorch, zero to all. HKUST

49
PyTorch

Dataloader

50
PyTorch

Iterate through Dataloader

51
PyTorch

Loss Function

52
PyTorch

Optimizer

53
Visualisation
PyTorch

Visualisation

Visdom

https://github.com/facebookresearch/visdom 54
PyTorch

Visualisation

TensorBoard

https://www.tensorflow.org/get_started/summaries_and_tensorboard
55
Save & Load
PyTorch

Saving and Loading Model Weights

56
PyTorch

Saving and Loading Models with Shapes

57
PyTorch

Conclusion
- PyTorch | Scientific computing package based on python
- Tensor | Fundamental datatype or most basic elemt in PyTorch equivalent to numpy.ndarray
- FloatTensor | Float Tensor datatype
- Autograd | Automatic differentiation of arbitrary scalar valued functions
- Variable | Wrapper around Tensor that contains three attributes of data, grad and grad_fn
- Computational Graph | Type of graph that can be used to represent mathematical expressions
- Torch.nn | PyTorch Class to help you create and train neural network
- Torch.nn.functional | PyTorch Class to. Create and train net that uses functions
- GPU | Graphical interface to accelerate multiple computations simultaneously
- GRU | Gating mechanism in RNN
- Dataloader | Iterable that combines a dataset and a sampler over the given dataset
- Batch size | Number of training examples in one forward/backward pass
- Iterations | Number of passes
- Epoch | Number of forward and backward passes of all training examples
- Loss Function | Method of evaluating how well the algorithm is modeling the dataset
- Optimizer | Minimizer of the loss function
- Visdom | Visualisation tool that generates rich visualizations of live data
- TensorBoard | Visualisation tool for machine learning experimentation

58
PyTorch

Resources
- Official resources
- Documentation http://pytorch.org/docs/master/
- Tutorials http://pytorch.org/tutorials/index.html
- Example projects https://github.com/pytorch/examples
- Github code
- fairseq-py
- OpenNMT-py
- Open course & tutorial used in this presentation
- Stanford CS231N 2017, Lecture 8 “Deep Learning Software”
- https://pytorch.org/tutorials/beginner/basics/intro.html