AI VIETNAM

All-in-One Course

Pytorch
Deep Learning Framework

Quang-Vinh Dinh
Ph.D. in Computer Science
 Outline
➢ Introduction to PyTorch
➢ Model Construction
➢ Model Training and Inference
➢ Applying Softmax for Image Data
 AI VIETNAM
All-in-One Course
Introduction
Create a tensor in PyTorch

2
 AI VIETNAM
All-in-One Course
Introduction
❖ Tensor
❖ Broadcasting

data

1 2
Axis 0 + 1
3 4

Axis 1

2 3

4 5
 AI VIETNAM
All-in-One Course
Introduction
❖ Tensor
❖ Broadcasting

data

1 2
Axis 0 + 2
3 4

Axis 1

3 4

5 6

4
 AI VIETNAM
All-in-One Course
Introduction
❖ Tensor
❖ Important functions

Squared Difference

𝑠𝑑 = 𝒙 − 𝒚 2

x sd

1 16

2 9
𝒙−𝑦 2 =
3 4

4 1

5
 AI VIETNAM
All-in-One Course
Introduction
❖ Tensor
❖ Important functions

Mean Squared Error

𝑠𝑑 = ෍ 𝒙𝑖 − 𝒚𝑖 2

x y sd

1 5 16

2 5 9
σ𝑖 𝒙𝑖 − 𝒚𝑖 2
= σ𝑖( ) = 7.5
3 5 4

4 5 1
6
 AI VIETNAM
All-in-One Course
Introduction
❖ Tensor
❖ Important functions

x y

1 2 3 4
Axis 0

3 4 5 6

Axis 1

tensor1

1 2
tensor2
3 4
1 2 3 4
3 4
3 4 5 6
5 6
 AI VIETNAM
All-in-One Course
Introduction
❖ Tensor
❖ Important functions

3 4
4 2 .argmax(axis=0) = 1 0
1 1

3 4
4 2 .argmax(axis=1) = 1 0 0
1 1

8
 AI VIETNAM
All-in-One Course
Introduction
❖ Gradient computation

𝑦 = 𝑥2

𝑧 = 3𝑦 + 2

𝑥 ∙ 2 ∗ + 𝑧

3 2

9
Example 1
𝑥 ∗ + 𝑦ො

1 output
𝑤 𝑏
Input 𝑥 = 6.7

1+1 variables
Model
Parameters
𝑏 = −0.0393 w = -0.655

Label

𝑦ො = 𝑥𝑤 + 𝑏 𝑦 = 9.1

Loss
2
𝐿𝑜𝑠𝑠 = 𝑦ො − 𝑦
10
Example 1
𝑥 ∗ + 𝑦ො

1 output
𝑤 𝑏
Input 𝑥 = 6.7
1+1 variables

Model
Parameters
𝑏 = 0.04 w = −0.34

Label

𝑦ො = 𝑥𝑤 + 𝑏 𝑦 = 9.1

Loss
2
𝐿𝑜𝑠𝑠 = 𝑦ො − 𝑦
Example 1 𝑦

𝐿′𝑤 𝐿′𝑏 𝑥 ∗ + 𝑦ො 𝐿

Input 𝑥 = 6.7 𝑤 𝑏

Model
Parameters
𝑏 = 0.04 w = −0.34

Label

𝑦ො = 𝑥𝑤 + 𝑏 𝑦 = 9.1

Loss
2
𝐿𝑜𝑠𝑠 = 𝑦ො − 𝑦
Example 1 3 𝑦
𝐿′𝑤 = 0 𝐿′𝑏 = 0
𝑥 ∗ + 𝑦ො 𝐿

Input 𝑥 = 6.7 4
𝑤 𝑏
𝐿′𝑤 = ⋯ 𝐿′𝑏 = ⋯ 5
Model
Parameters
𝑏 = −0.0393 w = -0.655
1
1
Label
2
𝑦ො = 𝑥𝑤 + 𝑏 𝑦 = 9.1

3
4
Loss
2
2
𝐿𝑜𝑠𝑠 = 𝑦ො − 𝑦
5 13
Example 1

Initialize b and
Input 𝑥 = 6.7
w randomly

Model
Parameters
𝑏 = 0.04 w = -0.34

Label

𝑦ො = 𝑥𝑤 + 𝑏 = -2.238 𝑦 = 9.1

Loss
2
𝑦ො − 𝑦 = 128.55
𝑥 ∗ + 𝜎() 𝑦ො Logistic Regression
1 output
𝑤 𝑏

1+1 variables
𝑥

Model
𝑏 𝑤
0.1 -0.1

𝑧 = 𝑤𝑥 + 𝑏

1 𝑦=0
𝑦ො = 𝜎(𝑧) =
1 + 𝑒 −𝑧
𝑦
Loss
−ylogොy−(1−y)log(1−ොy )
 1 1.5 0.2
𝒙=
1 4.1 1.3
Logistic Regression
Model
0.1
𝜽 = 0.5
−0.1
0
𝒚=
0.6963 1
ෝ = 𝜎 𝒙𝜽 =
𝒚
0.8828

Loss
3
𝐿(𝜽) = 0.65815

4 0.28961
𝛻𝜽 𝐿 = 0.28217
−0.0064
5 0.1 0.28961 0.0971
𝛉 − ηL′𝛉 = 0.5 − η 0.28217 = 0.4971
−0.1 −0.0064 −0.099
 AI VIETNAM
All-in-One Course
Loss Functions
❖ Cross-entropy 𝑥 𝒙=
1.4
4.5
𝑏0 𝑤0 𝑏1 𝑤1
0.1 0.2 0.05 −0.1
𝐿(ෝ
𝒚, 𝒚) = ෍ −𝑦 𝑖 log 𝑦ො 𝑖
𝑖
0.38 −0.09
𝒛0 = 𝑧0 = 𝑤0 𝑥 + 𝑏0 𝑧1 = 𝑤1 𝑥 + 𝑏1 𝒛1 =
1.0 −0.4

0.615 𝑒 𝑧0 𝑒 𝑧1 0.385
ෝ0 =
𝒚 𝑦ො0 = 1 𝑦ො1 = 1 ෝ1 =
𝒚
0.802 σ𝑖=0 𝑒 𝑧𝑖 σ𝑖=0 𝑒 𝑧𝑖 0.198

[1 0]
L = −𝑦0 log𝑦ො0 − 𝑦1 log𝑦ො1 𝑦 𝒚=
[0 1]

𝐿 = 1.0530
17
 AI VIETNAM
All-in-One Course
nn.Sequential()
Model Model
𝑥1 𝑥1
𝑧1 𝑦1
𝑥2 𝑥2
𝑧2 Softmax 𝑦2
𝑧
𝑥3 𝑥3
𝑧3 𝑦3
𝑥4
1

1 softmax

input_shape = (3,) input_shape = (4,)

dense(units=1) dense(units=3)

18
 AI VIETNAM
All-in-One Course
nn.Sequential()
𝑥1
Model 𝑥1 𝑧1
𝑥2
𝑥2 𝑧2
… 𝑥3
𝑧 𝑧3
𝑥3
𝑥4

1 1
nn.Sequential()
 AI VIETNAM
All-in-One Course
nn.Sequential()
Model
input_shape = (4,)
𝑥1
…
𝑧1 𝑦1
𝑥2
𝑧2 Softmax 𝑦2
𝑥3
nn.Sequential()
𝑧3 𝑦3
𝑥4

1 softmax

dense(units=3)

20
 Outline
➢ Introduction to PyTorch
➢ Model Construction
➢ Model Training and Inference
➢ Applying Softmax for Image Data
 AI VIETNAM
All-in-One Course
Model Construction
❖ Linear regression
Input layer a layer
area w price
Feature Label

𝑏
1
input_shape = (1,) dense(units=1)

House price data

Model

price = w ∗ 𝑎𝑟𝑒𝑎 + 𝑏
𝑦ො = 𝑤𝑥 + 𝑏

21
 AI VIETNAM
All-in-One Course
Model Construction
❖ Linear regression Input layer a layer
𝑥1
Features Label
𝑥2

𝑧
𝑥3

input_shape = (3,) dense(units=1)

Advertising-based sale data

Model

Sale = 𝑤1 ∗ 𝑇𝑉 + 𝑤2 ∗ 𝑅𝑎𝑑𝑖𝑜 + 𝑤3 ∗ 𝑁𝑒𝑤𝑠𝑝𝑎𝑝𝑒𝑟 + 𝑏

𝑦ො = 𝑤1 𝑥1 + 𝑤2 𝑥2 + 𝑤3 𝑥3 + 𝑏
 AI VIETNAM
All-in-One Course
Model Construction
❖ Linear regression

Features Label

Boston House
Price Data

Model
medv = 𝑤1 ∗ 𝑥1 + ⋯ + 𝑤13 ∗ 𝑥13 + 𝑏

23
 AI VIETNAM
All-in-One Course
Model Construction
❖ Linear regression

Model
medv = 𝑤1 ∗ 𝑥1 + ⋯ + 𝑤13 ∗ 𝑥13 + 𝑏

Input layer a layer

𝑥1
…

…
𝑧
𝑥13

input_shape = (13,) dense(units=1)

24
 AI VIETNAM
All-in-One Course
Model Construction
Feature Label ❖ Logistic regression
Model
z = 𝜽𝑇 𝒙
1
𝑦ො =
1 + 𝑒 −𝑧

Input layer A layer

𝑥

Sigmoid
𝑧 function
yො

1
criterion = nn.BCEWithLogitsLoss()
input_shape = (1,)
dense(units=1)
25
 AI VIETNAM
All-in-One Course
Model Construction
Feature Label ❖ Logistic regression
Model
𝑧 = 𝜽𝑇 𝒙
1
𝑦ො =
1 + 𝑒 −𝑧

Input layer A layer

𝑥1

𝑥2
Sigmoid
𝑧 function
yො

1
criterion = nn.BCEWithLogitsLoss()
input_shape = (2,)
dense(units=1)
26
 AI VIETNAM
All-in-One Course
Model Construction
Feature Label ❖ Logistic regression
Model
𝑧 = 𝜽𝑇 𝒙
1
𝑦ො =
1 + 𝑒 −𝑧

Input layer A layer

𝑥1

𝑥2
𝑧 Sigmoid yො
function
𝑥3

𝑥4 criterion = nn.BCEWithLogitsLoss()
dense(units=1)
1
input_shape = (4,) 27
 AI VIETNAM
All-in-One Course
Model Construction
Feature Label 𝑧1 = 𝑥𝑤1 + 𝑏1 ❖ Softmax regression
𝑧2 = 𝑥𝑤2 + 𝑏2
𝑒 𝑧1
yො 1 = 2
σ𝑗=1 𝑒 𝑧𝑗
𝑒 𝑧1
Iris Classification Data yො 2 = 2
σ𝑗=1 𝑒 𝑧𝑗

Input layer A layer Activation layer

𝑥 𝑤1
𝑧11
𝑤 yො 1
𝑏1 Softmax
𝑤2 function
1 𝑧2 yො 2
𝑏2
criterion = nn.CrossEntropyLoss()
input_shape = (1,) softmax

dense(units=2) 28
 AI VIETNAM
All-in-One Course
Model Construction
❖ Softmax regression

Input layer A layer

𝑥
𝑧1 yො 1

𝑧2 Softmax yො 2
criterion = nn.CrossEntropyLoss()

1 𝑧3 yො 3

input_shape = (1,)
dense(units=3)
29
 AI VIETNAM
All-in-One Course Model Construction
❖ Softmax regression
Forward computation
𝑒𝒛
𝒛 = 𝜽𝑇 𝒙 𝐲ො = 𝑘
σ𝑖=1 𝑒 𝑧𝑖

Input layer A layer

𝑥1
𝑧1 yො 1
𝑥2
𝑧2 Softmax yො 2
𝑥3
𝑧3 yො 3
𝑥4
dense(units=3)
1
input_shape = (4,)
 Outline
➢ Introduction to PyTorch
➢ Model Construction
➢ Model Training and Inference
➢ Applying Softmax for Image Data
 AI VIETNAM
All-in-One Course
Training
❖ Logistic regression
Declare optimizer and loss function

→ Tính output 𝑦ො
𝒛 = 𝜽𝑇 𝒙
1
𝑦ො = 𝜎(𝑧) =
1 + 𝑒 −𝒛
→ Tính loss (binary cross-entropy)
𝐿(𝜽) = −𝐲 T log𝐲−(1−y)
ො T log(1−𝐲
ො)

→ Tính đạo hàm

Start training
𝐿′𝜽 = 𝐱 T (𝐲ො − 𝒚)

→ Cập nhật tham số (Stochastic

gradient descent)
𝜽 = 𝜽 − 𝜂𝐿′𝜽

31
 AI VIETNAM
All-in-One Course
Training
❖ Softmax regression Delaration and Training

→ Tính output 𝑦ො
𝑒𝒛
𝒛 = 𝜽𝑇 𝒙 𝐲ො = 𝑘
σ𝑖=1 𝑒 𝑧𝑖
→ Tính loss (cross-entropy)
𝑘

𝐿(𝜽) = − ෍ 𝑦𝑖 log𝑦ො𝑖
𝑖=1

→ Tính đạo hàm

𝜕𝐿
= 𝒙 𝑦ො𝑖 − 𝑦𝑖
𝜕𝜽𝑖

→ Cập nhật tham số (Stochastic

gradient descent)
𝜽 = 𝜽 − 𝜂𝐿′𝜽
32
 AI VIETNAM
All-in-One Course
Training
❖ Softmax regression

Model
𝒛 = 𝜽𝑇 𝒙
𝑒𝒛
𝐲ො = 𝑘
σ𝑖=1 𝑒 𝑧𝑖

33
 AI VIETNAM
All-in-One Course
Training
❖ Softmax regression

Model
𝒛 = 𝜽𝑇 𝒙
𝑒𝒛
𝐲ො = 𝑘
σ𝑖=1 𝑒 𝑧𝑖

34
 Outline
➢ Introduction to PyTorch
➢ Model Construction
➢ Model Training and Inference
➢ Applying Softmax for Image Data
 AI VIETNAM
All-in-One Course
Image Classification: Image Data
❖ Grayscale images

(Height, Width)
Pixel p = scalar Resolution: #pixels
Resolution = HeightxWidth
0 ≤ p ≤ 255
35
 AI VIETNAM
All-in-One Course
Image Classification: Image Data
❖ Color images

(Channel, Height, Width,)

𝑟
RGB color image Pixel p= 𝑔 Resolution: #pixels
𝑏 Resolution = HeightxWidth

0 ≤ r,g,b ≤ 255 36
Important Packages
❖ Some functions

To download a file
import urllib.request as req
req.urlretrieve(url, name)

To open an image
from PIL import Image
img = Image.open(name)

To show an image
import matplotlib.pyplot as plt
plt.imshow(img)

37
Image Data

MNIST dataset

Grayscale images

Resolution=28x28

Training set: 60000 samples

Testing set: 10000 samples

38
Image Data T-shirt

Trouser

Pullover
Fashion-MNIST dataset
Dress

Grayscale images Coat

Resolution=28x28 Sandal

Training set: 60000 samples Shirt

Testing set: 10000 samples

Sneaker

Bag

Ankle
Boot
 Using 𝑧1
Fully
Softmax 28
784 connect Output

...
Softmax

...
28
activation
Regression 𝑧10
flatten data

784 Nodes 10 Nodes

Input layer Output layer

Data Sets
 AI VIETNAM
All-in-One Course
Demo

Test Accuracy
91.97%