Artificial Intelligence

Artificial Neural
Network
Pham Viet Cuong
Dept. Control Engineering & Automation, FEEE
Ho Chi Minh City University of Technology
 Artificial Neural Networks
✓ Computing systems inspired by biological neural networks
✓ Consists of several processing elements that receive inputs and deliver
outputs
✓ "Learn" to perform tasks by considering examples
✓ Be able to model nonlinear processes

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 2
 Artificial Neural Networks
✓ Applications:
❖ Natural Language Processing
▪ Text classification
▪ Language generation
▪ Document summarization
▪ Machine translation
▪ Speech recognition
▪ Character recognition
▪ Spell checking

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 3
 Artificial Neural Networks
✓ Applications:
❖ System identification and control
▪ Vehicle control
▪ Process control
▪ Auto-piloting
▪ Autonomous driving cars
▪ Robot navigation
▪ Component fault detection and simulation
▪ Chip failure analysis
▪ Trajectory prediction
▪ Petroleum exploration

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 4
 Artificial Neural Networks
✓ Applications:
❖ Time series forecasting and prediction
▪ Weather forecasting
▪ Stock market prediction
▪ Cost models
❖ Medical
▪ Cancers detection
▪ Medical image analysis
▪ EEGs
▪ Drug design

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 5
 Artificial Neural Networks
✓ Applications:
❖ Image/video/audio analysis
▪ Image recognition
▪ Image compression
▪ Radar and sonar image classification
▪ Signature verification

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 6
 Artificial Neural Networks
✓ Applications:
❖ Education:
▪ Adaptive learning software
▪ Student performance modeling
▪ Personality profiling
❖ Bussiness analytics
▪ Customer behavior modeling
▪ Customer segmentation
▪ Market research
❖ Banking
▪ Credit and loan application evaluation
▪ Fraud and risk evaluation
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 7
 Artificial Neural Networks
✓ Biological neural network

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 8
 Artificial Neural Networks
✓ Artificial neural network

w1
w2

wn
out = f(w1in1 + w2in2 + . . . + wninn + bias)
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 9
 Artificial Neural Networks
✓ Activation function

sigmoid

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 10
 Artificial Neural Networks

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 11
 Artificial Neural Networks
✓ Example: self-driving car
❖ Input (raw data/features)
❖ Output (steering angle, throttle, brake)
❖ Hidden layers

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 12
 Artificial Neural Networks
✓ Why nonlinear activation function?
✓ Why bias?

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 13
 Artificial Neural Networks
✓ Useful links
❖ Link 1
❖ Link 2
❖ Link 3

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 14
 Artificial Neural Networks
✓ Error back propagation Forward
❖ Framework

❖ Loss function:
Backward
➢ Squared L2 norm
➢ Angle between 2 vectors
➢ Cross entropy
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 15
 Artificial Neural Networks
✓ Error back propagation
❖ Loss function:
➢ Euclidean norm (2-norm)

➢ Angle between 2 vectors

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 16
 Artificial Neural Networks
✓ Error back propagation
❖ Loss function:
➢ Euclidean norm (2-norm) Cross entropy

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 17
 Artificial Neural Networks
✓ Error back propagation
❖ Optimization problem
❖ Gradient descent
𝑑𝑓
𝑥←𝑥−
𝑑𝑥
𝑑𝑓
𝑥 ←𝑥−𝜇
𝑑𝑥
𝜇: learning rate
𝑥2
𝜕𝐿
𝑤 ←𝑤−𝜇 𝑥1
𝜕𝑤

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 18
 Artificial Neural Networks
✓ Error back propagation
❖ Example Forward

𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤
Backward
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 19
 Artificial Neural Networks
✓ Error back propagation
❖ Example

𝑛𝑒𝑡ℎ1 = 𝑤1 𝑖1 + 𝑤2 𝑖2 + 𝑏1

𝑜𝑢𝑡ℎ1 = 𝑓 𝑛𝑒𝑡ℎ1

1
𝑓 𝑥 =
1 + 𝑒 −𝑥

1 2 2
𝐸= 𝑡𝑎𝑟𝑔𝑒𝑡𝑜1 − 𝑜𝑢𝑡𝑜1 + 𝑡𝑎𝑟𝑔𝑒𝑡𝑜2 − 𝑜𝑢𝑡𝑜2
2
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 20
 Artificial Neural Networks
✓ Error back propagation
𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤
𝜕𝐸
𝜕𝑤5

❖ Chain rule
𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1
=
𝜕𝑤5 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑤5

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 21
 Artificial Neural Networks
✓ Error back propagation
❖ Example
𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1
=
𝜕𝑤5 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑤5

𝑛𝑒𝑡𝑜1 = 𝑤5 𝑜𝑢𝑡ℎ1 + 𝑤6 𝑜𝑢𝑡ℎ2 + 𝑏2

𝑜𝑢𝑡𝑜1 = 𝑓 𝑛𝑒𝑡𝑜1
1
𝑓 𝑥 =
1 + 𝑒 −𝑥
1 2 2
𝐸= 𝑡𝑎𝑟𝑔𝑒𝑡𝑜1 − 𝑜𝑢𝑡𝑜1 + 𝑡𝑎𝑟𝑔𝑒𝑡𝑜2 − 𝑜𝑢𝑡𝑜2
2
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 22
 Artificial Neural Networks
✓ Error back propagation
❖ Chain rule

𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1
=
𝜕𝑤5 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑤5

w5 netO1 outO1 E

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 23
 Artificial Neural Networks
✓ Error back propagation
❖ Chain rule

𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1
=
𝜕𝑤5 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑤5

𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 24
 Artificial Neural Networks
✓ Error back propagation
❖ Chain rule
𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1
=
𝜕𝑤1 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1 𝜕𝑤1

w1 neth1 outh1 netO1 outO1 E

netO2 outO2

𝜕𝐸 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝐸 𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 25
 Artificial Neural Networks
❖ Homework 2: Find derivative formulas for a NN
▪ 2 inputs
▪ 2 hidden layers, each layer has 3 neurals
▪ 3 outputs
w7 w16
w1

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 26
 Artificial Neural Networks
❖ Vanishing gradient problem 𝜕𝐸
𝜕𝑤1
=
𝜕𝐸 𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1
𝜕𝑜𝑢𝑡𝑜1 𝜕𝑛𝑒𝑡𝑜1 𝜕𝑜𝑢𝑡ℎ1
+
𝜕𝐸 𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1
𝜕𝑜𝑢𝑡𝑜2 𝜕𝑛𝑒𝑡𝑜2 𝜕𝑜𝑢𝑡ℎ1 𝜕𝑛𝑒𝑡ℎ1 𝜕𝑤1

𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 27
 Artificial Neural Networks
✓ Overfitting: high variance
✓ Underfitting: high bias

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 28
 Artificial Neural Networks
✓ Overfitting vs. Underfitting

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 29
 Artificial Neural Networks
✓ Overfitting vs. Underfitting

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 30
 Artificial Neural Networks
✓ Overfitting vs. Underfitting

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 31
 Artificial Neural Networks
✓ Reasons
❖ Underfitting (simplifying assumption)
▪ Too simple model
▪ Not enough training
❖ Overfitting
▪ Too complicated model
▪ Not enough training data

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 32
 Artificial Neural Networks
✓ Countermeasures for Underfitting
❖ Better model
❖ More training

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 33
 Artificial Neural Networks
✓ Countermeasures for Overfitting
❖ Simplifying model
❖ Removing features from data
❖ More data
▪ ImageNet Large Scale Visual Recognition Challenge
▪ 1000 classes
▪ 1.2 M images

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 34
 Artificial Neural Networks
✓ Countermeasures for Overfitting
❖ Data augmentation
▪ Extracting patches

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 35
 Artificial Neural Networks
✓ Countermeasures for Overfitting
❖ Data augmentation
▪ Extracting patches
▪ Reflection

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 36
 Artificial Neural Networks
✓ Countermeasures for Overfitting
❖ Data augmentation
▪ Extracting patches
▪ Reflection
▪ Rotation

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 37
 Artificial Neural Networks
✓ Countermeasures for Overfitting
❖ Data augmentation
▪ Extracting patches
▪ Reflection
▪ Rotation
▪ Altering intensities of RGB channels

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT
?
38
 Artificial Neural Networks
✓ Countermeasures for Overfitting
❖ Data augmentation
▪ Extracting patches
▪ Reflection
▪ Rotation
▪ Altering intensities of RGB channels
▪ and more
▪ AlexNet: 2048 times

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 39
 Artificial Neural Networks
✓ Countermeasures
❖ Early stopping
▪ Training set
▪ Test set
▪ Validation set

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 40
 Artificial Neural Networks
✓ Reasons
✓ Countermeasures
❖ Early stopping
▪ Training set
▪ Test set
▪ Validation set

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 41
 Artificial Neural Networks
✓ Countermeasures
❖ Drop out
▪ Co-adaptation

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 42
 Artificial Neural Networks
✓ Gradient descent
❖ 2D illustration
❖ Stochastic gradient descent Forward
❖ Batch gradient descent

L
𝜕𝐿
𝑤 ←𝑤−𝜇
𝜕𝑤

Backward

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 43
 Artificial Neural Networks
✓ Gradient descent
❖ Stochastic gradient descent
▪ Update: each training example
▪ Online
▪ Unstable
▪ Can avoid local minima
❖ Batch gradient descent
▪ Update: all training dataset
▪ More stable
▪ May prematurely converge

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 44
 Artificial Neural Networks
✓ Gradient descent
❖ 2D illustration
❖ Stochastic gradient descent
❖ Batch gradient descent
❖ Mini-batch gradient descent
▪ Update: subset of training set
▪ Mini-batch size: 16 - 256
▪ Best of both worlds

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 45
 Artificial Neural Networks
✓ Error back propagation
❖ Data point/example/sample/instance/input vector/feature
vector/observation
❖ Batch size
❖ Epoch: 10, 100, 500, 1000, and larger

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 46
 Artificial Neural Networks
✓ References
❖ https://en.wikipedia.org/wiki/Neural_circuit
❖ https://en.wikipedia.org/wiki/Artificial_neural_network
❖ https://www.securityinfowatch.com/video-surveillance/video-
analytics/article/21069937/deep-learning-to-the-rescue
❖ and others

Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 47