Neural Network

What is Deep Learning

label
dataset (with label)

Cat

Dog

It is called deep learning if we use neural network as a model in supervise learning

 What is Deep Learning

Neural network

We can have million of neuron to analyse and learn the pattern (features) of a given data
and memorize the pattern.
NN Basic and Concept
Neural Network components

Neural network compose of 4 main components

1) Layers
2) Input and output
3) Loss function
4) optimizer
Layer
Layers

Node aka neuron

(e.g. layer has 4
neurons)

Fully connected layer/ dense

Input & Output

Input layer
(features)
Output layer
(classes/labels)
 Input & Output
n
Class 1
m
Class 2

Xmn

The whole image is input

to first layer at once.

E.g. If you have 100 images, each image

will be inserted into NN one by one
 Input & Output

Sentosa
X3 others

X4

The four features at first row are input to blue layer at once
 Loss Function

4 main components

1) Neuron
2) Weight and Biases
3) Activation function
4) Feedforward
Neuron

Artificial
b1 neuron is
w1
referred
S F perceptron
 Weight & Bias

Input layer Layer 1 x : input

w :weight
b b : bias
x F(s) : output
w s F F(s)

s = w*x + b
 Activation function

Input layer Layer 1 x : input

w :weight
b b : bias
x F(s) : output
w s F F(s)

F(s) = 1
s = w*x + b 1+ e-x
 Feed forward

Input layer Layer 1 x : input

x=1
w :weight
w=0.3
b b : bias
b= - 0.3 x F(s) : output
s= 0 w s F F(s)
F(s) = ?

F(s) = 1
s = w*x + b 1+ e-x
 Loss
Forward direction to
reach F(S)

x=1
w=0.3 x b
w
b= - 0.3
s F
s=
F(s) = F(S)
Target value (label)
target value =
Loss =
Loss = target value – F(S)

Loss is the different between target value and F(s). Also called error
MSE
 Loss
Forward direction to
reach F(S)

x=1
x b
w=0.3
w
w=0.3, b=-0.3
b= - 0.3
s F s = w*x +b
s= F(S) w x b s target value loss
F(s) =
Target value (label) 0.3*1 + (-0.3) = 0 0 0
target value =
Loss =
0.3*2 + (-0.3) = 0.3 -1 1.68
Loss = target value – F(S) 0.3*3 + (-0.3) = 0.6 -2 6.75
0.3*4 + (-0.3) = 0.9 -3 15.21

Lets x = 1, 2, 3, 4
 Optimizer

4 main components

1) Backpropagation
2) Optimizer
3) Learning rate
4) Epoch & Accuracy
 Back propagation
Backward direction

We go
backward in b
effort to x w
minimize s F Loss
the loss F(S)
Target value (label)

Loss = target value – F(S)

 Optimizer (Reducing the loss)
Backward direction

We go backward
Change in value
in effort to b
minimize x w
s= (x*w) + b What are w and b so that loss is zero?
the loss using s F
F(s) w=-1, b=1
Optimizer. It is a
function to Target value (label) s = w*x +b
change w x b s target value loss
w and b so that -1*1 + 1 = 0 0 0
loss is zero
-1*2 + 1 = -1 -1 0
-1*3 + 1 = -2 -2 0
Loss = target value – F(s) = 0 0
-1*4 + 1 = -3 -3
 Learning rate
Backward direction

Optimizer
updates Change in value
weight and x
b
w
bias toward s= (x*w) + b
s F
zero loss. F(s)
Learning rate Target value (label)
is the rate of
optimizer
changes
weights and Loss = target value – F(s) = 0
biases
 Epoch
Backward direction

Forward direction to
One epoch Reach target value
Input layer 1 output
consists of (Dataset)
one forward
direction and b
then one x1, x2, w Optimizer
x3, x4,
backward x5, x6 s F Loss
F(s)
direction and Target value (label)
optimizer is
executed 6 inputs are fetched into neuron in one cycle is one epoch
once for each
samples in You need run several epoch until loss is zero
dataset
Now let adding more layers (multilayer)
 Adding more neuron

w1, w2 .. W4 are weights

(Every input must x1 (Every path has weight)
w1 b1
has path to neuron(node))
w2 1
S1 F F(S1) =
1+ e-s
w3 b1and b2 is bias
x2
b2 (Node has bias)
w4

S2 F 1
F(S2) =
1+ e-s Every node has function F(s)
 Adding more layers
Input layer 1 layer 2 output
2 inputs
b3 Model has 2 hidden layers
Layer 1 has 2 nodes
x1 w1 b1 Layer 2 has 3 nodes
S F w11
w5 Layer output has two labels
w2
S F w6 w12
w7 b4 S F Target value 1
w3
x2 w13
w4 b2 S F
w8 w14
Target value 2
S F w15 S F
w9
b5 w16
w10 Every input has path to every node
Every node has path to every node
S F
Every path has weight and value are different
Every node has different bias (b) except the output nodes
Every node has same activation function (F)
Every node has different output F(s)
Every node has to do summation (s)
 F(s) is Activation function
Layer 1
Final value F(s) come out from nodes
determine by activation function. In
this example we use sigmoid function as
wn+1 activation function.
S F F(s)
1
F(S) =
1+ e-s

F(s)

1
Sigmoid function: F(S) = 1+ e-x
Sigmoid vs. Tanh ReLU Leaky ReLU
 Feed forward
From x forward direction to
reach F(S)

Input layer 1 layer 2 output

(Dataset)

x1
S F

S F
S F F1(S)

x2 S F

S F S F F2(S)

S F
 Loss
Forward direction to
reach F(S)

Input layer 1 layer 2 output

(Dataset)

x1
S F

S F
S F Target value (label)

x2 S F Loss = target value – F

S F Every path has loss

S F
Average of loss

S F
 Back propagation
From F(s) backward direction
To reach x

We go Input layer 1 layer 2 output

backward in b3
effort to x1 b1
w1 S F
minimize w5 w11
w2
the loss by S F w6 w12
w7 b4 S F Target value 1
changing x2
w3
w13
value of w4 b2 S F
w8 w14
weight &, Target value 2
S F w15 S F
biases w9
b5
w10 w16

S F
 Optimizer
From x forward direction to
reach F(S)
We go backward Input layer 1 layer 2 output
in effort to b3
minimize x1 b1
w1 S F
the loss. w5 w11
w2
Optimizer is a S F w6 w12
w7 b4 S F Target value 1
function to x2
w3
w13
change w4 b2 S F
w8 w14
weights and Target value 2
S F w15 S F
biases so that w9
b5
w10 w16
loss is zero
S F
 Optimizer
Backward direction

Change in value

x1
w1 b1 b3 Loss

S F
w2 b2 S F Target value (label)
w3
S F

Optimizer works at every path

Objective : Loss = 0
 Types of optimizer

GradientDescentOptimizer

AdadeltaOptimizer

MomentumOptimizer

AdamOptimizer

FtrlOptimizer

RMSPropOptimizer
 Learning rate
Backward direction

Optimizer Change the

updates value according
to learning rate
weight and
bias toward
x1
zero loss. w1 b1 b3 Loss
Learning rate
S F
is the rate of w2 b2 S F Target value (label)
w3
optimizer
S F
changes
weights and Loss = target value - F
biases
Learning rate : rule that optimizer has to follow in changing w and b
 Epoch
Backward direction

Forward direction to
One epoch Reach target value
Input layer 1 layer 2 output
consists of (Dataset)
one forward
direction and x1
S F Loss
then one
backward S F
Target value (label)
S F
direction and
optimizer is S F
Loss = target value - F
executed
S F S F
once for all
sample in
dataset. S F
 Epoch, batch & iterations
Epoch

Batch

Iteration
 Epoch, batch & iterations

This approach called “Mini batch gradient descent”

 Epoch, batch & iterations

Dataset is 100 samples

Epoch = 40
Num_of_batch (iteration) = 5
Batch_size = 20

for i less than or equal to Epoch

for j less than or equal to Num_of_batch
compute loss and optimized Batch_size
 Epoch, batch & iterations

What is happening during epoch?

Dataset is 1 sample

Epoch = 4
Num_of_batch = 1
Batch_size = 1

for i less than or equal to Epoch

for j less than or equal to Num_of_batch
After 4 epoch the optimizer achieves 0 error
compute loss and optimized Batch_size
 Parameter and hyperparameter

Parameter Any value that change by computer.

They are weight and biases. Automatically update by optimizer

Hyperparameter Any value that change by human.

They are learning rate, epoch, batch, number of layer, number of nodes
dropped out rate.
Tutorial
 How many parameters in this model
b3

w1 b1
X1 S F w11
w5
w2
S F w6 w12
w7 b4 S F
w3
x2 w13
w4 b2 S F
w8 w14
S F w15 S F
w9
b5 w16
w10
S F
 Layers How many layers?
Input
How many nodes?
Output How many inputs?
How many activation functions?
How many classes?
How many weights ?
How many biases
x4 How many optimizer?
How many parameters?
Assessing performance
 Assessing the performance
Train data Test data
(80%) (20%)
Dataset is 100 samples
Validation phase
Epoch = 40
Each single epoch → we run train data.
Num_of_batch = 5
End of each single epoch → we run test data
Batch_size = 20

for i less than or equal to Epoch

for j less than or equal to Num_of_batch Accuracy is the percentage of right prediction
compute loss and optimized Batch_size over number of sample in test data. It uses
during validation(of every epoch) or
testing phase(end of whole epoch).

Loss is the percentage during 1 epoch.

 Assessing the performance
Train data Test data
(80%) (20%)
Dataset is 100 samples
Validation phase
Epoch = 40
Each single epoch → we run train data.
Num_of_batch = 5
End of each single epoch → we run test data
Batch_size = 20

for i less than or equal to Epoch

for j less than or equal to Num_of_batch Overfitting is when loss in validation phase is much
compute loss and optimized Batch_size bigger than in training phase.

Underfitting is simply the loss is much bigger

during training phase.
 Assessing the performance
Overfitting is when training is so good but then when validation/testing phase is bit worsts

Dropped out

Randomly pick any nodes and disable it.

We gives every nodes a probability for being alive.

E.g. say probability is 0.5. So every node will be
50% alive or 50% dead.

Dropped out is always related to overcome overfitting.

Thank you