OFFICIAL (CLOSED) \ NON-SENSITIVE

Computer Vision
Mr Hew Ka Kian
[email protected]
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Artificial Neural Network

(ANN)

2
 OFFICIAL (CLOSED) \ NON-SENSITIVE

What is an Artificial Neural Network (ANN)

• A neuron may have a threshold, also called bias, such that there would not be
output signal if the sum of the input signal is below the threshold.
• Neurons and edges have a weight that adjusts as during learning. The weight
increases or decreases the strength of the signal.
• Typically, neurons are aggregated into layers. Signals travel from the first layer
(the input layer), to the last layer (the output layer). Layers in between are called
the hidden layers.
connection or edge node or neuron
 OFFICIAL (CLOSED) \ NON-SENSITIVE

A Neural Node
x1 w1 z = x1w1+ x2w2+ x3w3+b

w2 activation f(z)
x2
function
(f)
w3
b
x3 1

4
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Activation function
• Sigmoid (𝛔𝛔)
• Limit to 0 to 1
1
𝑓𝑓 𝑧𝑧 =
1 + 𝑒𝑒 −𝑧𝑧
• Tanh – Hyperbolic tangent
• Limit to -1 and 1
sinh(𝑧𝑧) 𝑒𝑒 2𝑥𝑥 − 1
𝑡𝑡𝑡𝑡𝑡𝑡𝑡 𝑧𝑧 = = 2𝑥𝑥
cosh(𝑧𝑧) 𝑒𝑒 + 1

• Relu – Rectified Linear Unit

• linear positive values
• Ignore negative value
0, 𝑧𝑧 < 0
𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 𝑧𝑧 = �
𝑧𝑧, 𝑧𝑧 ≥ 05
 OFFICIAL (CLOSED) \ NON-SENSITIVE

An example of a node
• When Sigmoid is used, the output equation of a neuron is similar to
the Logistic Regression. Hence, a neuron node is also interpreted as a
classifier.
.9 2 z = .9(2)+ .2(3)+ .3(-1)+.5 = 2.6

(sigmoid) f(z) = f(2.6) = 1/(1+exp(-2.6)) = 0.93

.2 3
activation
function
-1 Neuron would output a value of 0.93
.5
.3 1
6
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Forming a hidden layer

• Placing several nodes side by side, a layer is formed

𝑤𝑤11
𝑥𝑥1
𝑓𝑓 𝑎𝑎1 = 𝑓𝑓 𝑥𝑥1 𝑤𝑤11 +𝑥𝑥2 𝑤𝑤12 + 𝑥𝑥3 𝑤𝑤13 + 𝑏𝑏1
𝑤𝑤12
𝑥𝑥2
𝑤𝑤13 𝑓𝑓 𝑎𝑎2 = 𝑓𝑓 𝑥𝑥1 𝑤𝑤21 +𝑥𝑥2 𝑤𝑤22 + 𝑥𝑥3 𝑤𝑤23 + 𝑏𝑏2
𝑥𝑥3
𝑏𝑏1 𝑓𝑓 𝑎𝑎3 = 𝑓𝑓 𝑥𝑥1 𝑤𝑤31 +𝑥𝑥2 𝑤𝑤32 + 𝑥𝑥3 𝑤𝑤33 + 𝑏𝑏3
1

7
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Input and Output Layer

• Input layer allows the features to enters the neural network.
• Features are ‘things’ that will affect the output (prediction). For an image with
256 pixels, there are 256 features as every pixel may affect the prediction so
every pixel is a feature.
• Each input node is connected to a unique feature. So if there are 256 features,
there should be 256 input nodes.

• Output layer produces the predicted labels.

• The number of output nodes must be equal to the number of labels/classes
• The output of the output layer is calculated similar to the hidden layer

8
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Calculating Loss
• Loss Function calculates how poorly our model is performing by
comparing what the model is predicting with the actual value it is
supposed to output.
• Regression Loss – the model is predicting a continuous value.
• Mean Square Error

• Classification Loss (cross entropy loss) – the model is predicting a

discrete value or specific class.

9
Ref: https://deeplearningdemystified.com/article/fdl-3
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Calculating Loss
• Regression Loss – the model is predicting a continuous value.
• Mean Square Error

Height (X) Weight (Y) Predicted Weight (Y’) Error (Y-Y’) Squared Error
43 41 43.6 -2.6 6.76
44 45 44.4 0.6 0.36
47 44 46.8 -2.8 7.84

• MSE loss = (6.76 + 0.36 + 7.84) / 3 = 4.99

Ref: https://deeplearningdemystified.com/article/fdl-3
10
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Calculating Loss
• Classification Loss – the model is predicting a discrete value or specific class.
• Binary Cross Entropy
• Categorial Cross Entropy
1
• Loss = ∑(− log 2 𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜 𝑜𝑜𝑜𝑜 𝑡𝑡𝑡𝑡𝑡 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝)
𝑛𝑛
Sample Truth Output for Apple Output for Orange
1 Apple 0.89 0.11
2 Orange 0.21 0.79
3 Orange 0.01 0.99

1
• Loss = − log 2 0.89 + − log 2 0.79 + − log 2 0.99
3
1
Loss = 0.16812 + 0.34008 + 0.0145
3
0.522698
Loss = = 0.17
3

Ref: https://deeplearningdemystified.com/article/fdl-3
11
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Calculating Loss
• How do calculate log base 2?
1
• Loss = ∑(− log 2 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑜𝑜𝑜𝑜 𝑡𝑡𝑡𝑡𝑡 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝)
𝑛𝑛
1
• Loss = − log 2 0.89 + − log 2 0.79 + − log 2 0.99
3
• You can use a calculator to calculate log 2 0.89 or
• Open Excel and type into a cell =log(0.89,2)

2021 © Copyright, ACE@RP Ref: https://deeplearningdemystified.com/article/fdl-3

12
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Training of weights
• Iterate training
• A learning curve will be produced

Under-fitting Fit Over-fitting

‘Epoch’ refers to a single

pass of whole dataset.
13
Ref: https://www.kaggle.com/ryanholbrook/overfitting-and-underfitting
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Accuracy
• Accuracy
• Training accuracy – how accurate is the model when tested with data that is
used in the training.
• Accuracy may be artificially higher because it is tested on the data it is
optimized for.
• Validation or test accuracy – how accurate is the model when tested with data
that is NOT used in the training.
• The data can be from the testing data that is split from the dataset.
• Typically we split the dataset into a larger portion for training and a
smaller portion for testing/validation like 80:20 or 70:30.

14
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Accuracy
• Accuracy
• Although accuracy is inversely proportional to the loss (that is, less loss means
higher accuracy), accuracy is NOT equals to (1 – loss).
• Calculation of accuracy is straightforward.
Sample Truth Prediction Correct?
1 Car Car 1
2 Plane Car 0
3 Plane Plane 1

𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑜𝑜𝑜𝑜 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 1+0+1 2

• Accuracy = = = = 0.67
𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 𝑜𝑜𝑜𝑜 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 3 3

15
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Precision and Recall

• Positives and Negatives
• true positives (TP): These are cases in which we predicted a positive case
correctly.
• true negatives (TN): We predicted negative, and correctly.
• false positives (FP): We predicted positive for the actually negative cases.
• false negatives (FN): We predicted negative, but they actually are positive.
Predict Positive Predict Negative
Actual Positive 30 1

Actual Negative 3 9

Predict Positive Predict Negative

Actual Positive TP FN

Actual Negative FP TN

16
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Precision and Recall

• Confusion Matrix
• An intuitive way to present the performance of the model.
• Diagonal is TP and TN that are desirable

Predict Positive Predict Negative

Actual Positive TP FN
Actual Negative FP TN

17
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Precision and Recall

• Precision
• How correct is the positives predicted?
𝑇𝑇𝑇𝑇
•
𝑇𝑇𝑇𝑇 + 𝐹𝐹𝐹𝐹 Predict Positive Predict Negative
Actual Positive TP FN
Actual Negative FP TN
• Recall
• How many positive cases are picked up from all the positive cases?
𝑇𝑇𝑇𝑇
•
𝑇𝑇𝑇𝑇 + 𝐹𝐹𝐹𝐹

18
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Student Activity
Add a hidden layer with 10 nodes

model.add(Dense(8, input_shape=(13,),
activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

• Only the 2nd layer needs to state the input_shape (of the input
layer) as the input layer connects to the 2nd layer
• Add the hidden layer using Dense()

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Student Activity
Increase the learning rate

opt = Adam(learning_rate=0.010)
model.compile(optimizer=opt, loss='binary_crossentropy',
metrics=['accuracy'])
• Increase the value of the learning_rate parameter in the Adam
optimizer.

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Student Activity
Is the accuracy better?
• Accuracy before
Train accuracy of keras neural network: 87.5%
Test accuracy of keras neural network: 75.93%
• Accuracy after
Train accuracy of keras neural network: 99.07%
Test accuracy of keras neural network: 74.07%
• Train accuracy improved to an amazing rate of >99% but when tested with
data it had not seen before, test accuracy was worse at 74.07%
• There is no improvement and it is an example of overfitting where train
accuracy is much better than test accuracy. The gap between train and test
accuracy increases!

Source:
 OFFICIAL (CLOSED) \ NON-SENSITIVE

Student Activity
What are the precision and recall?

1
• Precision = =1
1+0

1
• Recall = = 0.5
1+1

Source: