Samsung

Innovation
Campus
Artificial Intelligence Course
 Chapter 8.

Neural Network and Deep Learn-

ing
Artificial Intelligence
Course

Samsung Innovation Campus 2

Chapter Description

Chapter objectives

 Build and train the deep neural networks.

 Optimize the deep learning neural networks preventing the overfitting and vanishing gradient
problems.
 Gain proficiency with deep learning libraries such as TensorFlow and Keras.

Chapter contents

 Unit 1. Basics of Neural Network

 Unit 2. TensorFlow Basics
 Unit 3. Deep Learning Methods with TensorFlow and Keras

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 3

 Unit 1.

Basics of Neural Network

1.1. Understanding Perceptron with Excel
1.2. Understanding Multilayer Perceptron with Excel
1.3. From Multilayer Perceptron to Deep Learning
1.4. Error Backpropagation and Gradient Descent to Reduce Errors
1.5. Activation Function
1.6. Deep Learning
1.7. Problems with Deep Learning and the Solutions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 4

1.1. Understanding Perceptron with Excel UNIT
01

Overview
Biological origin
‣ Human neuron is the smallest information processing unit that composes the neural system. It
consists of a cell body, dendrites, and an axon.
‣ A cell body transmits simple operations, a dendrite transmits received signals, and an axon
transmits the executed results.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 5

1.1. Understanding Perceptron with Excel UNIT
01

Overview
Biological origin
‣ They receive and store various information by exchanging chemical signals with adjacent neurons
through a structure named synapse.
‣ The number of neurons in the brain of a human being is about 1011 neurons. And the average
neuron has around 1,000 synapses. Thus, 100 trillion (1014) synapses are interconnected in the
human brain.
‣ In the 1940s, several researchers who studied biological neural networks began to investigate how
to emulate the mechanisms of the neural network. Perceptron is one of the early artificial neural
network models.
‣ From here on, the term ‘neural network’ refers to an ‘artificial’ neural network.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 6

1.1. Understanding Perceptron with Excel UNIT
01

History of Neural Network

‣ In 1943, McCulloch and Pitts first delineated the working mechanisms of neurons.
‣ Rosenblatt’s perceptron is based on McCulloch Pitts neuron model.
‣ In 1958, Rosenblatt first proposed perceptron.
‣ In 1969, Minsky and Papert mathematically proved the limitations of the perceptron
in their book 『 Perceptrons 』 . It was found that a perceptron could not solve the XOR
problem because it was only a linear classifier.

Frank Rosenblatt
(1928~1971)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 7

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
The Structure of a Perceptron

Input Output

𝜒 0=1
𝜒1 𝑊0
𝑊1
𝜒2 𝑊2 y
𝑊𝑑

𝜒𝑑
Structure of a perceptron

‣ Comprehend the structure of a perceptron by solving the OR function of the truth table.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 8

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ First, you need to understand the OR operation of the truth table. The results of the OR operation
are as below.
X1 X2 OR operation

False False F

True False T

False True T

True True T

‣ What are the weight values to solve the OR operation problem?

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 9

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ Suppose that we know the ‘w’ values to solve a problem. w0 is -0,5, w1 is 1, w2 is 1.
(w: a weight vector)
‣ Execute the Excel function after substituting x1 with 0 and x2 with 0.

x1 X2 w0 w1 w2
0 0 -0.5 1 1
1
x0 1
w0 x0 * w0 -0.5

+
0 w1 x0 * w1 0
x1 0 -0.5
+
x0 * w2 0
w2 Sum -0.5
0
x2 0

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 10

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ Put the value of the sum into a function.
‣ Here, that function is the step function.

1
𝑠
−1

Use a threshold function as an activation function (s).

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 11

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ A threshold function can be defined as follows, and we can create a graph by putting random values
in the Excel table.

-100 -1 Threshold
-90 -1 function
1.5
A threshold can be defined as follows, -80 -1
Here, -70 -1
1.0
-60 -1 0.5
-50 -1 0
-40 -1 -150 -100 -50 0 50 100 150
-0.5
-30 -1
-1
-20 -1
-10 -1 -1.5
0 1
10 1
20 1
30 1
40 1
50 1
60 1
70 1
80 1
90 1
100 1

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 12

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ Put the result from the previous slide, -0.5 as s, and run it through the threshold function. The return
is -1.
‣ Let’s learn about the truth table.

A truth table displays truth or false for all results of the propositions or the combination of their Boolean
functions.
For example, in conjunction with statements P and Q , the truth table can be constructed as below.
In addition, true∙false is also notated as T∙F or 1∙0.
Proposition P Proposition Q
True True True
True False False
False True False
False False False

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 13

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ A truth table can be expressed with 1 and 0 as follows.

x1 x2
0 0
1 0
0 1
1 1

‣ The figure from Slide 9 puts 0 for the first variable x1 and 0 for x2 as well.
‣ x0 is called a bias, and it initializes as 1.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 14

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ Substitute x1 with 1 and x2 with 0.

The bias is generally fixed as 1.

x0(bias) 1 1
w
x0*w0 -0.5
0
+
w
0 1 -0.5 x1*w1 1
x1 1
+

w
x2*w2 0
2
x2 0 0
Sum 0.5

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 15

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ The sum is 0.5. Run this through the threshold function, and the result is 1.
‣ Likewise, if we process all values in the truth table through the perceptron, the results are as
follows.

The bias is generally fixed as 1. • 4 results can be displayed in a

table.
x0(bias) 1 x1 x2
w 0 0 -0.5 -1
x0*w0 -0.5
0
1 0 0.5 1
+
w 0 1 0.5 1
1 -0.5 x1*w1 1
x1 1 1 1 1.5 1
+

w
x2*w2 1
2
x2 1
Sum 1.5

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 16

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ These results can be shown in the coordinate plane as follows.
‣ The results can be grouped into that of point (0,0) or the rest.

x1 x2

(0,1) (1.1) 0 0 -0.5 -1

1 0 0.5 1
0 1 0.5 1
1 1 1.5 1

(0,0) (1.0)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 17

1.1. Understanding Perceptron with Excel UNIT
01

Perceptron
‣ Such classification can be presented as a geometrical figure as follows.
‣ If you recall the basic notion of machine learning through linear regression, machine learning is a
process that builds regression equations with the predicted values of the slope and y-intercept. The
figure below shows how linear regression forms a line that separates the two groups.

x1 x2
(0,1) (1.1) 0 0 -0.5 -1
1 0 0.5 1
0 1 0.5 1
1 1 1.5 1

• This table displays the OR

operation of a truth table.

(0,0) (1.0)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 18

1.1. Understanding Perceptron with Excel UNIT
01

OR operation

Value before running Value after running

x1 x1 x2
threshold function threshold function
=$A$2*$C$7+B2*$C$8+C2*$C
0 0 0 =IF(D2>=0,1,-1)
$9 w1 -0.5
=$A$2*$C$7+B3*$C$8+C3*$C
0 1 =IF(D3>=0,1,-1) w2 1
$9
=$A$2*$C$7+B4*$C$8+C4*$C w3 1
1 0 =IF(D4>=0,1,-1)
$9

( 0) (0 ) ( 1) ()
=$A$2*$C$7+B8*$C$8+C5*$C
1
𝑥1 =$90 , 𝑦 1=− 1 , 𝑥 2= 1 , 𝑦 2=1=IF(D5>=0,1,-1)
1
, 𝑥 3 = 0 , 𝑦 3=1 , 𝑥 4 = 1 , 𝑦 =1
4
1
𝑥2 𝑥 0=1
𝑥3 𝑥4 -0.5
𝑥1 1.0 𝑦
𝑥1 𝑥2
1.0
𝑥1 𝑥2
(a) Train set (b) Perceptron

Example of a perceptron’s operation using the OR logic gate

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 19

1.1. Understanding Perceptron with Excel UNIT
01

OR operation

Let’s input four samples to the perceptron and check the results.

‣ As you’ve seen in the previous slide, the perceptron delivered correct results for all four samples.
‣ It can be said that this perceptron classifies the train set with 100% performance.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 20

1.1. Understanding Perceptron with Excel UNIT
01

AND operation
‣ To recap, you can solve the OR problem by perceptron with the appropriate w values.
‣ Now, the AND operation. The truth table is on the left, and the geometrical solution is the line that
separates the units into two groups.

AND opera-
x1 x2
tion
0 0 F
1 0 F
(0,1) (1.1)
0 1 F
1 1 T

(0,0) (1.0)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 21

1.1. Understanding Perceptron with Excel UNIT
01

AND operation
‣ Find the values of w0, w1, and w2 that return the result from the previous slide.
‣ First, let’s apply a random value, and then gradually change the values of w0, w1, and w2 to find
the solution.

The bias is generally fixed as 1. • 4 results can be displayed in a

table.
x0(bias) 1 x1 x2
w 0 0 0.5 1
0.5 x0*w0 0.5
0
1 0 0 1
+
w 0 1 0.4 1
-0.5 x1*w1 -0.5
x1 1 1 -0.5 1 1 -0.1 -1
+ • AND operation of a truth table
w
-0.1 x2*w2 -0.1
2
x2 1
Sum -0.1

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 22

1.1. Understanding Perceptron with Excel UNIT
01

AND operation

The bias is generally fixed as 1. • 4 results can be displayed in a

table.
x0(bias) 1 x1 x2
w 0 0 0.5 1
0.5 x0*w0 0.5
0
1 0 0 1
+
w 0 1 0.4 1
-0.5 x1*w1 -0.5
x1 1 1 -0.5 1 1 -0.1 -1
+ • AND operation of a truth table
w
-0.1 x2*w2 -0.1
2
x2 1
Sum -0.1

‣ By substituting w0 with 0.5, w1 with -0.5, and w2 with -0.1, the equation solves the AND operation.
‣ You found that a structure with appropriate values for the w vector solves a certain problem.
‣ Although the perceptron solved the OR and AND problems, it could not solve the XOR. Let’s find out
why.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 23

1.1. Understanding Perceptron with Excel UNIT
01

XOR operation
‣ XOR operation in a truth table
‣ In XOR, the result is true if two propositions are the opposite.

XOR Opera-
x1 x2
tion
(0,1) (1.1)
0 0 F
1 0 T
0 1 T
1 1 F

(0,0) (1.0)

‣ Since the perceptron is a linear classifier, you cannot find a line separating the blue and red dots.
‣ A multilayer perceptron solved the problem.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 24

1.1. Understanding Perceptron with Excel UNIT
01

Summary of Perceptron’s Structure

‣ It has an input layer and an output layer.
‣ The input layer does not operate, so the perceptron is considered a single-layer structure.
‣ The th node of the input layer takes from the feature vector
‣ The bias node always takes 1 as input.
‣ The output layer has a single node.
‣ The connection of the i th node of the input layer and the output layer has weight .

Input Output 𝑦 =𝜏 ( 𝑠 )
𝑥 0=1
1
𝑥1 𝑤0
𝑤1
𝑥2 𝑤2 𝑦 𝑆
𝑤𝑑
-1
𝑥𝑑
(b) Use threshold function as an activation function
(a) Structure of a perceptron
(s)
Mechanism of a perceptron

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 25

 Unit 1.

Basics of Neural Network

1.1. Understanding Perceptron with Excel
1.2. Understanding Multilayer Perceptron with Excel
1.3. From Multilayer Perceptron to Deep Learning
1.4. Error Backpropagation and Gradient Descent to Reduce Errors
1.5. Activation Function
1.6. Deep Learning
1.7. Problems with Deep Learning and the Solutions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 26

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Understand Multilayer Perceptron with Excel

Perceptron cannot compute the XOR function.
‣ Linear classification is impossible, and a certain amount of errors occur.
‣ 75% accuracy is the limit in the XOR function.

Linearly separable case and linearly inseparable case

OR XOR

(a) Linearly separable (b) Linearly inseparable

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 27

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Multilayer Perceptron
‣ In XOR, the inputs are not linearly separable.
‣ It could not solve a simple problem like XOR.

• This was claimed by Marvin Minsky, an MIT professor who was the pioneer of the AI field, in his
book <Perceptrons>, published in 1969.

 People realized that emulating the diagram of ‘neuron  neural network  intelligence’ through
the diagram of ‘perceptron  artificial neural network  artificial intelligence’ was not an easy
task.

After this publication, research on AI took a downturn.

After 10 years, this problem was solved by a ‘multilayer’ perceptron.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 28

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Multilayer Perceptron
‣ What are the ways you can separate the inputs that are not linearly separable?
‣ First, the inputs can be separated in a non-linear form.
‣ Second, a two-dimensional figure on a plane could be modified into a three-dimensional figure to
enable separation.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 29

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

‣ Artificial Intelligence scientists had to overcome the XOR problem to develop an artificial neural
network.
‣ An innovative idea provided hints for the solution.

‣ You had to bend the plane!

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 30

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

‣ Modify the coordinate plane itself.
‣ To solve the XOR, you need to run two perceptrons simultaneously.
‣ To make this possible, a hidden layer is necessary.

Perceptron to Multilayer Perceptron

Perceptro Multilayer
n Perceptron

Outpu
t

Input Input Hidden Output

layer layer layer

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 31

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

‣ If you divide the plane into sections of blue and red with regard to the inputs, no line can separate
the two. However, if a hidden layer warps the plane, the curve line separating the two sections
becomes a straight line.

Perceptron to Multilayer Perceptron

Inpu hidden output

t (2 (1
(2) sigmoid) sigmoid)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 32

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

Solving the XOR with 2 perceptrons

‣ If perceptron ① and perceptron ② are both +1, the unit is ● . If not, ◻︎.

𝑥2
Perceptron ② Perceptron ① Perceptron ②
C 𝑥 0=1 𝑥 0=1
Perceptron ① -
1.5
A 0.5
𝑥1 𝑧1 𝑥1 - 𝑧2
1.0
1.0
𝑥1 -
- + 1.0
B 1.0
+ - 𝑥2 𝑥2

(a) Separating the plane (b) Two perceptrons

by two perceptrons

Solution for the XOR problem

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 33

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

‣ Solve the XOR problem by applying a multilayer perceptron in Excel.

Solving XOR Sigmoid function

b1 b2
=1/(1+exp(-x))
Input layer Hidden layer Output layer
W1 W2
x1 n1
yout
x2 n2

x1 x2 W1 W2 n1 After running sigmoid n2 out- Desired

0 0 -2 function - 0.26894 0 put value
0 1 2 1 -1 0.268941 0 1 1 0 0
1 0 -2 1 1 0.731059 1 1 0.73105 1 1 1
1 1 2 1 0.731059 1 9 1 1
3 0.952574 1 1 0.73105 1 0 0
b1 b2 x1 x2 or and nan or nan
9
3 -1 d d
3 0.95257 1
-1 0 0 0 0 1 0 1 0
4
0 1 1 0 1 1 1 1
1 0 1 0 1 1 1 1
1 1 1 1 0 1 0 0

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 34

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

‣ Compute the XOR by applying a multilayer perceptron in Excel.

Solving XOR
b1 b2
Input layer Hidden layer Output layer
x W1 n W2
yout
x n

x1 x2 W1 W2 n1 After running sigmoid function n2

0 0 -2 =C12*$F$12+D12*$F$14 =1/(1+EXP(- 1 =C12*$F$13+D12*$F$15+ =1/(1+EXP(- 0
+$F$18 K12)) $F$19 O12))
0 1 2 1 =C13*$F$12+D13*$F$14 =1/(1+EXP(- 1 =C13*$F$13+D13*$F$15+ =1/(1+EXP(- 1
1 0 -2 1 +$F$18 K13)) $F$19 O13))
1 1 2 =C14*$F$12+D14*$F$14 =1/(1+EXP(- 1 =C14*$F$13+D14*$F$15+ =1/(1+EXP(- 1
+$F$18 K14)) $F$19 O14))
b1 b2 x1 x2 or
=C15*$F$12+D15*$F$14 and nan
=1/(1+EXP(- or
0 nan
=C15*$F$13+D14*$F$15+ =1/(1+EXP(- 1
+$F$18 K15)) $F$19 O15))
3 -1 d d
-1 0 0 0 0 1 0 1 0
0 1 1 0 1 1 1 1
1 0 1 0 1 1 1 1
1 1 1 1 0 1 0 0

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 35

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

‣ Compute the XOR by applying a multilayer perceptron in Excel.

Solving XOR
b1 b2
Input layer Hidden layer Output layer
x W1 n W2
yout
x n

x1 x2 W1 W2 n1 After running sigmoid function n2

0 0 -2 =C12*$F$12+D12*$F$14 =1/(1+EXP(- 1 =C12*$F$13+D12*$F$15+ =1/(1+EXP(- 0
+$F$18 K12)) $F$19 O12))
0 1 2 1 =C13*$F$12+D13*$F$14 =1/(1+EXP(- 1 =C13*$F$13+D13*$F$15+ =1/(1+EXP(- 1
1 0 -2 1 +$F$18 K13)) $F$19 O13))
1 1 2 =C14*$F$12+D14*$F$14 =1/(1+EXP(- 1 =C14*$F$13+D14*$F$15+ =1/(1+EXP(- 1
+$F$18 K14)) $F$19 O14))
b1 b2 x1 x2 or
=C15*$F$12+D15*$F$14 and nan
=1/(1+EXP(- or
0 nan
=C15*$F$13+D14*$F$15+ =1/(1+EXP(- 1
+$F$18 K15)) $F$19 O15))
3 -1 d d
-1 0 0 0 0 1 0 1 0
0 1 1 0 1 1 1 1
1 0 1 0 1 1 1 1
1 1 1 1 0 1 0 0

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 36

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Solution for the XOR problem

‣ The hidden layer warps the coordinate plane.

x1 x2 or and nand or nand xor

0 0 0 0 1 0 1 0
0 1 1 0 1 1 1 1
1 0 1 0 1 1 1 1
1 1 1 1 0 1 0 0

Input layer Hidden layer Output layer

‣ Hidden layer warps the plane. (https://goo.gl/8qEGHD for more reference)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 37

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

『 Perceptrons 』 by Minsky
‣ Minsky pointed out the limitations of the perceptron and suggested a multilayer structure as a
solution. The level of technology at the time was not advanced enough to materialize his idea.
‣ In 1974, Paul Werbos proposed a backpropagation algorithm in his doctoral thesis.
‣ In 1986, Rumelhart established the multilayer theory in his book 『 Parallel Distributed Processing 』
and revived interest in the neural network.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 38

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Central Idea to the Multilayer Perceptron

‣ Insert a hidden layer. It transforms a given feature space into a new one that is more favorable for
classification.
‣ Use a sigmoid activation function. Perceptron used a step function as an activation function.
This function is categorized as a hard decision decoder. However, a multilayer perceptron utilizes a
sigmoid function, which is a soft decision decoder, as an activation function. In soft-decision
decoding, the output is continuous and indicates confidence. This enables more flexible decision-
making.
‣ Use a backpropagation algorithm. A multilayer perceptron consists of many layers in consecutive
order. Thus, a backpropagation algorithm is effective that progresses in reverse order, computes
gradients, and renews weights in each layer.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 39

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Central Idea to the Multilayer Perceptron

‣ Parallel combine two perceptrons
• Transform a space into a new feature space .
• The new feature space is linearly separable.
𝑧2
𝑥2
②
𝑥 0=1 -0.5
1.0 𝑧1
1.0 ① 𝑧
𝑥1
1.5 Ⓑ Ⓐ
-1.0 𝑧2 Ⓒ
𝑥2 - + 𝑥1
-1.0
+ -

(a) Parallel combine two percep- (b) Transform a feature space x into a new feature
trons space z
Transformation of a feature space

• Equivalent to a human-being manually executing feature learning

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 40

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Central Idea to the Multilayer Perceptron

‣ If you serial combine a single perceptron
• In feature space , if you serial combine perceptron 3, which performs linear classification, it
becomes a multilayer perceptron.

𝑧2 Perceptron ③
Perceptron ③
𝑥 0=1 1
-0.5 -1.0
𝑧1 𝑥1 1.0
1.0 1.0
1.0
-1.0 1.5
𝑥2
-1.0

(a) Separation in a new feature (b) Multilayer perceptron that com-

space bines three perceptrons
Multilayer Perceptron

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 41

1.2. Understanding Multilayer Perceptron with Excel UNIT
01

Central Idea to the Multilayer Perceptron

‣ A threshold function is a hard-decision decoder (transforms an area into a dot). Other activation
functions are soft-decision decoders (transform an area into an area).

1 1

0.5 0.5
0 0

-0.5 -0.5
-1 -1
-6 -4 -2 0 2 4 6 -6 -4 -2 0 2 4 6

(a) Threshold func- (b) Logistic sigmoid

tion
Activation function used in neural network

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 42

 Unit 1.

Basics of Neural Network

1.1. Understanding Perceptron with Excel
1.2. Understanding Multilayer Perceptron with Excel
1.3. From Multilayer Perceptron to Deep Learning
1.4. Error Backpropagation and Gradient Descent to Reduce Errors
1.5. Activation Function
1.6. Deep Learning
1.7. Problems with Deep Learning and the Solutions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 43

1.3. From Multilayer Perceptron to Deep Learning UNIT
01

Artificial Neural Network (ANN)

About the artificial neural network (ANN)

Input Hidden Output

Layer Layers Layer

𝑋 𝑌

‣ Mimics the neural connections of a biological brain.

‣ There can be several hidden layers.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 44

1.3. From Multilayer Perceptron to Deep Learning UNIT
01

Why Artificial Neural Network (ANN)?

‣ For a logistic regression (even for the multi-class variant), the decision boundaries are linear.

‣ How about the cases that require non-linear decision boundaries?  ANN with hidden layers!

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 45

1.3. From Multilayer Perceptron to Deep Learning UNIT
01

ANN training: forward propagation

1) When the values of are given, apply the weights and propagate the signal forward to the next
layer.

Example
,

𝑥1𝑤
𝑤11
1
𝑤31 𝑥1
21

𝑥 2𝑤𝑤1222
𝑤32 1
𝑤13 𝑥 2
𝑥 3𝑤
𝑤33
23
1
𝑤𝑤
𝑤
14 24 𝑥 3
𝑥4 34

[ ] ([ ][ ] [ ])
1 𝑥1
𝑥1 𝑤 11 𝑤 12 𝑤 13 𝑤 14 𝑏1
𝑥2
𝑥1
2
= 𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 𝑤 21 𝑤 22 𝑤 23 𝑤 24 ∙ + 𝑏2
𝑥3
𝑥1 𝑤 31 𝑤 32 𝑤 33 𝑤 34 𝑏3
3
𝑥4

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 46

1.3. From Multilayer Perceptron to Deep Learning UNIT
01

ANN training: forward propagation

2) Propagate forward to the next layer.

Example
𝑏1 ,𝑏2 ,𝑏3 ,𝑏4

𝑤1 1𝑥 2
𝑤
1 𝑤 12 1
𝑥1 𝑤 13 2
𝑤2221
1 𝑤23
𝑥2
𝑥 2 𝑤𝑤3 1
2
𝑤33𝑥 3
32

𝑥1 𝑤𝑤 41
3 42
𝑤 43 2
𝑥4

[ ] ([ ] [ ] [ ])
2
𝑥1 𝑤 11 𝑤 12 𝑤 13 1 𝑏1
2 𝑥1
𝑥2 𝑤 21 𝑤 22 𝑤 23 𝑏2
2
= 𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 ∙ 𝑥
1
2
+
𝑥3 𝑤 31 𝑤 32 𝑤 33 1 𝑏3
𝑥
𝑥2
4
𝑤 42 𝑤 42 𝑤 43 3
𝑏4

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 47

 Unit 1.

Basics of Neural Network

1.1. Understanding Perceptron with Excel
1.2. Understanding Multilayer Perceptron with Excel
1.3. From Multilayer Perceptron to Deep Learning
1.4. Error Backpropagation and Gradient Descent to Reduce Errors
1.5. Activation Function
1.6. Deep Learning
1.7. Problems with Deep Learning and the Solutions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 48

1.4. Error Backpropagation and Gradient Descent to Reduce Errors UNIT
01

ANN training: forward propagation

Propagate all the way; the difference between the estimated value 𝑦 ̂ and the true value 𝑦 is
the error.

Example
𝑏
2
𝑥1𝑤
1

𝑤
2
𝑥 22
𝑤3
2
^
𝑦 ?
𝑥 3
𝑤4 Error
2
𝑥

( [ ] )
4 𝑥2
1

𝑥2
^
𝑦 = 𝐴𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 [ 𝑤1 𝑤2 𝑤3 𝑤4 ]∙ 2
+b
𝑥2
3

𝑥2
4

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 49

1.4. Error Backpropagation and Gradient Descent to Reduce Errors UNIT
01

ANN training: backward propagation

Propagate the error backward and update the parameters by gradient descent algorithm.
Repeat from 1).

Example

𝐸𝑟𝑟𝑜𝑟𝑜𝑟𝐿𝑜𝑠𝑠

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 50

 Unit 1.

Basics of Neural Network

1.1. Understanding Perceptron with Excel
1.2. Understanding Multilayer Perceptron with Excel
1.3. From Multilayer Perceptron to Deep Learning
1.4. Error Backpropagation and Gradient Descent to Reduce Errors
1.5. Activation Function
1.6. Deep Learning
1.7. Problems with Deep Learning and the Solutions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 51

1.5. Activation Function UNIT
01

Activation Function
Activation function: Linear threshold or Step
‣ )

‣ An activation function that was frequently used in the early days of neural networks is the step
function.

Step

0 𝑥0 𝑥

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 52

1.5. Activation Function UNIT
01

Activation function: Sigmoid

‣ The Sigmoid is another widely used activation function. Note that the output value ranges
between 0 and 1.
Sigmoid

+1

0 𝑥
‣ When we have more than two classes, it can be generalized to the “Softmax."

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 53

1.5. Activation Function UNIT
01

Activation function: Hyperbolic tangent or tanh

𝑥 − 𝑥
𝑒 − 𝑒
𝑡𝑎𝑛h ( 𝑥 ) = 𝑥 − 𝑥
𝑒 +𝑒
‣ The hyperbolic tangent (tanh) looks similar to the Sigmoid. However, the output value ranges between -
1 and +1.

tanh

+1

-1

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 54

1.5. Activation Function UNIT
01

Activation function: Linear

𝐿𝑖𝑛𝑒𝑎𝑟 ( 𝑥 )= 𝑥
‣ The linear activation’s output is the same as the input. The linear activation function is used in the
regression.

Linear

0 𝑥

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 55

1.5. Activation Function UNIT
01

Activation function: ReLu (Rectifier Linear Unit)

‣ ReLu is another commonly used activation function. It can be expressed in a simple functional form.

ReLu

0 𝑥

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 56

1.5. Activation Function UNIT
01

Activation function: Leaky ReLu

‣ Leaky ReLu is a slightly “tweaked” version of ReLu.

Leawky ReLu

0 𝑥

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 57

1.5. Activation Function UNIT
01

Activation function (summary)

‣ This summarizes some of the most commonly used activation functions.
‣ Softmax is often used at the output layer of neural networks that do classification.

Name Formula TensorFlow

Step

Sigmoid tf.math.sigmoid(x)

Tanh tf.math.tanh(x)

ReLu tf.nn.relu(x)

Softmax tf.nn.softmax(x)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 58

 Unit 1.

Basics of Neural Network

1.1. Understanding Perceptron with Excel
1.2. Understanding Multilayer Perceptron with Excel
1.3. From Multilayer Perceptron to Deep Learning
1.4. Error Backpropagation and Gradient Descent to Reduce Errors
1.5. Activation Function
1.6. Deep Learning
1.7. Problems with Deep Learning and the Solutions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 59

1.6. Deep Learning UNIT
01

Artificial neural networks: a recommended site

‣ Access the website below to check artificial neural networks.

Website:
https://playground.tensorflow.o
rg

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 60

1.6. Deep Learning UNIT
01

Artificial neural networks: a recommended site

‣ The GUI of the website (https://playground.tensorflow.org) is intuitive.
‣ Select data from the left side panel of the website.
‣ Adjust the features, layers, and nodes.
‣ From the top dropdown menus, customize the learning rate, activation function, regularization, etc.
‣ Press the Play button to start training  training loss decreases as the training progresses.
‣ The thickness of an edge line represents the weight that can also be changed manually.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 61

1.6. Deep Learning UNIT
01

About the Deep Neural Network (DNN)

‣ There may be several hidden layers.
‣ With more hidden layers, we have a higher degree of abstraction.

Input Layer Hidden Layers Output Layer

⋯
𝑋 ⋯ 𝑌
⋯

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 62

 Unit 1.

Basics of Neural Network

1.1. Understanding Perceptron with Excel
1.2. Understanding Multilayer Perceptron with Excel
1.3. From Multilayer Perceptron to Deep Learning
1.4. Error Backpropagation and Gradient Descent to Reduce Errors
1.5. Activation Function
1.6. Deep Learning
1.7. Problems with Deep Learning and the Solutions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 63

1.7. Problems of Deep Learning and the Solution UNIT
01

Common problems with the DNN

‣ As the ‘signal’ propagates through the layers, it can die out (vanish) or grow uncontrollably.
a) We should avoid using the Sigmoid 𝜎(𝑥) as the activation function but use the ReLu instead.
→ The derivative 𝜎′(𝑥) becomes very small as the absolute value of 𝑥 or |𝑥| increases.
→ Small 𝜎′(𝑥) may cause the so-called vanishing gradient problem.

+1

0 𝑥

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 64

1.7. Problems of Deep Learning and the Solution UNIT
01

Common problems with the DNN

‣ As the ‘signal’ propagates through the layers, it can die out (vanish) or grow uncontrollably.
b) We should initialize the weights such that the variance of the internal nodes is roughly constant.
→ Initializing as a constant such as 1 is not optimal.
→ Better to randomly initialize the weights with center at 0 and standard deviation ~.(*)
Here, = average number of nodes in the neighboring layers (before and after).

‣ (*) Refer to the research paper at http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 65

1.7. Problems of Deep Learning and the Solution UNIT
01

Common problems with the DNN

‣ Also, DNNs are prone to overfitting. To alleviate this problem, we can do the following:
a) Regularization of the weights: L1 or L2 regularization similar to the Lasso or Ridge
b) Dropout: randomly exclude certain nodes during the training step to avoid over-dependency
c) Data augmentation: add noise, create new data by transformation, etc.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 66

1.7. Problems of Deep Learning and the Solution UNIT
01

Dropout
‣ Randomly exclude certain nodes during the training step to avoid over dependency.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 67

1.7. Problems of Deep Learning and the Solution UNIT
01

Dropout
‣ Randomly exclude certain nodes during the training step to avoid over dependency.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 68

1.7. Problems of Deep Learning and the Solution UNIT
01

Dropout
‣ When predicting, use all the nodes and edges without excluding.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 69

 Unit 2.

TensorFlow Basics
2.1. Introduction to TensorFlow
2.2. TensorFlow dataset
2.3. Machine Learning with TensorFlow

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 70

2.1. Introduction to TensorFlow UNIT
02

Introduction to TensorFlow
About TensorFlow
‣ TensorFlow can utilize both CPU and GPU to improve the training performance of a machine learning
model.

‣ It can produce maximum performance only with GPU.

‣ TensorFlow officially supports CUDA-capable GPU.

‣ Support for OpenCL GPU is still in progress.

‣ It’s likely that OpenCL will be officially supported soon.

‣ Tensoflow is a front-end interface that supports various programming languages.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 71

2.1. Introduction to TensorFlow UNIT
02

About TensorFlow
‣ Since TensorFlow’s Python API is fully mature, it is popular to deep learning and machine learning
technologists.

‣ TensorFlow has its official C ++ API as well.

‣ In addition, two new TensorFlow-based libraries, TensorFlow.js and TensorFlow Lite, were introduced.

‣ APIs for Java, Haskell, Node.js, or Go are not yet stable.

‣ TensorFlow developers and open source community are constantly working to resolve such issues.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 72

2.1. Introduction to TensorFlow UNIT
02

What is TensorFlow?
‣ TensorFlow generates a computational graph composed of a series of nodes.

‣ Each node expresses an operation that can denote more than 0 input or output.

‣ Tensor was created as a symbolic handle to refer to inputs and outputs of such operation.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 73

2.1. Introduction to TensorFlow UNIT
02

Why TensorFlow?
‣ TensorFlow immensely expedites the machine learning process.

• Performance issue

→ Since the performance of computer processors has consistently improved in the past years, it can
train more complex and powerful machine learning systems.

→ It can enhance the machine learning model’s prediction accuracy.

• Even the cheapest computer hardware carries processors with a number of cores.

• Numerous functions from Scikit-learn can disperse an operation into a number of processes.

• Generally, Python can only run one core due to GIL (Global Interpreter Lock).

• Multiproceessing libraries can disperse an operation into multiple cores. Still, even the most
advanced desktop PCs do not carry more than 8 or 16 cores.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 74

2.1. Introduction to TensorFlow UNIT
02

Why TensorFlow?
‣ Even the simplest multilayer perceptron has a hundred units in a single hidden layer.
‣ You need about 80,000 weight parameters ([784×100 + 100] + [100×10 + 10] = 79,510) to train a
model to classify even the simplest images.

• MNIST is a relatively small image (28×28 pixels).

‣ The number of weight parameters will radically increase when processing high-pixel images or
adding hidden layers.
‣ Such tasks are difficult to operate on a single process.
‣ How can you effectively resolve such an issue?

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 75

2.1. Introduction to TensorFlow UNIT
02

Why TensorFlow?

‣ A foolproof solution to this issue is a GPU of exceptional performance level.

‣ A graphic card could be considered a small computer cluster within a computer.

‣ You can buy a GPU that carries 290 times more core and executes 10 times more floating point
operations per second, with 70% price of the latest CPU.

‣ Writing code for a certain CPU is not as easy as running code on a Python interpreter.

‣ There are packages such as CUDA or OpenCL that enable the usage of designated GPUs.

‣ However, CUDA or OpenCL does not offer the best environment to write code for machine learning
algorithms.

‣ TensorFlow was developed from such a background.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 76

2.1. Introduction to TensorFlow UNIT
02

What is TensorFlow?
‣ Refer to the figure below to precisely understand the notion of a tensor.
‣ The first row shows rank 0 and 1 tensors. The second row shows rank 2 and 3 tensors.

Ex Tensors

Rank Rank
0: 1:
(scalar (vecto
) r)

Rank 2: (matrix) Rank

3:

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 77

2.1. Introduction to TensorFlow UNIT
02

What is TensorFlow?

‣ TensorFlow creates a computational graph composed of a series of nodes.

‣ Each node expresses an operation that can denote more than 0 input or output.

‣ Tensor was created as a symbolic handle to refer to inputs and outputs of such operation.

‣ A tensor can be understood in mathematical forms of scalar, vector, matrix, etc.

‣ To be more precise, a scalar can be defined as a rank 0 tensor, a vector as a rank 1 tensor, a matrix
as a rank 2 tensor, and a matrix with three dimensions as a rank 3 tensor.

‣ The actual values are stored as NumPy arrays, and the tensor provides a reference for this array.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 78

2.1. Introduction to TensorFlow UNIT
02

What is TensorFlow?

‣ Let’s discuss the programming mechanisms of TensorFlow.

‣ Learn how to create and modify tensors.

‣ Learn how to load data and use TensorFlow Dataset objects which efficiently circulate datasets.

‣ Study the datasets in the TensorFlow datasets module.

‣ Let’s get acquainted with tf.keras API and build a machine learning model.

‣ Learn how to compile and train this model and store the trained model in a drive.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 79

2.1. Introduction to TensorFlow UNIT
02

What is TensorFlow?

‣ In TensorFlow version 1.x, it supported a static computational graph that expressed the flow of data.

‣ Many users had a difficult time utilizing the static computational graph. This issue has been resolved
in the recent 2.0 version, and the user can create and train neural network models more easily.

‣ TensorFlow 2.0 still supports static computational graphs, but dynamic computational graphs are the
default.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 80

2.1. Introduction to TensorFlow UNIT
02

Installing TensorFlow
‣ Use Python’s pip installer to install TensorFlow from PyPI.
‣ Run the following command on the terminal.

> pip install tensorflow

‣ This command installs the most recent, stable version.
‣ Version 2.5.0 at the time of writing.
‣ Recommend using TensorFlow 2.5.0 for codes (in this chapter) to show expected results.
‣ The command below installs a designated version.

> pip install tensorflow==[desired-

version]

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 81

2.1. Introduction to TensorFlow UNIT
02

Checking TensorFlow version

‣ Run the code below to check the version.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 82

2.1. Introduction to TensorFlow UNIT
02

Installing TensorFlow
‣ For GPU, you need an NVIDIA-compatible graphic card and an installation of CUDA Toolkit and
cuDNN library
(GPU is recommended for neural network training).
‣ If the above conditions are fulfilled, you can install the TensorFlow GPU version.
> pip install tensorflow-
gpu
‣ Refer to the official website for more information on installation (
https://www.tensorflow.org/install/gpu).

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 83

2.1. Introduction to TensorFlow UNIT
02

TensorFlow learning procedure

‣ New version is released every few months due to TensorFlow’s fast developing rate.
‣ Let’s study several ways to create a tensor.
‣ Then, study the features of TensorFlow and methods for modification.
‣ First, use tf.convert to tensor to create a tensor in a list or NumPy array.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 84

2.1. Introduction to TensorFlow UNIT
02

TensorFlow learning procedure

‣ New version is released every few months due to TensorFlow’s fast developing rate.
‣ Let’s study several ways to create a tensor.
‣ Then, study the features of TensorFlow and methods for modification.
‣ First, use tf.convert to tensor to create a tensor in a list or NumPy array.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 85

2.1. Introduction to TensorFlow UNIT
02

Creating a tensor in TensorFlow

‣ The tf.convert to tensor function supports tf.Variable objects unlike the tf.constant function (which
will be discussed soon).
‣ The tf.fill function and the tf.one hot function also create a tensor.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 86

2.1. Introduction to TensorFlow UNIT
02

Creating a tensor in TensorFlow

‣ The tf.fill function generates a tensor composed of scalar inputs.
‣ The first parameter delivers the tensor’s shape like the tf.ones function.
‣ The second parameter delivers the desired scalar input. The code below returns the following result
identical to that of tf.ones ((2,3)).

‣ The tf.fill function is more efficient than the tf.ones to generate tensors of larger size.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 87

2.1. Introduction to TensorFlow UNIT
02

Creating a tensor in TensorFlow

‣ tf.one_hot is a useful function to generate a one-hot encoding matrix.
‣ First parameter: an index that displays the location for one-hot encoding
‣ Second parameter: the length of the one-hot encoding vector
‣ The size of the generated matrix is (length of the first parameter x second parameter).

Ex The code below generates a one-hot encoding matrix of (3 x 4) size.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 88

2.1. Introduction to TensorFlow UNIT
02

Modifying shape and data type of a tensor

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 89

2.1. Introduction to TensorFlow UNIT
02

Modifying shape and data type of a tensor

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 90

2.1. Introduction to TensorFlow UNIT
02

Applying the math operation on a tensor

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 91

2.1. Introduction to TensorFlow UNIT
02

Applying the math operation on a tensor

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 92

2.1. Introduction to TensorFlow UNIT
02

Applying the math operation on a tensor

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 93

2.1. Introduction to TensorFlow UNIT
02

Applying the math operation on a tensor

‣ NumPy function loads __array__() method of the pertaining object before processing the input
parameter. This part of the process makes the object compatible with NumPy.
‣ For example, you can use the Series object from pandas in NumPy API. Since this method is also
embodied in the tensor, you can put a tensor as an input in a NumPy function.
‣ Many mathematical functions are referrable at the most advanced level. For example, they can be
referred to as tf.multiply(), tf.reduce_mean(), tf.reduce_sum(), and tf.matmul(). For Python 3.5
version or above, it can perform matrix operations using @ operator. For example, The computation
below returns results identical to that of the previous slide.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 94

2.1. Introduction to TensorFlow UNIT
02

split(), stack(), concatenate() functions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 95

2.1. Introduction to TensorFlow UNIT
02

split(), stack(), concatenate() functions

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 96

2.1. Introduction to TensorFlow UNIT
02

Coding Exercise #0601

Follow practice steps on ‘ex_0601.ipynb' file

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 97

 Unit 2.

TensorFlow Basics
2.1. Introduction to TensorFlow
2.2. TensorFlow dataset
2.3. Machine Learning with TensorFlow

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 98

2.2. TensorFlow dataset UNIT
02

Building TensorFlow dataset in a tensor

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 99

2.2. TensorFlow dataset UNIT
02

Connecting two tensors in one dataset (1/2)

Approach 1

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 100
2.2. TensorFlow dataset UNIT
02

Connecting two tensors in one dataset (2/2)

Approach 2

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 101
2.2. TensorFlow dataset UNIT
02

shuffle(), batch(), repeat() methods (1/6)

batch

batch

batch

batch

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 102
2.2. TensorFlow dataset UNIT
02

shuffle(), batch(), repeat() methods (2/6)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 103
2.2. TensorFlow dataset UNIT
02

shuffle(), batch(), repeat() methods (3/6)

## Step
1

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 104
2.2. TensorFlow dataset UNIT
02

shuffle(), batch(), repeat() methods (4/6)

## Step
1

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 105
2.2. TensorFlow dataset UNIT
02

shuffle(), batch(), repeat() methods (4/6)

## Step
2

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 106
2.2. TensorFlow dataset UNIT
02

shuffle(), batch(), repeat() methods (5/6)

## Step
2

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 107
2.2. TensorFlow dataset UNIT
02

shuffle(), batch(), repeat() methods (6/6)

## Step
3

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 108
2.2. TensorFlow dataset UNIT
02

Building dataset from a file stored in a local drive (1/6)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 109
2.2. TensorFlow dataset UNIT
02

Building dataset from a file stored in a local drive (2/6)

'Image
Size:

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 110
2.2. TensorFlow dataset UNIT
02

Building dataset from a file stored in a local drive (3/6)

Image
size:
Image
size:
Image
size:
Image
size:
Image
size:
Image
size:

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 111
2.2. TensorFlow dataset UNIT
02

Building dataset from a file stored in a local drive (4/6)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 112
2.2. TensorFlow dataset UNIT
02

Building dataset from a file stored in a local drive (5/6)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 113
2.2. TensorFlow dataset UNIT
02

Building dataset from a file stored in a local drive (6/6)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 114
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (1/10)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 115
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (2/10)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 116
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (3/10)

The following command generates the entire list.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 117
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (4/10)

Downloading CelebA dataset

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 118
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (5/10)

Downloading and saving data in a local drive

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 119
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (6/10)

Loading data from the drive by tf.data.Datasets

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 120
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (7/10)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 121
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (8/10)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 122
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (9/10)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 123
2.2. TensorFlow dataset UNIT
02

Loading datasets from tensorflow.datasets library (10/10)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 124
2.2. TensorFlow dataset UNIT
02

Another approach to loading datasets (1/3)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 125
2.2. TensorFlow dataset UNIT
02

Another approach to loading datasets (2/3)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 126
2.2. TensorFlow dataset UNIT
02

Another approach to loading datasets (3/3)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 127
 Unit 2.

TensorFlow Basics
2.1. Introduction to TensorFlow
2.2. TensorFlow Dataset
2.3. Machine Learning with TensorFlow

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 128
2.3. Machine Learning with TensorFlow UNIT
02

Keras
A high-level API that allows users to build, train, evaluate, and execute all kinds of neural
networks

Code Code

Keras API TensorFlow

Multi- Keras API specific
backend implementation
tf.keras
Keras s

코드 코드 코드

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 129
2.3. Machine Learning with TensorFlow UNIT
02

Building a neural network with TensorFlow

Linear regression with TensorFlow

‣ There is one or more explanatory variables: , ,…,

‣ There is one response variable: 𝑌
‣ The variables and 𝑌 are connected by a linear relation:

𝑌 =𝑏+𝑤1 𝑋 1 +𝑤 2 𝑋 2 +⋯+𝑤 𝑘 𝑋 𝑘 + 
‣ The coefficients previously denoted as 𝛽_ 𝑖 will be called “weights” 𝑤_ 𝑖 from now on.
‣ The intercept previously denoted as 𝛽_0 will be called “bias” 𝑏 from now on.
‣ We will use the “linear” activation function.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 130
2.3. Machine Learning with TensorFlow UNIT
02

Linear regression with TensorFlow

1
𝑏
Linear

𝑋𝑤
1
1
Activation

𝑋𝑤 2 𝜺
Input 2  Output 𝑌
⋮
⋮ 𝑤
𝑘
Summe
𝑋𝑘 d over
Parameters

‣ The disagreement between the predicted and the true 𝑦 has to be minimized by training.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 131
2.3. Machine Learning with TensorFlow UNIT
02

Linear regression with TensorFlow

‣ As we have an over-determined system of linear equations, the exact solution does not exist.
‣ We can minimize the loss (cost) defined as or and get the “best” solution (𝑏,𝑾).
‣ If we define the loss as , the solution (𝑏,𝑾) can be calculated using matrices.
‣ However, we will take a different approach with TensorFlow, which is the gradient descent
algorithm.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 132
2.3. Machine Learning with TensorFlow UNIT
02

Logistic regression with TensorFlow

‣ The linear combinations of variables 𝑋_𝑖 is the so-called “Logit” denoted here as :

𝑆=𝑏+𝑤1 𝑋 1 +𝑤 2 𝑋 2 +⋯+𝑤 𝑘 𝑋 𝑘
‣ The conditional probability of 𝑌 being equal to 1 is denoted as 𝑃(𝑌=1|{𝑥𝑖}).
‣ A “Sigmoid” function connects the probability with the logit:

𝑆
𝑒
𝜎 ( 𝑆 )= 𝑆
0.

1+𝑒
5

‣ This Sigmoid is the “activation function”  the biggest difference from the linear regression.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 133
2.3. Machine Learning with TensorFlow UNIT
02

Logistic regression with TensorFlow

1
𝑏
Sigmoid 0, 0, 1, 1,
𝑋𝑤
1
1
Activation 0, …

𝑋𝑤 2 𝜺
Input 2  Output 𝑌
⋮
⋮ 𝑤
𝑘 Sum = 𝑆 𝑃 ( 𝑌 =1|𝑑𝑎𝑡𝑎 ) → ^𝑦
𝑋𝑘
Parameters

‣ The mismatch between the predicted 𝑦 ̂ and the true has to be minimized by training.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 134
2.3. Machine Learning with TensorFlow UNIT
02

Logistic regression with TensorFlow

‣ We can recall the negative of logarithmic likelihood with the redefined or . (*)
𝑛
− ∑ 𝐿𝑜𝑔 ( 1+ 𝑒− 𝑦 𝑖 𝑆
)
𝑖 =1
‣ We now rename this as the “entropy” and express it in terms of the predicted probability .

𝑛
1
𝐿=− ∑ ( 𝑦 𝑖 𝑙𝑜𝑔 ( 𝑝 ^ 𝑖 ))
^ 𝑖 ) + ( 1− 𝑦 𝑖 ) 𝑙𝑜𝑔(1 − 𝑝
𝑛 𝑖=1 0 or .
‣ We went back to the definition
‣ This is the loss function we’d like to minimize by the gradient descent algorithm.

‣ Binary logistic regression can be generalized to the multi-class version using the Softmax activation
and multi-class cross entropy as the loss function.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 135
2.3. Machine Learning with TensorFlow UNIT
02

Loss (cost) functions in TensorFlow

Name Formula TensorFlow Expression

Squared Error
tf.reduce_sum(tf.square(y_true – y_model))
(L2 Error)
Absolute Er-
ror tf.reduce_sum(tf.abs(y_true – y_model))
(L1 Error)
tf.reduce_mean(tf.nn.sigmoid_cross_entrop
Binary
y_with_logits(labels=y_true,
Cross Entropy
logits=y_model))
tf.reduce_mean(tf.nn.softmax_cross_entrop
Multi-Class
y_with_logits_v2(labels=y_true,
Cross Entropy
logits=y_model))

‣ Cross entropy is the appropriate loss function for the classification.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 136
2.3. Machine Learning with TensorFlow UNIT
02

Gradient descent algorithm

‣ 𝐿(𝑏,𝑾) is minimized iteratively in small “steps” pushing (𝑏,𝑾) along the direction −𝜵𝐿(𝑏,𝑾):

1) is randomly initialized.
2) Calculate the gradient .
3) Update by one step: .
Convergence speed is controlled by the “learning rate” .
4) Repeat from step 2) a fixed number of times (epochs).

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 137
2.3. Machine Learning with TensorFlow UNIT
02

Gradient descent algorithm

‣ 𝐿(𝑏,𝑾) is minimized iteratively in small “steps” pushing (𝑏,𝑾) along the direction −𝜵𝐿(𝑏,𝑾):

𝐿(𝑏,𝑾 )

Optimized

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 138
2.3. Machine Learning with TensorFlow UNIT
02

Batch learning (or mini-batch gradient descent)

‣ Instead of using the whole training dataset for calculating the gradient, only a subset (batch) is
used.
‣ Batch is randomly sampled at every gradient descent step.
‣ Batch size is small and independent of the whole training data size.
‣ It is advantageous for generalization (testing).
‣ It requires less computing power.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 139
2.3. Machine Learning with TensorFlow UNIT
02

Gradient descent algorithms

Algorithm Explanation TensorFlow

Gradient Gradient descent algorithm with batch
tf.train.GradientDescentOptimizer()
Descent learning
Takes into account the gradient from tf.train.MomentumOptimizer(nesterov=Fa
Momentum
the previous step lse)
Nesterov tf.train.MomentumOptimizer(nesterov=Tr
Minimizes unnecessary movements
Momentum ue)

Adagrad Adaptative step size tf.train.AdagradOptimizer()

RMSProp Improves upon the Adagrad tf.train.RMSPropOptimizer()

Combines the best properties of

Adam tf.train.AdamOptimizer()
Adagrad and RMSProp

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 140
2.3. Machine Learning with TensorFlow UNIT
02

Training work flow

Define the variables.

Define the placeholders.

Define the machine learning model.

Define the loss function and the optimization method.

global_variables_initializer()

Session instance.
Send data to the placeholders and train the model.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 141
2.3. Machine Learning with TensorFlow UNIT
02

MNIST data of handwritten digits

‣ Handwritten images of digits (0~9). Image size is 28×28 (748 pixels).
‣ There are 55,000 training and 10,000 testing cases.
‣ The digit label is one-hot-encoded.

X0 X1 X2 X3 X4 X5 X6 X7 X8 X9

0 0 0 0 0 1 0 0 0 0

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 142
2.3. Machine Learning with TensorFlow UNIT
02

Coding Exercise #0602

Follow practice steps on ‘ex_0602.ipynb' file

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 143
2.3. Machine Learning with TensorFlow UNIT
02

Coding Exercise #0603

Follow practice steps on ‘ex_0603.ipynb' file

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 144
2.3. Machine Learning with TensorFlow UNIT
02

Coding Exercise #0604

Follow practice steps on ‘ex_0604.ipynb' file

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 145
 Unit 3.

Deep Learning Methods with Tensor-

Flow and Keras
3.1. Main Features of TensorFlow
3.2. Keras Basics
3.3. AI with Keras

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 146
3.1. Main Features of TensorFlow UNIT
03

Main features of TensorFlow (1/3)

TensorFlow is a programming interface used to build and execute machine learning algorithms. It supports
many platforms, and it is flexible for expansion.
TensorFlow API has become more stable and mature after its initial release of 1.0 in 2017.
It was renewed after the release of 2.0 in 2019.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 147
3.1. Main Features of TensorFlow UNIT
03

Main features of TensorFlow (2/3)

Based on user feedback, the TensorFlow team decided to support dynamic computational graphs for
TensorFlow 2.0.
Thus, building and training neural networks became much easier.
Since dynamic computational graphs iterate graph execution and declaration, TensorFlow 2.0 will be much
more natural and comfortable for Python and NumPy users.
TensorFlow 2.0 offers TensorFlow 1.x API in the tf.combat module.
Using the above model, users will be able to gradually migrate their codes to TensorFlow v.2 API.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 148
3.1. Main Features of TensorFlow UNIT
03

Main features of TensorFlow (3/3)

Another central feature of TensorFlow is the capability to use one or multiple GPUs.
You can efficiently train deep learning models in large-scale datasets and systems.
TensorFlow is an open-source library for the public, but it is sponsored and supported by Google.
A large team of software engineers is consistently expanding and improving the library.
Since it is an open-source library, many developers support TensorFlow.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 149
3.1. Main Features of TensorFlow UNIT
03

Summary of the entire Python API in TensorFlow

tf.keras High-level tf.distribute

tf.estimator Deep Learning tf.saved_model
API tf.autograph
tf.graph_util Distribution and
tf.nn tf.lite
tf.losses optimization
Low-level tf.quantization
tf.metrics tf.tpu
Deep
tf.optimizers tf.xla
Learning API
tf.train
tf.initializers tf.lookup
tf.nest
tf.GradientTape Automatic tf.ragged Specific data
tf.gradients () differentiation tf.sets structures
tf.sparse
tf.data th.strings
tf.feature_colum
tf.math
n Math operations including
Input, output, tf.linalg
tf.audio Linear algebra and signal
preprocessing tf.signal
tf.image processing
tf.random
tf.io
tf.bitwise
tf.queue
tf.compat
TensorBoard tf.config Others
tf.summary
visualization &more

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 150
 Unit 3.

Deep Learning Methods with Tensor-

Flow and Keras
3.1. Main Features of TensorFlow
3.2. Keras Basics
3.3. AI with Keras

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 151
3.2. Keras Basics UNIT
03

Keras basics
The most popular backend up to this day was TensorFlow which was the default backend for Keras.
Since more and more TensorFlow users worked with Keras for high-level APIs, TensorFlow developers had
to consider containing the Keras project in a separate module (tf.keras).
In TensorFlow 2.0, Keras and tf.keras are synchronized.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 152
3.2. Keras Basics UNIT
03

About the Keras library

‣ A high-end deep learning library
‣ Does not support the low-end tensor operations.
‣ Runs on top of TensorFlow.

High level APIs Keras

Framework TensorFlow, Theano, …

Linear Algebra, Math CUDA Blas, Eigen

Hardware GPU CPU

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 153
3.2. Keras Basics UNIT
03

About the Keras library

‣ It is easy to model deep neural networks with Keras.

Input Layer Hidden Layers Output

Layer

Node (activation)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 154
3.2. Keras Basics UNIT
03

Training work flow

‣ It is easy to model deep neural networks with Keras.

Define a model’s layer structure (specify the activation functions).

Define the loss function and the optimization method.

Compile the model.

Specify the batch learning method and start the training (gradient descent).

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 155
3.2. Keras Basics UNIT
03

Loss (cost) functions in Keras

‣ It is easy to model deep neural networks with Keras.

Name Formula Keras Expression

Squared Error
loss = "mse"
(L2 Error)

Absolute Error
loss = "mae"
(L1 Error)

Binary
loss = "binary_crossentropy"
Cross Entropy

Multi-Class
loss = "categorical_crossentropy"
Cross Entropy

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 156
3.2. Keras Basics UNIT
03

Gradient descent algorithms

Algorithm Explanation Keras

Stochastic
Gradient De- Gradient descent algorithm with batch learning SGD(lr=0.1)
scent

Takes into account the gradient from the previous

Momentum SGD(momentum=0.9)
step
Nesterov SGD(lr=0.01, momentum=0.9,
Minimizes unnecessary movements
Momentum nesterov=True)
Adagrad Adaptative step size Adagrad(lr=0.01)
RMSProp Improves upon the Adagrad RMSProp(lr=0.01)
Combines the best properties of Adagrad and
Adam Adam(lr=0.01)
RMSProp

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 157
3.2. Keras Basics UNIT
03

Activation functions

Name Formula Keras

Linear activation="linear"

Sigmoid activation="sigmoid"

Tanh activation="tanh"

ReLu activation="relu"

Softmax activation="softmax"

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 158
3.2. Keras Basics UNIT
03

Layers

Keras Explanation
Dense() Dense layer
Conv1D() 1D convolution layer
Conv2D() 2D convolution layer
MaxPooling1D() 1D max pooling layer
MaxPooling2D() 2D max pooling layer
Dropout() Applies dropout to the input
Flatten() Flattens the input
Embedding() Turns positive integers (label encoding) into dense vectors
SimpleRNN() RNN where the output is fed back to input
LSTM() LSTM layer
TimeDistributed() A wrapper that applies a layer to every time slice
Input() The first layer for a functional API model

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 159
3.2. Keras Basics UNIT
03

Two ways of building models with Keras

1) Sequential API:
‣ Layers are added to a “Sequential” object.
‣ Easy but less flexible.
2) Functional API:
‣ The first layer must be defined as “Input,” where the data shape is specified.
‣ Each layer is defined as a function that takes in as arguments the previous layer.
‣ A “Model” object takes in as arguments the first and the last layers.
‣ More flexible.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 160
3.2. Keras Basics UNIT
03

Datasets provided by Keras

‣ Useful for learning/practicing purpose
‣ Provided by the “keras.datasets” module

Name Description
cifar10 color images labeled over 10 categories. 50000 training and 10000 testing images.
cifar100 color images labeled over 100 categories. 50000 training and 10000 testing im-
ages.
imdb 25000 movie reviews from IMDB with binary labeling (positive or negative).
reuters 11228 newswires from Reuters, labeled over 46 topics.
mnist grayscale images of handwritten digits. 60000 training and 10000 testing images.
fashion_mnist grayscale images of 10 fashion categories. 60000 training and 10000 testing im-
ages.
boston_housing 13 attributes of houses at different locations around Boston suburbs in the late
1970s.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 161
 Unit 3.

Deep Learning Methods with Tensor-

Flow and Keras
3.1. Main Features of TensorFlow
3.2. Keras Basics
3.3. AI with Keras

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 162
3.3. AI with Keras UNIT
03

AI with Keras
Linear/Logistic regression with Keras

1
𝑏 Linear
or

𝑋𝑤
1
1 Sigmoid

𝑤
𝑋2 2
Input  Output
⋮
⋮𝑤
𝑘
Summed
𝑋𝑘 over

‣ The major difference between linear and logistic regression lies in the choice of activation function.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 163
3.3. AI with Keras UNIT
03

Keras Sequential API model

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 164
3.3. AI with Keras UNIT
03

Reading the data and exploring

‣ read

‣ view

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 165
3.3. AI with Keras UNIT
03

Reading the data and exploring

‣ Scale the X data.

‣ Spit the data into training and testing.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 166
3.3. AI with Keras UNIT
03

Defining a Sequential API model

‣ Add layers to a sequential object.

Line 91
• Add an output layer for linear regression.
Line 92
• Summary of the model

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 167
3.3. AI with Keras UNIT
03

Defining the hyperparameters and optimizer

Line 93
• Hyperparameters
Line 94
• Define the optimizer and then compile.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 168
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

‣ Train the model.

Line 95
• verbose = 0 means no output. verbose = 1 to view the epochs.
Line 96
• View the keys.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 169
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

‣ Visualize the training history.

Line 97-1
• Skip the first few steps.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 170
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

‣ Visualize the training history.

Training History
Train
100 Validation

80
MSE

60

40

20
0 250 500 750 1000125015001750
Epoch

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 171
3.3. AI with Keras UNIT
03

Testing
‣ Predict and test using a formula.

‣ Use the evaluate() method.

Line
100
• Returns the 0 = loss value and 1 = metrics value.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 172
3.3. AI with Keras UNIT
03

Keras Functional API model

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 173
3.3. AI with Keras UNIT
03

Defining a Functional API model

Line
102-1
• Input layer
Line
102-2
• Output layer

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 174
3.3. AI with Keras UNIT
03

Defining a Functional API model

‣ The model and its summary

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 175
3.3. AI with Keras UNIT
03

Defining a Functional API model

‣ Define the optimizer and then compile.

Line
105
• Loss = MAE (L1) and Metrics = MSE (L2)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 176
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

‣ Train the model.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 177
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

‣ Visualize the training history.

Training History
Train
100 Validation

80
MSE

60

40

20
0 250 500 750 1000 1250 1500 1750
Epoch

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 178
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

‣ Use the evaluate() method.

Line
109
• Returns the 0 = loss value and 1 = metrics value.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 179
3.3. AI with Keras UNIT
03

Softmax regression with Keras

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 180
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ Read the data and prepare:

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 181
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ One-hot-encoding for y

‣ View as DataFrame

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 182
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ Scale the X data.

‣ Split the data into training and testing.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 183
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ Define a Sequential API model.

Line 123
• Add layers to a Sequential object.
• units = N# of output variables
Line 124
• Summary of the model

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 184
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ Define the hyperparameters and optimizer.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 185
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ Train the model and visualize the history.

Line 128
• Verbose = 0 means no output. Verbose = 1 to view the epochs.
Line 129
• View the keys.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 186
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ Visualize the training history.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 187
3.3. AI with Keras UNIT
03

Softmax regression with Keras

‣ Testing

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 188
3.3. AI with Keras UNIT
03

Diagram of a Convolutional Neural Network (CNN)

‣ The convolution and pooling layers are repeated several times.
‣ There is a fully connected layer just before the output.

Input Feature Classification Output

Extraction 0
0
Convolution

Pooling layer

Dropout layer

Fully connected
0.02
layer

0.95

layer
0.03
0
0
0
0
0

Repeated several
times

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 189
3.3. AI with Keras UNIT
03

Reading the data

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 190
3.3. AI with Keras UNIT
03

Reading the data

‣ Bring in the data.

‣ View the shapes.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 191
3.3. AI with Keras UNIT
03

Visualizing the data

Line 135-1
• You may change this at will.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 192
3.3. AI with Keras UNIT
03

Preparing the data

‣ Scaling

‣ Reshaping

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 193
3.3. AI with Keras UNIT
03

Preparing the data

‣ One-hot-encoding

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 194
3.3. AI with Keras UNIT
03

Defining a CNN model

Line 140-1
• 1 channel of grayscale

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 195
3.3. AI with Keras UNIT
03

Defining a CNN model

‣ View the summary.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 196
3.3. AI with Keras UNIT
03

Defining the hyperparameters and optimizer

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 197
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

Line
144
• verbose = 0 means no output. verbose = 1 to view the epochs.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 198
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

Training History

0.99

0.98
Accuracy

0.97

0.96

0.95
Train
0.94 Validation

0 2 4 6 8
Epoch

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 199
3.3. AI with Keras UNIT
03

Testing

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 200
3.3. AI with Keras UNIT
03

Convolutional Neural Network with Keras (color images)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 201
3.3. AI with Keras UNIT
03

Reading the data

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 202
3.3. AI with Keras UNIT
03

Visualizing the data

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 203
3.3. AI with Keras UNIT
03

Preparing the data

‣ Scaling

‣ Reshaping

‣ One-hot-encoding

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 204
3.3. AI with Keras UNIT
03

Defining a CNN model

Line 155
• 3 channels of color

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 205
3.3. AI with Keras UNIT
03

Defining a CNN model

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 206
3.3. AI with Keras UNIT
03

Defining the hyperparameters and optimizer

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 207
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

Line
159
• verbose = 0 means no output. verbose = 1 to view the epochs.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 208
3.3. AI with Keras UNIT
03

Training the model and visualizing the history

Training History

0.8

0.7
Accuracy

0.6

0.5
Train
0.4 Validation

0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5

Epoch

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 209
3.3. AI with Keras UNIT
03

Testing

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 210
3.3. AI with Keras UNIT
03

Recurrent Neural Network (RNN)

𝑌 𝑡− 1 𝑌 𝑡 𝑌 𝑡 +1 Output

h 𝑡 −2 h 𝑡 −1 h𝑡 h 𝑡+ 1

Hidden

𝑋 𝑡 𝑋 𝑡 +1 Input
time

‣ This is a “Sequence in and Sequence out” model.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 211
3.3. AI with Keras UNIT
03

Long Short Term Memory (LSTM) network

‣ LSTM cells can form a sequence just like the regular RNN cells.
‣ The following is a “Sequence in and Sequence out” model.

𝑌 𝑡− 1 𝑌𝑡 𝑌 𝑡 +1

𝑋 𝑡− 1 𝑋𝑡 𝑋 𝑡 +1
time

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 212
3.3. AI with Keras UNIT
03

Reading the data and exploring

‣ Display the dataframe.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 213
3.3. AI with Keras UNIT
03

Reading the data and exploring

‣ Visualize the time series.

10

2
0 5 10 15 20 25 30 35 40

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 214
3.3. AI with Keras UNIT
03

Reading the data and exploring

‣ Multiplicative exponential smoothing model

Data
10 Fit

2
0 10 20 30 40 50

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 215
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ Pre-processing
‣ Scaling (please pay attention to the vertical scale of the plot)

1.0

0.8

0.6

0.4

0.2

0.0
0 5 10 15 20 25 30 35 40

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 216
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ Reshaping (batch_size, time series length, n_input)

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 217
3.3. AI with Keras UNIT
03

Necessary definitions
‣ Hyperparameters

Line 170-2
• There is only 1 time series data. No other choice but 1.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 218
3.3. AI with Keras UNIT
03

Necessary definitions
‣ Hyperparameters

Line 171-1
• Scalar input

Line 171-2
• N# of neurons per layer

Line 171-3
• Scalar output

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 219
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ RNN or LSTM network

Line 172
• Return sequences = True: means “Sequence to Sequence."
• Input shape = (None, n inputs): the variable length of the time series.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 220
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ View the summary.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 221
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ Define the optimizer and compile.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 222
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ Train the model.

Line 176
• No validation.
• CAUTION: y is X shifted by +1.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 223
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

Training History
1.4
1.2
1.0
MSE

0.8
0.6
0.4
0.2
Train
0.0
0 200 400 600 800 1000
Epoch

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 224
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ Predict the future.

Line 180-1
• Seed length
Line 180-2
• Prediction length

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 225
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ Prediction loop

Line 181-3
• Reshape.
Line 181-5
• The last output is the predicted y.

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 226
3.3. AI with Keras UNIT
03

Recurrent neural network (RNN)

‣ Do the inverse transform and show the times series.

12 Data
Fit
10

2
0 10 20 30 40 50 60

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 227
 End of
Document

Samsung Innovation Campus Chapter 8. Neural Network and Deep Learning 228/44
ⓒ2022 SAMSUNG. All rights reserved.
Samsung Electronics Corporate Citizenship Office holds the copyright of book.
This book is a literary property protected by copyright law so reprint and reproduction without permission are prohibited.
To use this book other than the curriculum of Samsung Innovation Campus or to use the entire or part of this book, you must receive written
consent from copyright holder.