Embedded Systems &

IoT Applications
AI on Embedded Device
Woojin Jeong
CNN (Convolutional Neural Network)

2
Convolutional Neural Network (CNN or ConvNet)

● “Convolutional Neural Network” indicates that the network employs a

mathematical operation called convolution. Convolutional networks are
a specialized type of neural networks that use convolution in place of
general matrix multiplication in at least one of their layers.
● CNN(ConvNet) was first introduced in the 1980s by Yann LeCun.
(LeNet-5)
● In 2012, computer vision took a quantum leap with AlexNet(Alex
Krizhevsky & Geoffrey Hinton,).

Source : https://en.wikipedia.org/wiki/Convolutional_neural_network
https://bdtechtalks.com/2020/01/06/convolutional-neural-ne
tworks-cnn-convnets/

3
CNN : LeNet-5

Source : http://vision.stanford.edu/cs598_spring07/papers/Lecun98.pdf

4
CNN : AlexNet

Source :
https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolut
ional-neural-networks.pdf

5
CNN Structure

● Use Convolution, Pooling layer to extract features in image.

Source : https://developersbreach.com/convolution-neural-network-deep-learning/ 6
CNN : Differences

● For e.g.) CIFAR-10 image shape is 32x32x3 (32x32 with RGB

color channel)
○ If we use normal NN, input layer has 32*32*3 = 3072 weights
to a neuron of next layer

● If we use 200x200x3 images, we need 200*200*3 = 120,000

weights for a neuron of next layer.

● Use 3D input (Height x Width x (RGB Channel)

7
CNN : Layers

● Convolutional Layer
● Pooling Layer
● Fully-Connected Layer (Regular NN)

8
CNN : Convolutional Layers
● Convolution

data kernel/filter

Source : https://mlnotebook.github.io/post/CNN1/

9
CNN : Convolutional Layers
● Convolution

Source : https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1

10
CNN : Convolutional Layers
● Convolution

Source : https://mlnotebook.github.io/post/CNN1/

11
CNN : Convolutional Layers
● Convolution for Multiple channels

0~255 Reds 0~255 Greens 0~255 Blues

16,777,216 colors in computer

True Color

Source : https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1

12
CNN : Convolutional Layers
● Convolution for Multiple channels

Source : https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1

13
CNN : Convolutional Layers
● Convolution for Multiple channels

Source : https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1

14
CNN : Multiple Convolutional Layers

● CNN has multiple convolutional layers.

● The first (or bottom) layer of the CNN usually
detects basic features such as horizontal, vertical,
and diagonal edges.
● The output of the first layer is fed as input of the
next layer, which extracts more complex features,
such as corners and combinations of edges.
● As you move deeper into the convolutional neural
network, the layers start detecting higher-level
features such as objects, faces, and more.

Source : https://bdtechtalks.com/2020/01/06/convolutional-neural-networks-cnn-convnets/

15
CNN : Multiple Convolutional Layers

● The top layer of the CNN

determines the class of the image
based on features extracted by
convolutional layers

Source : https://bdtechtalks.com/2020/01/06/convolutional-neural-networks-cnn-convnets/

16
CNN : Pooling Layers
● To progressively reduce the spatial size of the representation to reduce the
amount of parameters and computation in the network.

Source : https://bdtechtalks.com/2020/01/06/convolutional-neural-networks-cnn-convnets/

17
CNN : Pooling Layers

Source : https://cs231n.github.io/convolutional-networks/

18
CNN : Pooling Layers

max pooling

Source : https://cs231n.github.io/convolutional-networks/

19
CNN : Pooling Layers

max pooling

Source : https://cs231n.github.io/convolutional-networks/

20
CNN : Pooling Layers

max pooling

Source : https://cs231n.github.io/convolutional-networks/

21
CNN : Pooling Layers

Source : https://cs231n.github.io/convolutional-networks/

22
CNN

Good reference to understand CNN

CS231n Convolutional Neural Networks for Visual Recognition

(https://cs231n.github.io/)

● CNN : https://cs231n.github.io/convolutional-networks/

To understand the convolution

https://towardsdatascience.com/intuitively-understanding-convolutions-for-
deep-learning-1f6f42faee1

23
CIFAR-10, CNN, Keras, Tensorflow

24
CIFAR-10

● The CIFAR-10 dataset (Canadian Institute For Advanced Research) is a collection

of images that are commonly used to train machine learning and computer vision
algorithms. It is one of the most widely used datasets for machine learning
research. The CIFAR-10 dataset contains 60,000 32x32 color images in 10
different classes. The 10 different classes represent airplanes, cars, birds, cats,
deer, dogs, frogs, horses, ships, and trucks. There are 6,000 images of each
class.
● http://www.cs.toronto.edu/~kriz/cifar.html

25
CIFAR-10

Source : http://www.cs.toronto.edu/~kriz/cifar.html
26
CIFAR-10 with CNN in Tensorflow & Keras
import tensorflow as tf
from tensorflow.keras import datasets, layers, models
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images=train_images/255.0
test_images=test_images/255.0
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
SparseCategoricalCrossentropy(from_logits=True),
model.compile(optimizer='adam', loss=tf.keras.losses.
metrics=['accuracy'])
history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

27
CIFAR-10 with CNN in Tensorflow & Keras
import tensorflow as tf
Import modules
from tensorflow.keras import datasets, layers, models
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images=train_images/255.0
test_images=test_images/255.0
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True),
metrics=['accuracy'])
history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

28
CIFAR-10 with CNN in Tensorflow & Keras
import tensorflow as tf
from tensorflow.keras import datasets, layers, models
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images=train_images/255.0
Prepare data for training
test_images=test_images/255.0
& testing
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

29
CIFAR-10 with CNN in Tensorflow & Keras

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

train_images.shape
train_labels.shape

30
CIFAR-10 with CNN in Tensorflow & Keras

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

import matplotlib.pyplot as plt

class_names = ['airplains', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

plt.figure(figsize=(10,10))
for i in range(25) :
plt.subplot(5,5,i+1) Display first 25 images
plt.xticks([])
plt.yticks([])
plt.grid(False)
plt.imshow(train_images[i], cmap=plt.cm.binary)
plt.xlabel(class_names[train_labels[i][0]])
plt.show()

31
CIFAR-10 with CNN in Tensorflow & Keras

(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()

class_names = ['airplains', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

plt.figure(figsize=(10,10))
for i in range(25) :
plt.subplot(5,5,i+1)
plt.xticks([])
plt.yticks([]) Display first 25 images
plt.grid(False)
plt.imshow(train_images[i], cmap=plt.cm.binary)
plt.xlabel(class_names[train_labels[i][0]])
plt.show()

32
CIFAR-10 with CNN in Tensorflow & Keras
import tensorflow as tf
from tensorflow.keras import datasets, layers, models
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images=train_images/255.0
test_images=test_images/255.0 Normalization
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

33
CIFAR-10 with CNN in Tensorflow & Keras
import tensorflow as tf
from tensorflow.keras import datasets, layers, models
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images=train_images/255.0
test_images=test_images/255.0
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
Build network model
model.add(layers.Conv2D(64, (3,3), activation='relu'))
architecture
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

34
CIFAR-10 with CNN in Tensorflow & Keras

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
Build network model
model.add(layers.Conv2D(64, (3,3), activation='relu'))
architecture
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])

35
CIFAR-10 with CNN in Tensorflow & Keras

Convolutional Layers
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

36
CIFAR-10 with CNN in Tensorflow & Keras

Number of filters(kernels) to use

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

37
CIFAR-10 with CNN in Tensorflow & Keras

Shape of filter(kernel)
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

38
CIFAR-10 with CNN in Tensorflow & Keras

Input shape (width : 32, Height : 32, Depth(Channel) : 3)

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

39
CIFAR-10 with CNN in Tensorflow & Keras

Max Pooling Layers

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

40
CIFAR-10 with CNN in Tensorflow & Keras

Pooling window, kernel shape

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))

41
CIFAR-10 with CNN in Tensorflow & Keras

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.summary()

42
CIFAR-10 with CNN in Tensorflow & Keras

model.add(layers.Flatten()) Flatten the input

model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))

43
CIFAR-10 with CNN in Tensorflow & Keras

model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10)) Full connected layers (Regular NN)

44
CIFAR-10 with CNN in Tensorflow & Keras

model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.summary()

45
CIFAR-10 with CNN in Tensorflow & Keras

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])

Specify optimizer (ADAM) & loss function(SparseCategoricalCrossentropy)

● Do not need one-hot-encoding

46
CIFAR-10 with CNN in Tensorflow & Keras

alternative code of page 46

model.add(layers.Dense(10, activation='softmax'))
train_labels = tf.keras.utils.to_categorical(train_labels, 10)
test_labels = tf.keras.utils.to_categorical(test_labels, 10)
model.compile(optimizer='adam', loss=tf.keras.losses.CategoricalCrossentropy(), metrics=['accuracy'])

Use one-hot-encoding to use Categorical Cross Entropy for loss function.

● Should use ‘softmax’ for the activation function of output layer

47
CIFAR-10 with CNN in Tensorflow & Keras

alternative code of page 46

model.add(layers.Dense(10, activation='softmax'))
train_labels = tf.keras.utils.to_categorical(train_labels, 10)
test_labels = tf.keras.utils.to_categorical(test_labels, 10)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

Use one-hot-encoding to use Categorical Cross Entropy for loss function.

● Should use ‘softmax’ for the activation function of output layer

48
CIFAR-10 with CNN in Tensorflow & Keras
import tensorflow as tf
from tensorflow.keras import datasets, layers, models
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images=train_images/255.0
test_images=test_images/255.0
model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.MaxPooling2D(2,2))
model.add(layers.Conv2D(64, (3,3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))
model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(
from_logits=True),
metrics=['accuracy'])
history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

Training
49
CIFAR-10 with CNN in Tensorflow & Keras

● Repeat 10 epochs for whole 50,000 images

● batch_size is 32 if not specified
● Use 10,000 test_images for validation

history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

50
CIFAR-10 with CNN in Tensorflow & Keras

history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

51
CIFAR-10 with CNN in Tensorflow & Keras

history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

plt.plot(history.history['accuracy'], label='accuracy')
plt.plot(history.history['val_accuracy'], label = 'val_accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.ylim([0.5, 1])
plt.legend(loc='lower right')

52
CIFAR-10 with CNN in Tensorflow & Keras

history=model.fit(train_images, train_labels, epochs=10, validation_data=(test_images, test_labels))

test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)

print(test_acc)

53
Assignment

54
Assignment
● Make CIFAR-10 CNN in this material
○ Train it 10 epochs → You will get similar accuracy & loss with this material
○ Train it 50 epochs
■ Get the training accuracy & validation accuracy graph.
■ Get the test loss & accuracy
■ SUBMIT : Training accuracy & validation accuracy graph screenshot
■ SUBMIT : Test accuracy screenshot

● Improve CIFAR-10 CNN model. You can try to :

○ Add more Conv2D, MaxPooling2D layers
○ Modify Number of kernels or shape of kernel
○ Modify Full-connected NN structure (Add or change number of neurons)
○ SUBMIT : Source code of the best accurate model you build
○ SUBMIT : Training accuracy & validation accuracy graph screenshot
○ SUBMIT : Test loss & accuracy screenshot

55
Assignment
● Zip all files in one zip file, due date Apr. 2(Tue).
● Do not submit early!!!
● Test & evaluate as may as possible models you can create.
● Assignment will be open from Monday!!! (Do not email me!!)

56