10 DAYS
NO CODE AI/ML
CHALLENGE
DAY 4
TASK 1
PROJECT CARD
AND DEMO
DAY 4
EASY ADVANCED
PROJECT CARD
GOAL:
• Understand deep Neural Networks and
Visualize their layers/architecture.
4
TOOL:
• Google Tensorflow Playground
• Tensorspace.JS
• Ryerson 3D CNN Visualizations
PRACTICAL APPLICATION:
• This project can be effectively used to
understand how AI/ML/DL models work.
DATA:
• INPUTS:
o Images of Numbers
• OUTPUT:
o Class (digits 0 to 9)
Image Source: https://tensorspace.org/html/playground/lenet.html
PROJECT DEMO
4
TASK 2
ARTIFICIAL NEURAL
NETWORKS 101
DAY 4
EASY ADVANCED
AI POWER
“The least scary future I can think of is one where we have at
least democratized AI because if one company or small group
of people manages to develop godlike digital
superintelligence, they could take over the world,” Elon
Musk. Link to article:
https://www.cnbc.com/2018/04/06/elon-musk-warns-ai-could-
create-immortal-dictator-in-documentary
.html
RECALL SINGLE ARTIFICIAL NEURON MODEL
• Let’s assume an activation function of Unit Step.
• The activation functions is used to map the input between (0, 1).
4
𝑏=0
Input #1=1 𝑿𝟏 =0.7
𝑊 2=0.1
Input #2=3
𝑿𝟐 F
𝑊 3=0.3
Input #3=4
𝑿𝟑
RECALL MULTI-LAYER ARTIFICAL NEURAL
NETWORK
• Let’s connect multiple of these neurons in a multi-layered fashion.
• The more hidden layers, the more “deep” the network will get. 4
MATRIX OF WEIGHTS
[ ]
𝑊 11 𝑊 12 𝑊1, 𝑁
⋯ 1
𝑊 21 𝑊 22 𝑊 2, 𝑁 1
⋮ ⋱ ⋮
𝑊 𝑚 − 1 ,1 𝑊 𝑚 −1 ,2 𝑊 𝑚− 1 , 𝑁
⋯ 1
𝑊 𝑚 ,1 𝑊 𝑚, 2 𝑊 𝑚, 𝑁 1
DURING ARTIFICIAL NEURAL NETWORKS
TRAINING, OUR GOAL IS TO FIND THE
OPTIMAL VALUES OF WEIGHTS
m: number of neurons in the hidden layer
: number of inputs
TYPES OF ARTIFICIAL NEURAL
NETWORKS
• Feedforward Neural Networks
o One to one mapping
LET’S VISUALIZE THESE TWO TYPES OF
o Classification or regression
o information flows from input layer directly
NETWORKS USING GOOGLE TENSORFLOW
PLAYGROUND AND TENSORSPACE.JS
4
through hidden layers then to the output
layer without cycles/loops
• Recurrent Neural Networks
o Considers temporal (time) information
o Used for sequences prediction (future
stock prices, translation, text.
o Long short term memory (LSTM) and
GRU (Gated Recurrent Unit)
• Convolutional Neural Networks
o Image classification
Photo Credit: https://commons.wikimedia.org/wiki/File:RecurrentLayerNeuralNetwork_english.png
Photo Credit: https://commons.wikimedia.org/wiki/File:Artificial_neural_network.svg
DEEP LEARNING HISTORY
• There are many trained off the shelf convolutional neural networks that
are readily available such as:
4
o LeNet-5 (1998): 7 level convolutional neural network developed by
LeCun that works in classifying handwriting numbers.
o AlexNet (2012): Offered massive improvement, error reduction from
26% to 15.3%
o ZFNEt (2013): achieved error of 14.8%
o Googlenet/Inception (2014): error reduction to 6.67% which is at par
with human level accuracy.
o VGGNet (2014)
o ResNet (2015): Residual Neural Network includes “skip connection”
feature and therefore enabled training of 152 layers without vanishing
gradient issues. Error of 3.57% which is superior to humans.
o Check out this great resource:
https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet
-vgg-googlenet-resnet-and-more-666091488df5
READING TIME & QUIZ: CONVOLUTIONAL
4
NEURAL NETWORKS ARCHITECTURES
• Please read the article below and answer the following quiz.
o Link to article:
https://towardsdatascience.com/illustrated-10-cnn-architectures-9
5d78ace614d
12 MINS
5 MINS
TASK 3
VISUALIZE FEEDFORWARD
ANNs in GTP
DAY 4
EASY ADVANCED
TENSORFLOW PLAYGROUND
• Let’s choose a linear activation function. The network loss is very high.
4
LINK TO TENSORFLOW PLAYGROUND HERE: HTTPS://PLAYGROUND.TENSORFLOW.ORG/
TENSORFLOW PLAYGROUND
• Let’s choose a sigmoid activation function. The network loss is still very high.
4
TENSORFLOW PLAYGROUND
• Let’s choose a Tanh activation function. The network loss is dramatically reduced.
4
TENSORFLOW PLAYGROUND
• Let’s pick a more complex network.
4
ACTIVATION FUNCTIONS
• SIGMOID:
o Takes a number and sets it between 0 and 1
4
o Converts large negative numbers to 0 and
large positive numbers to 1.
o Generally used in output layer.
• Photo credit: https://commons.wikimedia.org/wiki/File:Sigmoid-function.svg
• Photo Credit: https://fr.m.wikipedia.org/wiki/Fichier:MultiLayerNeuralNetworkBigger_english.png
• Photo Credit: https://commons.wikimedia.org/wiki/File:Logistic-curve.svg
ACTIVATION FUNCTIONS
• RELU (RECTIFIED LINEAR UNITS):
4
o if input x < 0, output is 0 and if x > 0 the output is x.
o RELU does not saturate so it avoids vanishing gradient problem.
o It uses simple thresholding so it is computationally efficient.
o Generally used in hidden layers.
• Photo credit: https://commons.wikimedia.org/wiki/File:ReLU_and_Nonnegative_Soft_Thresholding_Functions.svg
• Photo Credit: https://fr.m.wikipedia.org/wiki/Fichier:MultiLayerNeuralNetworkBigger_english.png
ACTIVATION FUNCTIONS
• HYPERBOLIC TANGENT ACTIVATION FUNCTION:
4
o “Tanh” is similar to sigmoid, converts number between -1 and 1.
o Unlike sigmoid, tanh outputs are zero-centered (range: -1 and 1).
o Tanh suffers from vanishing gradient problem so it kills gradients
when saturated.
o In practice, tanh is preferable over sigmoid.
• Photo credit: https://commons.wikimedia.org/wiki/File:Hyperbolic_Tangent.svg
• Photo Credit: https://fr.m.wikipedia.org/wiki/Fichier:MultiLayerNeuralNetworkBigger_english.png
TASK 4
FINAL PROJECT –
PART A
DAY 4
EASY ADVANCED
FINAL PROJECT PART A
1.
2.
3.
Choose 80%/20% training vs. testing data split.
Select a high level noise ~30
Set the batch size to 20
4
4. Select the spiral dataset as an input
5. Modify the network architecture
6. Tune the network hyperparameters.
7. Assess network performance and comment on your results.
https://developers.google.com/machine-learning/crash-course/DPE/tp-ss-neural-net-intro-spiral
FINAL PROJECT SOLUTION
4
FINAL PROJECT SOLUTION
4
TASK 5
CONVOLUTIONAL
NEURAL NETWORKS
DAY 4
EASY ADVANCED
WHAT IS AN IMAGE?
•
•
•
A greyscale image is system of 256 tones with values ranging from 0-255.
'0' represents black and '255' represents white.
Numbers in-between represents greys between black and white.
4
• Binary systems use digits '0' and '1‘ where '00000000' for black, to '11111111'
for white (8-bit image).
• Note: binary value of '11111111' is equal to decimal value of '255‘.
255 255 255 11111111 11111111 11111111
155 155 155 10011011 10011011 10011011
0 0 0 00000000 00000000 00000000
Photo Credit: https://commons.wikimedia.org/wiki/file:monalisa_in_antique_color-
oil_painting_by_rajasekharan.jpg
CONVOLUTIONAL NEURAL NETWORKS:
4
ENTIRE NETWORK OVERVIEW
OUTPUT
INPUT
HEALTHY
CONVOLUTION POOLING FLATTENING COVID-19
VIRAL
BACTERIAL
KERNELS/ POOLING
FEATURE FILTERS
DETECTORS
CONVOLUTIONAL LAYER POOLING LAYER
(DOWNSAMPLING)
Photo Credit: https://commons.wikimedia.org/wiki/File:Artificial_neural_network.svg
FEATURE DETECTORS
4
• Convolutions use a kernel matrix to scan a given image and apply a filter to obtain a
certain effect.
• An image Kernel is a matrix used to apply effects such as blurring and sharpening.
• Kernels are used in machine learning for feature extraction to select most important pixels
of an image.
• Convolution preserves the spatial relationship between pixels.
FEATURE MAPS
KERNELS/
FEATURE
DETECTORS
FEATURE DETECTORS
• Live Convolution: http://setosa.io/ev/image-kernels/
FEATURE DETECTOR
4
0 1 1 0 1
1 0 0 1 0 0 0 1 1 1 1
0 1 0 1 1 0 0 0 3 1 1
0 1 0 0 1 1 1 0 2 3 1
0 0 1 0 1
FEATURE MAP
IMAGE
POOLING (DOWNSAMPLING)
• Pooling or down sampling layers are placed after convolutional layers to
reduce feature map dimensionality.
• This improves the computational efficiency while preserving the features.
4
• Pooling helps the model to generalize by avoiding overfitting.
• If one of the pixel is shifted, the pooled feature map will still be the same.
• Max pooling works by retaining the maximum feature response within a
given sample size in a feature map.
• Live illustration : http://scs.ryerson.ca/~aharley/vis/conv/flat.html
1 1 3 4 6
3 6 2 8 MAX POOLING
6 8
FLATTENING 8
3 9 1 0 2x2 9 4 9
STRIDE = 2
1 3 3 4 4
Photo Credit: https://commons.wikimedia.org/wiki/File:Artificial_neural_network.svg
TASK 6
CNNS
VISUALIZATION
DAY 4
EASY ADVANCED
CONVOLUTIONAL NEURAL NETWORKS IN
ACTION!
4
Resource: https://www.cs.ryerson.ca/~aharley/vis/conv/flat.html
MINI CHALLENGE
1. Using the 2D CNN data visualization link below, experiment with at least 3
numbers and assess the performance of trained model:
4
https://www.cs.ryerson.ca/~aharley/vis/conv/flat.html
2. Use 3D CNN visualization instead of 2D, try at least 3 numbers, and assess
model performance: https://www.cs.ryerson.ca/~aharley/vis/conv/
TASK 7
LE-NET CNN
DAY 4
EASY ADVANCED
LENET ARCHITECTURE
• The network used is called LeNet that was presented by Yann LeCun
• Reference and photo credit: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf
• C: Convolution layer, S: subsampling layer, F: Fully Connected layer
4
TASK 8
3D VISUALIZATION IN
TENSORSPACE.JS
DAY 4
EASY ADVANCED
VISUALIZATION IN TENSORSPACE.JS
4
VISUALIZATION IN TENSORSPACE.JS
4
VISUALIZATION IN TENSORSPACE.JS
4
TASK 9
RESNETS
DAY 4
EASY ADVANCED
RESNET (RESIDUAL NETWORK)
• ResNet is a powerful CNN that won the ImageNet challenge in
2015. ResNet contains 150+ layers and can overcome the
4
problem of vanishing gradients.
• As CNNs grow deeper, vanishing gradient tend to occur which
negatively impact network performance.
• Vanishing gradient problem occurs when the gradient is back-
propagated to earlier layers which results in a very small gradient.
• ResNets include “skip connection” feature which enables training
of multiple deep layers (152 layers) without vanishing gradient
issues.
• Check out ResNet architecture here:
https://www.researchgate.net/figure/The-structure-of-the-used-resi
dual-neural-network-Shortcuts-for-residual-blocks-12-are_fig2_332
927869
Photo Credit: https://en.wikipedia.org/wiki/Residual_neural_network
RESNET VISUALIZATION IN
TENSORSPACE.JS
4
TASK 10
FINAL PROJECT
DAY 4
EASY ADVANCED
FINAL PROJECT
• Research AlexNet and explore the network architecture?
• Visual AlexNet in Tensorflow.JS 4
FINAL PROJECT Source: https://www.researchgate.net/figure/AlexNet-architecture-
4
Includes-5-convolutional-layers-and-3-fullyconnected-
layers_fig3_322592079
