Demystifying Deep Learning

Dr. Amod Anandkumar

Senior Team Lead – Signal Processing & Communications

© 2015 The MathWorks, Inc.

1
What is Deep Learning?

2
Deep learning is a type of machine learning in which a model learns
to perform classification tasks directly from images, text, or sound.

Deep learning is usually implemented using a neural network.

The term “deep” refers to the number of layers in the network—the

more layers, the deeper the network.

3
Deep Learning is Versatile MATLAB Examples Available Here

4
Many Network Architectures for Deep Learning
Directed Acyclic
Series Network Recurrent Network
Graph Network

and more
(GAN, DQN,…)

AlexNet ResNet
LSTM
YOLO R-CNN
5
Convolutional Neural Networks

cake ✓
cat 

…
…

…
dog 
Convolution + Convolution + Flatten Fully Softmax
Input ReLU Pooling ReLU Pooling Connected

Feature Learning Classification

6
 Deep Learning Inference in 4 Lines of Code

• >> net = alexnet;

• >> I = imread('peacock.jpg')
• >> I1 = imresize(I,[227 227]);
• >> classify(net,I1)
• ans =
• categorical
• peacock

7
What is Training?

Images
Network
Training
Labels
Trained Deep
Neural Network
Large Data Set

During training, neural network architectures learn features directly

from the data without the need for manual feature extraction
8
 What Happens During Training?
AlexNet Example

Training Data Labels

Layer weights are learned

during training
9
 Visualize Network Weights During Training
AlexNet Example

Training Data Labels

First Convolution
Layer

10
Visualization Technique – Deep Dream

deepDreamImage(...
net, 'fc5', channel,
'NumIterations', 50, ...
'PyramidLevels', 4,...
'PyramidScale', 1.25);

Synthesizes images that strongly activate

a channel in a particular layer

Example Available Here

11
Visualize Features Learned During Training
AlexNet Example

Sample Training Data Features Learned by Network

12
Visualize Features Learned During Training
AlexNet Example

Sample Training Data Features Learned by Network

13
Deep Learning Challenges
Data
Not a deep learning expert
▪ Handling large amounts of data

▪ Labeling thousands of images & videos

Training and Testing Deep Neural Networks

▪ Accessing reference models from research
▪ Optimizing hyperparameters

▪ Training takes hours-days

Rapid and Optimized Deployment

▪ Desktop, web, cloud, and embedded hardware

14
Example – Semantic Segmentation Available Here

▪ Classify pixels into 11 classes

– Sky, Building, Pole, Road,
Pavement, Tree, SignSymbol,
Fence, Car, Pedestrian, Bicyclist

▪ CamVid dataset Brostow, Gabriel J., Julien Fauqueur, and Roberto Cipolla. "Semantic object classes in
video: A high-definition ground truth database." Pattern Recognition Letters Vol 30, Issue
2, 2009, pp 88-97. 15
Label Images Using Image Labeler App

16
 Accelerate Labeling With Automation Algorithms

Learn More

17
 Perform Bootstrapping to Label Large Datasets

Images
Videos Train
Network
Labeling Apps Ground Truth

Improve More Propose Labels

Network Ground Truth

Automation
Algorithm

18
Example – Semantic Segmentation Available Here

19
Access Large Sets of Images

Handle Large Sets of Images

Organize Images in Folders

(~ 10,000 images , 5 folders)

Easily manage large sets of images imageData =

- Single line of code to access images
imageDataStore(‘vehicles’)
- Operates on disk, database, big-data file system
Easily manage large sets of images
- Single line of code to access images
- Operates on disk, database, big-data file system

20
Handle Big Image Collections without Big Changes

Images in local directory

Images on HDFS

21
Import Pre-Trained Models and Network Architectures
Pretrained Models
Import Models from Frameworks
▪ alexnet
▪ Caffe Model Importer
▪ vgg16 (including Caffe Model Zoo)
▪ vgg19 – importCaffeLayers

▪ googlenet – importCaffeNetwork
▪ TensorFlow-Keras Model Importer
▪ inceptionv3
– importKerasLayers
▪ resnet50
– importKerasNetwork
▪ resnet101
▪ inceptionresnetv2
Download from within MATLAB
▪ squeezenet
22
Example – Semantic Segmentation Available Here

23
Augment Training Images

Rotation
Reflection
Scaling
Shearing
Translation

Colour pre-processing
Resize / Random crop / Centre crop 24
Tune Hyperparameters to Improve Training

Many hyperparameters
▪ depth, layers, solver options,
learning rates, regularization,
…

Techniques
▪ Parameter sweep
▪ Bayesian optimization

25
Training Performance

TensorFlow
MATLAB
MXNet

26
 NVIDIA Tesla V100 32GB
The Fastest and Most Productive GPU for AI and HPC
Volta Architecture Tensor Core Improved NVLink & Volta MPS Improved SIMT Model
HBM2

125 Programmable
Most Productive GPU TFLOPS Deep Learning Efficient Bandwidth Inference Utilization New Algorithms

5120 CUDA cores,

Core
640 Tensor cores
Compute 7.8 TF DP ∙ 15.7 TF SP ∙ 125 TF DL
Memory HBM2: 900 GB/s ∙ 32 GB/16 GB
NVLink (up to 300 GB/s) +
Interconnect
PCIe Gen3 (up to 32 GB/s)
Visit NVIDIA booth
to learn more
Deep Learning on CPU, GPU, Multi-GPU & Clusters
HOW TO TA R G E T ?

Single Single CPU

CPU Single GPU

Single CPU, Multiple GPUs

On-prem server with Cloud GPUs

GPUs (AWS)
28
Multi-GPU Performance Scaling

Ease of scaling
Seconds per epoch

▪ MATLAB “transparently”
scales to multiple GPUs

▪ Runs on Windows!

1 GPU 2 GPUs 4 GPUs

29
Examples to Learn More

30
Example – Semantic Segmentation Available Here

31
Accelerate Using GPU Coder

Running in MATLAB Generated Code from GPU Coder

32
Prediction Performance: Fast with GPU Coder
Images/Sec

~2x ~2x faster than

TensorFlow

~2x
TensorFlow
~2x MATLAB
MXNet
AlexNet ResNet-50 VGG-16 GPU Coder
33
Deploying Deep Learning Application
Embedded Hardware Desktop, Web, Cloud
Application
logic

Code Application
Generation Deployment
34
Next Session

35
Addressing Deep Learning Challenges

✓Perform deep learning without being an expert

✓Automate ground truth labeling
✓Create and visualize models with just a few lines of code
✓Seamless scale training to GPUs, clusters and cloud
✓Integrate & deploy deep learning in a single workflow

36
Framework Improvements
▪ Architectures / layers ▪ Performance
– Regression LSTMs – CPU performance optimizations
– Bidirectional LSTMs – Optimizations for zero learning-rate
– Multi-spectral images
– Custom layer validation ▪ Network training
– ADAM & RMSProp optimizers
▪ Data pre-processing – Gradient clipping
– Custom Mini-Batch Datastores – Multi-GPU DAG network training
– DAG network activations

38
Deep Learning Network Analyzer

39
 Creating Custom Architectures
MATLAB provides a simple programmatic
interface to create layers and form a network

• addLayers
• removeLayers
• connectLayers
• disconnectLayers

… but are all these layers

compatible?

40
41
8 x 12 x 64

42
8 x 12 x 128

43
7 x 11 x 32

44
Change the padding
from zero to one

45
46
Call to Action – Deep Learning Onramp
Free Introductory Course

Available Here

47
MATLAB Deep Learning Framework

Access Data Design + Train Deploy

▪ Manage large image sets ▪ Easy access to models ▪ Optimized inference

▪ Automate image labeling ▪ Acceleration with GPU’s deployment
▪ Scale to GPUs & clusters ▪ Target NVIDIA, Intel, ARM
platforms

48
 Speaker Details Contact MathWorks India
Email: [email protected] Products/Training Enquiry Booth
LinkedIn: https://in.linkedin.com/in/ajga2 Call: 080-6632-6000
Twitter: @_Dr_Amod Email: [email protected]

• Share your experience with MATLAB & Simulink on Social Media

▪ Use #MATLABEXPO

• Share your session feedback:

Please fill in your feedback for this session in the feedback form

49