Deep Learning
and
Pytorch Tutorial
Outline
Environment Settings on Ubuntu 16.04
Why Pytorch
Installation
Code explanation
Convolutional neural networks
Recurrent neural networks
Variational autoencoder
Generative adversarial networks
Conclusion
Environment Settings on Ubuntu 16.04
GPU drivers
CUDA
cuDNN
Virtual Environment
GPU drivers
軟體目錄下新增ppa地址，然後更新資料庫資訊
利用ppa來管理GPU driver’s version
$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo apt-get update
安裝 dirver 410版
安裝前必須先關閉桌面環境 lightdm
$ sudo service lightdm stop
$ sudo apt-get install nvidia-410 nvidia-settings
重開機後，輸入 nvidia-smi，安裝成功後會顯示顯卡資訊
查看 driver 版本
$ cat /proc/driver/nvidia/version
nvidia-smi
下載 CUDA 10.0
https://developer.nvidia.com/cuda-10.0-download-archive
安裝前必須先關閉桌面環境 lightdm
$ sudo service lightdm stop
接著安裝CUDA
$ sudo sh cuda_10.0.130_410.48_linux.run
下載 CUDA 10.0
步驟中有選擇安裝 CUDA Sample，可以選擇安裝在哪
將以下加入 ~/.bashrc : (Global setting, 設定在 /etc/profile 下)
export CUDA_HOME=/usr/local/cuda
export CUDA_ROOT=$CUDA_HOME
export CUDA_INC_DIR=$CUDA_HOME/include/
export LD_LIBRARY_PATH=${CUDA_HOME}/lib64
PATH=${CUDA_HOME}/bin:${PATH}
export PATH
下載 CUDA 10.0
設定完後，記得 $ source ~/.bashrc 可以透過 $ nvcc --version 查看 cuda 是否安裝成功
驗證 CUDA 是否安裝成功
安裝 CUDA Samples 目的主要是驗證 CUDA 安裝是否成功
如果編譯 CUDA Samples 沒有 Errors (Warning 忽略), 就表示沒問題
# 切換到 NVIDIA CUDA-10.0 Samples
$ cd /usr/local/cuda/samples
# 編譯 CUDA Sample
$ make –j8
# 編譯完後，切換到 release 目錄
$ cd /usr/local/cuda/samples/bin/x86_64/linux/release
# 驗證是否成功, 會顯示 NVIDIA 顯卡資訊
$ ./deviceQuery
ldconfig
進入 /etc/ld.so.conf.d 下, 建立新文件 cuda.conf：
/usr/local/cuda/lib64
/lib
/usr/lib
/usr/lib32
Install cuDNN
https://developer.nvidia.com/rdp/cudnn-download
Install cuDNN
1.Install the runtime library, for example:
sudo dpkg -i libcudnn7_7.5.0.56-1+cuda10.0_amd64.deb
2.Install the developer library, for example:
sudo dpkg -i libcudnn7-dev_7.5.0.56-1+cuda10.0_amd64.deb
3.Install the code samples and the cuDNN Library User Guide, for example:
sudo dpkg -i libcudnn7-doc_7.5.0.56-1+cuda10.0_amd64.deb
測試 cuDNN
1.Copy the cuDNN sample to a writable path.
$ cp -r /usr/src/cudnn_samples_v7/ $HOME
2.Go to the writable path.
$ cd $HOME/cudnn_samples_v7/mnistCUDNN
3.Compile the mnistCUDNN sample.
$ make clean && make
4.Run the mnistCUDNN sample.
$ ./mnistCUDNN
Deep learning platforms
There are many open source deep learning platforms.
Theano
Tensorflow
Mxnet
Torch
Caffe
Pytorch
Theano
Python framework developed by the MILA Lab, University of Montreal
Provide computational graph
Pros
Computational graph is a nice abstraction
High level wrappers
Cons
Long compile time
Fewer support
Tensorflow
Supported by Google Brain
Pros
Strong support
Distributed computation
Implement on edge devices
Cons
Hard to learn
A bit different from Numpy and Python
Intstall tensorflow
$ pip3 install tensorflow-gpu
Mxnet
Adopted by Amazon Web Services, AWS
Pros
Less memory usage
Variety of interfaces
Cons
Upgrade slowly
Torch
Has its API in Lua
Python version of Torch was open-sourced by Facebook in January, 2017
Pros
Flexibility and fast
Faster compile times
Caffe
Caffe was created by Yangqing Jia during his PhD at UC Berkeley
Used by computer vision people because of the fast convolutional nets
Pros
Good for feedforward networks and image processing
Good for fintuning existing networks
Cons
Not good for recurrent networks
Pytorch
Supported by Facebook AI Research(FAIR)
Has user coming from Torch and Caffe
More and more projects are written in Pytorch
More Python style compared to Tensorflow
Easier to learn
Installation
Quick start in https://pytorch.org/
The stable version is Pytorch 1.0
Can be installed in Linux, Mac and Windows systems
Can be implemented in different python versions
Supported by different version of Cuda
Prerequisites
Operating systems, especially Linux distributions
Python versions
python/python3
virtualenv
Package manager
Numpy if installed via pip
CUDA if using GPU
Tensors
Tensors are multi-dimensional arrays
1-D tensors are vectors
2-D tensors are matrices
Similar to ndarrays in Numpy
Can be accelerated by GPU
Uninitialized matrix
Randomly initialized matrix
Zero matrix
Tensor directly from data
Tensor based on existing tensor
Get the size
Operations
Operations do calculations on tensors
May have different syntaces
Addition syntax 1
Addition syntax 2
Addition with output tensor
Addition in-place
Indexing
Resizing
Retrieve one element tensor
Torch tensor to Numpy array
Numpy array to Torch tensor
CUDA Tensors
CNN experiment
CIFAR 10 classification
10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck
The images are 3x32x32
Train an image classifier
Implementation steps
1. Load and normalizing the CIFAR 10 training and test datasets using
torchvision
2. Define a Convolutional Neural Network
3. Define a loss function
4. Train the network on the training data
5. Test the network on the test data
Loading data
load with torchvision
output of torchvision are PILImage of range [0, 1],
transform to [-1, 1]
CNN
Loss function and optimizer
Train the network
Testing
Accuracy
Accuracy by classes
Training on GPU
