Deep Learning Basics - I

PGS.TS. Nguyễn Phương Thái

Adapted from slides of Andrew Ng

What is a
Neural Network?
Housing Price Prediction
price

size of house
Housing Price Prediction
Housing Price Prediction

size 𝑥1

#bedroom 𝑥2
s
y
zip code 𝑥3

wealth 𝑥4
Introduction to
Deep Learning
Supervised Learning
with Neural Networks
Supervised Learning
Input(x) Output Application
(y)
Home features Price Real Estate

Ad, user info Click on ad? (0/1) Online Advertising

Image Object (1,…,1000) Photo tagging

Audio Text transcript Speech recognition

English Chinese Machine translation

Image, Radar info Position of other cars Autonomous driving

Standard NN Convolutional NN Recurrent

User Ad Id … Click
Age Four scores and
41 93242 1
seven years ago…
80 93287 0
18 87312 1 Text
⋮ ⋮ ⋮
27 71244 1
 Introduction
to Neural
Networks

Why is Deep
Learning taking off?
Andrew Ng
Scale drives deep learning progress
Performance

Amount of data
Andrew Ng
Scale drives deep learning progress

Idea
• Data

• Computation

• Algorithms
Experiment Code

Andrew Ng
Basics of Neural
Network
Programming
Binary Classification
Binary Classification

1 (cat) vs 0 (non
cat)

Blue
Green
Red
Notation
Basics of Neural
Network
Programming
Logistic Regression
Logistic Regression
Basics of Neural
Network
Programming
Logistic Regression
cost function
 Logistic Regression cost function
, where

Given want .
Loss (error)
function:
Basics of Neural
Network
Programming
Gradient Descent
Gradient Descent
Recap: ,
𝑚 𝑚

𝐽 ( 𝑤 , 𝑏 ) =¿
1
∑
𝑚 𝑖=1
−
1
∑
𝑚 𝑖=1
+

Want to find that minimize

𝑏
𝑤
Gradient Descent

𝑤
Basics of Neural
Network
Programming
Derivatives
Intuition about derivatives
𝑓 ( 𝑎 )=3 𝑎

𝑎
Basics of Neural
Network
Programming
More derivatives
examples
Intuition about derivatives
2
𝑓 ( 𝑎 )=𝑎

𝑎
More derivative examples
Basics of Neural
Network
Programming
Computation Graph
Computation Graph
Basics of Neural
Network
Programming
Derivatives with a
Computation Graph
Computing derivatives
𝑎=5 11 33
𝑏=3 6
=
𝑐=2
Computing derivatives
𝑎=5 11 33
𝑏=3 6
=
𝑐=2
Basics of Neural
Network
Programming
Logistic Regression
Gradient descent
Logistic regression recap
𝑇
𝑧 =𝑤 𝑥 +𝑏
^
𝑦 =𝑎=𝜎 ( 𝑧 )
ℒ ( 𝑎 , 𝑦 )= − ¿
Logistic regression derivatives
𝑥1
𝑤1
𝑥2
𝑤2
b
Basics of Neural
Network
Programming
Gradient descent
on m examples
Logistic regression on m examples
Logistic regression on m examples
Basics of Neural
Network
Programming
Vectorization
What is vectorization?
Basics of Neural
Network
Programming
More vectorization
examples
Neural network programming
guideline
Whenever possible, avoid explicit for-loops.
Neural network programming
guideline
Whenever possible, avoid explicit for-loops.
Vectors and matrix valued functions
Say you need to apply the exponential operation on every element of a
matrix/vector.

[ ]
𝑣1
𝑣= ⋮
𝑣𝑛

u = np.zeros((n,1))
for i in range(n):
u[i]=math.exp(v[i])
Logistic regression derivatives
J = 0, dw1 = 0, dw2 = 0, db = 0
for i = 1 to n:
=
=
+=
d =
d+=
d+=
db += d
J = J/m, d = d/m, d = d/m, db = db/m
Basics of Neural
Network
Programming
Vectorizing Logistic
Regression
 Vectorizing Logistic Regression
(1) 𝑇 (1) (2) 𝑇 (2) (3) 𝑇 (3 )
𝑧 =𝑤 𝑥 +𝑏 𝑧 =𝑤 𝑥 + 𝑏 𝑧 =𝑤 𝑥 +𝑏
(1) (2 ) (3 )
𝑎 =𝜎 (𝑧¿¿(1))¿ 𝑎 =𝜎 (𝑧 ¿¿(2))¿ 𝑎 =𝜎 (𝑧 ¿¿ (3))¿
 Basics of Neural
Network
Programming
Vectorizing Logistic
Regression’s Gradient
Computation
Vectorizing Logistic Regression
 Implementing Logistic Regression
J = 0, d = 0, d = 0, db = 0
for i = 1 to m:
=
=
+=
d =
d+=
d+=
db += d
J = J/m, d = d/m, d = d/m
db = db/m
Basics of Neural
Network
Programming
Broadcasting in
Python
Broadcasting example
Calories from Carbs, Proteins, Fats in 100g of different foods:
Apples Bee Eggs Potatoes

[ ]
Carb 56.0 0.0 4.4 f68.0
Protein 1.2 104.0 52.0 8.0
Fat 1.8 135.0 99.0 0.9

cal = A.sum(axis = 0)
percentage = 100*A/(cal.reshape(1,4))
Broadcasting example

[] [ ]
1 1 01
2 100 =
1 02
3 + 103
4 104

+ [ 1 00 200 300 ] = [ 1 01
104
202
205
303
306 ]
+ [ 1 00
200 ] = [ 1 01
204
102
205
103
206 ]
General Principle
Basics of Neural
Network
Programming
A note on python/
numpy vectors
Python Demo
Python / numpy vectors

import numpy as np

a = np.random.randn(5)

a = np.random.randn((5,1))

a = np.random.randn((1,5))

assert(a.shape = (5,1))