Scribd
Upload
7 views20 pages

Winter1516 Lecture51

Uploaded by

rsethi3
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views20 pages

Winter1516 Lecture51

Uploaded by

rsethi3
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Lecture 5:

Training Neural Networks,


Part I

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 1 20 Jan 2016
Administrative
A1 is due today (midnight)
I’m holding make up office hours on today: 5pm @ Gates 259

A2 will be released ~tomorrow. It’s meaty, but educational!

Also:
- We are shuffling the course schedule around a bit
- the grading scheme is subject to few % changes

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 2 20 Jan 2016
Things you should know for your Project Proposal

“ConvNets need a lot


of data to train”

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 3 20 Jan 2016
Things you should know for your Project Proposal

“ConvNets need a lot


of data to train”
finetuning! we rarely ever
train ConvNets from scratch.

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 4 20 Jan 2016
1. Train on ImageNet 2. Finetune network on
your own data

ImageNet data your


data

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 5 20 Jan 2016
Transfer Learning with CNNs
1. Train on 2. If small dataset: fix 3. If you have medium sized
ImageNet all weights (treat CNN dataset, “finetune” instead:
as fixed feature use the old weights as
extractor), retrain only initialization, train the full
the classifier network or only some of the
higher layers
i.e. swap the Softmax
layer at the end retrain bigger portion of the
network, or even all of it.

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 6 20 Jan 2016
E.g. Caffe Model Zoo: Lots of pretrained ConvNets
https://github.com/BVLC/caffe/wiki/Model-Zoo

...
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 7 20 Jan 2016
Things you should know for your Project Proposal

“We have infinite


compute available
because Terminal.”

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 8 20 Jan 2016
Things you should know for your Project Proposal

“We have infinite


compute available
because Terminal.”
You have finite compute.
Don’t be overly ambitious.
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 9 20 Jan 2016
Where we are now...

Mini-batch SGD
Loop:
1. Sample a batch of data
2. Forward prop it through the graph, get loss
3. Backprop to calculate the gradients
4. Update the parameters using the gradient

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 10 20 Jan 2016
Where we are now...

(image credits
to Alec Radford)

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 11 20 Jan 2016
Neural Turing Machine

input tape

loss

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 12 20 Jan 2016
activations

“local gradient”

gradients

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 13 20 Jan 2016
Implementation: forward/backward API
Graph (or Net) object. (Rough psuedo code)

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 14 20 Jan 2016
Implementation: forward/backward API

x
z
*
y

(x,y,z are scalars)

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 15 20 Jan 2016
Example: Torch Layers

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 16 20 Jan 2016
Neural Network: without the brain stuff

(Before) Linear score function:

(Now) 2-layer Neural Network


or 3-layer Neural Network

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 17 20 Jan 2016
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 18 20 Jan 2016
Neural Networks: Architectures

“3-layer Neural Net”, or


“2-layer Neural Net”, or “2-hidden-layer Neural Net”
“1-hidden-layer Neural Net” “Fully-connected” layers

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 19 20 Jan 2016
Training Neural Networks
A bit of history...

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 5 - 20 20 Jan 2016

You might also like