Scribd
Upload
3 views29 pages

Issues in Backpropagation Algorithm

Uploaded by

meghana pandita
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
3 views29 pages

Issues in Backpropagation Algorithm

Uploaded by

meghana pandita
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 29

Remarks on Backpropagation

Algorithm
Common heuristic to attempt to
alleviate the problem of local
minima include:
Representation Power of
FeedFarward Networks
Hypothesis Space Search and
Inductive Bias
Hidden Layer Representations
• One intriguing property of Backpropagation ability to discover useful
intermediate representation at the hidden unit layers inside the
network.
Generalization, Overfitting, and
Stopping Criterion
Advance Topics In ANN
Alternative Error Minimization
Procedure
c) Recurrent network
Dynamically Modify Network
Structure
Example: Face
Recognition
●
From Machine Learning by Tom M. Mitchell
● Input: 30 by 32 pictures of people with the
following properties:
 Wearing eyeglasses or not
 Facial expression: happy, sad, angry,
neutral
 Direction in which they are looking: left, right, up,
● straight ahead
Output: Determine which category it fits into
for
one of these properties (we will talk about
direction)
Input
Encoding
●
Each pixel is an input
 30*32 = 960 inputs
●
The value of the pixel (0 – 255) is linearly mapped
onto the range of reals between 0 and 1
Output
Encoding
●
Could use a single output node with the
classifications assigned to 4 values (e.g. 0.2, 0.4,
0.6, and 0.8)
●
Instead, use 4 output nodes (one for each value)
 1-of-N output encoding
 Provides more degrees of freedom to the network
●
Use values of 0.1 and 0.9 instead of 0 and 1
The sigmoid function can never reach 0 or 1!
● Example: (0.9, 0.1, 0.1, 0.1) = left, (0.1, 0.9, 0.1,
0.1) = right, etc.
Network structure
Inputs
3 Hidden Units
3 Hidden Units
x1

x2

. 3 Hidden Units
Outputs
.
.

x960
Other Parameters
•●training rate:η = 0.3 momentum: α = 0.3
•●Used full gradient descent (as opposed to stochastic)
•●Weights in the output units were initialized to
•small random variables, but input weights were initialized to 0

 Yields better visualizations


●
Result: 90% accuracy on test set!
Try it
yourself!
●
Get the code from
http://www.cs.cmu.edu/~tom/mlbook.html
 Go to the Software and Data page, then
follow the
“Neural network learning to recognize faces”
● link
 Follow the documentation
You can also copy the code and data from my
ACM account (provide you have one too),
although you will want a fresh copy of facetrain.c
and imagenet.c from the website
 /afs/acm.uiuc.edu/user/jcander1/Public/
NeuralNetwork

You might also like