NEURAL NETWORKS, DATA

CLASSIFICATION AND MNIST

JEFFREY L. POPYACK
 THE PROBLEM: RECOGNITION /
CLASSIFICATION

• Classification (the simple version):

• Given a collection of data {x0, x1, …, xn-1},
where each datum can be classified as one of a set of possible
values
{y0, y1, …, yM-1},
• Create an algorithm that will
• classify each sample item correctly
• learn features from the samples that can be applied correctly to new
items
 LEARNING FROM EXAMPLES

• General Learning:
• Given a collection of sample data {x0, x1, …, xn-1},
where each datum can be classified as one of a set of possible
values
{y0, y1, …, yM-1}:
• Create an algorithm that will
• classify each sample item correctly
• learn features from the samples that can be applied correctly to new
items
• Examples :
• Given a set of emails, which we have designated as spam or not
spam,
learn to classify future items correctly as spam or not spam.
• Given a collection of handwritten digits, learn to classify
handwritten digits.
 LEARNING FROM EXAMPLES

• General approach:
“Ground truth”: data we will use that has already been classified
(correctly, we hope). Training data + Testing data
“Training data”: Used to train your algorithm.
• Need to know the answers in order to train the algorithm correctly.
• Hope/expect that nearly all will be categorized correctly by
algorithm.
“Testing data”: Used to test how well your algorithm works on data that
was not used in the training.
• Pretend not to know the answers, use answers to determine if
correct.
• Examples:
• Spam detection:
use last month’s emails to train your algorithm; use this week’s to test
your algorithm.
• Handwritten digits: with a set of 500 samples, use 400 to train, 100 to test.
 LEARNING FROM EXAMPLES

• General approach, continued:

• Use results of testing to tweak your algorithm
• Try again
• Repeat…
• When satisfied with results, ready to try with new, unclassified data
• Issues:
• How will your algorithm differ with different choices of training,
testing data?
• What to do with examples for which algorithm fails?
• Can results from several variations of the basic algorithm be used
together?
THE MNIST DATABASE OF HANDWRITTEN DIGITS

NIST (National Institute of Standards and Technology):

• Idea: Train a learning algorithm to recognize handwritten
digits, using samples from Census Bureau employees
(adult professionals who spent a lot of time writing
numbers, expected to be legible). Test algorithm on
data collected from less reliable sources.
• Ground Truth: images have been labeled with correct
values
• Collected binary images of handwritten digits
• Training data:
• Special Database 3 (SD-3): collected from Census Bureau employees.
• Testing data:
• Special Database 1 (SD-1): collected from high-school students. What could
• 58,527 digital images written by 500 different writers. possibly go
wrong?
THE MNIST DATABASE OF HANDWRITTEN DIGITS

MNIST (Modified NIST database):

• Yann LeCun, Courant Institute, NYU
Corinna Cortes, Google Labs, New York
Christopher J.C. Burges, Microsoft Research, Redmond
• Modified, normalized to 28x28 pixel images
• Mixture of original training/testing data:
• Training Set: 30,000 patterns each from SD-3 and SD-1 (from ~250 writers)
• Testing Set:
• 60,000 patterns from SD-3 and SD-1 (~250 writers, different from Training Set
writers)
• Typical benchmark Testing Set is 5,000 patterns each from SD-3 and SD-1

Website contains 60,000 element Training and 10,000

element Test set.
http://yann.lecun.com/exdb/mnist/
THE MNIST DATABASE OF HANDWRITTEN DIGITS

• Training set: 60,000 examples

Test set: 10,000 examples
• Digits have been size-normalized
and centered, fixed size.
• “It is a good database for people
who want to try learning techniques
and pattern recognition methods on
real-world data while spending
minimal efforts on preprocessing and
formatting.”
http://yann.lecun.com/exdb/mnist/

Image: http://neuralnetworksanddeeplearning.com/chap1.html
 NEURAL NETWORKS AND DEEP LEARNING

• Based on Neural Networks and Deep Learning

• By Michael Nielsen
• http://neuralnetworksanddeeplearning.com/
 SAMPLE NEURAL NETWORK

Three-Layer Neural Network:

• Input Layer:
28x28 pixels = 784 input values
0.0=black, 1.0=white, 0.xxx = greyscale
• Hidden Layer:
15 neurons
• Output Layer:
10 neurons, one for each digit
0-9.
Whichever has highest value is
the selected output.

Image: http://neuralnetworksanddeeplearning.com/chap1.html
 SAMPLE NEURAL NETWORK

Sample Results:
Training for 30 epochs, learning rate 3.0
>>> net = network.Network([784, 15, 10])

Epoch 0: 8870 / 10000 Inputs

Outputs
Epoch 1: 9094 / 10000
Hidden
Epoch 2: 9112 / 10000
. . .
Epoch 27: 9275 / 10000
Epoch 28: 9283 / 10000
Epoch 29: 9257 / 10000
92.6% accuracy

Can we do better with more hidden units?

Image: http://neuralnetworksanddeeplearning.com/chap1.html
 SAMPLE NEURAL NETWORK

Sample Results:
Training for 30 epochs, learning rate 3.0
>>> net = network.Network([784, 30, 10])

Epoch 0: 9057 / 10000 Inputs

Outputs
Epoch 1: 9222 / 10000
Hidden
Epoch 2: 9259 / 10000
. . .
Epoch 27: 9462 / 10000
Epoch 28: 9482 / 10000
Epoch 29: 9482 / 10000
94.8% accuracy

Can we do better with more hidden layers?

Image: http://neuralnetworksanddeeplearning.com/chap1.html
 SAMPLE NEURAL NETWORK

Sample Results: Sample Results:

Training for 30 epochs, learning rate 3.0 Training for 30 epochs, learning rate 3.0
>>> net = network.Network([784, 30, 10]) >>> net = network.Network([784,196,36,10])

Epoch 0: 9057 / 10000 Epoch 0: 9098 / 10000

Epoch 1: 9222 / 10000 Epoch 1: 9165 / 10000
Epoch 2: 9259 / 10000 Epoch 2: 9368 / 10000
Hidden Hidden
. . . . . . Layer Layer
1 2
Epoch 27: 9462 / 10000 Epoch 26: 9628 / 10000
Epoch 28: 9482 / 10000 Epoch 27: 9675 / 10000
Epoch 29: 9482 / 10000 Epoch 28: 9650 / 10000
94.8% accuracy 96.5% accuracy

Can we do better with more hidden layers?

Image: http://neuralnetworksanddeeplearning.com/chap1.html
 SAMPLE NEURAL NETWORK

Sample Results: Sample Results:

Training for 30 epochs, learning rate 3.0 Training for 30 epochs, learning rate 3.0
>>> net = network.Network([784, 30, 10]) >>> net = network.Network([784,196,36,10])

Epoch 0: 9057 / 10000 Epoch 0: 9098 / 10000

Epoch 1: 9222 / 10000 Inputs Epoch 1: 9165 / 10000
Outputs
Hidden Hidden
Epoch 2: 9259 / 10000 Hidden Epoch 2: 9368 / 10000
Layer Layer
. . . . . . 1 2
Epoch 27: 9462 / 10000 Epoch 26: 9628 / 10000
Epoch 28: 9482 / 10000 Epoch 27: 9675 / 10000
Epoch 29: 9482 / 10000 Epoch 28: 9650 / 10000

94.8% accuracy 96.5% accuracy

Can we do better with more hidden layers?

 SAMPLE NEURAL NETWORK

Why not 100% accuracy?

• Model is not detailed enough to handle intricacies of data set.
• Some sample data values too hard to recognize – even for humans:
 DEEP LEARNING
CONVOLUTIONAL NEURAL NETWORKS

Some basic concepts:

• Not beneficial/meaningful to have all input units
connected to all hidden units
• Let a hidden unit be assigned to a subsection of
the original data, extracting features of that
subsection (convolutional layer, feature map)
• Follow each convolutional layer with a pooling convolutional layer
layer, that simplifies/summarizes the features in
the convolutional layer (e.g., “was a particular
feature found somewhere?”)
• This means the exact position of the object in
pooling layer
the frame is not as important.

http://neuralnetworksanddeeplearning.com/chap6.html
DEEP LEARNING EXAMPLE
Recognizes as “5”

https://www.cs.ryerson.ca/~aharley/vis/conv/flat.html