Introduction to

Machine Learning

Slides of Dr. Humayoun, Prof. Dr. Andrew Ng, Stanford

Advice for applying machine learning

Deciding what to try next

2
Debugging a learning algorithm:
Suppose you have implemented regularized linear regression to predict housing
prices.

However, when you test your hypothesis on a new set of houses, you find that it
makes unacceptably large errors in its predictions. What should you try next?

- Get more training examples

- Try smaller sets of features
- do it by hand or use some dimensionality reduction technique?
- Try getting additional features
- Try adding polynomial features Aprox. 4-6 month time can be lost
- Try decreasing on every bullet if not careful
- Try increasing
3
Machine learning diagnostic:
Diagnostic: A test that you can run to gain insight what
is/isn’t working with a learning algorithm,
and gain insight for the improvement of its performance.

Diagnostics can take time to implement, but doing so

can be a very good use of your time.

4
Evaluating a hypothesis

5
Evaluating your hypothesis
Try to minimize the training error
Fitting parameters to training data
doesn’t always work
price

Fails to generalize to new

examples not in training set.
size of house
no. of bedrooms
size no. of floors
age of house
average income in neighborhood
kitchen size

6
Standard way to evaluate a hypothesis
Dataset:
Size Price
2104 400
1600 330
2400 369
1416 232
3000 540
1985 300
1534 315
1427 199
1380 212
1494 243
7
Training/testing procedure for linear regression

- Learn parameter from training data (minimizing

training error )

- Compute test set error:

- Average squared error as measured on test set

8
Training/testing procedure for logistic regression
- Learn parameter from training data
- Compute test set error:

- Misclassification error (0/1 misclassification error)[Alternative]:

Its the fraction in the test set the hypothesis mislabels 9

 Model selection and
training/validation/test sets
Model Selection: How to chose degree of polynomial or
regularization parameter

About three sets: training, validation and test

10
 Overfitting example

Once parameters
price

were fit to some set of data

(training set), the error of the
parameters as measured on
size that data (the training error
xxxxx) is likely to be lower
than the actual generalization
error.
Training set error is a poor indictor of hypothesis accuracy for
new data (generalization) 11
Model selection
1.
2.
3.

10.
Choose
How well does the model generalize? Report test set
error .
Problem: is likely to be an optimistic estimate of
generalization error. i.e. our extra parameter ( = degree of
polynomial) is fit to test set. So not a good way to evaluate if it will generalize 12
Improved model selection
Dataset:
Size Price
2104 400
1600 330
2400 369
1416 232
3000 540
1985 300
1534 315
1427 199
1380 212
1494 243
13
Train/validation/test error
Training error:

Cross Validation error:

Test error:

14
Model selection
1.
2.
3.

10.

Pick
Estimate generalization error for test set

15
 Train/validation/test sets
• In machine learning as practiced today
– Many people will select the model using the test set
and then check the model is OK for generalization using
the test error
– We've said is bad because it gives a bias analysis
• With a MASSIVE test set this is maybe OK
• But considered much better practice to have
separate training and validation sets
Diagnosing bias vs. variance
Underfitting vs. overfitting

17
 •
Bias/variance • One of the main reasons for getting bad results
Important to work out if it is bias or variance
• Knowing it will help you to improve the algorithm
Price

Price

Price
Size Size Size

High bias “Just right” High variance

(underfit) (overfit)

18
Bias/variance
Training error:

Cross validation error:

19
Bias/variance
Training error:

Cross validation error:

error

degree of polynomial d
20
 Diagnosing bias vs. variance
Suppose your learning algorithm is performing less well than
you were hoping. ( or is high.) Is it a bias
problem or a variance problem?
Bias (underfit):
(cross validation
error

error)

Variance (overfit):
(training error)

degree of polynomial d
21
 Diagnosing bias vs. variance
Suppose your learning algorithm is performing less well than
you were hoping. ( or is high.) Is it a bias
problem or a variance problem?
Bias (underfit):
(cross validation
error

error)

Variance (overfit):
(training error)

degree of polynomial d
22
 Diagnosing bias vs. variance
Suppose your learning algorithm is performing less well than
you were hoping. ( or is high.) Is it a bias
problem or a variance problem?
Bias (underfit):
(cross validation
will be high
error

error)

Variance (overfit):
(training error)
will be low
degree of polynomial d
23
Neural networks and overfitting
“Small” neural network “Large” neural network
(fewer parameters; more (more parameters; more prone
prone to underfitting) to overfitting)

Computationally cheaper Computationally more expensive.

Use regularization ( ) to address overfitting.

24
End

25