AIL 7310: MACHINE LEARNING FOR ECONOMICS

Lecture 4

8th August, 2023

AIL 7310: ML for Econ Lecture 4 1/9

Assessing Model Accuracy

There is no universal statistical learning technique that dominates others.

The norm in ML literature is to try out several ML methods and find the
one with the highest model accuracy.

AIL 7310: ML for Econ Lecture 4 2/9

Assessing Model Accuracy

There is no universal statistical learning technique that dominates others.

The norm in ML literature is to try out several ML methods and find the
one with the highest model accuracy.

How do we do it?

AIL 7310: ML for Econ Lecture 4 2/9

Assessing Model Accuracy

There is no universal statistical learning technique that dominates others.

The norm in ML literature is to try out several ML methods and find the
one with the highest model accuracy.

How do we do it?

We measure model accuracy using several ways of comparing how close

the predicted values are to the actual observed data.

AIL 7310: ML for Econ Lecture 4 2/9

Assessing Model Accuracy
In a regression setting,the most commonly used measure is mean-squared
error(MSE).
n
1X
MSE = (yi − fˆ(x))2
n
i=1

where fˆ(x) is the prediction that fˆ gives for the ith observations.

AIL 7310: ML for Econ Lecture 4 3/9

Assessing Model Accuracy
In a regression setting,the most commonly used measure is mean-squared
error(MSE).
n
1X
MSE = (yi − fˆ(x))2
n
i=1

where fˆ(x) is the prediction that fˆ gives for the ith observations.

The MSE will be small if the predicted responses are very close to the true
responses, and will be large if for some of the observations, the predicted
and true responses differ substantially.

AIL 7310: ML for Econ Lecture 4 3/9

Assessing Model Accuracy
In a regression setting,the most commonly used measure is mean-squared
error(MSE).
n
1X
MSE = (yi − fˆ(x))2
n
i=1

where fˆ(x) is the prediction that fˆ gives for the ith observations.

The MSE will be small if the predicted responses are very close to the true
responses, and will be large if for some of the observations, the predicted
and true responses differ substantially.

In general, we care about how accurately our model predicts on previously

unseen data - mimic reality.

AIL 7310: ML for Econ Lecture 4 3/9

Assessing Model Accuracy
In a regression setting,the most commonly used measure is mean-squared
error(MSE).
n
1X
MSE = (yi − fˆ(x))2
n
i=1

where fˆ(x) is the prediction that fˆ gives for the ith observations.

The MSE will be small if the predicted responses are very close to the true
responses, and will be large if for some of the observations, the predicted
and true responses differ substantially.

In general, we care about how accurately our model predicts on previously

unseen data - mimic reality.

This is the rationale behind dividing the data into training or test or
conducting cross-validation. To assess model accuracy we calculate MSE
on test data.
AIL 7310: ML for Econ Lecture 4 3/9
Assessing Model Accuracy

AIL 7310: ML for Econ Lecture 4 4/9

Assessing Model Accuracy

In the previous figure, as the flexibility of the statistical learning method

increases, we observe a monotone decrease in the training MSE and a
U-shape in the test MSE. This is a fundamental property of statistical
learning that holds regardless of the particular data set at hand and
regardless of the statistical method being used.

AIL 7310: ML for Econ Lecture 4 5/9

Assessing Model Accuracy

In the previous figure, as the flexibility of the statistical learning method

increases, we observe a monotone decrease in the training MSE and a
U-shape in the test MSE. This is a fundamental property of statistical
learning that holds regardless of the particular data set at hand and
regardless of the statistical method being used.

As model flexibility increases, training MSE will decrease, but the test MSE
may not. When a given method yields a small training MSE but a large
test MSE, we might be overfitting the data. This happens because our
statistical learning procedure is working too hard to find patterns in the
training data, and may be picking up some patterns that are just caused by
random chance rather than by true properties of the unknown function f.

AIL 7310: ML for Econ Lecture 4 5/9

Assessing Model Accuracy

When we overfit the training data, the test MSE will be very large because
the supposed patterns that the method found in the training data simply
don’t exist in the test data. Note that regardless of whether or not
overfitting has occurred, we almost always expect the training MSE to be
smaller than the test MSE because most statistical learning methods either
directly or indirectly seek to minimize the training MSE.

AIL 7310: ML for Econ Lecture 4 6/9

Assessing Model Accuracy

When we overfit the training data, the test MSE will be very large because
the supposed patterns that the method found in the training data simply
don’t exist in the test data. Note that regardless of whether or not
overfitting has occurred, we almost always expect the training MSE to be
smaller than the test MSE because most statistical learning methods either
directly or indirectly seek to minimize the training MSE.

Overfitting refers specifically to the case in which a less flexible model

would have yielded a smaller test MSE.

AIL 7310: ML for Econ Lecture 4 6/9

Assessing Model Accuracy

In the classification setting, model accuracy is assessed using the

misclassification error rate.
n
1X
I (yi ̸= yˆi )
n
i=1

Here I (yi ̸= yˆi ) is an indicator variable which equals 1 if yi ̸= yˆi and 0 if

yi = yˆi

AIL 7310: ML for Econ Lecture 4 7/9

Assessing Model Accuracy

In the classification setting, model accuracy is assessed using the

misclassification error rate.
n
1X
I (yi ̸= yˆi )
n
i=1

Here I (yi ̸= yˆi ) is an indicator variable which equals 1 if yi ̸= yˆi and 0 if

yi = yˆi

A good classifier is one for which this error is smallest on test data.

AIL 7310: ML for Econ Lecture 4 7/9

Supervised Vs Unsupervised Algorithms

Most traditional ML algorithms fall into 2 categories: Supervised Learning

and Unsupervised Learning.

AIL 7310: ML for Econ Lecture 4 8/9

Supervised Vs Unsupervised Algorithms

Most traditional ML algorithms fall into 2 categories: Supervised Learning

and Unsupervised Learning.

In supervised learning, for each observation xi = 1, 2, ..., n there is an

associated response measurement yi . We fit a model that relates the
response to the predictor with either the aim of accurately predicting the
response for unseen observations(prediction), or better understanding the
relationship between the response and the predictors (inference).

AIL 7310: ML for Econ Lecture 4 8/9

Supervised Vs Unsupervised Algorithms

In contrast, in unsupervised learning for every observation i = 1, ..., n, we

observe a vector of measurements xi but no associated response yi .The
situation is referred to as unsupervised because we lack a response variable
that can supervise our analysis.

AIL 7310: ML for Econ Lecture 4 9/9

Supervised Vs Unsupervised Algorithms

In contrast, in unsupervised learning for every observation i = 1, ..., n, we

observe a vector of measurements xi but no associated response yi .The
situation is referred to as unsupervised because we lack a response variable
that can supervise our analysis.

We can seek to understand the relationships between the variables or

between the observations. One statistical learning tool that we may use in
this setting is cluster analysis, or clustering. The goal of cluster analysis is
to ascertain, on the basis of x1 , ..., xn , whether the observations fall into
analysis relatively distinct groups.

AIL 7310: ML for Econ Lecture 4 9/9