Scribd
Upload
14 views17 pages

Curve Fitting

The document discusses curve fitting and regularization in machine learning, focusing on polynomial curve fitting. It highlights the importance of balancing model complexity to avoid overfitting, where higher-order polynomials may fit noise rather than the underlying data. Regularization techniques are introduced to discourage large coefficients and improve generalization performance.

Uploaded by

Omama Arshad
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views17 pages

Curve Fitting

The document discusses curve fitting and regularization in machine learning, focusing on polynomial curve fitting. It highlights the importance of balancing model complexity to avoid overfitting, where higher-order polynomials may fit noise rather than the underlying data. Regularization techniques are introduced to discourage large coefficients and improve generalization performance.

Uploaded by

Omama Arshad
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 17

EC-332 Machine Learning

Nazar Khan

Department of Computer Science


University of the Punjab

Curve Fitting and Regularization


Curve Fitting Regularized Curve Fitting

Example: Polynomial Curve Fitting


Problem : Given N observations of input xi with corresponding observations of
output ti , nd function f (x ) that predicts t for a new value of x .

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Real-world Data
Real-world data has 2 important properties
1. underlying regularity,

2. individual observations are corrupted by noise.

Learning corresponds to discovering the underlying regularity of data (the


sin(·) function in our example).
Nazar Khan Machine Learning
Curve Fitting Regularized Curve Fitting

Polynomial curve tting


I We will t the points (x , t ) using a polynomial function
M
y (x , w) = w0 + w1 x + w2 x 2 + · · · + wM x M = wj x j
X

j =0
where M is the order of the polynomial.
I Function y (x , w) is a
I non-linear function of the input x , but
I a linear function of the parameters w.
I So our model y (x , w) is a linear model.

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Polynomial curve tting


I Fitting corresponds to nding the optimal w. We denote it as w∗ .
I Optimal w∗ can be found by minimising an error function

1X N
E (w) = {y (xn , w) − tn }2
2 n=1

I Why does minimising E (w) make sense?


I Can E (w) ever be negative?
I Can E (w) ever be zero?

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Geometric interpratation of the sum-of-squares error function.

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Over-tting
I Lower order polynomials can't capture the variation in data.
I Higher order leads to over-tting.
I Fitted polynomial passes exactly through each data point.
I But it oscillates wildly in-between.
I Gives a very poor representation of the real underlying function.
I Over-tting is bad because it gives bad generalization.

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Over-tting
I To check generalization performance of a certain w∗ , compute E (w∗ ) on
a new test set.
I Alternative performance measure: root-mean-square error (RMS)

2 E (w ∗ )
r
ERMS =
N
I Mean ensures datasets of dierent sizes are treated equally. (How?)
I Square-root brings the squared error scale back to the scale of the target
variable t .

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Root-mean-square error on training and test set for various polynomial orders
M.

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Paradox?
I A polynomial of order M contains all polynomials of lower order.
I So higher order should always be better than lower order.
I But, it's not better. Why?
I Because higher order polynomial starts tting the noise instead of the
underlying function.

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Over-tting

I Typical magnitude of the polynomial coecients is increasing dramatically


as M increases.
I This is a sign of over-tting.
I The polynomial is trying to t the data points exactly by having larger
coecients.
Nazar Khan Machine Learning
Curve Fitting Regularized Curve Fitting

Over-tting
I Large M =⇒ more exibility =⇒ more tuning to noise.
I But, if we have more data, then over-tting is reduced.

I Fitted polynomials of order M = 9 with N = 15 and N = 100 data points.


More data reduces the eect of over-tting.
I Rough heuristic to avoid over-tting: Number of data points should be
greater than k |w| where k is some multiple like 5 or 10.
Nazar Khan Machine Learning
Curve Fitting Regularized Curve Fitting

How to avoid over-tting


I Since large coecients =⇒ over-tting, discourage large coecents in
w.

1X N λ
Ẽ (w) = {y (xn , w) − tn }2 + ||w||2
2 n=1 2
where ||w||2 = wT w = w02 + w12 + · · · + wM2 and λ controls the relative
importance of the regularizer compared to the error term.
I Also called regularization, shrinkage, weight-decay.

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

For a polynomial of order 9

For λ = e −18 For λ = 1


No over-tting Too much smoothing (no tting)

Nazar Khan Machine Learning


Curve Fitting Regularized Curve Fitting

Eect of regularization

I As λ increases, the typical magnitude of coecients gets smaller.


I We go from over-tting (λ = 0) to no over-tting (λ = e −18 ) to poor
tting (λ = 1).
I Since M = 9 is xed, regularization controls the degree of over-tting.
Nazar Khan Machine Learning
Curve Fitting Regularized Curve Fitting

Eect of regularization

Graph of root-mean-square (RMS) error of tting the M = 9 polynomial as λ


is increased.

Nazar Khan Machine Learning

You might also like