Scribd
Upload
39 views43 pages

MTH408 Machine - Learning - Logistic - Regression

This document discusses logistic regression and regularization. It covers topics like overfitting, underfitting and penalizing parameters. Examples are provided for linear regression, classification and multiclass classification. Formulas and notation for logistic regression models are also presented.

Uploaded by

junzi2000
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
39 views43 pages

MTH408 Machine - Learning - Logistic - Regression

This document discusses logistic regression and regularization. It covers topics like overfitting, underfitting and penalizing parameters. Examples are provided for linear regression, classification and multiclass classification. Formulas and notation for logistic regression models are also presented.

Uploaded by

junzi2000
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 43

Acknowledgement: The lecture series are produced based on the

original contributions by Dr. Ahmet Concu (previously DFAM,XJTLU).


We thank him for his work.

1
Dr. David Liu
Department of Financial and Actuarial Mathematics
School of Science, Xi’an Jiaotong-Liverpool University

2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
x0 ⋯ x1
x= ⋮ ⋱ ⋮
x0 ⋯ x𝑚

Note: x0 = 1 because a
constant term is required.

18
ln
ln

Note: The above result is consistent with the negative log likelihood of the problem. This turns out to be a
convex function, so minimization can be achieved.
19
20
21
22
23
24
%% % Logistic regression using CostFunction.m
clc
clear all
load Commodity.mat
closep = Close(:,1); % the first column is the soybean prices.
LogRet = log( closep(2:end) ./ closep(1:end-1) ); % calculate log-returns
tmp1 = LogRet(6:end,:) ; % the day's return if we count from 6
Y = zeros(length(tmp1),1);
tmp2 = find( tmp1 > 0 ); % find positive returns
Y(tmp2) = 1;
ONEs = ones(length(Y),1); % for the constant term
X = [ones(length(Y),1), tmp1];
[rr,cc]=size(X);
initial_theta = zeros(cc, 1);
options = optimset('GradObj', 'on', 'MaxIter', 400);
[theta, cost] = fminunc(@(t)(costFunction(t, X, Y)), initial_theta, options)Ahmet

25
26
Example: Linear regression

Price ($) Price ($) Price ($)


in 1000’s in 1000’s in 1000’s

Size in feet^2 Size in feet^2 Size in feet^2

ℎ𝜃 𝑥 = 𝜃0 + 𝜃1 𝑥 ℎ𝜃 𝑥 = 𝜃0 + 𝜃1 𝑥 + 𝜃2 𝑥 2 ℎ𝜃 𝑥 = 𝜃0 + 𝜃1 𝑥 + 𝜃2 𝑥 2 +
𝜃3 𝑥 3 + 𝜃4 𝑥 4 + ⋯
Under-fitting Just right Overfitting

27
• If we have too many features (i.e. complex model), the learned hypothesis
may fit the training set very well
𝑚
1 𝑖 𝑖 2
𝐽 𝜃 = ෍ ℎ𝜃 𝑥 −𝑦 ≈0
2𝑚
𝑖=1

but fail to generalize to new examples (predict prices on new examples).

28
Price ($) Price ($)
in 1000’s in 1000’s

Size in feet^2 Size in feet^2

ℎ𝜃 𝑥 = 𝜃0 + 𝜃1 𝑥 + 𝜃2 𝑥 2 ℎ𝜃 𝑥 = 𝜃0 + 𝜃1 𝑥 + 𝜃2 𝑥 2 + 𝜃3 𝑥 3 + 𝜃4 𝑥 4

• Suppose we penalize and make 𝜃3 , 𝜃4 really small.


𝑚
1 𝑖 2
min 𝐽 𝜃 = ෍ ℎ𝜃 𝑥 𝑖
−𝑦 + 1000 𝜃32 + 1000 𝜃42
𝜃 2𝑚
𝑖=1
29
Example: classification

Age Age Age

Tumor Size Tumor Size Tumor Size


ℎ𝜃 𝑥 = 𝑔(𝜃0 + 𝜃1 𝑥 + 𝜃2 𝑥2 ) ℎ𝜃 𝑥 = 𝑔(𝜃0 + 𝜃1 𝑥 + 𝜃2 𝑥2 + ℎ𝜃 𝑥 = 𝑔(𝜃0 + 𝜃1 𝑥 + 𝜃2 𝑥2 +
𝜃3 𝑥12 + 𝜃4 𝑥22 + 𝜃5 𝑥1 𝑥2 ) 𝜃3 𝑥12 + 𝜃4 𝑥22 + 𝜃5 𝑥1 𝑥2 +
𝜃6 𝑥13 𝑥2 + 𝜃7 𝑥1 𝑥23 + ⋯ )

Under-fitting Overfitting
30
31
32
33
34
Binary classification Multiclass classification

𝑥2 𝑥2

𝑥1 𝑥1
35
𝑥2
ℎ𝜃1 𝑥
𝑥1
𝑥2
2 𝑥2
ℎ𝜃 𝑥

𝑥1 𝑥1
Class 1:
Class 2: 3 𝑥2
Class 3: ℎ𝜃 𝑥

𝑖
ℎ𝜃 𝑥 = 𝑃 𝑦 = 𝑖 𝑥; 𝜃 (𝑖 = 1, 2, 3) 36
𝑥1
37
38
39
40
:,

41
42
43

You might also like