Scribd
Upload
12 views3 pages

Notes On Logistic Regression

Logistic regression

Uploaded by

And i
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views3 pages

Notes On Logistic Regression

Logistic regression

Uploaded by

And i
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Logistic Regression

The Logistic Regression is classified under classification algorithms. For classification we need answer
in terms of yes or no i.e. 1 or 0. For example, whether a tumor is benign or malign.

We cannot use linear regression here as it is not suitable for this problem so we use a special function
called the Sigmoid function or the Logistic function.

The Sigmoid function


The output of the sigmoid function is always between 0 and 1. The function is,
1
𝑔(𝑧) = , 0 < 𝑔(𝑧) < 1
1 + 𝑒 −𝑧

The Logistic regression model


Simply the model can be defined as:

⃗⃗ ,𝑏 (𝑥 ) = 𝑧 = 𝑤
𝑓𝑤 ⃗⃗ • 𝑥 + 𝑏
1
𝑔(𝑧) =
1 + 𝑒 −𝑧
1
⟹ 𝑔(𝑤
⃗⃗ • 𝑥 + 𝑏) = ⃗⃗ •𝑥+𝑏)
1+ 𝑒 −(𝑤
This can be interpreted as the probability that y = 1 or the output is equal to 1.

Cost function for Logistic regression


1. The Logistic loss function:
(𝑖)
− log (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) 𝑖𝑓 𝑦 (𝑖) = 1
(𝑖) (𝑖)
𝐿 (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = {
(𝑖)
− log (1 − 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) 𝑖𝑓 𝑦 (𝑖) = 0

(𝑖) (𝑖)
As 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 1 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → 0 and as 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 0 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → ∞

(𝑖) (𝑖)
Loss is lowest when 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) predicts close to true label 𝑦 .

(𝑖) (𝑖)
As 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 0 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → 0 and as 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) → 1 𝑡ℎ𝑒𝑛 𝑙𝑜𝑠𝑠 → ∞

(𝑖) (𝑖)
The further prediction 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ) is from target 𝑦 , the higher the loss.

Simplified loss function


The loss function can simply be written in just one single expression i.e.
(𝑖) (𝑖) (𝑖) (𝑖) (𝑖) (𝑖)
𝐿 (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = − 𝑦 log (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) − (1− 𝑦 ) log (1 − 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ))
If 𝑦 (𝑖) = 1:
(𝑖) (𝑖)
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = − log(𝑓(𝑥 ))
𝐿 (𝑓𝑤

If 𝑦 (𝑖) = 0 :
(𝑖) (𝑖)
⃗⃗ ,𝑏 (𝑥 , 𝑦 )) = − log(1 − 𝑓(𝑥 ))
𝐿 (𝑓𝑤

The cost function


The cost function is simply the average of all the loss functions, i.e. for every feature.
𝑚
1 (𝑖) (𝑖)
𝐽(𝑤
⃗⃗ , 𝑏) = ∑ [𝐿 (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 , 𝑦 ))]
𝑚
𝑖=1
𝑚
−1 (𝑖) (𝑖) (𝑖)
= ∑ [𝑦 (𝑖) log (𝑓𝑤
⃗⃗ ,𝑏 (𝑥 )) + (1− 𝑦 ) log (1 − 𝑓𝑤
⃗⃗ ,𝑏 (𝑥 ))]
𝑚
𝑖=1

You might also like