Logistic Regression

Part 1

Manoj Kumar

Youtube

18th October, 2024

Manoj Kumar Youtube

Logistic Regression 1 / 36
Given data

Blood Pressure Level (mm Hg) Diabetes Status

80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic

1: Blood Pressure Levels and Diabetes Status

Manoj Kumar Youtube

Logistic Regression 2 / 36
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Why is Linear Regression not good for classification?

• The output ŷ can be outside of the label range {0, 1}.

• What does a response of value of -2 mean?
Possible classification: assign ŷ > 0.5 to 1, and 0 otherwise.
• Boundary very sensitive to outliers.

Manoj Kumar Youtube

Logistic Regression 4 / 36
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Blood Diabetes
Pressure Status
Level
(mm Hg)
80 Not Diabetic
90 Not Diabetic
100 Not Diabetic
110 Not Diabetic
120 Diabetic
130 Diabetic
140 Diabetic
150 Diabetic
Sigmoid/Logistic Function

Manoj Kumar Youtube

Logistic Regression 6 / 36
Sigmoid/Logistic Function

Sigmoid Function

σ(a)

0.5

Manoj Kumar Youtube

Logistic Regression 6 / 36
Sigmoid/Logistic Function

• Bounded:

1
Sigmoid Function σ(a) = ∈ (0, 1)
1 + exp(−a)
σ(a)

0.5

Manoj Kumar Youtube

Logistic Regression 6 / 36
Sigmoid/Logistic Function

• Bounded:

1
Sigmoid Function σ(a) = ∈ (0, 1)
1 + exp(−a)
σ(a)
• Symmetric:

exp(−a) 1
1 − σ(a) = = = σ(−a)
0.5 1 + exp(−a) exp(a) + 1

Manoj Kumar Youtube

Logistic Regression 6 / 36
Sigmoid/Logistic Function

• Bounded:

1
Sigmoid Function σ(a) = ∈ (0, 1)
1 + exp(−a)
σ(a)
• Symmetric:

exp(−a) 1
1 − σ(a) = = = σ(−a)
0.5 1 + exp(−a) exp(a) + 1

• Gradient:

exp(−a)
a σ ′ (a) = = σ(a)(1 − σ(a))
(1 + exp(−a))2

Manoj Kumar Youtube

Logistic Regression 6 / 36
Interpretation of the Sigmoid Function

Manoj Kumar Youtube

Logistic Regression 7 / 36
Interpretation of the Sigmoid Function

Note

For a given input x , the probability that the

Sigmoid Function
class label Y equals 1 given x , is represented
by: 1
σ(x )
1 0.8
P(Y = 1|x ) =
1 + e −(w0 +w1 x )
0.6

• P(Y = 0|x ) + P(Y = 1|x ) = 1 0.4

• P(Y = 0|x ) = 1 − 1
1+e −(w0 +w1 x ) 0.2
x
By default, we take threshold = 0.5: −5 5
• Y = 1 if P(Y = 1|x ) ≥ 0.5
• Y = 0 if P(Y = 1|x ) < 0.5

Manoj Kumar Youtube

Logistic Regression 7 / 36
Question
 
3
Given a logistic regression model with a parameter vector θ = 4, and a test data point
1
 
1
x = 1, what is the probability of Y = 1 for the given data point x?
1
Question

Suppose that you have trained a logistic regression classifier hθ (x ) = σ(1 − x ) where σ(·) is the
logistic/sigmoid function. What does its output on a new example x = 2 mean? Check all that
apply.
• □ Your estimate for P(y = 1|x ; θ) is about 0.73.
• □ Your estimate for P(y = 0|x ; θ) is about 0.27.
• □ Your estimate for P(y = 1|x ; θ) is about 0.27.
• □ Your estimate for P(y = 0|x ; θ) is about 0.73.
Question

Consider the sigmoid function f (x ) = 1

1+e −x
. The derivative f ′ (x ) is:
• f (x ) ln f (x ) + (1 − f (x )) ln(1 − f (x ))
• f (x )(1 − f (x ))
• f (x ) ln(1 − f (x ))
• f (x )(1 + f (x ))
Question

Let σ(a) = 13 . Using the properties of sigmoid function, calculate the value of the expression:
σ ′ (−a), where ′ represents derivative.
2
1 9
2 − 29
1
3 9
4 − 19
Question

Q3-2: Which of the following statement is true about outliers in Linear regression?
1 Linear regression is sensitive to outliers
2 Linear regression is NOT sensitive to outliers
3 Can’t say
4 None of these
Question

Suppose we have trained a logistic regression classifier for a binary classification task. The table
below provides the true labels y and the predicted probabilities P(Y = 1 | x ) for a set of data
points. We want to evaluate the accuracy of the classifier for the following thresholds:
• Model A: T = 0.25
• Model B: T = 0.5
• Model C: T = 0.75
Calculate the accuracy for each model and determine which threshold results in the highest
accuracy.

True Label y P(Y = 1 | x )

1 0.9
1 0.6
0 0.6
0 0.55
1 0.4
0 0.3
Decision Boundary (Using Logistic)

Manoj Kumar Youtube

Logistic Regression 14 / 36
Decision Boundary (Using Logistic)
• Y = 1 if
P(Y = 1|X ) ≥ 0.5
• For 1 input feature (X ):
z = w0 + w1 X

1 1
≥
1 + e −z 2

1 + e −(w0 +w1 X ) ≤ 2

w0 + w1 X ≥ 0
Y = 1 if X ≥ − ww10

Y =0 Y =1

Decision Boundary

Manoj Kumar Youtube

Logistic Regression 14 / 36
• For 2 input features (X1 , X2 ):
z = w0 + w1 X1 + w2 X2

P(Y = 1|X1 , X2 ) ≥ 0.5

1
≥ 0.5
1 + e −z
Decision Boundary

Y = 1 if w0 + w1 X1 + w2 X2 ≥ 0

X2
Y=1

Y=0

X1
Question

Suppose you train a logistic regression classifier and the learned hypothesis function is:

hθ (x ) = σ(θ0 + θ1 x1 + θ2 x2 ),
where θ0 = 6, θ1 = 0, θ2 = −1. Which of the following represents the decision boundary for
hθ (x )?

A B C D
10 x2 10 x2 10 x2 10 x2
8 y =1 8 y =1 y =0 8 y =0 y =1 8 y =0
6 6 6 6
4 y =0 4 4 4 y =1
2 x1 2 x1 2 x1 2 x1
2 4 6 8 10 2 4 6 8 10 2 4 6 8 10 2 4 6 8 10