Department of Electrical Engineering

Faculty Member: LE Munadi Sial Date:

Semester: Group:

CS471 Machine Learning

Lab 8: Logistic Regression

PLO4 - PLO4 - PLO5 - PLO8 - PLO9 -

CLO4 CLO4 CLO5 CLO6 CLO7
Name Reg. No Viva /Quiz / Analysis Modern Ethics Individual
Lab of data in Tool and Team
Performance Lab Usage Work
Report

5 Marks 5 Marks 5 Marks 5 Marks 5 Marks

Machine Learning
Introduction

This laboratory exercise will focus on the python implementation of logistic

regression. Logistic regression is a supervised learning technique that
incorporates a sigmoid function activation with the linear regression algorithm
to implement classification. Unlike regression, classification involves discrete
labels such 0/1, true/false, cat/not a cat, benign/malignant etc. The sigmoid
function causes the hypothesis values to take place between 0 or 1. Similar to
regression, weight parameters are trained on a dataset so as to fit a model that
can make accurate predictions from that dataset. To prevent overfitting,
regularization can be used in logistic regression.

Objectives

The following are the main objectives of this lab:

• Extract and prepare the training and validation datasets

• Use feature scaling to ensure uniformity among the feature columns
• Implement the sigmoid activation function
• Implement cost function to get the overall loss
• Implement gradient descent algorithm to update weight parameters
• Perform binary classification using logistic regression
• Plot the training and validation losses
• Make scatter plots of the classification

Lab Conduct

• Respect faculty and peers through speech and actions

• The lab faculty will be available to assist the students. In case some aspect
of the lab experiment is not understood, the students are advised to seek
help from the faculty.
• In the tasks, there are commented lines such as #YOUR CODE STARTS
HERE# where you have to provide the code. You must put the

Machine Learning
 code/screenshot/plot between the #START and #END parts of these
commented lines. Do NOT remove the commented lines.
• Use the tab key to provide the indentation in python.
• When you provide the code in the report, keep the font size at 12

Theory
Logistic Regression is another basic supervised learning technique besides
Linear Regression. In logistic regression, the linear regression algorithm is
modified by applying a sigmoid function to the predicted value. This causes the
prediction to fall between 0 to 1 values. Thus, logistic regression is actually a
classification technique built from the linear regression. The sigmoid function is
a type of an activation function. Aside from loss and accuracy, logistic
regression also at times, requires calculation of precision and recall. This is
needed when the dataset is skewed; the two class labels are not equally
distributed in the dataset.

A brief summary of the relevant keywords and functions in python is provided

below:

print() output text on console

input() get input from user on console
range() create a sequence of numbers
len() gives the number of characters in a string
if contains code that executes depending on a logical condition
else connects with if and elif, executes when conditions are not met
elif equivalent to else if
while loops code as long as a condition is true
for loops code through a sequence of items in an iterable object
break exit loop immediately
continue jump to the next iteration of the loop
def used to define a function

Machine Learning
Lab Task 1 - Dataset Preparation, Feature Scaling ______________________
Download a dataset containing several columns. You will need to select at least
3 of the feature columns to make your own dataset. Also, you will need to
choose the label column that your model will predict. Ensure that the label
column has discrete, binary values. Specify the features and the label that you
choose. You will need to attach your dataset in the submission.

The dataset examples are to be divided into 2 separate portions: training and
validation datasets (choose from 80-20 to 70-30 ratios). Load the dataset into
your python program, split them into the portions and store them as NumPy
arrays (Xtrain , ytrain, Xval, yval,). Next, use feature scaling to rescale the feature
columns for both datasets so that their values range from 0 to 1. Finally, print
both of the datasets (you need to show any 5 rows of the datasets).

### TASK 1 CODE STARTS HERE ###

### TASK 1 CODE ENDS HERE ###

### TASK 1 SCREENSHOT STARTS HERE ###

### TASK 1 SCREENSHOT ENDS HERE ###

Lab Task 2 – Sigmoid Activation ____________________________________________

For logistic regression, you will implement the following hypothesis:
h(x) = g(b + w1x1 + w2x2 + w3x3 + …)
g(z) = 1 / (1 + e-z)
The w represents the weights, the b represents the bias and the x represent the
features. h(x) is to be calculated for each training example and its difference with
the label y of that training example will represent the loss. The g(z) function

Machine Learning
 represents the sigmoid activation function. In this task, you will write a function
that takes in a value z as argument and outputs the result of the sigmoid
activation g(z). Provide the code for this task. For the screenshot, provide the
plot of the sigmoid activation function.

### TASK 2 CODE STARTS HERE ###

### TASK 2 CODE ENDS HERE ###

### TASK 2 SCREENSHOT STARTS HERE ###

### TASK 2 SCREENSHOT ENDS HERE ###

Lab Task 3 - Cost Function __________________________________________________

In this task, you will write a cost function that calculates the overall loss across
a set of training examples. This cost function will be useful to calculate the
losses in both the training and validation phases of the program.

cost_function(X, y)

The X and y are the features and labels of the training/validation dataset. The
function will return the cost value. For binary classification, use the given cost
function:
m−1
1
J (w)= ∑ ¿ ¿
m i=0

The m is the number of the training/validation examples in the dataset.

Remember that the hypothesis requires sigmoid activation. Write the code for
the cost function and implement it to print out the cost. You will need to
initialize the weights to some random values in order to calculate the

Machine Learning
 hypothesis. Provide the code and all relevant screenshots showcasing the use of
your cost function.

### TASK 3 CODE STARTS HERE ###

### TASK 3 CODE ENDS HERE ###

### TASK 3 SCREENSHOT STARTS HERE ###

### TASK 3 SCREENSHOT ENDS HERE ###

Lab Task 4 – Gradient Descent ______________________________________________

In this task, you will write a function that uses gradient descent to update the
weight parameters:

gradient_descent(X, y, alpha)

The X and y are the features and labels of the training dataset, alpha is the
learning rate which is a tuning hyperparameter. The gradient descent algorithm
is given as follows:
m
∂J 1
d w j= = ∑ (h( x (i ))– y (i )) x j(i)
∂ w j m i=1

m
∂J 1
db= = ∑ (h(x ( i)) – y (i) )
∂ b m i=1

∂J
w j :=w j−α
∂wj

∂J
b :=b−α
∂wj

Machine Learning
 The gradient descent for logistic regression may seem identical to that in linear
regression, however, it should be noted that they are not the same formulas as
the cost function for the logistic regression is different from that used in linear
regression.

For the submission, you will need to run the gradient descent algorithm once to
update the weights. You will need to print the weights, training cost and
validation cost both before and after the weight update. Provide the code and
all relevant screenshots of the final output.

### TASK 4 CODE STARTS HERE ###

### TASK 4 CODE ENDS HERE ###

### TASK 4 SCREENSHOT STARTS HERE ###

### TASK 4 SCREENSHOT ENDS HERE ###

Lab Task 5 – Training and Validation Implementation _________________

In this task, you will use the functions from the previous two tasks to write a
“main” function that performs the actual training and validation. Use the cost
function and gradient descent function on the training examples to determine
the training loss and update the weights. Then, use the cost function on the
validation examples to determine the validation loss. This single iteration over
the entire dataset (both training and validation) marks completion of one
epoch. You will need to perform the training and validation over several epochs
(the epoch number is another hyperparameter that must be chosen). Ensure
that at the end of each epoch, the training loss, validation loss, precision value
and recall value are stored for plotting purposes. To get the precision and recall,
use the following:

Machine Learning
 TP
Precision=
TP+ FP

TP
Recall=
TP+ FN

Start the training at some value of alpha. Try multiple training attempts with
various alpha values and find the best value of the step size. Once you have
found the value for the step size, showcase the output by making three plots:

1. training loss and validation loss vs. the epoch number

2. precision and recall vs. the epoch number
3. precision vs. recall

Ensure all axes are labeled appropriately. Provide the code (excluding function
definitions) and all plots of the final output.

### TASK 5 CODE STARTS HERE ###

### TASK 5 CODE ENDS HERE ###

### TASK 5 PLOT 1 STARTS HERE ###

### TASK 5 PLOT 1 ENDS HERE ###

### TASK 5 PLOT 2 STARTS HERE ###

### TASK 5 PLOT 2 ENDS HERE ###

### TASK 5 PLOT 3 STARTS HERE ###

### TASK 5 PLOT 3 ENDS HERE ###

Machine Learning
Lab Task 6 – Prediction and Scatter Plot ________________________________
Save the weights that fit the best model and use them to create a prediction
function. The prediction function will take the features as input and output the
predicted class of the label. To convert the output to 0 or 1, a threshold of 0.5
needs to be applied to the predicted value h(x). Call your prediction function by
giving it some input values to make at least three predictions. Print your
predictions and take a screenshot. Additionally, your program must make a
scatter plot showing the training and validation examples. The coordinates in
the scatter plot correspond to the inputs (x). The class is denoted by:

 red “o” for zero class (training and validation examples)

 blue “x” for one class (training and validation examples)
 black “O” for zero class (user prediction)
 black “X” for one class (user prediction)

Provide the code and screenshot of it being used.

### TASK 6 CODE STARTS HERE ###

### TASK 6 CODE ENDS HERE ###

### TASK 6 SCREENSHOT STARTS HERE ###

### TASK 6 SCREENSHOT ENDS HERE ###

Machine Learning