SEN 503/339 Artificial Intelligence

Lec 5.4: Regression Python Examples

Halûk Gümüşkaya
Professor of Computer Engineering

web: http://www.gumuskaya.com
e-mail: [email protected], [email protected]

: https://tr.linkedin.com/in/halukgumuskaya
: https://www.facebook.com/2haluk.gumuskaya

Regression Python Examples

1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 2

 The Normal Equation
 To find the value of θ that minimizes the cost function, there is
a closed-form solution in other words, a mathematical equation
that gives the result directly.

 This is called the Normal Equation.

Haluk Gümüşkaya @ www.gumuskaya.com 3

Generate Data
 Let’s generate some linear-looking data to test this equation.

Haluk Gümüşkaya @ www.gumuskaya.com 4

 Compute θ using the Normal Equation
 Use the inv() function from NumPy’s linear algebra module
(np.linalg) to compute the inverse of a matrix, and the
dot() method for matrix multiplication:

Let’s see what the equation found:

 We would have hoped for θ0 = 4 and θ1 = 3 instead of θ0 = 4.215

and θ1 = 2.770.
 Close enough, but the noise made it impossible to recover the exact
parameters of the original function.
Haluk Gümüşkaya @ www.gumuskaya.com 5

Make Predictions using θ:

Let’s plot this model’s predictions

Haluk Gümüşkaya @ www.gumuskaya.com 6

 The figure with a legend and axis labels:

Haluk Gümüşkaya @ www.gumuskaya.com 7

Linear Regression using Scikit-Learn

 Performing Linear Regression using Scikit-Learn is simple:

Haluk Gümüşkaya @ www.gumuskaya.com 8

 Linear Regression using Scikit-Learn
 The LinearRegression class is based on the
scipy.linalg.lstsq() function (the name stands for
“least squares”), which you could call directly:

Haluk Gümüşkaya @ www.gumuskaya.com 9

Regression Python Examples

1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient
Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 10

Gradient Decent for More than One Feature
This was for
one feature J(θ)
j

x1(i)

More than 2 features

update rules

hθ(x) = θ T X

Haluk Gümüşkaya @ www.gumuskaya.com 11

Batch Gradient Descent

You need to calculate how much the cost function will change if you
change θj just a little bit. This is called a partial derivative.

Partial derivatives of
the cost function

Instead of computing these partial derivatives individually,

you can use
This formula involves calculations over the
full training set X, at each Gradient
Descent step!

This is why the algorithm is called Batch

Gradient Descent: it uses the whole batch
of training data at every step (actually, Full
Gradient Descent would probably
be a better name). As a result it is terribly
slow on very large training sets
Haluk Gümüşkaya @ www.gumuskaya.com 12
 Batch Gradient Descent
Once you have the gradient vector, which points uphill, just go in the
opposite direction to go downhill.

This means subtracting ∇θ MSE(θ) from θ. This is where the

learning rate η comes into play: multiply the gradient vector by η to
determine the size of the downhill step:

Gradient Descent step

Haluk Gümüşkaya @ www.gumuskaya.com 13

Linear Regression using Batch Gradient Descent

Haluk Gümüşkaya @ www.gumuskaya.com 14

 Theta found

 That’s exactly what the Normal Equation found! Gradient Descent

worked perfectly.

Haluk Gümüşkaya @ www.gumuskaya.com 15

Gradient Descent with various Learning Rates

Haluk Gümüşkaya @ www.gumuskaya.com 16

 Regression Python Examples
1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 17

Batch Gradient Descent vs

Stochastic Gradient Descent (SGD)
 The main problem with Batch Gradient Descent is the fact
that it uses the whole training set to compute the gradients at
every step, which makes it very slow when the training set is
large.
 At the opposite extreme, Stochastic Gradient Descent (SGD)
picks a random instance in the training set at every step and
computes the gradients based only on that single instance.
 Obviously, working on a single instance at a time makes the
algorithm much faster because it has very little data to
manipulate at every iteration.
 It also makes it possible to train on huge training sets, since
only one instance needs to be in memory at each iteration.

Haluk Gümüşkaya @ www.gumuskaya.com 18

 An Example Code for SGD

Look at the jupyter notebook file for the full implementation:

04_training_linear_models.ipynb
Haluk Gümüşkaya @ www.gumuskaya.com 19

Iterations and Epoch

 By convention we iterate by rounds of m iterations; each round is
called an epoch.
 While the Batch Gradient Descent code iterated 1,000 times through
the whole training set, this code goes through the training set only
50 times and reaches a pretty good solution:

Haluk Gümüşkaya @ www.gumuskaya.com 20

The first 20 steps of Stochastic Gradient Descent

Notice how
irregular the
steps are.

 Note that since instances are picked randomly, some instances may be
picked several times per epoch, while others may not be picked at all.
 If you want to be sure that the algorithm goes through every instance at
each epoch, another approach is to shuffle the training set (making sure to
shuffle the input features and the labels jointly), then go through it instance
by instance, then shuffle it again, and so on.
 However, this approach generally converges more slowly.
Haluk Gümüşkaya @ www.gumuskaya.com 21

Linear Regression using SGD with Scikit-Learn

 Once again, you find a solution quite close to the one returned
by the Normal Equation:

Haluk Gümüşkaya @ www.gumuskaya.com 22

 Regression Python Examples
1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 23

Mini-batch Gradient Descent

 Mini-batch GD computes the gradients on small random sets
of instances called mini-batches.
 The main advantage of Mini-batch GD over Stochastic GD is
that you can get a performance boost from hardware
optimization of matrix operations, especially when using
GPUs.
 The algorithm’s progress in parameter space is less erratic
than with Stochastic GD, especially with fairly large mini-
batches.
 As a result, Mini-batch GD will end up walking around a bit
closer to the minimum than Stochastic GD—but it may be
harder for it to escape from local minima (in the case of
problems that suffer from local minima, unlike Linear
Regression).
Haluk Gümüşkaya @ www.gumuskaya.com 24
 Gradient Descent Paths in Parameter Space

 They all end up near the minimum, but Batch GD’s path actually
stops at the minimum, while both Stochastic
 GD and Mini-batch GD continue to walk around.
 However, don’t forget that Batch GD takes a lot of time to take each
step, and Stochastic GD and Mini-batch GD would also reach the
minimum if you used a good learning schedule.
Haluk Gümüşkaya @ www.gumuskaya.com 25

Comparison of Algorithms for Linear Regression

m is the number of training instances and n is the number of features

 There is almost no difference after training: all these

algorithms end up with very similar models and make
predictions in exactly the same way.

Haluk Gümüşkaya @ www.gumuskaya.com 26

 Regression Python Examples
1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 27

Polynomial Regression
 What if your data is more complex than a straight line?
 Surprisingly, you can use a linear model to fit nonlinear data.
 A simple way to do this is to add powers of each feature as
new features, then train a linear model on this extended set of
features.
 This technique is called Polynomial Regression.

 Let’s look at an example. First, let’s generate some nonlinear

data, based on a simple quadratic equation (plus some noise).

Haluk Gümüşkaya @ www.gumuskaya.com 28

 Generate Some Nonlinear Data

Haluk Gümüşkaya @ www.gumuskaya.com 29

PolynomialFeatures Class
 Use Scikit-Learn’s PolynomialFeatures class to transform our
training data, adding the square (second-degree polynomial) of each
feature in the training set as a new feature (in this case there is just
one feature):

X_poly now contains the original feature of X

plus the square of this feature.

fit a LinearRegression model to this extended

training data

Haluk Gümüşkaya @ www.gumuskaya.com 30

 Fit a LinearRegression Model

 Not bad: the model estimates:

 The original function was:

Haluk Gümüşkaya @ www.gumuskaya.com 31

Note about PolynomialFeatures

Haluk Gümüşkaya @ www.gumuskaya.com 32

 High-degree Polynomial Regression

severely overfitting

The model that will

generalize best

underfitting

Haluk Gümüşkaya @ www.gumuskaya.com 33

Cross-Validation
 You used cross-validation to get an estimate of a model’s
generalization performance.
 If a model performs well on the training data but generalizes
poorly according to the cross-validation metrics, then your
model is overfitting.
 If it performs poorly on both, then it is underfitting.
 This is one way to tell when a model is too simple or too
complex.

 Another way to tell is to look at the learning curves.

Haluk Gümüşkaya @ www.gumuskaya.com 34

 Regression Python Examples
1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 35

Learning Curves
 These are plots of the model’s performance on the training set
and the validation set as a function of the training set size (or
the training iteration).

 To generate the plots, train the model several times on

different sized subsets of the training set.

 The following code defines a function that, given some training

data, plots the learning curves of a model:

Haluk Gümüşkaya @ www.gumuskaya.com 36

 Learning Curves: Simplified Code

Haluk Gümüşkaya @ www.gumuskaya.com 37

Learning Curves: Underfitting

 These learning curves are typical of a model that’s underfitting.

 Both curves have reached a plateau; they are close and fairly high.
 If your model is underfitting the training data, adding more training
examples will not help. You need to use a more complex model or
come up with better features.
Haluk Gümüşkaya @ www.gumuskaya.com 38
 Learning Curves: Overfitting

Haluk Gümüşkaya @ www.gumuskaya.com 39

Learning Curves: Overfitting

These learning curves look a bit like the previous ones, but there
are 2 very important:
 The error on the training data is much lower than with the
Linear Regression model.
 There is a gap between the curves. This means that the model
performs significantly better on the training data than on the
validation data, which is the hallmark of an overfitting model. If
you used a much larger training set, however, the two curves
would continue to get closer.

 One way to improve an overfitting model is to feed it more

training data until the validation error reaches the training error.

Haluk Gümüşkaya @ www.gumuskaya.com 40

 Regression Python Examples
1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 41

Regularization
 A good way to reduce overfitting is to regularize the model
(i.e., to constrain it): the fewer degrees of freedom it has, the
harder it will be for it to overfit the data.
 A simple way to regularize a polynomial model is to reduce the
number of polynomial degrees.
 For a linear model, regularization is typically achieved by
constraining the weights of the model.
 We will now look at Ridge Regression, Lasso Regression, and
Elastic Net, which implement 3 different ways to constrain the
weights.

Haluk Gümüşkaya @ www.gumuskaya.com 42

 Ridge Regression
 A regularized version of Linear Regression: a regularization
term equal to is added to the cost function.

 This forces the learning algorithm to not only fit the data but
also keep the model weights as small as possible.

 The regularization term should only be added to the cost

function during training.

 Once the model is trained, you want to use the unregularized

performance measure to evaluate the model’s performance.

Haluk Gümüşkaya @ www.gumuskaya.com 43

Ridge Regression Cost Function

 Itis important to scale the data (e.g., using a

StandardScaler) before performing Ridge Regression, as it
is sensitive to the scale of the input features. This is true of
most regularized models.

Haluk Gümüşkaya @ www.gumuskaya.com 44

Several Ridge Models Trained on Some Linear Data

A linear model (left) and a polynomial model (right), both with various
levels of Ridge regularization

Haluk Gümüşkaya @ www.gumuskaya.com 45

Ridge Regression Implementation: Closed-form

 As with Linear Regression, we can perform Ridge Regression either
by computing a closed-form equation or by performing Gradient
Descent.

Ridge Regression
closed-form solution
(n+1) x (n+1)
 Here is how to perform Ridge Regression with Scikit-Learn using a
closed-form solution (a variant of the above equation that uses a
matrix factorization technique by André-Louis Cholesky):

Haluk Gümüşkaya @ www.gumuskaya.com 46

Ridge Regression Implementation: Gradient Decent

Haluk Gümüşkaya @ www.gumuskaya.com 47

Lasso Regression
 Another regularized version of Linear Regression.
 Just like Ridge Regression, it adds a regularization term to the cost
function, but it uses the ℓ1 norm of the weight vector instead of half
the square of the ℓ2 norm.
The same thing as before, but replaces Ridge models
with Lasso models and uses smaller α values.

A linear model (left) and a polynomial model (right), both using various levels of
Lasso regularization
Haluk Gümüşkaya @ www.gumuskaya.com 48
Important Characteristic of Lasso Regression
 It tends to eliminate the weights of the least important features
(i.e., set them to zero).
 For example, the dashed line in the righthand plot in the the
figure (with α = ) looks quadratic, almost linear:
 All the weights for the high-degree polynomial features are
equal to zero.
 In other words, Lasso Regression automatically performs
feature selection and outputs a sparse model (i.e., with few
nonzero feature weights).

Haluk Gümüşkaya @ www.gumuskaya.com 49

Lasso Cost Function and Gradient Decents

 The Lasso cost function is not differentiable at θi = 0 (for i = 1, 2, ⋯, n),
but Gradient Descent still works fine if you use a subgradient vector
instead when any θi = 0.
 The equation below shows a subgradient vector equation you can use
for Gradient Descent with the Lasso cost function.

Lasso Regression
subgradient vector

Haluk Gümüşkaya @ www.gumuskaya.com 50

 Elastic Net
 ElasticNet is a middle ground between Ridge Regression and
Lasso Regression.
 The regularization term is a simple mix of both Ridge and
Lasso’s regularization terms, you can control the mix ratio r.
 When r = 0, Elastic Net is equivalent to Ridge Regression, and
when r = 1, it is equivalent to Lasso Regression.

l1_ratio corresponds to
the mix ratio r):

Haluk Gümüşkaya @ www.gumuskaya.com 51

Early Stopping
 A very different way to regularize iterative learning algorithms
such as Gradient Descent is to stop training as soon as the
validation error reaches a minimum.
 This is called early stopping.

Haluk Gümüşkaya @ www.gumuskaya.com 52

 Example
 A complex model (in this case, a high-degree Polynomial
Regression model) being trained with Batch Gradient Descent.
 As the epochs go by the algorithm learns, and its prediction
error (RMSE) on the training set goes down, along with its
prediction error on the validation set.
 After a while though, the validation error stops decreasing and
starts to go back up.
 This indicates that the model has started to overfit the training
data.
 With early stopping you just stop training as soon as the
validation error reaches the minimum.
 It is such a simple and efficient regularization technique that
Geoffrey Hinton called it a “beautiful free lunch.”

Haluk Gümüşkaya @ www.gumuskaya.com 53

A Basic Implementation of Early Stopping

Haluk Gümüşkaya @ www.gumuskaya.com 54

 Regression Python Examples
1. Linear Regression using the Normal Equation
2. Linear Regression using Batch Gradient Descent
3. Stochastic Gradient Descent
4. Mini-batch Gradient Descent
5. Polynomial Regression
6. Learning Curves
7. Regularized Linear Models
8. Logistic Regression

Haluk Gümüşkaya @ www.gumuskaya.com 55

Logistic Regression: Binary classifier

 Commonly used to estimate the probability that an instance
belongs to a particular class (e.g., what is the probability that
this email is spam?).
 If the estimated probability is greater than 50%, then the model
predicts that the instance belongs to that class (called the
positive class, labeled “1”), and otherwise it predicts that it
does not (i.e., it belongs to the negative class, labeled “0”).
 This makes it a binary classifier.

Haluk Gümüşkaya @ www.gumuskaya.com 56

 Estimating Probabilities
 Just like a Linear Regression model, a Logistic Regression model
computes a weighted sum of the input features (plus a bias term),
but instead of outputting the result directly like the Linear Regression
model does, it outputs the logistic of this result:

Logistic Regression model estimated

probability (vectorized form)
The logistic—noted σ(・)—is a sigmoid function (i.e., S-shaped) that
outputs a number between 0 and 1.

Logistic Function
Haluk Gümüşkaya @ www.gumuskaya.com 57

Logistic Regression Model Prediction

 Once the Logistic Regression model has estimated the probability p
= hθ(x) that an instance x belongs to the positive class, it can make
its prediction ŷ easily.

Haluk Gümüşkaya @ www.gumuskaya.com 58

 Training and Cost Function
Cost function of a single training instance

The cost function over the whole training set is the average cost
over all training instances. It can be written in a single expression
called the log loss, shown in:

Haluk Gümüşkaya @ www.gumuskaya.com 59

Bad and good News for Cost Function

 The bad news: There is no known closed-form equation to
compute the value of θ that minimizes this cost function (there is
no equivalent of the Normal Equation).
 The good new: This cost function is convex, so Gradient Descent
(or any other optimization algorithm) is guaranteed to find the
global minimum (if the learning rate is not too large and you wait
long enough).
Logistic cost function
partial derivatives

• Once you have the gradient vector containing all the partial derivatives,
you can use it in the Batch Gradient Descent algorithm. That’s it: You
now know how to train a Logistic Regression model.
• For Stochastic GD you would take one instance at a time, and
• For Mini-batch GD you would use a minibatch at a time.

Haluk Gümüşkaya @ www.gumuskaya.com 60

 Logistic Regression Code

Haluk Gümüşkaya @ www.gumuskaya.com 61

Decision Boundaries: Iris Dataset

 This is a famous dataset that contains the sepal and petal
length and width of 150 iris flowers of three different species:
Iris setosa, Iris versicolor, and Iris virginica.

Flowers of three iris plant species

Haluk Gümüşkaya @ www.gumuskaya.com 62
 Load Iris Dataset and Print its Description

Haluk Gümüşkaya @ www.gumuskaya.com 63

Training a Logistic Regression Model

Now let’s train a Logistic Regression model:

Haluk Gümüşkaya @ www.gumuskaya.com 64

 Simple Plot

Haluk Gümüşkaya @ www.gumuskaya.com 65

Fancier Plot
Estimated probabilities and decision boundary

• The petal width of Iris virginica flowers (represented by triangles) ranges from 1.4 cm
to 2.5 cm, while the other iris flowers (represented by squares) generally have a
smaller petal width, ranging from 0.1 cm to 1.8 cm.
• Notice that there is a bit of overlap.
• Above about 2 cm the classifier is highly confident that the flower is an Iris virginica (it
outputs a high probability for that class), while below 1 cm it is highly confident that it
is not an Iris virginica (high probability for the “Not Iris virginica” class).
Haluk Gümüşkaya @ www.gumuskaya.com 66
 Linear Decision Boundary
 The same dataset, displaying 2 features: petal width and length.
 Once trained, the Logistic Regression classifier can, based on these
2 features, estimate the probability that a new flower is an Iris
virginica.
 The dashed line represents the points where the model estimates a
50% probability: this is the model’s decision boundary.
 Note that it is a linear boundary.***

***
It is the the set of points x such that θ0 + θ1x1 + θ2x2 = 0, which defines a straight line.
Haluk Gümüşkaya @ www.gumuskaya.com 67

Regularized Logistic Regression

 Each parallel line represents the points where the model outputs a
specific probability, from 15% (bottom left) to 90% (top right).
 All the flowers beyond the top-right line have an over 90% chance of
being Iris virginica, according to the model.

Regularized Logistic Regression

 Just like the other linear models, Logistic Regression models can be
regularized using ℓ1 or ℓ2 penalties. Scikit-Learn actually adds an ℓ2
penalty by default.

 The hyperparameter controlling the regularization strength of a

Scikit-Learn LogisticRegression model is not alpha (as in other
linear models), but its inverse: C. The higher the value of C, the less
the model is regularized.

Haluk Gümüşkaya @ www.gumuskaya.com 68

 References
All examples from our main practical book:
 Hands-On Machine Learning with Scikit-Learn, Keras and
TensorFlow, Aurélien Geron, O’Reilly, 2019.

Other References:
 Fit a Linear Regression Model with Gradient Descent from Scratch,
Chris I., Dec 2019,
https://towardsdatascience.com/fit-a-linear-regression-model-with-
gradient-descent-from-scratch-d9bb41bc821e

Haluk Gümüşkaya @ www.gumuskaya.com 69