Supervised machine learning:

regression and classification

Lê Ngọc Quang Hưng

Contents
1. Introduction ........................................................................................................................ 1
2. The need for machine learning ............................................................................................ 1
3. Linear regression ................................................................................................................. 1
3.1. Cost function ................................................................................................................ 2
3.2. Gradient descent algorithm .......................................................................................... 2
3.3. Multiple linear regression ............................................................................................. 4
4. Feature engineering ............................................................................................................. 4
5. Polynomial regression ......................................................................................................... 5

1. Introduction
This report will tackle the basics of supervised machine learning, mainly about regression
problems. It delves into fundamental concepts such as linear regression, cost function,
gradient descent, feature scaling and feature engineering.

2. The need for machine learning

While any programmer can design a program to handle different scenarios of a problem by
using tools like conditional statements, many problems such as housing price prediction,
tumor detection, image recognition are very hard to solve using those tools alone. Take image
recognition for example, for a small image with a resolution of 32x32, each pixel in the grid
can have a different color represented by a combination of 3 three numbers with each number
ranging from 0 to 255 which adds up to about 17 billion combinations that the programmer
has to account for, even more, these standards will always change with time as technology
evolve, so there arise a need for a solution that grow and adapt with the change in the real
world, hence, the need for machine to learn from examples and grow with time. The main
concept behind machine learning is that given a feature or an input 𝑥, the model needs to
output an estimated result 𝑦̂ as close to the desired result 𝑦 as possible. In supervised
machine learning, the set of input 𝑋 and desired result 𝑌 has already been given beforehand
while unsupervised machine learning only gets access to the set of input 𝑋. This report will
be about supervised machine learning and addressing one of the two main types of supervised
learning, which is regression.

3. Linear regression
The goal of machine learning is to find a model 𝑓(𝑥) that can roughly fit the training data
and from that model, the machine can predict future outcomes when it encounters data it
has never seen before. A regression problem requires the machine to predict a value from an
infinite amount of choices. The simplest method to tackle regression problems is called linear
regression. In linear regression, this model take the form of:

1
 𝑓𝑤,𝑏 (𝑥(𝑖) ) = 𝑤 ∗ 𝑥(𝑖) + 𝑏
𝑤 and 𝑏 are called the weight and bias of the model, while the 𝑖 in 𝑥(𝑖) denotes the 𝑖th
training example.

3.1. Cost function

In order for the model to be able to predict future results accurately, the parameters 𝑤 and 𝑏
need to be tuned to fit the training data. One way of choosing the appropriate parameters is
using a cost function to evaluate the performance of the model, which calculates the
difference between the output 𝑦̂ and the actual result 𝑦. The smaller the cost function, the
better the model fits the data. One of the more popular cost function is the Mean Square
Error function:
2
𝐽 (𝑤, 𝑏) = ( 𝑛1 ) ∗ ∑𝑛𝑖=1 (𝑓𝑤,𝑏 (𝑥(𝑖) ) − 𝑦(𝑖) )
The cost function enables the machine to learn from training data, by trying to minimize the
value of the function through the tuning of parameters 𝑤 and 𝑏, the machine can arrive at a
model that can predict future outcomes when it is given some data.

3.2. Gradient descent algorithm

Gradient descent is a widely used algorithm for minimizing the cost function. In linear
regression, the cost function is the average sum of the square of the “loss” in training, which
means the plot of the function is convex like a soup bowl.

2
 Figure 1: The plot of the cost function 𝐽 (𝑤, 𝑏)
Since the cost function is convex, there exists at most one global minimum. In linear
regression, the minimum can be achieved by starting at a random point (𝑤, 𝑏) and calculate
𝜕 𝜕
the respective partial derivative 𝜕𝑤 𝐽 (𝑤, 𝑏) and 𝜕𝑏 𝐽 (𝑤, 𝑏) at that point and nudge 𝑤 and 𝑏
toward a value where the direction of the slope at that point is going down, which means the
function is heading toward the global minimum. Repeat this step until the cost function is
around the area of the global minimum or the partial derivatives are approaching 0. Here is
how 𝑤 and 𝑏 are calculated at each step:

𝜕
𝑤 = 𝑤 − 𝛼 𝜕𝑤 𝐽 (𝑤, 𝑏) = 𝑤 − 𝛼 𝑛1 ∑𝑛𝑖=1 (𝑓𝑤,𝑏 (𝑥(𝑖) − 𝑦(𝑖) )𝑥(𝑖)
𝜕
𝑏 = 𝑏 − 𝛼 𝜕𝑏 𝐽 (𝑤, 𝑏) = 𝑏 − 𝛼 𝑛1 ∑𝑛𝑖=1 (𝑓𝑤,𝑏 (𝑥(𝑖) − 𝑦(𝑖) )
The 𝛼 here is called a hyperparameter or the learning rate of the model, which is set before
the training starts. Notice that 𝛼 needs to be chosen carefully, if 𝛼 is too small, the model
will take a longer time to train, on the contrary, having an 𝛼 too big and the model will

3
actually get worse overtime as it makes the algorithm overshoots the minimum again and
again.

Figure 2: The plot of J(w,b) when 𝛼 is a big number

The cost function should decrease after each iteration of gradient descent, this means the
algorithm is working as intended and the chosen 𝛼 is appropriate and it can still be increased
to improve the performance of the algorithm.

3.3. Multiple linear regression

For some input 𝑋, the training data can contain multiple features, in which case, the input
can be represented as a vector of features and the linear regression model can be rewritten as:

𝑓𝑤,𝑏 (𝑥)⃗ = 𝑤⃗ ⋅ 𝑥⃗ + 𝑏
By using the vector to represent training data, the machine can use the GPU parallel
computing to efficiently calculate the dot product of 𝑤⃗ and 𝑥⃗ simultaneously.
For gradient descent implementation, there is a slight different with multiple linear
regression, where every features should have its own gradient descent process:

𝜕
𝑤𝑗 = 𝑤𝑗 − 𝛼 𝜕𝑤 𝐽 (𝑤, 𝑏) = 𝑤𝑗 − 𝛼 𝑛1 ∑𝑛𝑖=1 (𝑓𝑤,𝑏 (𝑥(𝑖) −
𝑗
(𝑖)
𝑦(𝑖) )𝑥𝑗 for 𝑗 = 0..(𝑛 − 1)
𝜕
𝑏 = 𝑏 − 𝛼 𝜕𝑏 𝐽 (𝑤, 𝑏) = 𝑏 − 𝛼 𝑛1 ∑𝑛𝑖=1 (𝑓𝑤,𝑏 (𝑥(𝑖) − 𝑦(𝑖) )
4. Feature engineering
Feature engineering is a crucial part of optimization in machine learning. Feature engineering
is the concept of organizing and processing data before training so that the final input X fit
the model best. One key concept of feature engineering is feature scaling, where features are
scaled so that all features have a similar impact on the final state of the model. For example,
a housing price input has 2 features, the size of the house in 𝑚2 ranging from 500 to 2000
and the number of bedrooms ranging from 1 to 5. In this case, the value of the first feature is

4
much higher than the value of the second one, this leads to an uneven influence on the model
where only a little change in the size of the house is already greater than any change in the
number of bedrooms. The model will try to compensate by choosing the appropriate weight
𝑤1 and 𝑤2 so that their influence cancel out, leading to a “skinny” contour plot of 𝑤1 when
compared to 𝑤2 and cause the gradient descent algorithm to run slower since it has a longer
path to go before reaching the global minimum.

Figure 3: The contour plot of 𝑤1 to 𝑤2

One way to implement feature scaling is Mean Normalization:

𝑥−𝜇
𝑥rescaled = max(𝑥)− min(𝑥)
where 𝜇 is the average value of the feature.

5. Polynomial regression
Sometimes, using linear regression is not enough to fit the input data, in this case,
polynomial regression can be used to fit a curve over the training data instead of just a
straight line. Polynomial regression can enable a model to better fit the data but might cause

5
some issues like calculation overflow since polynomial regression introduces variables with
exponential growth, this makes feature scaling even more important. A general form of
polynomial regression:

2 3 𝑛
𝑓𝑤,𝑏
⃗ (𝑥) = 𝑤1 𝑥 + 𝑤2 𝑥 + 𝑤3 𝑥 + … + 𝑤𝑛 𝑥 + 𝑏