Apex Institute of Technology

Department of Computer Science & Engineering

Bachelor of Engineering (Computer Science &
Engineering)
Python for Machine Learning – (20CST255)
Prepared By: Dr. Dinesh Vij

Lecture - 11
DISCOVER . LEARN . EMPOWER
Model Selection using sklearn - I
 Python for Machine Learning
Course Objective:
The Course aims to: :

CO Will be covered in
Title
Number this lecture
Make students understand the structure, semantics
CO1 and syntax of Python programming Languages.

Make students understand and apply various data

CO2
handling and visualization techniques.

Enable students to develop and implement the first

CO3 principles of data science.

DISCOVER . LEARN . EMPOWER

 Python for Machine Learning
Course Outcome:
Upon successful completion of this course, students will be able to:

CO
Title Will be covered in
Number
this lecture
Understand Python programming language by
CO1
navigating software documentation
Development of Python programs using Numpy and
CO2
Pandas.
Visualization of Data Models using Matplotlib and
CO3 Seaborn.
CO4 Implement simple learning strategies using data
science principles.

CO5 Optimize the evaluation results obtained after

applying machine learning model.

DISCOVER . LEARN . EMPOWER

Model Selection

Apex Institute of Technology- CSE

Model Selection
• Model selection is the process of selecting one final machine
learning model from among a collection of candidate machine
learning models for a training dataset.
• Model selection is a process that can be applied both across
different types of models (e.g. logistic regression, SVM, KNN,
etc.) and across models of the same type configured with
different model hyperparameters (e.g. different kernels in an
SVM).
• For example, we may have a dataset for which we are
interested in developing a classification or regression predictive
model. We do not know beforehand as to which model will
perform best on this problem, as it is unknowable. Therefore, we
fit and evaluate a suite of different models on the problem.
• Model selection is the process of choosing one of the models
as the final model that addresses the problem.

Apex Institute of Technology- CSE

Model Selection (contd.)
• All models have some predictive error, given the statistical
noise in the data, the incompleteness of the data sample, and
the limitations of each different model type. Therefore, the
notion of a perfect or best model is not useful. Instead, we must
seek a model that is “good enough.”

• What do we care about when choosing a final model?

• The project stakeholders may have specific requirements, such
as maintainability and limited model complexity. As such, a
model that has lower skill but is simpler and easier to
understand may be preferred.
• Alternately, if model skill is prized above all other concerns,
then the ability of the model to perform well on out-of-sample
data will be preferred regardless of the computational
complexity involved.

Apex Institute of Technology- CSE

Model Selection (contd.)
Therefore, a “good enough” model may refer to
many things and is specific to your project, such
as:
 A model that meets the requirements and
constraints of project stakeholders.
 A model that is sufficiently skillful given the
time and resources available.
 A model that is skillful as compared to naive
models.
 A model that is skillful relative to other tested
models.
 A model that is skillful relative to the state-of-
the-art.
Apex Institute of Technology- CSE
Linear method for
Regression

Apex Institute of Technology- CSE

 Regression?
• Generally, in regression analysis, you usually consider some
phenomenon of interest and have a number of observations.
Each observation has two or more features. Following the
assumption that (at least) one of the features depends on the
others, you try to establish a relation among them.
• In other words, you need to find a function that maps some
features or variables to others sufficiently well.
• The dependent features are called the dependent
variables, outputs, or responses.
• The independent features are called the independent
variables, inputs, or predictors.
Apex Institute of Technology- CSE
Regression? (contd.)
•Regression problems usually have one
continuous and unbounded dependent
variable. The inputs, however, can be
continuous, discrete, or even categorical data
such as gender, nationality, brand, and so on.

with 𝑦 and inputs with 𝑥. If there are two or

•It is a common practice to denote the outputs

represented as the vector 𝐱 = (𝑥₁, …, 𝑥ᵣ),

more independent variables, they can be

where 𝑟 is the number of inputs.

Apex Institute of Technology- CSE
Linear Regression
• Linear regression is probably one of the most important and widely
used regression techniques. It’s among the simplest regression
methods. One of its main advantages is the ease of interpreting
results.

When implementing linear regression of some dependent variable 𝑦

Problem Formulation

on the set of independent variables 𝐱 = (𝑥₁, …, 𝑥ᵣ), where 𝑟 is the

number of predictors, you assume a linear relationship between 𝑦 and
𝐱:
𝑦 = 𝛽₀ + 𝛽₁𝑥₁ + ⋯ + 𝛽ᵣ𝑥ᵣ + 𝜀. This equation is the regression
equation. 𝛽₀, 𝛽₁, …, 𝛽ᵣ are the regression coefficients, and 𝜀 is
the random error.
Apex Institute of Technology- CSE
Linear Regression (contd.)
• Linear regression calculates the estimators of the regression

𝑏₀, 𝑏₁, …, 𝑏ᵣ. They define the estimated regression

coefficients or simply the predicted weights, denoted with

function 𝑓(𝐱) = 𝑏₀ + 𝑏₁𝑥₁ + ⋯ + 𝑏ᵣ𝑥ᵣ. This function should

capture the dependencies between the inputs and output

• The estimated or predicted response, 𝑓(𝐱ᵢ), for each

sufficiently well.

observation 𝑖 = 1, …, 𝑛, should be as close as possible to the

corresponding actual response 𝑦ᵢ. The differences 𝑦ᵢ - 𝑓(𝐱ᵢ)
for all observations 𝑖 = 1, …, 𝑛, are called the residuals.

Apex Institute of Technology- CSE

 Linear Regression (contd.)
• Regression is about determining the best predicted
weights, that is the weights corresponding to the smallest
residuals.
• To get the best weights, you usually minimize the sum of
squared residuals (SSR) for all observations 𝑖 = 1, …, 𝑛:
SSR = Σᵢ(𝑦ᵢ - 𝑓(𝐱ᵢ))². This approach is called the method of
ordinary least squares.

Apex Institute of Technology- CSE

Regression Performance
• The coefficient of determination, denoted as 𝑅², tells you
which amount of variation in 𝑦 (actual responses) can be
explained by the dependence on 𝐱 (predictors) using the

• Larger 𝑅² indicates a better fit and means that the model can
particular regression model.

• The value 𝑅² = 1 corresponds to SSR = 0, that is to the perfect

better explain the variation of the output with different inputs.

fit since the values of predicted and actual responses fit

completely to each other.

Apex Institute of Technology- CSE

Simple Linear Regression
Simple or single-variate linear regression is the simplest

variable, 𝐱 = 𝑥.
case of linear regression with a single independent

The following figure illustrates simple linear regression:

Apex Institute of Technology- CSE

Simple Linear Regression (contd.)

start with a given set of input-output (𝑥-𝑦) pairs (blue

• When implementing simple linear regression, you typically

circles). These pairs are your observations. Where x is the

input and y is the actual output.

equation 𝑓(𝑥) = b₀ + b₁𝑥. Your goal is to calculate the

• The estimated regression function (orange line) has the

optimal values of the predicted weights 𝑏₀ and 𝑏₁ that

minimize SSR and determine the estimated regression
function.

where the estimated regression line crosses the 𝑦 axis. It is

• The value of b₀, also called the intercept, shows the point

the value of the estimated response 𝑓(𝑥) for 𝑥 = 0. The

value of b₁ determines the slope of the estimated regression
Apex Institute of Technology- CSE
Simple Linear Regression (contd.)
• The predicted responses are the points on the
regression line that correspond to the input values.

𝑦ᵢ - 𝑓(𝐱ᵢ) = 𝑦ᵢ - 𝑏₀ - 𝑏₁𝑥ᵢ for 𝑖 = 1, …, 𝑛. They are

• The residuals (vertical pink lines) can be calculated as

the distances between the blue circles and red

squares. When you implement linear regression, you
are actually trying to minimize these distances and
make the red squares as close to the predefined blue
circles as possible.

Apex Institute of Technology- CSE

 Multiple Linear Regression
• Multiple or multivariate linear regression is a case of linear
regression with two or more independent variables.

regression function is 𝑓(𝑥₁, 𝑥₂) = 𝑏₀ + 𝑏₁𝑥₁ + 𝑏₂𝑥₂. It

• If there are just two independent variables, the estimated

represents a regression plane in a three-dimensional space.

weights 𝑏₀, 𝑏₁, and 𝑏₂ such that this plane is as close as

• The goal of regression is to determine the values of the

possible to the actual responses and yield the minimal SSR.

Apex Institute of Technology- CSE

 Multiple Linear Regression (contd.)

but more general. The estimated regression function is 𝑓(𝑥₁,

• The case of more than two independent variables is similar,

…, 𝑥ᵣ) = 𝑏₀ + 𝑏₁𝑥₁ + ⋯ +𝑏ᵣ𝑥ᵣ, and there are 𝑟 + 1 weights to

be determined when the number of inputs is 𝑟.

Apex Institute of Technology- CSE

Suggestive Readings
• http://www.stat.yale.edu/Courses/1997-98/101/linreg.htm#
:~:text=Linear%20regression%20attempts%20to%20mod
el,linear%20equation%20to%20observed%20data.&text=
A%20linear%20regression%20line%20has,Y%20is%20the
%20dependent%20variable

• https://www.youtube.com/watch?v=E5RjzSK0fvY

• https://www.youtube.com/watch?v=OCwZyYH14uw

Apex Institute of Technology- CSE

THANK YOU

For queries
Email: [email protected]

Apex Institute of Technology- CSE