ML

Module No. 02
Learning with Regression and Trees

By Mr. Sachin Balawant Takmare

ML

Learning with Regression

By Mr. Sachin Balawant Takmare

ML Regression
Regression analysis is a statistical method to model the relationship between a dependent

(target) and independent (predictor) variables with one or more independent variables.
More specifically, Regression analysis helps us to understand how the value of the dependent

variable is changing corresponding to an independent variable when other independent

variables are held fixed. It predicts continuous/real values such as temperature, age, salary,

price, etc.
We can understand the concept of regression analysis using the below example:

By Mr. Sachin Balawant Takmare

ML Regression
Example: Suppose there is a marketing company A, who does various advertisement every

year and get sales on that. The below list shows the advertisement made by the company in

the last 5 years and the corresponding sales:

Now, the company wants to do the

advertisement of $200 in the year

2019 and wants to know the

prediction about the sales for this

year. So to solve such type of

prediction problems in machine

learning, we need regression analysis.

By Mr. Sachin Balawant Takmare
ML Regression
Regression analysis is a statistical method that helps us to analyze and understand the

relationship between two or more variables of interest.

Regression is a supervised learning technique which helps in finding the correlation between

variables and enables us to predict the continuous output variable based on the one or more

predictor variables.
It is mainly used for prediction, forecasting, time series modeling, and determining the

causal-effect relationship between variables.

By Mr. Sachin Balawant Takmare

ML Regression
In Regression, we plot a graph between the variables which best fits the given datapoints,

using this plot, the machine learning model can make predictions about the data.
In simple words, "Regression shows a line or curve that passes through all the

datapoints on target-predictor graph in such a way that the vertical distance

between the datapoints and the regression line is minimum."

The distance between datapoints and line tells whether a model has captured a strong

relationship or not.

Example1: Prediction of rain using temperature and other factors

Example2: Determining Share Market trends

Example3: Prediction of road accidents due to rash driving.

By Mr. Sachin Balawant Takmare
ML Regression
Terminologies Related to the Regression Analysis:

Dependent Variable: The main factor in Regression analysis which we want to predict or

understand is called the dependent variable. It is also called target variable.

Independent Variable: The factors which affect the dependent variables or which are used to

predict the values of the dependent variables are called independent variable, also called as a

predictor.
Outliers: Outlier is an observation which contains either very low value or very high value in

comparison to other observed values. An outlier may hamper the result, so it should be

avoided.
Underfitting and Overfitting
By Mr. Sachin Balawant Takmare
ML Regression
Types of Regression
There are various types of regressions which are used in data science and machine learning. Each type has

its own importance on different scenarios, but at the core, all the regression methods analyze the effect

of the independent variable on dependent variables.

1. Linear Regression

2. Logistic Regression

3. Polynomial Regression

4. Support Vector Regression

5. Decision Tree Regression

6. Random Forest Regression

7. Ridge Regression
By Mr. Sachin Balawant Takmare
ML Linear Regression
Linear regression is one of the easiest and most popular Machine Learning algorithms. It is a

statistical method that is used for predictive analysis.

Linear regression makes predictions for continuous/real or numeric variables such as sales,

salary, age, product price, etc.

Linear regression algorithm shows a linear relationship between a dependent (x) and one or

more independent (y) variables, hence called as linear regression.

Since linear regression shows the linear relationship, which means it finds how the value of

the dependent variable is changing according to the value of the independent variable.

By Mr. Sachin Balawant Takmare

ML Linear Regression
The linear regression model provides a sloped straight line representing the

relationship between the variables. Consider the below image:

Mathematically, we can represent a linear regression as:

The values for x and y variables are training datasets for Linear Regression

model representation.
By Mr. Sachin Balawant Takmare
ML Linear Regression
Types of Linear Regression

Linear regression can be further divided into two types of the algorithm:

Simple Linear Regression:

If a single independent variable is used to predict the value of a numerical dependent

variable, then such a Linear Regression algorithm is called Simple Linear Regression.

Multiple Linear regression:

If more than one independent variable is used to predict the value of a numerical dependent

variable, then such a Linear Regression algorithm is called Multiple Linear Regression.

By Mr. Sachin Balawant Takmare

ML Linear Regression
Simple Linear Regression

Simple Linear Regression is a type of Regression algorithms that models the relationship

between a dependent variable and a single independent variable.

The relationship shown by a Simple Linear Regression model is linear or a sloped straight

line, hence it is called Simple Linear Regression.

The key point in Simple Linear Regression is that the dependent variable must be a

continuous/real value. However, the independent variable can be measured on continuous or

categorical values.

By Mr. Sachin Balawant Takmare

ML Linear Regression
Simple Linear regression algorithm has mainly two objectives:

Model the relationship between the two variables. Such as the relationship between

Income and expenditure, experience and Salary, etc.

Forecasting new observations. Such as Weather forecasting according to temperature,

Revenue of a company according to the investments in a year, etc.

By Mr. Sachin Balawant Takmare

ML Linear Regression
EXAMPLE:01
Suppose we are given a dataset. Given is a Salary vs Experience dataset of a company and the
task is to predict the salary of an employee based on his / her work experience. Prepare the
equation for linear regression model and predict the salary of an employee if he has an
experience of 4 years

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML Linear Regression
EXAMPLE:02
Given is a Size of a flat in Sq.ft. vs Flat Prize dataset of a real estate company and the task is to
predict prize of a flat based on the size. Prepare the equation for linear regression model and
predict the prize of a flat of size 700 sq. ft.

By Mr. Sachin Balawant Takmare

ML

By Mr. Sachin Balawant Takmare

ML

The equation of regression line is

By Mr. Sachin Balawant Takmare

ML Linear Regression
Multiple Linear Regression
In the Simple Linear Regression a single Independent/Predictor(X) variable is used to model the

response variable (Y). But there may be various cases in which the response variable is affected by

more than one predictor variable; for such cases, the Multiple Linear Regression algorithm is used.
Moreover, Multiple Linear Regression is an extension of Simple Linear regression as it takes more

than one predictor variable to predict the response variable. We can define it as:
“Multiple Linear Regression is one of the important regression algorithms which

models the linear relationship between a single dependent continuous variable

and more than one independent variable.”

e.g. Prediction of CO2 emission based on engine size and number of cylinders in a car.

By Mr. Sachin Balawant Takmare

ML Linear Regression
EXAMPLE:01
Predict the value of y

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML

By Mr. Sachin Balawant Takmare

ML Linear Regression

By Mr. Sachin Balawant Takmare

ML

By Mr. Sachin Balawant Takmare

ML Logistic regression

• Logistic regression is one of the most popular Machine Learning algorithms, which comes

under the Supervised Learning technique. It is used for predicting the categorical dependent

variable using a given set of independent variables.

• Logistic regression predicts the output of a categorical dependent variable. Therefore the

outcome must be a categorical or discrete value. It can be either Yes or No, 0 or 1, true or

False, etc. but instead of giving the exact value as 0 and 1, it gives the probabilistic values

which lie between 0 and 1.

By Mr. Sachin Balawant Takmare

ML Logistic regression
• Logistic Regression is much similar to the Linear Regression except that how they are used.

Linear Regression is used for solving Regression problems, whereas Logistic regression is

used for solving the classification problems.

• In Logistic regression, instead of fitting a regression line, we fit an "S" shaped logistic

function, which predicts two maximum values (0 or 1).

• The curve from the logistic function indicates the likelihood of something such as whether the

cells are cancerous or not, a mouse is obese or not based on its weight, etc.

• Logistic Regression is a significant machine learning algorithm because it has the ability to

provide probabilities and classify new data using continuous and discrete datasets.
By Mr. Sachin Balawant Takmare
ML Logistic regression
• Logistic Regression can be used to classify the

observations using different types of data and

can easily determine the most effective

variables used for the classification. The below

image is showing the logistic function:

• Logistic regression uses the concept of

predictive modeling as regression; therefore, it

is called logistic regression, but is used to

classify samples; Therefore, it falls under the

classification algorithm. By Mr. Sachin Balawant Takmare

ML Logistic regression
• logistic regression could be used to predict whether a political candidate will win or lose an

election or whether a high school student will be admitted or not to a particular college.

By Mr. Sachin Balawant Takmare

ML Logistic regression
Logistic Function (Sigmoid Function):

• The sigmoid function is a mathematical function used to map the predicted values to

probabilities.

• It maps any real value into another value within a range of 0 and 1.

• The value of the logistic regression must be between 0 and 1, which cannot go beyond this

limit, so it forms a curve like the "S" form. The S-form curve is called the Sigmoid function or

the logistic function.

• In logistic regression, we use the concept of the threshold value, which defines the probability

of either 0 or 1. Such as values above the threshold value tends to 1, and a value below the

threshold values tends to 0.

By Mr. Sachin Balawant Takmare
ML Logistic regression
Equation for Logistic Regression

By Mr. Sachin Balawant Takmare

ML Logistic regression
Type of Logistic Regression:

On the basis of the categories, Logistic Regression can be classified into three types:

• Binomial: In binomial Logistic regression, there can be only two possible types of the

dependent variables, such as 0 or 1, Pass or Fail, etc.

• Ordinal: In ordinal Logistic regression, there can be 3 or more possible ordered types of

dependent variables, such as "low", "Medium", or "High".

• Multinomial: In multinomial Logistic regression, there can be 3 or more possible unordered

types of the dependent variable, such as "cat", "dogs", or "sheep"

By Mr. Sachin Balawant Takmare

ML

Learning with Trees

By Mr. Sachin Balawant Takmare

ML Classification
• Classification is a process of categorizing a given set of data into
predefined classes.

• Classification is a form of data analysis that extracts models describing

important data classes.

• Such models, called classifiers, predict categorical class labels

• E.g. we can build a classification model to categorize bank loan applications

as either safe or risky

• A bank loans officer need to analysis from the data to learn which loan
applicants are “safe” and which are “risky” for the bank to sanction loan
By Mr. Sachin Balawant Takmare
ML Classification
General approach
General approach of classification is divided into two-steps

In the first step, we build a classification model based on previous data.

In the second step, we determine if the model’s accuracy is acceptable,

and if so, we use the model to classify new data.

By Mr. Sachin Balawant Takmare

ML Classification
First Step

By Mr. Sachin Balawant Takmare

ML Classification
Second Step

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification
• Decision Tree is a Supervised learning technique that can be used for both
classification and Regression problems, but mostly it is preferred for
solving Classification problems. It is a tree-structured classifier, where
internal nodes represent the features of a dataset, branches represent the
decision rules and each leaf node represents the outcome.
• In a Decision tree, there are two nodes, which are the Decision Node and
Leaf Node. Decision nodes are used to make any decision and have
multiple branches, whereas Leaf nodes are the output of those decisions
and do not contain any further branches.
• The decisions or the test are performed on the basis of features of the
By Mr. Sachin Balawant Takmare
given dataset.
ML Decision Tree Classification
• It is a graphical representation for getting all the possible solutions to a
problem/decision based on given conditions.
• It is called a decision tree because, similar to a tree, it starts with the root
node, which expands on further branches and constructs a tree-like
structure.
• In order to build a tree, we use the CART algorithm, which stands for
Classification and Regression Tree algorithm.
• A decision tree simply asks a question, and based on the answer (Yes/No),
it further split the tree into subtrees.
• Note: A decision tree can contain categorical data (YES/NO) as well as
By Mr. Sachin Balawant Takmare
numeric data.
ML Decision Tree Classification
• General structure of a decision tree

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification
Why use Decision Trees?

• There are various algorithms in Machine learning, so choosing the

best algorithm for the given dataset and problem is the main point to
remember while creating a machine learning model. Below are the
two reasons for using the Decision tree:
• Decision Trees usually mimic human thinking ability while making a
decision, so it is easy to understand.
• The logic behind the decision tree can be easily understood because it
shows a tree-like structure.
By Mr. Sachin Balawant Takmare
ML Decision Tree Classification
Decision Tree Terminologies

• Root Node: Root node is from where the decision tree starts. It represents the
entire dataset, which further gets divided into two or more homogeneous sets.
• Leaf Node: Leaf nodes are the final output node, and the tree cannot be
segregated further after getting a leaf node.
• Splitting: Splitting is the process of dividing the decision node/root node into
sub-nodes according to the given conditions.
• Branch/Sub Tree: A tree formed by splitting the tree.
• Pruning: Pruning is the process of removing the unwanted branches from the
tree.
By Mr. Sachin Balawant Takmare
• Parent/Child node: The root node of the tree is called the parent node, and other
ML Decision Tree Classification
Example

• Suppose there is a candidate

who has a job offer and wants
to decide whether he should
accept the offer or Not.

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification
Attribute Selection Measures

• While implementing a Decision tree, the main issue arises that how to
select the best attribute for the root node and for sub-nodes. So, to
solve such problems there is a technique which is called as Attribute
selection measure or ASM. By this measurement, we can easily select
the best attribute for the nodes of the tree. There are two popular
techniques for ASM, which are:
▪ Information Gain
▪ Gini Index
By Mr. Sachin Balawant Takmare
ML Classification
Rule Extraction from

a Decision Tree

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification
A simple example from the stock market involving only discrete ranges has profit as
categorical attribute with values (UP, DOWN) and the training data is as follows:

Generate classification rules for the given data by applying decision tree classifier.
By Mr. Sachin Balawant Takmare
ML Decision Tree Classification
A simple example from the stock market involving only discrete ranges has profit as
categorical attribute with values (UP, DOWN) and the training data is as follows:

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification
Apply decision tree algorithm on following dataset using information gain

By Mr. Sachin Balawant Takmare

ML

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

 Consider _____________ attribute/feature

ML
First calculate Gini Index of attribute values

Now calculate Gini Index of attribute _________

By Mr. Sachin Balawant Takmare

 Consider _____________ attribute/feature

ML
First calculate Gini Index of attribute values

Now calculate Gini Index of attribute _________

By Mr. Sachin Balawant Takmare

 Consider _____________ attribute/feature

ML
First calculate Gini Index of attribute values

Now calculate Gini Index of attribute _________

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML Decision Tree Classification

By Mr. Sachin Balawant Takmare

ML • Bayes Classification Method : Naive Bayes classification
• Bayesian classifiers are statistical classifiers.
• They can predict class membership probabilities such as the
probability that a given tuple belongs to a particular class.
• A simple Bayesian classifier known as the naive Bayesian
classifier to be comparable in performance with decision tree
• Naive Bayesian classifiers assume that the effect of an
attribute value on a given class is independent of the values of
the other attributes. This assumption is called class
conditional independence.
• It is made to simplify the computations involved and, in this
sense, is considered “naive.”

By Mr. Sachin Balawant Takmare

ML • Predicting a class label using naive Bayesian classification

By Mr. Sachin Balawant Takmare

72
73
ML

By Mr. Sachin Balawant Takmare

74
• Predicting a class label using naive Bayesian classification

75
76
77
78
79
80
ML

Performance Metrics

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Understand some basic terminologies

• Now we know what is classification, how classifiers works so we may built a

classification model
• For example, suppose you used sales data to build a classifier to predict customer
purchasing behaviour
• In this example we would like to analyse how our model can predict the purchasing
behaviour of future customers.(data on which classifier has not been trained)
• We may built different classifiers and we can compare their accuracy/performance by
applying various evaluation matrices

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Understand some basic terminologies

• MODEL : a model is created by applying an algorithms(or statistical

calculations) to data to generate predictions/classifications of new
data.
• Given data set is partitioned into subsets
Training data set
Testing data set
• Training data set: training data set is used to derive the model or train
the model
• Testing data set: the models accuracy is estimated by using testing
data set
By Mr. Sachin Balawant Takmare
ML Confusion Matrix
Understand some basic terminologies

• Positive tuples : positive tuples of the class attribute (in our last
example positive tuples are buys_computer= yes)
• Negative tuples : negative tuples of the class attribute (in our last
example negative tuples are buys_computer= no)
• Suppose we use our classifier on a test set of labeled tuples.
• P is the number of positive tuples and N is the number of negative
tuples.
• For each tuple, we compare the classifier’s class attribute prediction
with the tuple’s known class attribute value.

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
There are four additional terms we need to know that are

• True positives (TP): These refer to the positive tuples that were correctly
labeled by the classifier. Let TP be the number of true positives.
• True negatives (TN): These are the negative tuples that were correctly
labeled by the classifier. Let TN be the number of true negatives.
• False positives (FP): These are the negative tuples that were incorrectly
labeled as positive (e.g., tuples of class buys_computer=no for which the
classifier predicted buys_computer=yes). Let FP be the number of false
positives.
• False negatives (FN): These are the positive tuples that were mislabeled as
negative (e.g., tuples of class buys_computer=yes for which the classifier
predicted buys_computer=no). Let FN be the number of false negatives.
By Mr. Sachin Balawant Takmare
ML Confusion Matrix

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Confusion Matrix

• The confusion matrix is a useful tool for analyzing how well your
classifier can recognize tuples of different classes.
• TP and TN tell us when the classifier is getting things right, while FP
and FN tell us when the classifier is getting things wrong.

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Confusion Matrix
• E.g. suppose in a data set of the customers who buys the computer, there are total
10000 tuples, out of that 7000 are positive and 3000 are negative and our model
has predicated 6954 are positive and 2588 are negative, so prepare confusion
matrix

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Confusion Matrix

• E.g. suppose in a data set of the customers who buys the computer, there are total 10000
tuples, out of that 7000 are positive and 3000 are negative and our model has predicated
6954 are positive and 2588 are negative, so the confusion matrix will be

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Classifiers performance evaluation measures

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Find all evaluation measures for the following confusion matrix

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Find all evaluation measures for the following confusion matrix

• E.g. suppose in a data set of the cancer, there are total 10000 tuples, out of that 300 are
positive and 9700 are negative and our model has predicated 90 are positive and 9560
are negative, so prepare confusion matrix and Find all evaluation measures for the
confusion matrix

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Find all evaluation measures for the following confusion matrix

• We have a data-set where we are predicting number of people who have more than Rs
1000 in their bank account. Consider a data-set with 200 observations i.e. n=200

• Out of 200 cases, our classification model predicted “YES” 135 times, and “NO” 65 times *
• Out of 200 cases, our classification model predicted “YES” 125 times, and “NO” 65 times.
• Out of 200 cases, our classification model predicted “YES” 135 times, and “NO” 60 times.
• Out of 200 cases, our classification model predicted “YES” 135 times, and “NO” 5 times.

By Mr. Sachin Balawant Takmare

ML Confusion Matrix
Evaluation measures for the confusion matrix

1. Accuracy:
2. Error rate:
3. Sensitivity: ability to correctly label the positive as positive
4. Specificity: ability to correctly label the negative as negative
5. Precision: % of positive tuples labelled as positive

By Mr. Sachin Balawant Takmare

ML ROC Curves
ROC Curves (Receiver operating characteristic curves)
• Useful visual tool for comparing two classification models
• An ROC curve for a given model shows the trade-off between the true
positive rate (TPR: Sensitivity) and the false positive rate (FPR:
Specificity)
• TPR is the proportion of positive (or “yes”) tuples that are correctly
labeled by the model; FPR is the proportion of negative (or “no”)
tuples that are mislabelled as positive

By Mr. Sachin Balawant Takmare

ML ROC Curves
ROC Curves (Receiver operating characteristic curves)
• The vertical axis represents TPR
• The horizontal axis represents FPR
• If we have a true positive then TPR
increase we move up and plot a point
• If the model classifies a negative
tuple as positive, FPR increase
we move right and plot a point
• The area under the ROC curve
is a measure of the accuracy
of the model
By Mr. Sachin Balawant Takmare
ML ROC Curves
ROC Curves (Receiver operating characteristic curves)

By Mr. Sachin Balawant Takmare

ML ROC Curves
ROC Curves

(Receiver operating

characteristic curves)

By Mr. Sachin Balawant Takmare

ML
ROC Curves

(Receiver operating characteristic

curves)

By Mr. Sachin Balawant Takmare