MACHINE LEARNING

ALGORITHM
Unit-II
 Linearregression
Before we start Linear Regression
we have to perform cleaning and initial data
Country Salary Purchased
analysis by France 44 72000 No

Spain 27 48000 Yes

● Look at the summary of numerical Germany 30 54000 No
variables. Spain 38 61000 No

● See the distribution ofvariables Germany 40 Yes

● Look for possible correlation France 35 58000 Yes

● Explore any possible outliers Spain 52000 No

● Look for data errors with data sanity. France 48 79000 Yes

● Make sure data types are correct. Germany 50 83000 No

France 37 67000 Yes

 Linear regression

• Regression gives us simply the linear Country Salary

Purchas
ed

relationship of two or more variables France

Spain
44

27
72000

48000
No

Yes

within a dataset. Germany 30 54000 No

• We have a dependent variable and

Spain 38 61000 No

Germany 40 56400 Yes

independent variables. France 35 58000 Yes

Spain 50 52000 No

France 48 79000 Yes

Germany 50 83000 No

France 37 67000 Yes

• Linear relationship between variables means that when the value of one
or more independent variables will change (increase or decrease), the
value of dependent variable will also change accordingly (increase or
decrease).
• Mathematically the relationship can be represented with the help of
following equation :
Y = mX + c
• Here, Y is the dependent variable we are trying to predict.
• X is the independent variable we are using to make predictions.
• m is the slop of the regression line which represents the effect X has on Y
• c is a constant.
 Linear
regression
Linear regression model represents
the linear relationship between a
dependent variable and
independent variable(s) via a
sloped straight line.
 Types of Linear regression

Positive Linear
Relationship

A linear relationship will be

called positive if both
independent and dependent
variable increases.
 Types of Linear regression
Negative Linear
Relationship

A linear relationship will be

called negative if
independent increases and
dependent variable
decreases.
 Simple Linear Regression(SLR)

● It is the most basic version of linear regression,

which predicts a response using a single feature.
The assumption in SLR is that the two variables
are linearly related.
● In simple linear regression, the dependent
variable depends only on a single independent
variable
● For simple linear regression, the form of the model is :
Y = β0 + β1X
Here,
• Y is a dependent variable.
• X is an independent variable.
• β0 and β1 are the regression
coefficients.
• β0 is the intercept point
• β1 is the slope of line
 Simple Linear Regression

• There are following 3 cases

possible-
Case-01: β1 < 0
• It indicates that variable X
has negative impact on Y.
• If X increases, Y will
decrease and vice-versa.
 Simple Linear Regression

Case-02: β1 = 0
• It indicates that variable X has
no impact on Y.
• If X changes, there will be no
change in Y
 Simple Linear Regression

Case-03: β1 > 0
• It indicates that variable X
has positive impact on Y.
• If X increases, Y will
increase and vice-versa.
• Ex: the weight of a person
is depend on height of a
person.
 Multiple Linear Regression
• In multiple linear regression, the dependent variable
depends on more than one independent variables.
• For multiple linear regression, the form of the model is-
Y = β0 + β1X1 + β2X2 + β3X3 + …… + βnXn
Here,
• Y is a dependent variable.
• X1, X2, …., Xn are independent variables.
• β0, β1,…, βn are the regression coefficients
• βj (1<=j<=n) is the slope or weight that specifies the
factor by which Xj has an impact on Y.
 Multiple LinearRegression
Y = β0 + β1X1 + β2X2 + β3X3 + …… +
βnXn
Exp: Price of Flat depend on size of flat, floor ,
location, module kitchen etc.
Y= 0.9+ 1.2 . X1+ 2 . X2 + 4. X3 + 1 . X4

It is indication that which factor is more

important for predicting price of flat (Y).
 Multiple LinearRegression
Y = β0 + β1X1 + β2X2 + β3X3 + …… +
βnXn

Let , X3 is most important factor for this

prediction , so keeping the regression
coefficients value 4 for X3.
Linear regression
Linear regression
 Linear regression

On the basis of size of house to predict selling price of a house

Linear regression
Linear regression
Linear regression
 Polynomial Regression
● If your linear regression model cannot model the relationship between the
target variable and the predictor variable?

● In other words, what if they don’t have a linear relationship?

● Polynomial regression is a special case of linear regression where we fit a

polynomial equation on the data with a curvilinear relationship between the
target variable and the independent variables.

● Polynomial Regression is a form of linear regression in which the relationship

between the independent variable x and dependent variable y is modeled as
an nth degree polynomial.
 Polynomial Regression

● Polynomial regression fits a nonlinear relationship between the value of x and the
corresponding conditional mean of y, denoted E(y |x)
Finding theminimum

loss

How can we do this for a function?

 One approach: gradientdescent
Gradient descent is an optimization algorithm
used to find the values of parameters
(coefficients) of a function (f) that minimizes a
cost function (cost).

Gradient descent is best used when the

parameters cannot be calculated analytically
(e.g. using linear algebra) and must be
searched for by an optimization algorithm.
 One approach: gradientdescent

Partial derivatives give

us the slope loss

(i.e. direction to move)

in that dimension w
 One approach: gradientdescent
Approach:
• pick a starting point (w) loss

• repeat:
• pick a dimension
• move a small amount in w

that dimension towards decreasing

loss (using the derivative)
Gradient Descent Procedure
● The derivative of the cost is calculated.
● The derivative is a concept from calculus and refers
to the slope of the function at a given
point.
● We need to know the slope so that we know the
direction (sign) to move the coefficient values in
order to get a lower cost on the next iteration.
delta = derivative(cost)
○ pick a starting
Gradient point (w)
descent
○ repeat until loss doesn’t decrease in all dimensions:

■ pick a dimension
■ move a small amount in that dimension towards decreasing
loss (using the derivative)
d
wn = w o −η L
dw

What does this do?

Gradient descent
○ pick a starting point (w)

○ repeat until loss doesn’t decrease in all dimensions:

■ pick a dimension

■ move a small amount in that dimension towards decreasing loss

(using the derivative)

d
wj = w j −η loss(w)
dw j

learning rate (how much we want to move in the

error direction, often this will change over time)
 Decision trees

● In decision analysis, a decision tree can be used to

visually and explicitly
represent decisions and decision making.
● Decision trees can be constructed by an
algorithmic approach that can split the dataset
in different ways based on different conditions.
● Decisions trees are the most powerful algorithms
that falls under the category of supervised
algorithms.
● The two main entities of a tree are decision nodes,
where the data is split and leaves, where we got
outcome.
● The example of a binary tree for predicting
whether a person will get Loan or Not based on
his Income, Credit Score etc.
Before we started to design Decision tree,
following four steps are important
1. To find the Target / class attribute ( Profit )
2.To Find the Information Gain of Target
attribute
3.To find the Entropy ( for deciding root of tree)
4.At the end find the Gain of each attribute
 Data Set
Sr. Age Competition Type Profit
No

1 Old Yes SW Down

2 Old No SW Down

3 Old No HW Down

4 Mid Yes SW Down

5 Mid Yes HW Down

6 Mid No HW Up

7 Mid No SW Up

8 New Yes SW Up

9 New No HW Up

10 new No SW Up
● Now, Find the Information Gain of target attribute

IG = - [ 𝑃𝑃/(𝑃𝑃+𝑁𝑁) 〖 log〗_2 (𝑃𝑃/(𝑃𝑃+𝑁𝑁))− 𝑁𝑁/(𝑃𝑃+𝑁𝑁) 〖 log〗_2 (𝑁𝑁/(𝑃𝑃+𝑁𝑁)) ]

where P=down , N= Up

● Then find the Entropy of given attribute

i.e. Information gain of Attribute X Probability of that

attribute

● Finally , find the Gain for all attribute ( here for 3 attributes ), those Gain will be
greatest , we should called it as Root of Tree.

Gain =IG- E(A)

Implementing Decision TreeAlgorithm
Now, Find Information Gain of target attribute

Sr. Age Competiti Type Profit

No on
1 Old Yes SW Down

2 Old No SW Down

3 Old No HW Down

4 Mid Yes SW Down

5 Mid Yes HW Down

6 Mid No HW Up

7 Mid No SW Up

8 New Yes SW Up

9 New No HW Up

10 New No SW Up
 Decision Tree
Down UP

Old 3 0
Now, Find the Entropy of each attributes Age = Mid 2 2
Lets , start with attribute New 0 3

Sr Age Competiti Type Profit

.N on
o

1 Old Yes SW Down

2 Old No SW Down

3 Old No HW Down

4 Mid Yes SW Down

5 Mid Yes HW Down

6 Mid No HW Up

7 Mid No SW Up

8 New Yes SW Up

9 New No HW Up

10 New No SW Up
 Decision Tree
Down UP

Old 3 0
Now, find the Gain of all attribute
Mid 2 2

New 0 3
Where, Sr Age Competiti Type Profit
.N on
o
Gain (Age) = IG- E(Age) 1 Old Yes SW Down

= 1- 0.4 2 Old No SW Down

= 0.6 3 Old No HW Down

Gain( Competition) = IG- E(Competition) 4 Mid Yes SW Down

= 0.124 5 Mid Yes HW Down

6 Mid No HW Up

Gain( Type ) = IG- E(Type) 7 Mid No SW Up

=0 8 New Yes SW Up

9 New No HW Up

10 New No SW Up
Implementing Decision TreeAlgorithm
 Decision Tree

Sr Age Competiti Type Profit

.N on
o

1 Old Yes SW Down

2 Old No SW Down
Age
3 Old No HW Down
OLD NEW
4 Mid Yes SW Down
MID 5 Mid Yes HW Down

6 Mid No HW Up
Down UP
Competition 7 Mid No SW Up

8 New Yes SW Up
YES NO
9 New No HW Up

10 New No SW Up

Down UP
Decision Tree
Over fitting

Over Fitting Under Fitting

01 02
Too MUCH DATA given so LESS DATA given to
to Machine so that It Machine that it NOT
become CONFUSED in ABLE to Understand
things! Things.
 Over fitting
“OverFitting" : A hypothesis h is said to overfed the
training data , if there is another hypothesis h’ , such
that h’ has more error then h on training data but, h’
has less error than h on test data.
Exp: If we have small decision tree and it has higher
error in training data and lower error on test data
compare to larger decision tree , which has smaller
error in training data and higher error on test data ,
then we say that overftting has occurred
Over fitting
Over fitting : When model is so complex. Here , your model is trying to cover
every data points of output on X , Y plot.
Under fitting
Underfitting : When model is so simple . Here , your model is trying to cover
very few data points of output on X ,Y plot.
 Under fitting
Example :Let us consider that , you have to train your model that if any object
looks Sphere in shape , called it is as a Ball.
It’s
Ball

Sphere

Here , we are providing only one attribute to identify the object , i.e. Shape = Sphere
Example :Let us consider that , you have provide large number of attributes
like , Sphere, Play, Not Eat, Radius=5 cm.

Sphere

Play

Not Eat

Radius =5 cm

Here , we are providing lots of attributes to identify theobject.

 Instance Based Learning /LazyAlgorithm
K Nearest Neighbor Algorithm

● KNN algorithm can be used for both classification and regression predictive problems.
However, it is more widely used in classification problems in the industry.
● KNN algorithm at the training phase just stores the dataset and when it gets new data, then
it classifies that data into a category that is much similar to the new data.

● KNN works by finding the distances between a query and all the examples in the data,
selecting the specified number examples (K) closest to the query, then votes for the most
frequent label (in the case of classification) or avers the labels (in the case of regression).
Instance Based Learning /LazyAlgorithm
K Nearest Neighbor Algorithm

Let’s take a simple case to understand this algorithm. Following is a spread of

red circles (RC) and green squares (GS)
Instance Based Learning /LazyAlgorithm
K Nearest Neighbor Algorithm

● Let, to find out the class of the blue star (BS).

● BS can either be RC or GS and nothing else. The “K” is KNN algorithm is the nearest
neighbor we wish to take the vote from.
● Let’s say K = 3. Hence, we will now make a circle with BS as the center just as big as to
enclose only three datapoints on the plane.
● Refer to the following diagram for more details:

● The three closest points to BS is all RC.

● Hence, with a good confidence level, we can say that the BS should belong to the class
RC.
● Here, the choice became very obvious as all three votes from the closest neighbor
went to RC
 Instance Based Learning /LazyAlgorithm
K Nearest Neighbor Algorithm

● Let, us take an another example

Sr Math Comp Result
.N Sci
● Query : X=(Math=6, Comp Sci=8) , Is students Pass or Fail ?
o
● Here , we take K=3 any random value of K to find out nearest
1 4 3 Fail
neighbors
2 6 7 Pass
● To find the distance between these values , we use Euclidean
DistanceA 3 7 8 Pass

4 5 5 Fail

5 8 8 Pass

X 6 8 ????
𝑑𝑑= |𝑋𝑋01− 𝑋𝑋𝐴𝐴1|2+ |𝑋𝑋02− 𝑋𝑋𝐴𝐴2|2

Where , Xo is observed value

Xa is actual value
Instance Based Learning /LazyAlgorithm
K Nearest Neighbor Algorithm
Sr Math Comp Result
.N Sci
𝑑𝑑= |𝑋𝑋01− 𝑋𝑋𝐴𝐴1|2+ |𝑋𝑋02− 𝑋𝑋𝐴𝐴2|2 o

1 4 3 Fail
𝑑𝑑1= |6 − 4|2 + |8 − 3|2 = 29 = 5.38
2 6 7 Pass
𝑑𝑑2= |6 − 6|2 + |8 − 7|2 = 1 3 7 8 Pass
4 5 5 Fail
𝑑𝑑3= |6 − 7|2 + |8 − 8|2 = 1
5 8 8 Pass
X 6 8 ????
𝑑𝑑4= |6 − 5|2 + |8 − 5| 2 = 10 = 3.16

𝑑𝑑5= |6 − 8|2 + |8 − 8|2 =2

Instance Based Learning /LazyAlgorithm
K Nearest Neighbor Algorithm
Sr Math Comp Result
Three NNare (1,1,2,) .N Sci
o
Sr. Math Comp Result
No Sci
1 4 3 Fail

2 6 7 Pass
2 6 7 Pass
3 7 8 Pass 3 7 8 Pass

4 5 5 Fail
5 8 8 Pass
5 8 8 Pass

X 6 8 Pass
3 𝑃𝑃𝑎𝑎𝑠𝑠𝑠𝑠𝑎𝑎𝑛𝑛𝑑𝑑0 𝐹𝐹
𝑎𝑎
𝑖𝑖𝑙𝑙
3> 0
Instance Based Learning /LazyAlgorithm
K Nearest Neighbor Algorithm

Temp(X) in C Humidity (Y) % Rain Condition

27.8 76 Yes
28.2 76 Yes
28.7 80 No
28.6 81.6 Yes
27.7 89.4 Yes
30.5 89.9 No
26.7 81.4 Yes
25.9 85 No
36 90 No
31.8 88 Yes
35.7 70 No

Using KNN algorithm find the Rain Condition, Let K=3

When Temp: 29.6 C and Humidity: 78 %
KNN

01 Why do we needKNN? 02 What is KNN?

How do we choosethe
03 factor ‘K’? 04 When do we useKNN?

Use Case: Predict

How does KNN
05 Algorithm work?
06 whether a person will
need diabetes or not
Why KNN?
Why KNN?
Why KNN?
What isKNN?
What is KNN?
The KNN algorithm assumes that similar things exist in close proximity. In
other words, similar things are near to each other.
What is KNNAlgorithm?
What is KNNAlgorithm?
What is KNNAlgorithm?
What is KNNAlgorithm?
What is KNNAlgorithm?
What is KNNAlgorithm?
What is KNNAlgorithm?
How do we
choose ’k’?
How do we choose’k’?
How do we choose’k’?
How do we choose’k’?
How do we choose’k’?
How do we choose’k’?
When do we use
KNN?
When do we useKNN?
When do we useKNN?
When do we useKNN?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
How does KNN Algorithmwork?
Thank You