Simple Regression Model fitting 

https://datascienceplus.com/how-to-apply-linear-regression-in-r/ 
Machine Learning (ML) is a field of study that provides the capability to a Machine to
understand data and to learn from the data. ML is not only about analytics modeling but it
is end-to-end modeling that broadly involves following steps: 
– Defining problem statement 
– Data collection. 
– Exploring, Cleaning and transforming data. 
– Making the analytics model. 
– Dashboard creation & deployment of the model. 
Machine learning has two distinct field of study – supervised learning and unsupervised learning.
Supervised learning technique generates a response based on the set of input features. Unsupervised
learning does not have any response variable and it explores the association and interaction between
input features. In the following topic, we will discuss linear regression that is an example of
supervised learning technique. 
Supervised Learning & Regression 
Linear Regression is a supervised modelling technique for continuous data. The model fits a line that
is closest to all observation in the dataset. The basic assumption here is that functional form is the
line and it is possible to fit the line that will be closest to all observation in the dataset. Please note
that if the basic assumption about the linearity of the model is away from reality then there is bound
to have an error (bias towards linearity) in the model however best one will try to fit the model. 
Let’s analyse the basic equation for any supervised learning algorithm 
Y = F(x) + ε 
The above equation has three terms: 
Y – define the response variable. 
F(X) – defines the function that is dependent on set of input features. 
ε – defines the random error. For ideal model, this should be random and should not be dependent
on any input. 
In linear regression, we assume that functional form, F(X) is linear and hence we can write the
equation as below. Next step will be to find the coefficients (β0, β1..) for below model. 
Y = β0 + β1 X + ε ( for simple regression ) 
Y = β0 + β1 X1 + β2 X2+ β3 X3 + …. + βp Xp + ε ( for multiple regression ) 
How to apply linear regression 
The coefficient for linear regression is calculated based on the sample data. The basic assumption
here is that the sample is not biased. This assumption makes sure that the sample does not
necessarily always overestimate or underestimate the coefficients. The idea is that a particular
sample may overestimate or underestimate but if one takes multiple samples and try to estimate
the coefficient multiple times, then the average of co-efficient from multiple samples will be spot
on. 
Extract the data and create the training and testing sample 
For the current model, let’s take the Boston dataset that is part of the MASS library in R Studio.
Following are the features available in Boston dataset. The problem statement is to predict ‘medv’
based on the set of input features. 
 
library(MASS) 
library(ggplot2) 
Boston # calling data set 
names(Boston) # Names of the variables in a data set 
str(Boston) # Structure of the variables of a data set 
set.seed(1) # It is syntax used for getting same output 
Split the sample data and make the model 
Split the input data into training and evaluation set and make the model for the training dataset. It
can be seen that training dataset has 404 observations and testing dataset has 102 observations
based on 80-20 split. 
row.number=sample(1:nrow(Boston),0.8*nrow(Boston))# Splitting data 
train=Boston[row.number,] 
test=Boston[-row.number,] 
dim(train) 
dim(test) 
Explore the response variable 
Let’s check for the distribution of response variable ‘medv’. The following figure shows the three
distributions of ‘medv’ original, log transformation and square root transformation. We can see that
both ‘log’ and ‘sqrt’ does a decent job to transform ‘medv’ distribution closer to normal. In the
following model, we have selected ‘log’ transformation but it is also possible to try out ‘sqrt’
transformation. 
 
 
ggplot(train, aes(log(medv))) + geom_density(fill="blue") 
 
ggplot(train, aes(sqrt(medv))) + geom_density(fill="blue") 
 
Model Building – Model 1 
Now as a first step we will fit the multiple regression models. We will start by taking all input
variables in the multiple regression. 
#Let’s make default model. 
model1 = lm(log(medv)~., data=train) 
summary(model1) 
par(mfrow=c(2,2)) 
plot(model1) 
 
Observation from summary (model1) 
Is there a relationship between predictor and response variables? 
We can answer this using F stats. This defines the collective effect of all predictor variables on the
response variable. In this model, F=102.3 is far greater than 1, and so it can be concluded that there
is a relationship between predictor and response variable. 
Which of the predictor variables are significant? 
Based on the ‘p-value’ we can conclude on this. The lesser the ‘p’ value the more significant is the
variable. From the ‘summary’ dump we can see that ‘zn’, ‘age’ and ‘indus’ are less significant
features as the ‘p’ value is large for them. In next model, we can remove these variables from the
model. 
Is this model fit? 
We can answer this based on R2 (multiple-R-squared) value as it indicates how much variation is
captured by the model. R2 closer to 1 indicates that the model explains the large value of the
variance of the model and hence a good fit. In this case, the value is 0.7733 (closer to 1) and hence
the model is a good fit. 
Observation from the plot 
Fitted vs Residual graph 
 
Residuals plots should be random in nature and there should not be any pattern in the graph. The
average of the residual plot should be close to zero. From the above plot, we can see that the red
trend line is almost at zero except at the starting location. 
 
Normal Q-Q Plot 
 
Q-Q plot shows whether the residuals are normally distributed. Ideally, the plot should be on the
dotted line. If the Q-Q plot is not on the line then models need to be reworked to make the residual
normal. In the above plot, we see that most of the plots are on the line except at towards the end. 
 
Scale-Location 
 
This shows how the residuals are spread and whether the residuals have an equal variance or not. 
 
Residuals vs Leverage 
 
The plot helps to find influential observations. Here we need to check for points that are outside the
dashed line. A point outside the dashed line will be influential point and removal of that will affect
the regression coefficients. 
Model Building – Model 2 
As the next step, we can remove the four lesser significant features (‘zn’, age’ and ‘indus’ ) and
check the model again. 
# remove the less significant feature 
model2 = update(model1, ~.-zn-indus-age)  
summary(model2) 
 
Observation from summary (model1) 
Is there a relationship between predictor and response variable? 
F=131.2 is far greater than 1 and this value is more than the F value of the previous model. It can be
concluded that there is a relationship between predictor and response variable. 
Which of the variable are significant? 
Now in this model, all the predictors are significant. 
Is this model fit? 
R2 =0.7696 is closer to 1 and so this model is a good fit. Please note that this value has decreased a
little from the first model but this should be fine as removing three predictors caused a drop from
0.7733 to 0.7696 and this is a small drop. In other words, the contribution of three predictors
towards explaining the variance is an only small value (0.0037) and hence it is better to drop the
predictor. 
Observation of the plot 
All the four plots look similar to the previous model and we don’t see any major effect. 
Check for predictor vs Residual Plot 
In the next step, we will check the residual graph for all significant features from Model 2. We need
to check if we see any pattern in the residual plot. Ideally, the residual plot should be random plot
and we should not see a pattern. In the following plots, we can see some non-linear pattern for
features like ‘crim’, ‘rm’, ‘nox’ etc. 
attach(train) 
require(gridExtra) 
plot1 = ggplot(train, aes(crim, residuals(model2))) + geom_point() + geom_smooth() 
plot2=ggplot(train, aes(chas, residuals(model2))) + geom_point() + geom_smooth() 
plot3=ggplot(train, aes(nox, residuals(model2))) + geom_point() + geom_smooth() 
plot4=ggplot(train, aes(rm, residuals(model2))) + geom_point() + geom_smooth() 
plot5=ggplot(train, aes(dis, residuals(model2))) + geom_point() + geom_smooth() 
plot6=ggplot(train, aes(rad, residuals(model2))) + geom_point() + geom_smooth() 
plot7=ggplot(train, aes(tax, residuals(model2))) + geom_point() + geom_smooth() 
plot8=ggplot(train, aes(ptratio, residuals(model2))) + geom_point() + geom_smooth() 
plot9=ggplot(train, aes(black, residuals(model2))) + geom_point() + geom_smooth() 
plot10=ggplot(train, aes(lstat, residuals(model2))) + geom_point() + geom_smooth() 
grid.arrange(plot1,plot2,plot3,plot4,plot5,plot6,plot7,plot8,plot9,plot10,ncol=5,nrow=2) 
 
 
Model Building – Model 3 & Model 4 
We can now enhance the model by adding a square term to check for non-linearity. We can first try
model3 by introducing square terms for all features ( from model 2). And in the next iteration, we
can remove the insignificant feature from the model. 
#Model Building – Model 3 & Model 4 
#Lets  make default model and add square term in the model. 
model3 = lm(log(medv)~crim+chas+nox+rm+dis+rad+tax+ptratio+ 
              black+lstat+ I(crim^2)+ I(chas^2)+I(nox^2)+ I(rm^2)+ I(dis^2)+  
              I(rad^2)+ I(tax^2)+ I(ptratio^2)+ I(black^2)+ I(lstat^2), data=train) 
summary(model3) 
Output: 
(Intercept)   8.273e+00  8.749e-01   9.456  < 2e-16 *** 
crim         -3.291e-02  4.505e-03  -7.306 1.61e-12 *** 
chas          1.124e-01  3.223e-02   3.487 0.000546 *** 
nox          -6.286e-01  1.074e+00  -0.585 0.558693     
rm           -8.026e-01  1.324e-01  -6.063 3.20e-09 *** 
dis          -1.202e-01  2.452e-02  -4.900 1.41e-06 *** 
rad           1.628e-02  9.436e-03   1.726 0.085217 .   
tax          -3.393e-04  5.300e-04  -0.640 0.522477     
ptratio      -1.592e-01  7.163e-02  -2.222 0.026843 *   
black         1.314e-03  5.115e-04   2.568 0.010594 *   
lstat        -5.419e-02  5.487e-03  -9.876  < 2e-16 *** 
I(crim^2)     2.961e-04  6.690e-05   4.426 1.25e-05 *** 
I(chas^2)            NA         NA      NA       NA     
I(nox^2)     -2.450e-01  8.002e-01  -0.306 0.759664     
I(rm^2)       6.752e-02  1.036e-02   6.520 2.22e-10 *** 
I(dis^2)      6.899e-03  1.936e-03   3.564 0.000411 *** 
I(rad^2)      2.739e-04  3.730e-04   0.734 0.463258     
I(tax^2)     -4.613e-07  6.474e-07  -0.712 0.476601     
I(ptratio^2)  3.751e-03  2.040e-03   1.839 0.066742 .   
I(black^2)   -2.355e-06  1.129e-06  -2.085 0.037695 *   
I(lstat^2)    7.380e-04  1.520e-04   4.854 1.77e-06 *** 
--- 
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 0.1627 on 384 degrees of freedom 
Multiple R-squared:  0.8526,Adjusted R-squared:  0.8453  
F-statistic: 116.9 on 19 and 384 DF,  p-value: < 2.2e-16 
##Removing the insignificant variables. 
model4=update(model3, ~.-nox-rad-tax-I(crim^2)-I(chas^2)-I(rad^2)- 
                I(tax^2)-I(ptratio^2)-I(black^2)) 
summary(model4) 
par(mfrow=c(2,2)) 
 
plot(model4) 
 
pred1 <- predict(model4, newdata = test) 
rmse <- sqrt(sum((exp(pred1) - test$medv)^2)/length(test$medv)) 
c(RMSE = rmse, R2=summary(model4)$r.squared) 
Output 
RMSE        R2  
4.8235100 0.8265999 
 
par(mfrow=c(1,1)) 
plot(test$medv, exp(pred1)) 
 
Conclusion 
The example shows how to approach linear regression modelling. The model that is created still has
scope for improvement as we can apply techniques like Outlier detection, Correlation detection to
further improve the accuracy of more accurate prediction. One can as well use an advanced technique
like Random Forest and Boosting technique to check whether the accuracy can be further improved
for the model. A piece of warning is that we should refrain from overfitting the model for training
data as the test accuracy of the model will reduce for test data in case of overfitting.