0% found this document useful (0 votes)
4 views24 pages

Unit III Regression

The document discusses the Gauss-Markov theorem and its assumptions necessary for Ordinary Least Squares (OLS) to provide the best linear unbiased estimates in regression analysis. It explains the concept of regression, including linear and logistic regression, and highlights the importance of checking assumptions for accurate coefficient estimation. Additionally, it covers methods like Maximum Likelihood Estimation (MLE) and Multinomial Logistic Regression for modeling relationships among variables.

Uploaded by

Shaik Javeed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
4 views24 pages

Unit III Regression

The document discusses the Gauss-Markov theorem and its assumptions necessary for Ordinary Least Squares (OLS) to provide the best linear unbiased estimates in regression analysis. It explains the concept of regression, including linear and logistic regression, and highlights the importance of checking assumptions for accurate coefficient estimation. Additionally, it covers methods like Maximum Likelihood Estimation (MLE) and Multinomial Logistic Regression for modeling relationships among variables.

Uploaded by

Shaik Javeed
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 24

UNIT-III

• Regression
• The Gauss-Markov theorem is based on five assumptions,
or conditions, that must be met for the Ordinary Least
Squares (OLS) estimate to be the best linear unbiased
estimate (BLUE):
• Linearity: The parameters being estimated must be
BLUE linear.

Property • Random sampling: The data must be randomly sampled


from the population.
assumptio • No perfect multicollinearity: The independent variables
should not be perfectly correlated with each other.
ns • Exogeneity: The regressors should not be correlated with
the error term.
• Homoscedasticity: The error variance should be constant
regardless of the values of the regressors.
• Not to be confused with Gauss–Markov process. "BLUE"
redirects here. For queue management algorithm
• In statistics, the Gauss–Markov theorem, states that the
ordinary least squares (OLS) estimator has the lowest
sampling variance within the class of linear unbiased
Gauss– estimators, if the errors in the linear regression model are
uncorrelated, have equal variances and expectation value
Markov of zero.[2] The errors do not need to be normal, nor do
they need to be independent and identically distributed
theorem (only uncorrelated with mean zero and homoscedastic
with finite variance). The requirement that the estimator
be unbiased cannot be dropped, since biased estimators
exist with lower variance. See, for example, the
James–Stein estimator (which also drops linearity),
ridge regression, or simply any degenerate estimator..
• The Gauss Markov assumptions guarantee the
validity of ordinary least squares for estimating
regression coefficients.
• Checking how well our data matches these
assumptions is an important part of estimating
Purpose of regression coefficients.
• When you know where these conditions are
the violated, you may be able to plan ways to change
your experiment setup to help your situation fit
Assumptio the ideal Gauss Markov situation more closely.
• In practice, the Gauss Markov assumptions are
ns rarely all met perfectly, but they are still useful as
a benchmark, and because they show us what
‘ideal’ conditions would be.
• They also allow us to pinpoint problem areas that
might cause our estimated regression coefficients
to be inaccurate or even unusable.
The Gauss-
Markov
Assumptio
ns in
Algebra
Suppose we have, in
matrix notation, the
linear relationship
What is
Regression ?

• A statistical procedure used to find


relationships among a set of
variables
• In regression analysis, there is a
dependent variable, which is the
one you are trying to explain, and
one or more independent variables
that are related to it.
• You can express the relationship as a
linear equation, such as:
• y = a + bx
• •y is the dependent variable
• • x is the independent variable
• • a is a constant
• • b is the slope of the line
• • For every increase of 1 in x, y changes by
an amount equal to b
• Some relationships are perfectly linear and fit
this equation exactly
Regression
• It is a Predictive modeling technique
where the target variable to be estimated
is continuous
Examples of applications of regression

Forecasting the
Predicting a stock amount of
market index using precipitation in a
other economic region based on
indicators characteristics of the
jet stream

Estimating the age of


Projecting the total
a fossil according to
sales of a company
the amount of carbon-
based on the amount
14 left in the organic
spent for advertising
material.
Least Square
Estimation or
Least Square
Method
Regression (Definition)
• Regression is the task of learning a target function f that maps each attribute set x into a
continuous-valued output y.
The goal of regression:
To find a target function that can fit the input data with minimum error.
• The error function for a regression task can be expressed in terms of the sum of absolute or
squared error
Simple Linear Regression (example)
• Consider the physiological data shown in Figure D.1.
• The data corresponds to measurements of heat flux and skin temperature of a person during sleep.
• Suppose we are interested in predicting the skin temperature of a person based on the heat flux measurements
generated by a heat sensor.
• The two-dimensional scatter plot shows that there is a strong linear relationship between the two variables.
Logistic Regression

• Logistic regression, or Logit regression, or Logit model


• Is a regression model where the dependent variable (DV) is categorical.
• was developed by statistician David Cox in 1958.
• The response variable Y has been regarded as a continuous quantitative variable.
• There are situations, however, where the response variable is qualitative.
• The predictor variables, however, have been both quantitative, as well as qualitative.
• Indicator variables fall into the second category.
Example:
• Consider a procedure in which individuals are selected on the basis of their scores in a battery of tests.
• After five years the candidates are classified as "good" or "poor.”
• We are interested in examining the ability of the tests to predict the job performance of the candidates.
• Here the response variable, performance, is dichotomous.
• We can code "good" as 1 and "poor" as 0, for example.
• The predictor variables are the scores in the tests.
• In a study to determine the risk factors for cancer, health records of several people were studied.
• Data were collected on several variables, such as age, gender, smoking, diet, and the family's medical
history.
• The response variable was the person had cancer (Y = 1) or did not have cancer (Y = 0).
• The relationship between the probability π and X can often be represented by a logistic response function.
• It resembles an S-shaped curve.
• The probability π initially increases slowly with increase in X, and then the increase accelerates, finally
stabilizes, but does not increase beyond 1.
• Intuitively this makes sense.
• Consider the probability of a questionnaire being returned as a function of cash reward, or the probability of
passing a test as a function of the time put in studying for it.
• The shape of the S-curve can be reproduced if we model the probabilities as follows:
• A sigmoid function is a bounded differentiable real function that is defined for all real input values and has a
positive derivative at each point.
• It has an “S” shape. It is defined by below function:

• The process of linearization of logistic regression function is called Logit Transformation.

The process of linearization of logistic regression function is called Logit Transformation.


 Modeling the response probabilities by the logistic distribution and estimating the parameters of the
model given below constitutes fitting a logistic regression.
 In logistic regression the fitting is carried out by working with the logits.
 The Logit transformation produces a model that is linear in the parameters.
 The method of estimation used is the maximum likelihood method.
 The maximum likelihood estimates are obtained numerically, using an iterative procedure
OLS: Ordinary Least Squares
 The ordinary least squares, or OLS, can also be called the linear least squares.
 This is a method for approximately determining the unknown parameters located in a linear regression model.
 According to books of statistics and other online sources, the ordinary least squares is obtained by minimizing
the total of squared vertical distances between the observed responses within the dataset and the responses
predicted by the linear approximation.
 Through a simple formula, you can express the resulting estimator, especially the single regressor, located on
the right-hand side of the linear regression model.
 For example, you have a set of equations which consists of several equations that have unknown parameters.
 You may use the ordinary least squares method because this is the most standard approach in finding the
approximate solution to your overly determined systems.
 In other words, it is your overall solution in minimizing the sum of the squares of errors in your equation.
 Data fitting can be your most suited application. Online sources have stated that the data that best fits the
ordinary least squares minimizes the sum of squared residuals.
 “Residual” is “the difference between an observed value and the fitted value provided by a model.”
Maximum likelihood estimation, or MLE

 MLE is a method used in estimating the parameters of a statistical model, and for fitting a statistical
model to data.
 If you want to find the height measurement of every basketball player in a specific location, you can
use the maximum likelihood estimation.
 Normally, you would encounter problems such as cost and time constraints.
 If you could not afford to measure all of the basketball players’ heights, the maximum likelihood
estimation would be very handy.
 Using the maximum likelihood estimation, you can estimate the mean and variance of the height of your
subjects.
• The MLE would set the mean and variance as parameters in determining the specific parametric values
in a given model.
Multinomial Logistic Regression

 We have n independent observations with p explanatory variables.


 The qualitative response variable has k categories.
 To construct the logits in the multinomial case one of the categories is considered the base level and all
the logits are constructed relative to it. Any category can be taken as the base level.
 We will take category k as the base level in our description of the method.
 Since there is no ordering, it is apparent that any category may be labeled k. Let 7rjdenote the multinomial
probability of an observation falling in the jth category.
 We want to find the relationship between this probability and the p explanatory variables, Xl, X 2 , ... ,Xp.
The multiple logistic regression model then is
• Since all the 7r'S add to unity, this reduces to

For j = 1,2,···, (k - 1). The model parameters are estimated by the method of maximum likelihood. Statistical
software is available to do this fitting.

You might also like