Matrix MLE for Linear Regression

Joseph E. Gonzalez

Some people have had some trouble with the linear algebra form of the MLE for multiple regression.  I tried to find a nice online derivation but I could not find anything helpful.  So I have decide to derive the matrix form for the MLE weights for linear regression under the assumption of Gaussian noise.

The Model

Lets say we are given some set of data X and y.  The matrix X has n rows corresponding to each of the examples and d columns corresponding to each of the d features.  The column vector y consists has n rows corresponding to each of the examples and 1 column.  We want to "learn" the relationship between an individual feature vector x and an outcome y.  In some sense we want to learn the function f:(ℝ^d)->ℝ which satisfies:

y=f(x)

Linear Models

There are many functions f that we could chose from (I am sure you have some favorites).  To simplify our computation and to impose some assumptions (which often aids in generalization) we will restrict f to the class of linear functions.  That is for a choice of weights w we can express f as:

(Subscript[f, w])(x)=(∑_(j=1)^d)(Subscript[w, j]) (Subscript[x, j])

Nonlinear Features

Often people find this assumption to restrictive.  We can permit a more complex class of functions by creating new (nonlinear)  features from the original features (Subscript[x, j]).  For example:

(Subscript[f, w])(x)=(∑_(j=1)^d)(Subscript[w, j]) (Subscript[x, j]) + (∑_(j=d+1)^(2d))(Subscript[w, j]) Sin[(Subscript[x, j]^2)]

To formalize this notion we can rewrite equation 3 as:

(Subscript[f, w])(x)=(∑_(j=1)^m)(Subscript[w, j]) (Subscript[ϕ, j])[x]

Returning to the example in equation 3 we can use the notation of equation 4 by defining:

(Subscript[ϕ, j ])[x]=(Piecewise[{{(Subscript[x, j]), if 1≤j≤d}, {Sin[(Subscript[x, j]^2)], if d+1≤j≤2d}, {0, otherwise}}])

This technique allows us to lift our simple linear function (Subscript[f, w]) into a more complex space permitting a richer class of functions in our original space (ℝ^d).  With this transformation we can define a matrix Φ which is like X but consists of the transformed features.  If we do not want to transform our features then we simply define:

(Subscript[ϕ, j ])[x]=(Piecewise[{{(Subscript[x, j]), if 1≤j≤d}, {0, otherwise}}])

The matrix Φ is constructed by:

Φ=(({{(Subscript[ϕ, 1])[(Subscript[X, 11]),…, (Subscript[X, 1d])], …, (Subscript[ϕ, m])[(Subscript[X, 11]),…, (Subscript[X, 1d])]}, {..., ..., ...}, {(Subscript[ϕ, 1])[(Subscript[X, n1]),…, (Subscript[X, nd])], …, (Subscript[ϕ, m])[(Subscript[X, n1]),…, (Subscript[X, nd])]}}))

If we use the trivial transform in equation 5 equation 6 becomes:

Φ=(({{(Subscript[X, 11]), …, (Subscript[X, 1d])}, {..., ..., ...}, {(Subscript[X, n1]), …, (Subscript[X, nd])}}))=X

For the rest of these notes I will use the trivial feature space X. However feel free to substitute Φ where ever X is used if a nonlinear feature space is desired.

Noise

Sadly we live in the real world where there is random noise ϵ that gets mixed into our observations.  So a more natural model would be of the form:

y=(Subscript[f, w])(x) +ϵ

We have to pick what type of noise we expect to observe.  A common choice is 0 mean independent gaussian noise of the form:

ϵ~N(0,σ)

Which (Subscript[f, w])

Having selected the feature transformation ϕ and having decided to use a linear model we have reduced our hypothesis space (the space of functions we are willing to consider for f) from all the functions (and then some) to linear functions in the feature space determined by ϕ.  The functions in this space are indexed by w (the weight vector).  How do we pick f from this reduced hypothesis space?  We simply choose the "best" w.  For the remainder of these notes we will be describing how to choose the w that maximizes the likelihood of our data X and y.

Matrix Notation

Lets begin with some linear algebra.  We can apply our model to the data in the following ways:

y=(({{(Subscript[y, 1])}, {…}, {(Subscript[y, n])}}))=(({{(Subscript[f, w])(<(Subscript[X, 11]),…,(Subscript[X, 1d])>) +(Subscript[ϵ, 1])}, {…}, {(Subscript[f, w])(<(Subscript[X, n1]),…,(Subscript[X, nd])>)+(Subscript[ϵ, n])}}))=(({{(∑_(j=1)^d)(Subscript[w, j])(Subscript[X, 1j]) +(Subscript[ϵ, 1])}, {…}, {(∑_(j=1)^d)(Subscript[w, j])(Subscript[X, nj])+(Subscript[ϵ, n])}}))=X w + ϵ

where w is a d×1 column vector of weights and ϵ is a d×1 column vector of iid (Subscript[ϵ, i])~N(0,σ) gaussian noise.  Notice how we can compactly compute all the y at once by simply multiplying X w.  If we solve for the noise in equation 9 we obtain:

y-X w = ϵ ~ N(0, σ I);

(y-X w) ~ N(0, σ I);

We see that the residual of our regression model follows a multivariate gaussian with covariance σ I were I  is the identity matrix.  The density of the multivariate Gaussian takes the form:

p(V)=(1/((((2π))^(N/2))|Σ((|)^(1/2))))Exp[-(1/2)(V-μ)ᵀ (Σ^(-1))(V-μ)]

where V~N(μ,Σ) and V∈(ℝ^(N×1)) is a column vector of size N.

Likelihood

Using equation 10 and 11 we can express the likelihood of our data given our weights w as:

P(X,y|w) ∝L(w)∝Exp[-(1/2)(y-X w)ᵀ (1/σ)I (y-X w)]

We now want to maximize the likelihood of our data given the weights.   First we take the Log to make thinks easier

l(w) ∝(y-X w)ᵀ I (y-X w)

Notice that we can remove any additional multiplicative constants.  We now have

l(w)∝(Underscript[(y-X w)ᵀ, (Underscript[⏟, row vector])]) (Underscript[I, (Underscript[⏟, identity Matrix])]) (Underscript[(y-X w), (Underscript[⏟, col vector])])

You should be able to convince yourself that this is equivalent to:

l(w)∝(y-X w)ᵀ (y-X w)

Now lets take the gradient (row vector) derivative with respect to w:

(∂/(∂w)) l(w)∝(∂/(∂w))[(y-X w)ᵀ (y-X w)]

To compute this we will use the gradient of a quadratic matrix equation.
For more details see http://en.wikipedia.org/wiki/Matrix_calculus 
http://www.ee.ic.ac.uk/hp/staff/dmb/matrix/calculus.html#deriv_quad

(∂/(∂w)) l(w)∝-(y-X w)ᵀ  X- (y-X w)ᵀ X

Simplifying a little

(∂/(∂w)) l(w)∝-2(y-X w)ᵀ X

Removing extraneous constants

(∂/(∂w)) l(w)∝-(y-X w)ᵀ X

Apply the tranpose

(∂/(∂w)) l(w)∝-(yᵀ-wᵀ Xᵀ) X

Multiplying through by X:

(∂/(∂w)) l(w)∝-yᵀX+wᵀ XᵀX

Finally we set the derivative equal to zero and solve for w to obtain:

(∂/(∂w)) l(w)∝-yᵀX+wᵀ XᵀX= 0

wᵀ XᵀX=yᵀX

wᵀ =yᵀX (((XᵀX))^(-1))

Finally remvoing the transpose we have:

w= (((XᵀX))^(-1))Xᵀ y

Thus you have the matrix form of the MLE.