Matrices

David Staubli
[email protected]

HEC, University of Lausanne

2024

1 / 18
 Matrices: terminology

A matrix is a rectangular array of numbers, denoted

 
a11 a12 ··· a1K
 a21 a22 ··· a2K 
 
A= . .. .. ..  .
 .. . . . 
an1 an2 ··· anK

A is symmetric if and only if aik = aki for all i and k.

The transpose of a matrix A is A′ , where the kth row of A
corresponds to the kth column of A′ . If A is n × K, then A′ is K × n:

B = A′ ⇔ bik = aki for all i and k.

We have (A′ )′ = A.

2 / 18
 Vector multiplication
Consider two n × 1 vectors a and b such that:
   
a1 b1
 a2   b2 
   
a =  .  and b =  . 
 ..   .. 
an bn
The inner product of a and b is given by:
X
n
a′ b = a1 b1 + a2 b2 + . . . + an bn = ai bi . (a scalar)
i=1

where
a′ b = b′ a.
Similarly,
X
n
a′ a = a2i .
i=1

3 / 18
 Matrix multiplication

Consider two matrices 2 × 2 A and B such that:

   
a11 a12 b b12
A= and B = 11
a21 a22 b21 b22
The product of A and B is given by:
 
a11 b11 + a12 b21 a11 b12 + a12 b22
AB =
a21 b11 + a22 b21 a21 b12 + a22 b22
The logic extends to higher dimension matrices.
In terms of dimensions:
A |{z}
|{z} B
K×n n×K
| {z }
K×K

4 / 18
 Some useful matrix algebra rules

Some matrix algebra rules since we need them to derive the OLS
estimator and its properties

A + B = B + A;
(A + B)′ = A′ + B′ ;
If A is n × m, then AIm = A and In A = A;
A(BC) = (AB)C;
(A + B)C = AC + BC; and
(AB)′ = B′ A′ .

In general, AB ̸= BA.

5 / 18
 Symmetry, transpose and inverse of a matrix

A is symmetric if and only if aik = aki for all i and k.

The transpose of a matrix A is A′ , where the kth row of A′
corresponds to the kth column of A′ . If A is n × K, then A′ is K × n:

B = A′ ⇔ bik = aki for all i and k.

We have: (A′ )′ = A.
If the matrix B exists, then it is the inverse of A if

B = A−1 .

We have,
A−1 A = I.

6 / 18
 Rank and idempotence of a matrix

The rank of the n × K matrix A [rank(A)] is the number of linearly

independent columns of A. A has full rank if rank(A) = K.
An idempotent M is equal to its square: M = MM. It M is also
symmetric, then M = M′ M

7 / 18
 Trace of a matrix

The trace of a matrix is the sum of its diagonal elements:

X
K
tr(A) = akk .
k=1

Rule on cyclic permutations in a product:

tr(ABCD) = tr(BCDA) = tr(CDAB) = tr(DABC)

Obviously, we also have:

tr(IK ) = K

8 / 18
 Differentiating with respect to a vector
Let c and a be two k × 1 column vectors, then the product of c′ (the
transpose of c) and a is given by y = c′ a:

   
c1 a1
    X
K
c =  ... , a =  ... , y = c′ a = c1 a1 + · · · + cK aK = ci ai
|{z}
i=1
cK aK scalar
| {z } | {z }
K×1 K×1

Taking the derivative of y with respect to c, we get:

 dy   
a1
dc1
dy  .   . 
=  ..  =  ..  = a
dc dy
dc
aK
K

9 / 18
 Differentiating with respect to a vector (2)

Make sure the derivative has the same dimensions as the vector with
respect to which we differentiate.
Taking the derivative of y with respect to c′ , we get:

dy  

′
= dy
dc · · · dy
dc = a1 · · · aK = a′
dc 1 K

10 / 18
 Differentiating with respect to a vector (3)

   
c1 a11 a12
c= , A= .
c2 a12 a22
| {z } | {z }
2×1 2×2

Quadratic form of symmetric matrix A:

  
′

a11 a12 c1
Q = c Ac = c1 c2
|{z} a12 a22 c2
scalar
 

a11 c1 + a12 c2
= c1 c2
a12 c1 + a22 c2
| {z }
Ac
= a11 c21 + a12 c1 c2 + a12 c1 c2 + a22 c22
= a11 c21 + 2a12 c1 c2 + a22 c22

11 / 18
 Differentiating with respect to a vector (4)

Take derivative of Q w.r.t. c. Note: The derivative has dimension

2 × 1.
 
d(c′ Ac) dQ dQ/dc1
= =
dc dc dQ/dc2
 
2a11 c1 + 2a12 c2
=
2a12 c1 + 2a22 c2
= 2Ac
|{z}
2×1

12 / 18
 Differentiating with respect to a vector (5)

Take derivative of Q w.r.t. c′ . Note: The derivative has dimension

1 × 2.
 
d(c′ Ac) dQ dQ dQ
= ′ =
dc′ dc dc1 dc2
= 2(Ac)′
= 2c′ A′ ((AB)′ = (B′ A′ ))
= 2c′ A (A is symmetric)

13 / 18
 Review statistics

Let X and Y be random column (n × 1) vectors, with the

corresponding expected values.
       
X1 Y1 E(X1 ) E(Y1 )
       
X =  ...  , Y =  ...  , E(X) =  ...  , E(Y) =  ... 
Xn Yn E(Xn ) E(Yn )

14 / 18
 Review statistics (2)

The linearity of the expectation operator extends to vectors:

   
X1 + Y1 E(X1 + Y1 )
 ..   .. 
E(X + Y) = E  .  =  . 
Xn + Yn E(Xn + Yn )
   
E(X1 ) E(Y1 )
 ..   .. 
= . + . 
E(Xn ) E(Yn )

= E(X) + E(Y)

Let A be a constant m × n matrix.

E(AX) = AE(X)

15 / 18
 Review statistics (3)

To define the variance of the random n × 1 vector Y, we have to take

account of the covariances. The variance operator for the vector Y is
defined as:

V ar(Y) = E[(Y − µ Y )(Y − µ Y )′ ]

  
Y1 − µY 1
 .. 

= E  .  Y1 − µY 1 ··· Yn − µY n 
Yn − µY n
 
E[(Y1 − µY 1 )2 ] ··· E[(Y1 − µY 1 )(Yn − µY n )]
 .. .. .. 
= . . . 
E[(Yn − µY n )(Y1 − µY 1 )] · · · E[(Yn − µY n )2 ]

16 / 18
 Review statistics (4)

 
V ar(Y1 ) ··· Cov(Y1 , Yn )
 . .. .. 
V ar(Y) =  .
. . . 
Cov(Yn , Y1 ) · · · V ar(Yn )

1. Cov(Y1 , Y2 ) = Cov(Y2 , Y1 ) and, generally,

Cov(Yi , Yj ) = Cov(Yj , Yi ), which implies that V ar(Y) is a
symmetric n × n matrix;
2. if Cov(Yi , Yj ) = 0 for all i ̸= j, then all off-diagonal elements are
zero;
3. if V ar(Yi ) = σY for all i, then all diagonal elements are equal to σY ;
4. if 2. and 3. hold, then V ar(Y) = σY In .

17 / 18
 Review statistics (5)

The variance operator for the vector AY:

V ar(AY) = E[(AY − Aµ µY )′ ] (by def. of variance)

µY )(AY − Aµ
= E[A(Y − µ Y )(Y − µ Y )′ A′ ] ((AB)′ = (B′ A′ ))
= AE[(Y − µ Y )(Y − µ Y )′ ]A′ (linearity of E()-operator)
′
= AV ar(Y)A (by def. of variance)

18 / 18