Symmetric matrices and dot products

Proposition
An n × n matrix A is symmetric iff, for all x, y in Rn ,
(Ax) · y = x · (Ay).

Proof.
If A is symmetric, then (Ax) · y = xT AT y = xT Ay = x · (Ay).
If equality holds for all x, y in Rn , let x, y vary over the standard
basis of Rn .

Corollary
If A is symmetric and x, y are eigvecs corresponding to different
eigvals λ, µ, then x · y = 0.

Proof.
λx · y = (Ax) · y = x · (Ay) = x · (µy) = µx · y, so
(λ − µ)(x · y) = 0, but λ − µ 6= 0.
Proposition
Every eigenvalue of a symmetric matrix (with real entries) is real,
and we can pick the corresponding eigenvector to have real entries.

Proof.
Suppose A is symmetric and Ax = λx with x 6= 0 (maybe with
complex entries). Then Ax = Ax = λx, so

λ(x · x) = (Ax) · x = x · (Ax) = λ(x · x) ;

but x · x is real and positive, so λ = λ.

And if x is an eigvec with complex entries corr to λ, then we can
multiply by some complex number and get at least one real nonzero
entry, and then x + x is an eigvec corr to λ with real entries.
Lemma
If A is a symmetric matrix with (real) eigvec x, then if y is
perpendicular to x, then so is Ay (i.e., A multiplies the orthogonal
complement (Rx)⊥ into itself).

Proof.
Let λ be the corr eigval. Because y · x = 0, we have

(Ay) · x = y · (Ax) = y · (λx) = λ(y · x) = 0 .

Symmetrics have orthogonal diagonalization

Theorem (Spectral Theorem)

If A is symmetric, then there is an orthogonal matrix Q and a
diagonal matrix Λ for which A = QΛQ T .
Proof
Let λ1 , x1 be a real eigval and eigvec of A, with ||x|| = 1. Pick an
orthonormal basis B for (Rx)⊥ ; then x, B form an orthonormal
basis for Rn . Use them as columns of an orthogonal matrix Q1 ,
with x1 first; then because A multiplies (Rx)⊥ into itself, we have
 
λ1 0
A = Q1 Q1T .
0 A2
Because
   
λ1 0 λ1 0
Q1 Q1T T
= A = A = Q1 T Q1T ,
0 A2 0 A2

A2 is still symmetric; so we can repeat the process with A2 ; and if

 
λ2 0
A2 = Q2 Q2T
0 A3

then
 
  λ1 0 0  
1 0 1 0
A = Q1  0 λ2 0  Q1T .
0 Q2 0 Q2T
0 0 A3

Continuing, we get to the desired form.

Example

For a symmetic A, R gives orthonormal eigenvectors (and

eigenvalues in decreasing order).
  
Suppose we have A symmetric, Q = x1 . . . xn and
Λ = diag(λ1 , . . . , λn ) with A = QΛQ T . Write
   
λ1 0 . . . 0 0 0 ... 0
 0 0 ... 0   0 0 ... 0 
Λ= . . .  + · · · +  .. ..  .
   
. . . . . .. ..
 . . . .   . . . . 
0 0 ... 0 0 0 ... λn
Then
 
λ1 0 . . . 0
 0 0 ... 0 
 T
Q Q

.. .. . . ..
 . . . . 
0 0 ... 0
xT
  
λ1 0 . . . 0 1
  0 0 ... 0  xT
2

= x1 x2 . . . xn
  
 .. .. . . ..  .. 
 . . . .  . 
0 0 ... 0 xT
n
= λ1 x1 xT
1
And so on, so we get

A = λ1 x1 xT T T
1 + λ2 x2 x2 + · · · + λn xn xn .

The xi xT
i ’s are the projection matrices Pi ’s on the subspaces Rxi ’s.
(And they are rank 1 matrices.)

Because they are the orthogonal projections onto an orthonormal

basis, they add up to I — if we apply them all to the same vector
and add up the results, we’ll get back the original vector.
Example:
 
9 −2
A= :
−2 6
 √   √ 
1/√5 −2/√5
λ1 = 5, x1 = λ2 = 10, x2 =
2/ 5 1/ 5

√ 
1/√5  √ √ 

A=5 1/ 5 2/ 5
2/ 5
√ 
√ √ 

−2/√5 
+ 10 −2/ 5 1/ 5
1/ 5
   
1/5 2/5 4/5 −2/5
=5 + 10
2/5 4/5 −2/5 1/5
   
1/5 2/5 4/5 −2/5
and + = I.
2/5 4/5 −2/5 1/5
Definition
A quadratic form on Rn is a function q : Rn → R given Pby a
polynomial in which every term has degree 2: q(x) = i,j ai,j xi xj .
Because xi xj = xj xi , we can assume ai,j = aj,i , and rewrite
q(x) = xT Ax where A is symmetric.
I If q(x) > 0 for every nonzero x, then q and A are positive
definite.
I If q(x) ≥ 0 for every x, then they are positive semidefinite.
Example 1
 
1 −2 1/2
x12 + 2x22 − 3x32 − 4x1 x2 + x1 x3 + 2x2 x3 = xT  −2 2 1 x
1/2 1 −3

Because of the −3 on the main diagonal, this quadratic form is not

even pos semidef (nor is its associated matrix):
  
  1 −2 1/2 0
0 0 1  −2 2 1   0  = −3
1/2 1 −3 1

Moral: If any main diagonal entry is ≤ 0, the matrix is not pos def.
Example 2
Positive definite:

(x + 3y )2 + 4(2x − y )2 = 17x 2 − 10xy + 13y 2

  
  17 −5 x
= x y
−5 13 y

Example 3

(x + 3y )2 + 4(2y − z)2 + 2(2x + 3z)2

= 9x 2 + 25y 2 + 22z 2 + 6xy − 16yz + 24xz
  
  9 3 12 x
= x y z  3 25 −8   y 
12 −8 22 z

Certainly always nonnegative, but if x = −3y = −3z/2, all 3

squares are 0; so pos semidef.
Example 4
What about f (x, y , z) = 23x 2 + 14y 2 + 8z 2 − 28xy + 4xz + 32yz?

4 4 2 16 64
f (x, y ) = 23(x 2 + xz + z ) + 14(y 2 − yz + z 2 )
 23 529 7 49
4 128 2
+ 8− − z − 28xy
23 7
2 8 1684 2
= 23(x + z)2 + 14(y − z)2 − z − 28xy .
23 7 161

Can we make this negative for some x, y , z? Well, try setting

x = −(2/23)z and y = (8/7)z: That will make f into
−(1652/161)z 2 , which is negative for any z 6= 0. So f is not even
pos semidef.
Application: Second Derivative Test

Suppose z = f (x, y ) is a function of two variables, and at a point

(a, b) the first derivatives ∂f /∂x and ∂f /∂y are both 0. Then the
Taylor series for f at that point is

∂2f 2 ∂2f
f (a, b) + (a, b)(x − a) + (a, b)(x − a)(y − b)
∂x 2 ∂x ∂y
∂2f ∂2f
+ (a, b)(y − b)(x − a) + 2 (a, b)(y − b)2 + . . .
∂y ∂x ∂y

where the rest is higher-degree terms. So near (a, b), f behaves

like the quadratic part, which is a quadratic form. When this form
is positive definite, the smallest value f takes on in a small
neighborhood of (a, b) at (a, b). In other words, (a, b) is a local
minimum of f .
So it would be useful to have a quick way to decide whether the
matrix is positive definite:
∂2f ∂2f
" #
2
∂ x
(a, b) ∂x ∂y (a, b)
∂2f ∂2f
∂y ∂x (a, b) ∂y 2
(a, b)
Theorem
Let A be a symmetric matrix. TFAE:
(a) All pivots of A are positive.
(b) All upper left subdeterminants are positive. (This is
Sylvester’s criterion.)
(c) All eigenvalues of A are positive.
(d) The quadratic form xT Ax is positive definite. (This is the
energy-based definition.)
(e) A = R T R where R has independent columns.

Proof
(a) =⇒ (b): a1,1 is first pivot, hence pos, and first subdet. By
row ops, upper left subdets of A are equal to subdets of a matrix,
say B, with the rest of the first column 0’s. Second entry on B’s
main diagonal is A’s 2nd pivot; because it and a1,1 are pos, B’s
2nd upper left subdet is also pos. Etc.
(b) =⇒ (a): First subdet is a1,1 > 0, first pivot. Repeat the rest
of (a) =⇒ (b), except reversing causality: k-th pivot is pos
because k × k subdet is pos.
(c) ⇐⇒ (d): Write A = QΛQ T ; then
X
xT Ax = xT QΛQ T x = (Q T x)T Λ(Q T x) = λi yi2 ,
i

where the yi ’s are the entries of Q T x. So if all the λi ’s are

positive, xT Ax is positive definite; while if one is negative, then
there are x’s for which xT Ax is negative.
T
√
(c) =⇒ (e): Write A = QΛQ √ . TThen by (c), Λ makes sense
T
and is invertible. Set R = Q ΛQ . Then A = R R, and the
columns of R are independent.
(e) =⇒ (d): Given A = R T R, xT Ax = (Rx) · (Rx) = ||Rx||2 ≥ 0.
And if R has ind cols, then the only x with ||Rx||2 = 0 is x = 0.
(a) =⇒ (e): Because A is symmetric, (a) says it has an
LDU-decomposition A = LDLT where L is lower triangular with 1’s
on the main diagonal,
√ and D is diagonal with the pivots of A on its
diagonal. Set R = DL .T

(d) =⇒ (b): Let bk = the upper left k × k subdet of A, and

ckj = the cofactor of akj if we evaluate bk along its last row. Then
for i < k, kj=1 aij ckj is the value of the same det as bk except
P
that
Pk its last row is equal to its i-th row, so it is 0; but
j=1 akj ckj = bk . Note ckk = bk−1 . We prove by induction (and
pos defness) that all bk ’s are pos:
Because A is pos def, b1 = a11 > 0. Assume bk−1 > 0. Set

x = (ck1 , ck2 , . . . , ckk , 0, . . . , 0) .

Then

0 < xT Ax = xT (0, . . . , 0, bk , ?, . . . , ?) = bk−1 bk ,

so bk > 0.
Notes on proof:
(e): The proof of (c) =⇒ (e) gives one square, in fact symmetric,
R that works (the Cholesky decomposition), but any R with
independent columns, even if not square, works.

Example:  
  1 0  
T 1 2 −1  6 5
A=R R= 2 3 =  , and R
0 3 1 5 10
−1 1
has independent columns, so A is positive definite.

Therefore, the eigenvalues of −A are all negative, so the

solutions of
du
= −Au
dt
approach 0 as t → ∞.
(c): The (c) =⇒ (d) part of the proof shows us the right way to
complete squares in quadratic forms. (Example, next page.)
Example 4, revisited:
For f (x, y , z) = 23x 2 + 14y 2 + 8z 2 − 28xy + 4xz + 32yz, the
matrix is
 
23 −14 2
A =  −14 14 16  = QΛQ T where
2 16 8
   
1/3 2/3 −2/3 −9 0 0
Q =  2/3 1/3 2/3  and Λ =  0 18 0  , so
−2/3 2/3 1/3 0 0 36

xT Ax = xT QΛQ T x = (Q T x)T Λ(Q T x)

1 2 2 2 1 2
= −9( x + y − z)2 + 18( x + y + z)2
3 3 3 3 3 3
2 2 1 2
+ 36(− x + y + z) ;
3 3 3
again, not even pos semidef.
Proposition
A symmetric matrix is positive semidefinite iff its eigenvalues are
nonnegative, or equivalently iff it has the form R T R.

Proof.
Same as the positive definite case, except that, if the cols of R are
not ind, there are nonzero x’s for which Rx = 0.
Example 1 revisited:
 
1 −2 1/2
xT  −2 2 1 x
1/2 1 −3
 
  1 −2 1/2
1 −2
det[1] = 1, det = −2, det  −2 2 1  = 5/2
−2 2
1/2 1 −3
Not even positive semidefinite. Eigenvalues from R: 3.5978953,
−0.2047827, −3.3931126
Example 2 revisited:
Positive definite:
  
  17 −5 x
x y
−5 13 y
 
17 −5
det[17] = 17, det = 196
−5 13
Eigenvalues from R: 20.385165, 9.614835
Example 3 revisited:

(x + 3y )2 + 4(2y − z)2 + 2(2x + 3z)2

= 9x 2 + 25y 2 + 22z 2 + 6xy − 16yz + 24xz
  
  9 3 12 x
= x y z  3 25 −8   y 
12 −8 22 z
 
  9 3 12
9 3
det[9] = 9 , det = 216 , det  3 25 −8  = 0 ;
3 25
12 −8 22
pos semidef. (Eigvals from R: 33.29150, 22.70850, 0.)
Example 4 re-revisited:

23x 2 + 14y 2 + 8z 2 − 28xy + 4xz + 32yz = xT Ax where

 
23 −14 2
A =  −14 14 16 
2 16 8
 
23 −14
det[23] = 23 , det = 126 , det A = −5832 ,
−14 14
so not pos def. (We saw eigvals were −9, 18, 36.)
Application: Tilted Ellipses

The equation of an ellipse in standard position is

x 2 /a2 + y 2 /b 2 = 1, i.e.,

 1/a2
  
 0 x
x y =1.
0 1/b 2 y

The semiaxes are a and b (the positive square roots of the

reciprocals of the eigvals). But a rotation matrix Q gives a tilted
ellipse. The axes are in the directions of the columns of Q.
Example 5
   
1/16 0 4/5 3/5
Λ= , Q= :
0 1/9 3/5 −4/5
 
T 288/3600 −84/3600
A = QΛQ =
−84/3600 337/3600

1 2 1 2 288 2 168 337 2

x + y =1 x − xy + y =1
16 9 3600 3600 3600
Example 6
√
Describe the ellipse 13x 2 − 6 3xy + 7y 2 = 4.

Divide by 4: √
13 2 3 3 7
x − xy + y 2 = 1
4 2 4
The quadratic form has matrix
 √ 
√13/4 −3 3/4
A= = QΛQ T
−3 3/4 7/4

where  √   
Q= √1/2 3/2
, Λ=
1 0
.
3/2 −1/2 0 4
So the ellipse has axes tilted 60◦ from the xy -axes, with
half-lengths 1 and 1/2.
Example 7

Describe the ellipse x 2 + 6xy + 4y 2 = 16.

The matrix of the quadratic form is

 
1/16 3/16
A= ,
3/16 1/4

and R shows that its eigvals are 0.35688137 and −0.05338137.

The negative eigenvalue means that it is not an ellipse. The graph
of the quadratic form z = (x 2 + 6xy + 4y 2 )/16 is a saddle, and the
cross section z = 1 is a hyperbola.
Or,
 
1/16 3/16
det[1/16] = 1/16, det = −5/256 < 0
3/16 1/4