QR Factorization: Triangular Matrices QR Factorization Gram-Schmidt Algorithm Householder Algorithm

L.
Vandenberghe ECE133A (Fall 2022)
6. QR factorization
• triangular matrices
• QR factorization
• Gram–Schmidt algorithm
• Householder algorithm
6.1
Triangular matrix
a square matrix 𝐴 is lower triangular if 𝐴𝑖 𝑗 = 0 for 𝑗 > 𝑖
 𝐴11 0 ··· 0 0 
0 0
 
 𝐴21 𝐴22 ··· 

𝐴 =  .. .. ... 0 0


0

 𝐴𝑛−1,1
 𝐴𝑛−1,2 ··· 𝐴𝑛−1,𝑛−1 

 𝐴𝑛1
 𝐴𝑛2 ··· 𝐴𝑛,𝑛−1 𝐴𝑛𝑛 

𝐴 is upper triangular if 𝐴𝑖 𝑗 = 0 for 𝑗 < 𝑖 (the transpose 𝐴𝑇 is lower triangular)
a triangular matrix is unit upper/lower triangular if 𝐴𝑖𝑖 = 1 for all 𝑖
QR factorization 6.2
Forward substitution
solve 𝐴𝑥 = 𝑏 when 𝐴 is lower triangular with nonzero diagonal elements
Algorithm
𝑥1 = 𝑏 1/𝐴11
𝑥2 = (𝑏 2 − 𝐴21𝑥1)/𝐴22
𝑥3 = (𝑏 3 − 𝐴31𝑥1 − 𝐴32𝑥2)/𝐴33
..
𝑥𝑛 = (𝑏 𝑛 − 𝐴𝑛1𝑥1 − 𝐴𝑛2𝑥2 − · · · − 𝐴𝑛,𝑛−1𝑥 𝑛−1)/𝐴𝑛𝑛
Complexity: 1 + 3 + 5 + · · · + (2𝑛 − 1) = 𝑛2 flops
Back substitution
solve 𝐴𝑥 = 𝑏 when 𝐴 is upper triangular with nonzero diagonal elements
Algorithm
𝑥 𝑛 = 𝑏 𝑛 /𝐴𝑛𝑛
𝑥 𝑛−1 = (𝑏 𝑛−1 − 𝐴𝑛−1,𝑛 𝑥 𝑛 )/𝐴𝑛−1,𝑛−1
𝑥 𝑛−2 = (𝑏 𝑛−2 − 𝐴𝑛−2,𝑛−1𝑥 𝑛−1 − 𝐴𝑛−2,𝑛 𝑥 𝑛 )/𝐴𝑛−2,𝑛−2
..
𝑥1 = (𝑏 1 − 𝐴12𝑥2 − 𝐴13𝑥3 − · · · − 𝐴1𝑛 𝑥 𝑛 )/𝐴11
Complexity: 𝑛2 flops
Inverse of triangular matrix
a triangular matrix 𝐴 with nonzero diagonal elements is nonsingular:
𝐴𝑥 = 0 =⇒ 𝑥=0
this follows from forward or back substitution applied to the equation 𝐴𝑥 = 0
• inverse of 𝐴 can be computed by solving 𝐴𝑋 = 𝐼 column by column
= (𝑥𝑖 is column 𝑖 of 𝑋 )

𝐴 𝑥1 𝑥2 · · · 𝑥𝑛 𝑒1 𝑒2 · · · 𝑒𝑛
• inverse of lower triangular matrix is lower triangular

• inverse of upper triangular matrix is upper triangular
• complexity of computing inverse of 𝑛 × 𝑛 triangular matrix is
𝑛2 + (𝑛 − 1) 2 + · · · + 1 ≈ 13 𝑛3 flops
Outline
QR factorization
if 𝐴 ∈ R𝑚×𝑛 has linearly independent columns then it can be factored as
 𝑅11 𝑅12 · · · 𝑅1𝑛 

 0
 
𝑅22 · · · 𝑅2𝑛 
𝐴=

𝑞1 𝑞2 · · · 𝑞 𝑛  . .. ... .. 
 .

 0 0 ···

 𝑅𝑛𝑛 

• vectors 𝑞 1, . . . , 𝑞 𝑛 are orthonormal 𝑚 -vectors:
k𝑞𝑖 k = 1, 𝑞𝑇𝑖 𝑞 𝑗 = 0 if 𝑖 ≠ 𝑗
• diagonal elements 𝑅𝑖𝑖 are nonzero

• if 𝑅𝑖𝑖 < 0, we can switch the signs of 𝑅𝑖𝑖 , . . . , 𝑅𝑖𝑛 , and the vector 𝑞𝑖
• most definitions require 𝑅𝑖𝑖 > 0; this makes 𝑄 and 𝑅 unique
QR factorization in matrix notation
if 𝐴 ∈ R𝑚×𝑛 has linearly independent columns then it can be factored as
𝐴 = 𝑄𝑅
Q-factor
• 𝑄 is 𝑚 × 𝑛 with orthonormal columns (𝑄𝑇 𝑄 = 𝐼 )

• if 𝐴 is square (𝑚 = 𝑛), then 𝑄 is orthogonal (𝑄𝑇 𝑄 = 𝑄𝑄𝑇 = 𝐼 )
R-factor
• 𝑅 is 𝑛 × 𝑛, upper triangular, with nonzero diagonal elements

• 𝑅 is nonsingular (diagonal elements are nonzero)
Example
 −1 −1 1   −1/2 1/2 −1/2 

 1 3 3   1/2 1/2 −1/2   2 4 2 
    
=  0 2 8 
 −1 −1 5   −1/2 1/2 1/2  
 
0 0 4 

 1 3 7   1/2 1/2 1/2 
  
 
 
 𝑅11 𝑅12 𝑅13

 
𝑞 3  0

=

𝑞1 𝑞2 𝑅22 𝑅23 
 0 0

 𝑅33 

= 𝑄𝑅
Full QR factorization
the QR factorization is often defined as a factorization

𝑅
𝐴= 𝑄˜

𝑄
0
• 𝐴 = 𝑄𝑅 is the QR factorization as defined earlier (page 6.7)

• 𝑄˜ has size 𝑚 × (𝑚 − 𝑛) , the zero block has size (𝑚 − 𝑛) × 𝑛
˜
• the matrix 𝑄 𝑄 is 𝑚 × 𝑚 and orthogonal

• MATLAB’s function qr returns this factorization

• this is also known as the full QR factorization or QR decomposition
in this course we use the definition of page 6.7
Applications
in the following lectures, we will use the QR factorization to solve
• linear equations
• least squares problems
• constrained least squares problems
here, we show that it gives useful simple formulas for
• the pseudo-inverse of a matrix with linearly independent columns

• the inverse of a nonsingular matrix
• projection on the range of a matrix with linearly independent columns
QR factorization and (pseudo-)inverse
pseudo-inverse of a matrix 𝐴 with linearly independent columns (page 4.22)
𝐴† = ( 𝐴𝑇 𝐴) −1 𝐴𝑇
• pseudo-inverse in terms of QR factors of 𝐴:
𝐴† = ((𝑄𝑅)𝑇 (𝑄𝑅)) −1 (𝑄𝑅)𝑇

= (𝑅𝑇 𝑄𝑇 𝑄𝑅) −1 𝑅𝑇 𝑄𝑇
= (𝑅𝑇 𝑅) −1 𝑅𝑇 𝑄𝑇 (𝑄𝑇 𝑄 = 𝐼 )
= 𝑅 −1 𝑅 −𝑇 𝑅𝑇 𝑄𝑇 ( 𝑅 is nonsingular)
= 𝑅 −1𝑄𝑇
• for square nonsingular 𝐴 this is the inverse:
𝐴−1 = (𝑄𝑅) −1 = 𝑅 −1𝑄𝑇
Range
recall definition of range of a matrix 𝐴 ∈ R𝑚×𝑛 (page 5.16):
range( 𝐴) = {𝐴𝑥 | 𝑥 ∈ R𝑛 }
suppose 𝐴 has linearly independent columns with QR factors 𝑄 , 𝑅
• 𝑄 has the same range as 𝐴:
𝑦 ∈ range( 𝐴) ⇐⇒ 𝑦 = 𝐴𝑥 for some 𝑥

⇐⇒ 𝑦 = 𝑄𝑅𝑥 for some 𝑥
⇐⇒ 𝑦 = 𝑄𝑧 for some 𝑧
⇐⇒ 𝑦 ∈ range(𝑄)
• columns of 𝑄 are an orthonormal basis for range( 𝐴)
Projection on range
• combining 𝐴 = 𝑄𝑅 and 𝐴† = 𝑅 −1𝑄𝑇 (from page 6.11) gives
𝐴𝐴† = 𝑄𝑅𝑅 −1𝑄𝑇 = 𝑄𝑄𝑇
note the order of the product in 𝐴𝐴† and the difference with 𝐴† 𝐴 = 𝐼
• recall (from page 5.17) that 𝑄𝑄𝑇 𝑥 is the projection of 𝑥 on the range of 𝑄
𝑥
𝐴𝐴†𝑥 = 𝑄𝑄𝑇 𝑥
range( 𝐴) = range(𝑄)
QR factorization of complex matrices
if 𝐴 ∈ C𝑚×𝑛 has linearly independent columns then it can be factored as
𝐴 = 𝑄𝑅
• 𝑄 ∈ C𝑚×𝑛 has orthonormal columns (𝑄 𝐻 𝑄 = 𝐼 )

• 𝑅 ∈ C𝑛×𝑛 is upper triangular with real nonzero diagonal elements
• most definitions choose diagonal elements 𝑅𝑖𝑖 to be positive
• in the rest of the lecture we assume 𝐴 is real
Algorithms for QR factorization
Gram–Schmidt algorithm (section 5.4 in textbook and page 6.16)

• complexity is 2𝑚𝑛2 flops
• not recommended in practice (sensitive to rounding errors)
Modified Gram–Schmidt algorithm

• complexity is 2𝑚𝑛2 flops
• better numerical properties
Householder algorithm (page 6.26)

• complexity is 2𝑚𝑛2 − (2/3)𝑛3 flops
• represents 𝑄 as a product of elementary orthogonal matrices
• the most widely used algorithm (used by the function qr in MATLAB and Julia)
in the rest of the course we will take 2𝑚𝑛2 for the complexity of QR factorization
Outline
Gram–Schmidt algorithm
Gram–Schmidt QR algorithm computes 𝑄 and 𝑅 column by column
• after 𝑘 steps we have a partial QR factorization
 𝑅11 𝑅12 · · · 𝑅1𝑘 

 0
 
𝑅22 · · · 𝑅2𝑘 
=

𝑎1 𝑎2 · · · 𝑎𝑘 𝑞1 𝑞2 · · · 𝑞𝑘  .
 . .. ... .. 

 0 0 ···
 
 𝑅𝑘 𝑘 

this is the QR factorization for the first 𝑘 columns of 𝐴
• columns 𝑞 1, . . . , 𝑞 𝑘 are orthonormal

• diagonal elements 𝑅11, 𝑅22, . . . , 𝑅 𝑘 𝑘 are positive
• columns 𝑞 1, . . . , 𝑞 𝑘 have the same span as 𝑎 1, . . . , 𝑎 𝑘 (see page 6.12)
• in step 𝑘 of the algorithm we compute 𝑞 𝑘 , 𝑅1𝑘 , . . . , 𝑅 𝑘 𝑘
Computing the 𝑘 th columns of 𝑄 and 𝑅
suppose we have the partial factorization for the first 𝑘 − 1 columns of 𝑄 and 𝑅
• column 𝑘 of the equation 𝐴 = 𝑄𝑅 reads
𝑎 𝑘 = 𝑅1𝑘 𝑞 1 + 𝑅2𝑘 𝑞 2 + · · · + 𝑅 𝑘−1,𝑘 𝑞 𝑘−1 + 𝑅 𝑘 𝑘 𝑞 𝑘
• regardless of how we choose 𝑅1𝑘 , . . . , 𝑅 𝑘−1,𝑘 , the vector
𝑞˜ 𝑘 = 𝑎 𝑘 − 𝑅1𝑘 𝑞 1 − 𝑅2𝑘 𝑞 2 − · · · − 𝑅 𝑘−1,𝑘 𝑞 𝑘−1
will be nonzero: 𝑎 1, 𝑎 2, . . . , 𝑎 𝑘 are linearly independent and therefore
𝑎 𝑘 ∉ span{𝑎 1, . . . , 𝑎 𝑘−1 } = span{𝑞 1, . . . , 𝑞 𝑘−1 }
• 𝑞 𝑘 is 𝑞˜ 𝑘 normalized: choose 𝑅 𝑘 𝑘 = k 𝑞˜ 𝑘 k and 𝑞 𝑘 = (1/𝑅 𝑘 𝑘 ) 𝑞˜ 𝑘

• 𝑞˜ 𝑘 and 𝑞 𝑘 are orthogonal to 𝑞 1, . . . , 𝑞 𝑘−1 if we choose 𝑅1𝑘 , . . . , 𝑅 𝑘−1,𝑘 as
𝑅1𝑘 = 𝑞𝑇1 𝑎 𝑘 , 𝑅2𝑘 = 𝑞𝑇2 𝑎 𝑘 , ..., 𝑅 𝑘−1,𝑘 = 𝑞𝑇𝑘−1 𝑎 𝑘
Gram–Schmidt algorithm
Given: 𝑚 × 𝑛 matrix 𝐴 with linearly independent columns 𝑎 1, . . . , 𝑎 𝑛
Algorithm
for 𝑘 = 1 to 𝑛
𝑅1𝑘 = 𝑞𝑇1 𝑎 𝑘
𝑅2𝑘 = 𝑞𝑇2 𝑎 𝑘
..
𝑅 𝑘−1,𝑘 = 𝑞𝑇𝑘−1 𝑎 𝑘
𝑞˜ 𝑘 = 𝑎 𝑘 − (𝑅1𝑘 𝑞 1 + 𝑅2𝑘 𝑞 2 + · · · + 𝑅 𝑘−1,𝑘 𝑞 𝑘−1)
𝑅𝑘 𝑘 = k 𝑞˜ 𝑘 k
1
𝑞𝑘 = 𝑞˜ 𝑘
𝑅𝑘 𝑘
Example
example on page 6.8:
 −1 −1 1 
 1 3 3 
 
=

𝑎1 𝑎2 𝑎3  −1 −1 5 

 1 3 7 
 

 𝑅11 𝑅12 𝑅13 
𝑞 3  0
 
=

𝑞1 𝑞2 𝑅22 𝑅23 
 0 0

 𝑅33 

First column of 𝑄 and 𝑅
 −1   −1/2 
 1  1 1/2 
   
𝑞˜ 1 = 𝑎 1 =  𝑅11 = k 𝑞˜ 1 k = 2, 𝑞˜ 1 = 

, 𝑞1 = 
 −1  𝑅11  −1/2 
 
 1   1/2 
   
Example
Second column of 𝑄 and 𝑅
• compute 𝑅12 = 𝑞𝑇1 𝑎 2 = 4

• compute
 −1   −1/2   1 
 3   1/2   1 
     
𝑞˜ 2 = 𝑎 2 − 𝑅12 𝑞 1 =   −4 = 
 −1   −1/2   1 
    
 3   1/2   1 
     
• normalize to get
 1/2 
1 1/2
 
𝑅22 = k 𝑞˜ 2 k = 2, 𝑞˜ 2 = 
 
𝑞2 =
1/2

𝑅22 
1/2
 
 
 
Example
Third column of 𝑄 and 𝑅
• compute 𝑅13 = 𝑞𝑇1 𝑎 3 = 2 and 𝑅23 = 𝑞𝑇2 𝑎 3 = 8

• compute
 1   −1/2   1/2   −2 
3  1/2   1/2   −2 
       
𝑞˜ 3 = 𝑎 3 − 𝑅13 𝑞 1 − 𝑅23 𝑞 2 =   −2
 −1/2  − 8  1/2  =  2 
 
5
    

7  1/2   1/2   2 
       
 
       
• normalize to get
 −1/2 
1
 
−1/2 
𝑅33 = k 𝑞˜ 3 k = 4, 𝑞˜ 3 = 

𝑞3 =
 1/2 

𝑅33 
 1/2 
 
Example
Final result
 −1 −1 1 
 1 3 3   𝑅11 𝑅12 𝑅13
   
𝑞 3  0

=

 −1 −1 5 
 𝑞1 𝑞2 𝑅22 𝑅23 
 0 0

 1 𝑅33
3 7 
  
 

 −1/2 1/2 −1/2 
 1/2 1/2 −1/2   2 4 2 
  
=  0 2 8 
 −1/2 1/2 1/2  

0 0 4 

 1/2 1/2 1/2 

 

Complexity
Complexity of cycle 𝑘 (of algorithm on page 6.18)
• 𝑘 − 1 inner products with 𝑎 𝑘 : (𝑘 − 1) (2𝑚 − 1) flops

• computation of 𝑞˜ 𝑘 : 2(𝑘 − 1)𝑚 flops
• computing 𝑅 𝑘 𝑘 and 𝑞 𝑘 : 3𝑚 flops
total for cycle 𝑘 : (4𝑚 − 1) (𝑘 − 1) + 3𝑚 flops
Complexity for 𝑚 × 𝑛 factorization:
𝑛
𝑛(𝑛 − 1)
((4𝑚 − 1) (𝑘 − 1) + 3𝑚) (4𝑚 − 1) + 3𝑚𝑛
X
=
𝑘=1 2
≈ 2𝑚𝑛2 flops
Numerical experiment
• we use the following MATLAB implementation of the algorithm on page 6.18:

[m, n] = size(A);
Q = zeros(m,n);
R = zeros(n,n);
for k = 1:n
R(1:k-1,k) = Q(:,1:k-1)’ * A(:,k);
v = A(:,k) - Q(:,1:k-1) * R(1:k-1,k);
R(k,k) = norm(v);
Q(:,k) = v / R(k,k);
end;
• we apply this to a square matrix 𝐴 of size 𝑚 = 𝑛 = 50
• 𝐴 is constructed as 𝐴 = 𝑈𝑆𝑉 with 𝑈 , 𝑉 orthogonal, 𝑆 diagonal with
𝑆𝑖𝑖 = 10−10(𝑖−1)/(𝑛−1) , 𝑖 = 1, . . . , 𝑛
Numerical experiment
plot shows deviation from orthogonality between 𝑞 𝑘 and previous columns
𝑒 𝑘 = max |𝑞𝑇𝑖 𝑞 𝑘 |, 𝑘 = 2, . . . , 𝑛
1≤𝑖<𝑘
0.8
0.6
𝑒𝑘
0.4
0.2
0
0 10 20 30 40 50
𝑘
loss of orthogonality is due to rounding error

Outline
Householder algorithm
• the most widely used algorithm for QR factorization (qr in MATLAB and Julia)
• less sensitive to rounding error than Gram–Schmidt algorithm
• computes a “full” QR factorization (QR decomposition)

𝑅
𝐴= 𝑄˜ 𝑄˜ orthogonal

𝑄 , 𝑄
0
• the full Q-factor is constructed as a product of orthogonal matrices
𝑄˜ = 𝐻1 𝐻2 · · · 𝐻 𝑛

𝑄
each 𝐻𝑖 is an 𝑚 × 𝑚 symmetric, orthogonal “reflector” (page 5.10)
Reflector
𝐻 = 𝐼 − 2𝑣𝑣𝑇 with k𝑣k = 1
• 𝐻𝑥 is reflection of 𝑥 through hyperplane {𝑧 | 𝑣𝑇 𝑧 = 0} (see page 5.10)

• 𝐻 is symmetric
• 𝐻 is orthogonal
• matrix–vector product 𝐻𝑥 can be computed efficiently as
𝐻𝑥 = 𝑥 − 2(𝑣𝑇 𝑥)𝑣
complexity is 4𝑝 flops if 𝑣 and 𝑥 have length 𝑝
Reflection to multiple of unit vector
given nonzero 𝑝 -vector 𝑦 = (𝑦 1, 𝑦 2, . . . , 𝑦 𝑝 ) , define
 𝑦 1 + sign(𝑦 1) k𝑦k 
1
 
 𝑦2 
𝑤 =  .. , 𝑣= 𝑤


 k𝑤k

 𝑦𝑝 

• we define sign(0) = 1
• vector 𝑤 satisfies
k𝑤k 2 = 2 (𝑤𝑇 𝑦) = 2k𝑦k (k𝑦k + |𝑦 1 |)
• reflector 𝐻 = 𝐼 − 2𝑣𝑣𝑇 maps 𝑦 to multiple of 𝑒 1 = (1, 0, . . . , 0) :
2(𝑤𝑇 𝑦)
𝐻𝑦 = 𝑦 − 2
𝑤 = 𝑦 − 𝑤 = −sign(𝑦 1)k𝑦k𝑒 1
k𝑤k
Geometry
𝑤
𝑦
first coordinate axis

−sign(𝑦 1) k𝑦k𝑒 1
hyperplane {𝑥 | 𝑤𝑇 𝑥 = 0}
the reflection through the hyperplane {𝑥 | 𝑤𝑇 𝑥 = 0} with normal vector
𝑤 = 𝑦 + sign(𝑦 1) k𝑦k𝑒 1
maps 𝑦 to the vector −sign(𝑦 1) k𝑦k𝑒 1
Householder triangularization
• computes reflectors 𝐻1, . . . , 𝐻𝑛 that reduce 𝐴 to triangular form:

𝑅
𝐻𝑛 𝐻𝑛−1 · · · 𝐻1 𝐴 =
0
• after step 𝑘 , the matrix 𝐻 𝑘 𝐻 𝑘−1 · · · 𝐻1 𝐴 has the following structure:
𝑚−𝑘
𝑘 𝑛−𝑘
(elements in positions 𝑖, 𝑗 for 𝑖 > 𝑗 and 𝑗 ≤ 𝑘 are zero)
Householder algorithm

𝑅
the following algorithm overwrites 𝐴 with
0
Algorithm: for 𝑘 = 1 to 𝑛,
1. define 𝑦 = 𝐴 𝑘:𝑚,𝑘 and compute (𝑚 − 𝑘 + 1) -vector 𝑣 𝑘 :
1
𝑤 = 𝑦 + sign(𝑦 1) k𝑦k𝑒 1, 𝑣𝑘 = 𝑤
k𝑤k
2. multiply 𝐴 𝑘:𝑚,𝑘:𝑛 with reflector 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 :
𝐴 𝑘:𝑚,𝑘:𝑛 := 𝐴 𝑘:𝑚,𝑘:𝑛 − 2𝑣 𝑘 (𝑣𝑇𝑘 𝐴 𝑘:𝑚,𝑘:𝑛 )
(see page 109 in textbook for “slice” notation for submatrices)
Comments
• in step 2 we multiply 𝐴 𝑘:𝑚,𝑘:𝑛 with the reflector 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 :
(𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 ) 𝐴 𝑘:𝑚,𝑘:𝑛 = 𝐴 𝑘:𝑚,𝑘:𝑛 − 2𝑣 𝑘 (𝑣𝑇𝑘 𝐴 𝑘:𝑚,𝑘:𝑛 )
• this is equivalent to multiplying 𝐴 with 𝑚 × 𝑚 reflector

𝑇
0 0 0

𝐼
𝐻𝑘 = = 𝐼 −2
0 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 𝑣𝑘 𝑣𝑘
• algorithm overwrites 𝐴 with

𝑅
0
and returns the vectors 𝑣 1, . . . , 𝑣 𝑛 , with 𝑣 𝑘 of length 𝑚 − 𝑘 + 1
Example
example on page 6.8:
 −1 −1 1 
 1 3 3 
 
 𝑅
𝐴 =  = 𝐻 𝐻 𝐻
1 2 3
−1 −1 5  0
 1 3 7 
 

we compute reflectors 𝐻1, 𝐻2, 𝐻3 that triangularize 𝐴:
 𝑅11 𝑅12 𝑅13 

0
 
 𝑅22 𝑅23 
𝐻3 𝐻2 𝐻1 𝐴 = 
0 0

𝑅33 
0 0 0
 
 
 
Example
First column of 𝑅
• compute reflector that maps first column of 𝐴 to multiple of 𝑒 1:
 −1   −3   −3 
 1   1  1 1  1 
     

𝑦 =  , 𝑤 = 𝑦 − k𝑦k𝑒 1 =  , 𝑣1 = 𝑤= √ 
 −1   −1  k𝑤k 2 3  −1 
  
 1   1   1 
     
• overwrite 𝐴 with product of 𝐼 − 2𝑣 1 𝑣𝑇1 and 𝐴
 2 4 2 
0 4/3 8/3 

𝐴 := (𝐼 − 2𝑣 1 𝑣 1 ) 𝐴 = 
𝑇

0 2/3 16/3 
0 4/3 20/3 



Example
Second column of 𝑅
• compute reflector that maps 𝐴2:4,2 to multiple of 𝑒 1:
 4/3   10/3   5 
1 1  
𝑦 =  2/3  , 𝑤 = 𝑦 + k𝑦k𝑒 1 =  2/3  , 𝑤=√  1 
   
𝑣2 =
 4/3   4/3  k𝑤k 30  2 
     
• overwrite 𝐴2:4,2:3 with product of 𝐼 − 2𝑣 2 𝑣𝑇2 and 𝐴2:4,2:3:
 2 4 2 
1 0 0 −2

−8 
𝐴 :=

𝐴 = 
0 𝐼 − 2𝑣 2 𝑣𝑇2 0 0 16/5 
0 0 12/5 



Example
Third column of 𝑅
• compute reflector that maps 𝐴3:4,3 to multiple of 𝑒 1:
16/5 36/5 1 1 3

𝑦= , 𝑤 = 𝑦 + k𝑦k𝑒 1 = , 𝑣3 = 𝑤=√
12/5 12/5 k𝑤k 10 1
• overwrite 𝐴3:4,3 with product of 𝐼 − 2𝑣 3 𝑣𝑇3 and 𝐴3:4,3:
 2 4 2 
0 0 −2 −8 

𝐼
𝐴 :=

𝐴 = 
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 0 −4 
0 0 0 



Example
Final result
0 1 0

𝐼
𝐻3 𝐻2 𝐻1 𝐴 = (𝐼 − 2𝑣 1 𝑣𝑇1 ) 𝐴
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 𝐼 − 2𝑣 2 𝑣𝑇2
 2 4 2 
0 1 0 0 4/3 8/3 

𝐼 
=
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 𝐼 − 2𝑣 2 𝑣𝑇2 0 2/3 16/3 


0 4/3 20/3 



 2 4 2 
0 0 −2
 
𝐼  −8 
=
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 0 16/5
 
 
0 0 12/5
 
 
 
 2 4 2 
0 −2 −8 


=
0 0 −4 


0 0 0 



Complexity
Complexity in cycle 𝑘 (of algorithm on page 6.31): the dominant terms are
• (2(𝑚 − 𝑘 + 1) − 1)(𝑛 − 𝑘 + 1) flops for product 𝑣𝑇𝑘 ( 𝐴 𝑘:𝑚,𝑘:𝑛 )

• (𝑚 − 𝑘 + 1)(𝑛 − 𝑘 + 1) flops for outer product with 𝑣 𝑘
• (𝑚 − 𝑘 + 1)(𝑛 − 𝑘 + 1) flops for subtraction from 𝐴 𝑘:𝑚,𝑘:𝑛
sum is roughly 4(𝑚 − 𝑘 + 1) (𝑛 − 𝑘 + 1) flops
Total for computing 𝑅 and vectors 𝑣 1, . . . , 𝑣 𝑛 :
𝑛 ∫ 𝑛
4(𝑚 − 𝑘 + 1) (𝑛 − 𝑘 + 1) 4(𝑚 − 𝑡)(𝑛 − 𝑡)𝑑𝑡
X
≈
𝑘=1 0
2
= 2𝑚𝑛2 − 𝑛3 flops
3
Q-factor
the Householder algorithm returns the vectors 𝑣 1, . . . , 𝑣 𝑛 that define
𝑄˜ = 𝐻1 𝐻2 · · · 𝐻 𝑛

𝑄
• usually there is no need to compute the matrix [ 𝑄 𝑄˜ ] explicitly

• the vectors 𝑣 1, . . . , 𝑣 𝑛 are an economical representation of [ 𝑄 𝑄˜ ]
• products with [ 𝑄 𝑄˜ ] or its transpose can be computed as
𝑄˜ 𝑥 = 𝐻1 𝐻2 · · · 𝐻 𝑛 𝑥

𝑄
𝑄˜
𝑇
𝑦 = 𝐻𝑛 𝐻𝑛−1 · · · 𝐻1 𝑦

𝑄
Multiplication with Q-factor
• the matrix–vector product 𝐻 𝑘 𝑥 is defined as
0

𝐼 𝑥1:𝑘−1 𝑥1:𝑘−1
𝐻𝑘 𝑥 = =
0 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 𝑥 𝑘:𝑚 𝑥 𝑘:𝑚 − 2(𝑣𝑇𝑘 𝑥 𝑘:𝑚 )𝑣 𝑘
• complexity of multiplication 𝐻 𝑘 𝑥 is 4(𝑚 − 𝑘 + 1) flops:

• complexity of multiplication with 𝐻1 𝐻2 · · · 𝐻𝑛 or its transpose is
𝑛
4(𝑚 − 𝑘 + 1) ≈ 4𝑚𝑛 − 2𝑛2 flops
X
𝑘=1
• roughly equal to matrix–vector product with 𝑚 × 𝑛 matrix (2𝑚𝑛 flops)

QR Factorization: Triangular Matrices QR Factorization Gram-Schmidt Algorithm Householder Algorithm

Uploaded by

Copyright:

Available Formats

QR Factorization: Triangular Matrices QR Factorization Gram-Schmidt Algorithm Householder Algorithm

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

QR Factorization: Triangular Matrices QR Factorization Gram-Schmidt Algorithm Householder Algorithm

Uploaded by

Copyright:

Available Formats

L.

Vandenberghe ECE133A (Fall 2022)

a square matrix 𝐴 is lower triangular if 𝐴𝑖 𝑗 = 0 for 𝑗 > 𝑖

𝐴 is upper triangular if 𝐴𝑖 𝑗 = 0 for 𝑗 < 𝑖 (the transpose 𝐴𝑇 is lower triangular)

a triangular matrix is unit upper/lower triangular if 𝐴𝑖𝑖 = 1 for all 𝑖

solve 𝐴𝑥 = 𝑏 when 𝐴 is lower triangular with nonzero diagonal elements

𝑥𝑛 = (𝑏 𝑛 − 𝐴𝑛1𝑥1 − 𝐴𝑛2𝑥2 − · · · − 𝐴𝑛,𝑛−1𝑥 𝑛−1)/𝐴𝑛𝑛

Complexity: 1 + 3 + 5 + · · · + (2𝑛 − 1) = 𝑛2 flops

solve 𝐴𝑥 = 𝑏 when 𝐴 is upper triangular with nonzero diagonal elements

𝑥1 = (𝑏 1 − 𝐴12𝑥2 − 𝐴13𝑥3 − · · · − 𝐴1𝑛 𝑥 𝑛 )/𝐴11

a triangular matrix 𝐴 with nonzero diagonal elements is nonsingular:

this follows from forward or back substitution applied to the equation 𝐴𝑥 = 0

• inverse of 𝐴 can be computed by solving 𝐴𝑋 = 𝐼 column by column

• inverse of lower triangular matrix is lower triangular

if 𝐴 ∈ R𝑚×𝑛 has linearly independent columns then it can be factored as

 𝑅11 𝑅12 · · · 𝑅1𝑛 

• vectors 𝑞 1, . . . , 𝑞 𝑛 are orthonormal 𝑚 -vectors:

• diagonal elements 𝑅𝑖𝑖 are nonzero

if 𝐴 ∈ R𝑚×𝑛 has linearly independent columns then it can be factored as

• 𝑄 is 𝑚 × 𝑛 with orthonormal columns (𝑄𝑇 𝑄 = 𝐼 )

• 𝑅 is 𝑛 × 𝑛, upper triangular, with nonzero diagonal elements

 −1 −1 1   −1/2 1/2 −1/2 

 𝑅11 𝑅12 𝑅13

the QR factorization is often defined as a factorization

• 𝐴 = 𝑄𝑅 is the QR factorization as defined earlier (page 6.7)

• MATLAB’s function qr returns this factorization

in this course we use the definition of page 6.7

in the following lectures, we will use the QR factorization to solve

here, we show that it gives useful simple formulas for

• the pseudo-inverse of a matrix with linearly independent columns

pseudo-inverse of a matrix 𝐴 with linearly independent columns (page 4.22)

• pseudo-inverse in terms of QR factors of 𝐴:

𝐴† = ((𝑄𝑅)𝑇 (𝑄𝑅)) −1 (𝑄𝑅)𝑇

• for square nonsingular 𝐴 this is the inverse:

𝐴−1 = (𝑄𝑅) −1 = 𝑅 −1𝑄𝑇

recall definition of range of a matrix 𝐴 ∈ R𝑚×𝑛 (page 5.16):

suppose 𝐴 has linearly independent columns with QR factors 𝑄 , 𝑅

• 𝑄 has the same range as 𝐴:

𝑦 ∈ range( 𝐴) ⇐⇒ 𝑦 = 𝐴𝑥 for some 𝑥

• columns of 𝑄 are an orthonormal basis for range( 𝐴)

• combining 𝐴 = 𝑄𝑅 and 𝐴† = 𝑅 −1𝑄𝑇 (from page 6.11) gives

𝐴𝐴† = 𝑄𝑅𝑅 −1𝑄𝑇 = 𝑄𝑄𝑇

if 𝐴 ∈ C𝑚×𝑛 has linearly independent columns then it can be factored as

• 𝑄 ∈ C𝑚×𝑛 has orthonormal columns (𝑄 𝐻 𝑄 = 𝐼 )

Gram–Schmidt algorithm (section 5.4 in textbook and page 6.16)

Modified Gram–Schmidt algorithm

Householder algorithm (page 6.26)

Gram–Schmidt QR algorithm computes 𝑄 and 𝑅 column by column

• after 𝑘 steps we have a partial QR factorization

 𝑅11 𝑅12 · · · 𝑅1𝑘 

this is the QR factorization for the first 𝑘 columns of 𝐴

• columns 𝑞 1, . . . , 𝑞 𝑘 are orthonormal

• column 𝑘 of the equation 𝐴 = 𝑄𝑅 reads

𝑎 𝑘 = 𝑅1𝑘 𝑞 1 + 𝑅2𝑘 𝑞 2 + · · · + 𝑅 𝑘−1,𝑘 𝑞 𝑘−1 + 𝑅 𝑘 𝑘 𝑞 𝑘

• regardless of how we choose 𝑅1𝑘 , . . . , 𝑅 𝑘−1,𝑘 , the vector

𝑞˜ 𝑘 = 𝑎 𝑘 − 𝑅1𝑘 𝑞 1 − 𝑅2𝑘 𝑞 2 − · · · − 𝑅 𝑘−1,𝑘 𝑞 𝑘−1

will be nonzero: 𝑎 1, 𝑎 2, . . . , 𝑎 𝑘 are linearly independent and therefore

𝑎 𝑘 ∉ span{𝑎 1, . . . , 𝑎 𝑘−1 } = span{𝑞 1, . . . , 𝑞 𝑘−1 }

• 𝑞 𝑘 is 𝑞˜ 𝑘 normalized: choose 𝑅 𝑘 𝑘 = k 𝑞˜ 𝑘 k and 𝑞 𝑘 = (1/𝑅 𝑘 𝑘 ) 𝑞˜ 𝑘

• algorithm overwrites 𝐴 with