L.

Vandenberghe ECE133A (Fall 2024)

6. QR factorization

• triangular matrices
• QR factorization
• Gram–Schmidt algorithm
• modified Gram–Schmidt algorithm
• Householder algorithm

6.1
 Triangular matrix

a square matrix 𝐴 is lower triangular if 𝐴𝑖 𝑗 = 0 for 𝑗 > 𝑖

 𝐴11 0 ··· 0 0 
0 0
 
 𝐴21 𝐴22 ··· 

𝐴 =  .. .. ... 0 0


0

 𝐴𝑛−1,1
 𝐴𝑛−1,2 ··· 𝐴𝑛−1,𝑛−1 

 𝐴𝑛1
 𝐴𝑛2 ··· 𝐴𝑛,𝑛−1 𝐴𝑛𝑛 


𝐴 is upper triangular if 𝐴𝑖 𝑗 = 0 for 𝑗 < 𝑖 (the transpose 𝐴𝑇 is lower triangular)

a triangular matrix is unit upper/lower triangular if 𝐴𝑖𝑖 = 1 for all 𝑖

QR factorization 6.2
 Forward substitution

solve 𝐴𝑥 = 𝑏 when 𝐴 is lower triangular with nonzero diagonal elements

Algorithm

𝑥1 = 𝑏 1/𝐴11
𝑥2 = (𝑏 2 − 𝐴21𝑥1)/𝐴22
𝑥3 = (𝑏 3 − 𝐴31𝑥1 − 𝐴32𝑥2)/𝐴33
..

𝑥𝑛 = (𝑏 𝑛 − 𝐴𝑛1𝑥1 − 𝐴𝑛2𝑥2 − · · · − 𝐴𝑛,𝑛−1𝑥 𝑛−1)/𝐴𝑛𝑛

Complexity: 1 + 3 + 5 + · · · + (2𝑛 − 1) = 𝑛2 flops

QR factorization 6.3
 Back substitution

solve 𝐴𝑥 = 𝑏 when 𝐴 is upper triangular with nonzero diagonal elements

Algorithm

𝑥 𝑛 = 𝑏 𝑛 /𝐴𝑛𝑛
𝑥 𝑛−1 = (𝑏 𝑛−1 − 𝐴𝑛−1,𝑛 𝑥 𝑛 )/𝐴𝑛−1,𝑛−1
𝑥 𝑛−2 = (𝑏 𝑛−2 − 𝐴𝑛−2,𝑛−1𝑥 𝑛−1 − 𝐴𝑛−2,𝑛 𝑥 𝑛 )/𝐴𝑛−2,𝑛−2
..

𝑥1 = (𝑏 1 − 𝐴12𝑥2 − 𝐴13𝑥3 − · · · − 𝐴1𝑛 𝑥 𝑛 )/𝐴11

Complexity: 𝑛2 flops

QR factorization 6.4
 Inverse of triangular matrix

a triangular matrix 𝐴 with nonzero diagonal elements is nonsingular:

𝐴𝑥 = 0 =⇒ 𝑥=0

this follows from forward or back substitution applied to the equation 𝐴𝑥 = 0

• inverse of 𝐴 can be computed by solving 𝐴𝑋 = 𝐼 column by column

= (𝑥𝑖 is column 𝑖 of 𝑋 )
   
𝐴 𝑥1 𝑥2 · · · 𝑥𝑛 𝑒1 𝑒2 · · · 𝑒𝑛

• inverse of lower triangular matrix is lower triangular

• inverse of upper triangular matrix is upper triangular
• complexity of computing inverse of 𝑛 × 𝑛 triangular matrix is

𝑛2 + (𝑛 − 1) 2 + · · · + 1 ∼ 31 𝑛3 flops

QR factorization 6.5
 Outline

• triangular matrices
• QR factorization
• Gram–Schmidt algorithm
• modified Gram–Schmidt algorithm
• Householder algorithm
 QR factorization

if 𝐴 ∈ R𝑚×𝑛 has linearly independent columns then it can be factored as

 𝑅11 𝑅12 · · · 𝑅1𝑛 

 0
 
𝑅22 · · · 𝑅2𝑛 
𝐴=

𝑞1 𝑞2 · · · 𝑞 𝑛  . .. ... .. 
 .

 0 0 ···

 𝑅𝑛𝑛 


• vectors 𝑞 1, . . . , 𝑞 𝑛 are orthonormal 𝑚 -vectors:

∥𝑞𝑖 ∥ = 1, 𝑞𝑇𝑖 𝑞 𝑗 = 0 if 𝑖 ≠ 𝑗

• diagonal elements 𝑅𝑖𝑖 are nonzero

• if 𝑅𝑖𝑖 < 0, we can switch the signs of 𝑅𝑖𝑖 , . . . , 𝑅𝑖𝑛 , and the vector 𝑞𝑖
• most definitions require 𝑅𝑖𝑖 > 0; this makes 𝑄 and 𝑅 unique

QR factorization 6.6
 QR factorization in matrix notation

if 𝐴 ∈ R𝑚×𝑛 has linearly independent columns then it can be factored as

𝐴 = 𝑄𝑅

Q-factor

• 𝑄 is 𝑚 × 𝑛 with orthonormal columns (𝑄𝑇 𝑄 = 𝐼 )

• if 𝐴 is square (𝑚 = 𝑛), then 𝑄 is orthogonal (𝑄𝑇 𝑄 = 𝑄𝑄𝑇 = 𝐼 )

R-factor

• 𝑅 is 𝑛 × 𝑛, upper triangular, with nonzero diagonal elements

• 𝑅 is nonsingular (diagonal elements are nonzero)

QR factorization 6.7
 Example

 −1 −1 1   −1/2 1/2 −1/2 

 1 3 3   1/2 1/2 −1/2   2 4 2 
    
=  0 2 8 
 −1 −1 5   −1/2 1/2 1/2  
 
0 0 4 

 1 3 7   1/2 1/2 1/2 
  
 
 

 𝑅11 𝑅12 𝑅13 

𝑞 3  0
 
=

𝑞1 𝑞2 𝑅22 𝑅23 
 0 0

 𝑅33 


= 𝑄𝑅

QR factorization 6.8
 Full QR factorization

the QR factorization is often defined as a factorization

 
𝑅
𝐴= 𝑄˜
 
𝑄
0

• 𝐴 = 𝑄𝑅 is the QR factorization as defined earlier (page 6.7)

• 𝑄˜ has size 𝑚 × (𝑚 − 𝑛) , the zero block has size (𝑚 − 𝑛) × 𝑛
˜
• the matrix 𝑄 𝑄 is 𝑚 × 𝑚 and orthogonal
 

• MATLAB’s function qr returns this factorization

• this is also known as the full QR factorization or QR decomposition

in this course we use the definition of page 6.7

QR factorization 6.9
 Applications

in the following lectures, we will use the QR factorization to solve

• linear equations
• least squares problems
• constrained least squares problems

here, we show that it gives useful simple formulas for

• the pseudo-inverse of a matrix with linearly independent columns

• the inverse of a nonsingular matrix
• projection on the range of a matrix with linearly independent columns

QR factorization 6.10
 QR factorization and (pseudo-)inverse

pseudo-inverse of a matrix 𝐴 with linearly independent columns (page 4.22)

𝐴† = ( 𝐴𝑇 𝐴) −1 𝐴𝑇

• pseudo-inverse in terms of QR factors of 𝐴:

𝐴† = ((𝑄𝑅)𝑇 (𝑄𝑅)) −1 (𝑄𝑅)𝑇

= (𝑅𝑇 𝑄𝑇 𝑄𝑅) −1 𝑅𝑇 𝑄𝑇
= (𝑅𝑇 𝑅) −1 𝑅𝑇 𝑄𝑇 (𝑄𝑇 𝑄 = 𝐼 )
= 𝑅 −1 𝑅 −𝑇 𝑅𝑇 𝑄𝑇 ( 𝑅 is nonsingular)
= 𝑅 −1𝑄𝑇

• for square nonsingular 𝐴 this is the inverse:

𝐴−1 = (𝑄𝑅) −1 = 𝑅 −1𝑄𝑇

QR factorization 6.11
 Range

recall definition of range of a matrix 𝐴 ∈ R𝑚×𝑛 (page 5.16):

range( 𝐴) = {𝐴𝑥 | 𝑥 ∈ R𝑛 }

suppose 𝐴 has linearly independent columns with QR factors 𝑄 , 𝑅

• 𝑄 has the same range as 𝐴:

𝑦 ∈ range( 𝐴) ⇐⇒ 𝑦 = 𝐴𝑥 for some 𝑥

⇐⇒ 𝑦 = 𝑄𝑅𝑥 for some 𝑥
⇐⇒ 𝑦 = 𝑄𝑧 for some 𝑧
⇐⇒ 𝑦 ∈ range(𝑄)

• columns of 𝑄 are an orthonormal basis for range( 𝐴)

QR factorization 6.12
 Projection on range

• combining 𝐴 = 𝑄𝑅 and 𝐴† = 𝑅 −1𝑄𝑇 (from page 6.11) gives

𝐴𝐴† = 𝑄𝑅𝑅 −1𝑄𝑇 = 𝑄𝑄𝑇

note the order of the product in 𝐴𝐴† and the difference with 𝐴† 𝐴 = 𝐼
• recall (from page 5.17) that 𝑄𝑄𝑇 𝑥 is the projection of 𝑥 on the range of 𝑄
𝑥

𝐴𝐴†𝑥 = 𝑄𝑄𝑇 𝑥

range( 𝐴) = range(𝑄)

QR factorization 6.13
 QR factorization of complex matrices

if 𝐴 ∈ C𝑚×𝑛 has linearly independent columns then it can be factored as

𝐴 = 𝑄𝑅

• 𝑄 ∈ C𝑚×𝑛 has orthonormal columns (𝑄 𝐻 𝑄 = 𝐼 )

• 𝑅 ∈ C𝑛×𝑛 is upper triangular with real nonzero diagonal elements
• most definitions choose diagonal elements 𝑅𝑖𝑖 to be positive
• in the rest of the lecture we assume 𝐴 is real

QR factorization 6.14
 Algorithms for QR factorization

Gram–Schmidt algorithm (section 5.4 in textbook and page 6.16)

• complexity is 2𝑚𝑛2 flops
• not recommended in practice (sensitive to rounding errors)

Modified Gram–Schmidt algorithm (page 6.27)

• complexity is 2𝑚𝑛2 flops
• better numerical properties

Householder algorithm (page 6.34)

• complexity is 2𝑚𝑛2 − (2/3)𝑛3 flops
• represents 𝑄 as a product of elementary orthogonal matrices
• the most widely used algorithm (used by the function qr in MATLAB and Julia)

in the rest of the course we will take 2𝑚𝑛2 for the complexity of QR factorization
QR factorization 6.15
 Outline

• triangular matrices
• QR factorization
• Gram–Schmidt algorithm
• modified Gram–Schmidt algorithm
• Householder algorithm
 Gram–Schmidt algorithm

Gram–Schmidt QR algorithm computes 𝑄 and 𝑅 column by column

• after 𝑘 steps we have a partial QR factorization

 𝑅11 𝑅12 · · · 𝑅1𝑘 

 0
 
𝑅22 · · · 𝑅2𝑘 
=
  
𝑎1 𝑎2 · · · 𝑎𝑘 𝑞1 𝑞2 · · · 𝑞𝑘  .
 . .. ... .. 

 0 0 ···
 
 𝑅𝑘 𝑘 


this is the QR factorization for the first 𝑘 columns of 𝐴

• columns 𝑞 1, . . . , 𝑞 𝑘 are orthonormal
• diagonal elements 𝑅11, 𝑅22, . . . , 𝑅 𝑘 𝑘 are positive
• columns 𝑞 1, . . . , 𝑞 𝑘 have the same span as 𝑎 1, . . . , 𝑎 𝑘 (see page 6.12)
• in step 𝑘 of the algorithm we compute 𝑞 𝑘 , 𝑅1𝑘 , . . . , 𝑅 𝑘 𝑘

QR factorization 6.16
 Computing the 𝑘 th columns of 𝑄 and 𝑅

suppose we have the partial factorization for the first 𝑘 − 1 columns of 𝑄 and 𝑅

• column 𝑘 of the equation 𝐴 = 𝑄𝑅 reads

𝑎 𝑘 = 𝑅1𝑘 𝑞 1 + 𝑅2𝑘 𝑞 2 + · · · + 𝑅 𝑘−1,𝑘 𝑞 𝑘−1 + 𝑅 𝑘 𝑘 𝑞 𝑘

• regardless of how we choose 𝑅1𝑘 , . . . , 𝑅 𝑘−1,𝑘 , the vector

𝑞˜ 𝑘 = 𝑎 𝑘 − 𝑅1𝑘 𝑞 1 − 𝑅2𝑘 𝑞 2 − · · · − 𝑅 𝑘−1,𝑘 𝑞 𝑘−1

will be nonzero: 𝑎 1, 𝑎 2, . . . , 𝑎 𝑘 are linearly independent and therefore

𝑎 𝑘 ∉ span(𝑎 1, . . . , 𝑎 𝑘−1) = span(𝑞 1, . . . , 𝑞 𝑘−1)

• 𝑞 𝑘 is 𝑞˜ 𝑘 normalized: choose 𝑅 𝑘 𝑘 = ∥ 𝑞˜ 𝑘 ∥ and 𝑞 𝑘 = (1/𝑅 𝑘 𝑘 ) 𝑞˜ 𝑘

• 𝑞˜ 𝑘 and 𝑞 𝑘 are orthogonal to 𝑞 1, . . . , 𝑞 𝑘−1 if we choose 𝑅1𝑘 , . . . , 𝑅 𝑘−1,𝑘 as

𝑅1𝑘 = 𝑞𝑇1 𝑎 𝑘 , 𝑅2𝑘 = 𝑞𝑇2 𝑎 𝑘 , ..., 𝑅 𝑘−1,𝑘 = 𝑞𝑇𝑘−1 𝑎 𝑘

QR factorization 6.17
 Interpretation

on the previous page, 𝑞˜ 𝑘 = 𝑅 𝑘 𝑘 𝑞 𝑘 was computed as

𝑞˜ 𝑘 = 𝑎 𝑘 − 𝑅1𝑘 𝑞 1 − 𝑅2𝑘 𝑞 2 − · · · − 𝑅 𝑘−1,𝑘 𝑞 𝑘−1

= 𝑎 𝑘 − 𝑞 1 (𝑞𝑇1 𝑎 𝑘 ) − 𝑞 2 (𝑞𝑇2 𝑎 𝑘 ) − · · · − 𝑞 𝑘−1 𝑞𝑇𝑘−1 𝑎 𝑘
 
= 𝐼 − 𝑞 1 𝑞𝑇1 − 𝑞 2 𝑞𝑇2 − · · · − 𝑞 𝑘−1 𝑞𝑇𝑘−1 𝑎 𝑘
 𝑇 
= 𝐼 − 𝑞 1 𝑞 2 · · · 𝑞 𝑘−1 𝑞 1 𝑞 2 · · · 𝑞 𝑘−1
 
𝑎𝑘

this is the residual of 𝑎 𝑘 after subtracting its orthogonal projection on

span(𝑎 1, 𝑎 2, . . . , 𝑎 𝑘−1) = span(𝑞 1, 𝑞 2, . . . , 𝑞 𝑘−1)

= range( 𝑞 1 𝑞 2 · · · 𝑞 𝑘−1 )
 

QR factorization 6.18
 Gram–Schmidt algorithm

Given: 𝑚 × 𝑛 matrix 𝐴 with linearly independent columns 𝑎 1, . . . , 𝑎 𝑛

Algorithm

for 𝑘 = 1 to 𝑛
𝑅1𝑘 = 𝑞𝑇1 𝑎 𝑘
𝑅2𝑘 = 𝑞𝑇2 𝑎 𝑘
..

𝑅 𝑘−1,𝑘 = 𝑞𝑇𝑘−1 𝑎 𝑘
𝑞˜ 𝑘 = 𝑎 𝑘 − (𝑅1𝑘 𝑞 1 + 𝑅2𝑘 𝑞 2 + · · · + 𝑅 𝑘−1,𝑘 𝑞 𝑘−1)
𝑅𝑘 𝑘 = ∥ 𝑞˜ 𝑘 ∥
1
𝑞𝑘 = 𝑞˜ 𝑘
𝑅𝑘 𝑘

QR factorization 6.19
 Example

example on page 6.8:

 −1 −1 1 
 1 3 3 
 
=
 
𝑎1 𝑎2 𝑎3  −1 −1 5 

 1 3 7
 

 
 𝑅11 𝑅12 𝑅13 
𝑞 3  0
 
=

𝑞1 𝑞2 𝑅22 𝑅23 
 0 0

 𝑅33 


First column of 𝑄 and 𝑅

 −1   −1/2 
 1  1 1/2 
   
𝑞˜ 1 = 𝑎 1 =  𝑅11 = ∥ 𝑞˜ 1 ∥ = 2, 𝑞˜ 1 = 

, 𝑞1 = 
 −1  𝑅11  −1/2 
 
 1   1/2 
   

QR factorization 6.20
 Example

Second column of 𝑄 and 𝑅

• compute 𝑅12 = 𝑞𝑇1 𝑎 2 = 4

• compute
 −1   −1/2   1 
 3   1/2   1 
     
𝑞˜ 2 = 𝑎 2 − 𝑅12 𝑞 1 =   −4 = 
 −1   −1/2   1 
    
 3   1/2   1 
     
• normalize to get

 1/2 
1 1/2
 
𝑅22 = ∥ 𝑞˜ 2 ∥ = 2, 𝑞˜ 2 = 
 
𝑞2 =
1/2

𝑅22 
1/2
 
 
 

QR factorization 6.21
 Example

Third column of 𝑄 and 𝑅

• compute 𝑅13 = 𝑞𝑇1 𝑎 3 = 2 and 𝑅23 = 𝑞𝑇2 𝑎 3 = 8

• compute

 1   −1/2   1/2   −2 
3  1/2   1/2   −2 
       
𝑞˜ 3 = 𝑎 3 − 𝑅13 𝑞 1 − 𝑅23 𝑞 2 =   −2
 −1/2  − 8  1/2  =  2 
 
5
    

7  1/2   1/2   2 
       
 
       

• normalize to get

 −1/2 
1
 
−1/2 
𝑅33 = ∥ 𝑞˜ 3 ∥ = 4, 𝑞˜ 3 = 

𝑞3 =
 1/2 

𝑅33 
 1/2 
 

QR factorization 6.22
 Example

Final result

 −1 −1 1 
 1 3 3    𝑅11 𝑅12 𝑅13
   
𝑞 3  0

=

 −1 −1 5 
 𝑞1 𝑞2 𝑅22 𝑅23 
 0 0

 1 𝑅33
3 7 
  
 

 −1/2 1/2 −1/2 
 2 4 2 
 1/2 1/2 −1/2  

 0 2 8 

=  −1/2 1/2 1/2  

0 0 4 

 1/2 1/2 1/2 

 


QR factorization 6.23
 Complexity

Complexity of cycle 𝑘 (of algorithm on page 6.19)

• 𝑘 − 1 inner products with 𝑎 𝑘 : (𝑘 − 1) (2𝑚 − 1) flops

• computation of 𝑞˜ 𝑘 : 2(𝑘 − 1)𝑚 flops
• computing 𝑅 𝑘 𝑘 and 𝑞 𝑘 : 3𝑚 flops

total for cycle 𝑘 : (4𝑚 − 1) (𝑘 − 1) + 3𝑚 flops

Complexity for 𝑚 × 𝑛 factorization:

𝑛
𝑛(𝑛 − 1)
((4𝑚 − 1) (𝑘 − 1) + 3𝑚) (4𝑚 − 1) + 3𝑚𝑛
∑︁
=
𝑘=1 2
∼ 2𝑚𝑛2 flops

QR factorization 6.24
 Numerical experiment

• we use the following MATLAB implementation of the algorithm on page 6.19:

[m, n] = size(A);
Q = zeros(m,n);
R = zeros(n,n);
for k = 1:n
R(1:k-1,k) = Q(:,1:k-1)’ * A(:,k);
qtilde = A(:,k) - Q(:,1:k-1) * R(1:k-1,k);
R(k,k) = norm(qtilde);
Q(:,k) = qtilde / R(k,k);
end;
• we apply this to a square matrix 𝐴 of size 𝑚 = 𝑛 = 50
• 𝐴 is constructed as 𝐴 = 𝑈𝑆𝑉 with 𝑈 , 𝑉 orthogonal, 𝑆 diagonal with

𝑆𝑖𝑖 = 10−10(𝑖−1)/(𝑛−1) , 𝑖 = 1, . . . , 𝑛

QR factorization 6.25
 Numerical experiment

plot shows deviation from orthogonality between 𝑞 𝑘 and previous columns

𝑒 𝑘 = max |𝑞𝑇𝑖 𝑞 𝑘 |, 𝑘 = 2, . . . , 𝑛
1≤𝑖<𝑘

0.8

0.6
𝑒𝑘

0.4

0.2

0
0 10 20 30 40 50
𝑘

loss of orthogonality is due to rounding error

QR factorization 6.26
 Outline

• triangular matrices
• QR factorization
• Gram–Schmidt algorithm
• modified Gram–Schmidt algorithm
• Householder algorithm
 Modified Gram–Schmidt algorithm

a variation of the Gram–Schmidt algorithm for the QR factorization

 𝑅11 𝑅12 · · · 𝑅1𝑛 

 0
 
𝑅22 · · · 𝑅2𝑛 
=
  
𝑎1 𝑎2 · · · 𝑎𝑛 𝑞1 𝑞2 · · · 𝑞 𝑛  . .. ... .. 
 .

 0 0 ···

 𝑅𝑛𝑛 


• has better numerical properties than the Gram–Schmidt algorithm

• computes 𝑄 column by column, 𝑅 row by row
• computes vectors 𝑞˜ 𝑘 as

𝑞˜ 𝑘 = (𝐼 − 𝑞 𝑘−1 𝑞𝑇𝑘−1) · · · (𝐼 − 𝑞 2 𝑞𝑇2 ) (𝐼 − 𝑞 1 𝑞𝑇1 )𝑎 𝑘

(see exercise on page 5.20)

QR factorization 6.27
 Modified Gram–Schmidt algorithm

after 𝑘 − 1 steps, the algorithm has computed a partial factorization

𝐴 = 𝑎 1 · · · 𝑎 𝑘−1 𝑎 𝑘 · · · 𝑎𝑛
 

 𝑅 ··· 𝑅 𝑅1𝑘 · · · 𝑅1𝑛

 11 1,𝑘−1

.
  . ... .. .. ..

𝑄˜ 𝑘
 
=

𝑞 1 · · · 𝑞 𝑘−1  0 ··· 𝑅

𝑘−1,𝑘−1 𝑅 𝑘−1,𝑘 · · · 𝑅 𝑘−1,𝑛 
0
 
𝐼
 
 
 

• columns of 𝑄˜ 𝑘 are residuals of 𝑎 𝑘 , . . . , 𝑎 𝑛 after projection on span(𝑞 1, . . . , 𝑞 𝑘−1)

• 𝑞˜ 𝑘 is the first column of 𝑄˜ 𝑘
• we start with 𝑘 = 0 and 𝑄˜ 1 = 𝐴
• the factorization is complete when 𝑘 = 𝑛
• in step 𝑘 , we compute

𝑞𝑘, 𝑅𝑘 𝑘 , 𝑅 𝑘,𝑘+1, . . . , 𝑅 𝑘𝑛 , 𝑄˜ 𝑘+1

QR factorization 6.28
 Modified Gram–Schmidt update

careful inspection of the update at step 𝑘 shows that

" #
𝑅𝑘 𝑘 𝑅 𝑘,(𝑘+1):𝑛
𝑄˜ 𝑘 = 𝑄˜ 𝑘+1
 
𝑞𝑘
0 𝐼

˜ ˜
partition 𝑄 𝑘 as 𝑄 𝑘 = 𝑞˜ 𝑘 𝐵 with 𝑞˜ 𝑘 the first column and 𝐵 of size 𝑚 × (𝑛 − 𝑘) :
 

𝑞˜ 𝑘 = 𝑞 𝑘 𝑅 𝑘 𝑘 , 𝐵 = 𝑞 𝑘 𝑅 𝑘,(𝑘+1):𝑛 + 𝑄˜ 𝑘+1

• from the first equation, and the required properties ∥𝑞 𝑘 ∥ = 1 and 𝑅 𝑘 𝑘 > 0:

1
𝑅 𝑘 𝑘 = ∥ 𝑞˜ 𝑘 ∥, 𝑞𝑘 = 𝑞˜ 𝑘
𝑅𝑘 𝑘

• from the second equation, and the requirement that 𝑞𝑇𝑘 𝑄˜ 𝑘+1 = 0:

𝑅 𝑘,(𝑘+1):𝑛 = 𝑞𝑇𝑘 𝐵, 𝑄˜ 𝑘+1 = (𝐼 − 𝑞 𝑘 𝑞𝑇𝑘 )𝐵 = 𝐵 − 𝑞 𝑘 𝑅 𝑘,(𝑘+1):𝑛

QR factorization 6.29
 Summary: modified Gram–Schmidt algorithm

Algorithm ( 𝐴 is 𝑚 × 𝑛 with linearly independent columns)

define 𝑄˜ 1 = 𝐴; for 𝑘 = 1 to 𝑛,
• compute 𝑅 𝑘 𝑘 = ∥ 𝑞˜ 𝑘 ∥ and 𝑞 𝑘 = (1/𝑅 𝑘 𝑘 ) 𝑞˜ 𝑘 where 𝑞˜ 𝑘 is the first column of 𝑄˜ 𝑘
• compute

𝑅 𝑘,𝑘+1 · · · 𝑅 𝑘𝑛 = 𝑞𝑇𝑘 𝐵, ˜
𝑄 𝑘+1 = 𝐵 − 𝑞 𝑘 𝑅 𝑘,𝑘+1 · · · 𝑅 𝑘𝑛
   

where 𝐵 is 𝑄˜ 𝑘 with first column removed

MATLAB code (Q(:,k:n) is used to store 𝑄˜ 𝑘 )

Q = A; R = zeros(n,n);
for k = 1:n
R(k,k) = norm(Q(:,k));
Q(:,k) = Q(:,k) / R(k,k);
R(k,k+1:n) = Q(:,k)’ * Q(:,k+1:n);
Q(:,k+1:n) = Q(:,k+1:n) - Q(:,k) * R(k,k+1:n);
end;
QR factorization 6.30
 Example

example on page 6.8

 −1 −1 1 
  1 3 3 
 
𝑎3 = 

𝑎1 𝑎2
 −1 −1 5 


 1 3 7 
 

Step 1: first column of 𝑄 , first row of 𝑅

 −1/2 1 2 
 2 4 2 
 1/2 1 2  

 0 1 0 

=
 
𝑎1 𝑎2 𝑎3  −1/2 1 6  

 0 0 1 

 1/2 1 6  


 
 
 𝑅11 𝑅1,2:3
= 𝑞 1 𝑄˜ 2

0 𝐼

QR factorization 6.31
 Example

Step 2: second column of 𝑄 , second row of 𝑅

 −1/2 1/2 −2 
 1/2 1/2 −2   2 4 2 
  
=  0 2 8 
 
𝑎1 𝑎2 𝑎3  −1/2 1/2 2  

0 0 1 

 1/2 1/2 2 

 

 𝑅11 𝑅12 𝑅13 
𝑞 1 𝑞 2 𝑄˜ 3  0 𝑅22 𝑅23 
 
=

 0 0 1 


Step 3: third column of 𝑄 , third row of 𝑅

 −1/2 1/2 −1/2 
 1/2 1/2 −1/2   2 4 2
  
 0 2 8

=
 
𝑎1 𝑎2 𝑎3  −1/2 1/2 1/2  
 
0 0 4

 1/2 1/2 1/2 
 
  

 𝑅11 𝑅12 𝑅13 
𝑞 1 𝑞 2 𝑞 3  0 𝑅22 𝑅23
 
=


 0 0 𝑅33


 
QR factorization 6.32
 Complexity

Complexity of cycle 𝑘 (of algorithm on page 6.30)

• computing 𝑅 𝑘 𝑘 and 𝑞 𝑘 : 3𝑚 flops

• computing 𝑅 𝑘,𝑘+1, . . . , 𝑅 𝑘𝑛 : (𝑛 − 𝑘) (2𝑚 − 1) flops
• computing 𝑄˜ 𝑘+1: 2(𝑛 − 𝑘)𝑚 flops

total for cycle 𝑘 : (4𝑚 − 1) (𝑛 − 𝑘) + 3𝑚 flops

Complexity for 𝑚 × 𝑛 factorization:

𝑛
𝑛(𝑛 − 1)
((4𝑚 − 1) (𝑛 − 𝑘) + 3𝑚) (4𝑚 − 1) + 3𝑚𝑛
∑︁
=
𝑘=1 2
∼ 2𝑚𝑛2 flops

QR factorization 6.33
 Outline

• triangular matrices
• QR factorization
• Gram–Schmidt algorithm
• modified Gram–Schmidt algorithm
• Householder algorithm
 Householder algorithm

• the most widely used algorithm for QR factorization (qr in MATLAB and Julia)
• less sensitive to rounding error than Gram–Schmidt algorithm
• computes a “full” QR factorization (QR decomposition)
 
𝑅
𝐴= 𝑄˜ 𝑄˜ orthogonal
   
𝑄 , 𝑄
0

• the full Q-factor is constructed as a product of orthogonal matrices

𝑄˜ = 𝐻1 𝐻2 · · · 𝐻 𝑛
 
𝑄

each 𝐻𝑖 is an 𝑚 × 𝑚 symmetric, orthogonal “reflector” (page 5.10)

QR factorization 6.34
 Reflector

𝐻 = 𝐼 − 2𝑣𝑣𝑇 with ∥𝑣∥ = 1

• 𝐻𝑥 is reflection of 𝑥 through hyperplane {𝑧 | 𝑣𝑇 𝑧 = 0} (see page 5.10)

• 𝐻 is symmetric
• 𝐻 is orthogonal
• matrix–vector product 𝐻𝑥 can be computed efficiently as

𝐻𝑥 = 𝑥 − 2(𝑣𝑇 𝑥)𝑣

complexity is 4𝑝 flops if 𝑣 and 𝑥 have length 𝑝

QR factorization 6.35
 Reflection to multiple of unit vector

given nonzero 𝑝 -vector 𝑦 = (𝑦 1, 𝑦 2, . . . , 𝑦 𝑝 ) , define

 𝑦 1 + sign(𝑦 1) ∥𝑦∥ 
1
 
 𝑦2 
𝑤 =  .. , 𝑣= 𝑤


 ∥𝑤∥

 𝑦𝑝 


• we define sign(0) = 1
• vector 𝑤 satisfies
∥𝑤∥ 2 = 2 (𝑤𝑇 𝑦) = 2∥𝑦∥ (∥𝑦∥ + |𝑦 1 |)

• reflector 𝐻 = 𝐼 − 2𝑣𝑣𝑇 maps 𝑦 to multiple of 𝑒 1 = (1, 0, . . . , 0) :

2(𝑤𝑇 𝑦)
𝐻𝑦 = 𝑦 − 𝑤 = 𝑦 − 𝑤 = −sign(𝑦 1)∥𝑦∥𝑒 1
∥𝑤∥ 2

QR factorization 6.36
 Geometry

𝑤
𝑦

first coordinate axis

−sign(𝑦 1) ∥𝑦∥𝑒 1

hyperplane {𝑥 | 𝑤𝑇 𝑥 = 0}

the reflection through the hyperplane {𝑥 | 𝑤𝑇 𝑥 = 0} with normal vector

𝑤 = 𝑦 + sign(𝑦 1) ∥𝑦∥𝑒 1

maps 𝑦 to the vector −sign(𝑦 1) ∥𝑦∥𝑒 1

QR factorization 6.37
 Householder triangularization

• computes reflectors 𝐻1, . . . , 𝐻𝑛 that reduce 𝐴 to triangular form:

 
𝑅
𝐻𝑛 𝐻𝑛−1 · · · 𝐻1 𝐴 =
0

• after step 𝑘 , the matrix 𝐻 𝑘 𝐻 𝑘−1 · · · 𝐻1 𝐴 has the following structure:

𝑚−𝑘

𝑘 𝑛−𝑘

(elements in positions 𝑖, 𝑗 for 𝑖 > 𝑗 and 𝑗 ≤ 𝑘 are zero)

QR factorization 6.38
 Householder algorithm
 
𝑅
the following algorithm overwrites 𝐴 with
0

Algorithm: for 𝑘 = 1 to 𝑛,

1. define 𝑦 = 𝐴 𝑘:𝑚,𝑘 and compute (𝑚 − 𝑘 + 1) -vector 𝑣 𝑘 :

1
𝑤 = 𝑦 + sign(𝑦 1) ∥𝑦∥𝑒 1, 𝑣𝑘 = 𝑤
∥𝑤∥

2. multiply 𝐴 𝑘:𝑚,𝑘:𝑛 with reflector 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 :

𝐴 𝑘:𝑚,𝑘:𝑛 := 𝐴 𝑘:𝑚,𝑘:𝑛 − 2𝑣 𝑘 (𝑣𝑇𝑘 𝐴 𝑘:𝑚,𝑘:𝑛 )

QR factorization 6.39
 Comments

• in step 2 we multiply 𝐴 𝑘:𝑚,𝑘:𝑛 with the reflector 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 :

(𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 ) 𝐴 𝑘:𝑚,𝑘:𝑛 = 𝐴 𝑘:𝑚,𝑘:𝑛 − 2𝑣 𝑘 (𝑣𝑇𝑘 𝐴 𝑘:𝑚,𝑘:𝑛 )

• this is equivalent to multiplying 𝐴 with 𝑚 × 𝑚 reflector

𝑇
0 0 0
   
𝐼
𝐻𝑘 = = 𝐼 −2
0 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 𝑣𝑘 𝑣𝑘

• algorithm overwrites 𝐴 with  

𝑅
0

and returns the vectors 𝑣 1, . . . , 𝑣 𝑛 , with 𝑣 𝑘 of length 𝑚 − 𝑘 + 1

QR factorization 6.40
 Example

example on page 6.8:

 −1 −1 1 
 1 3 3 
   
 𝑅
𝐴 =  = 𝐻 𝐻 𝐻
1 2 3
−1 −1 5  0
 1 3 7 
 


we compute reflectors 𝐻1, 𝐻2, 𝐻3 that triangularize 𝐴:

 𝑅11 𝑅12 𝑅13 

0
 
 𝑅22 𝑅23 
𝐻3 𝐻2 𝐻1 𝐴 = 
0 0

𝑅33 
0 0 0
 
 
 

QR factorization 6.41
 Example
First column of 𝑅

• compute reflector that maps first column of 𝐴 to multiple of 𝑒 1:

 −1   −3   −3 
 1   1  1 1  1 
     
𝑦 =  , 𝑤 = 𝑦 − ∥𝑦∥𝑒 1 =  , 𝑣1 = 𝑤= √ 
 −1   −1  ∥𝑤∥ 2 3  −1 
  
 1   1   1 
     

• overwrite 𝐴 with product of 𝐼 − 2𝑣 1 𝑣𝑇1 and 𝐴

 2 4 2 
0 4/3 8/3 

𝐴 := (𝐼 − 2𝑣 1 𝑣 1 ) 𝐴 = 
𝑇

0 2/3 16/3 
0 4/3 20/3 




QR factorization 6.42
 Example
Second column of 𝑅

• compute reflector that maps 𝐴2:4,2 to multiple of 𝑒 1:

 4/3   10/3   5 
1 1  
𝑦 =  2/3  , 𝑤 = 𝑦 + ∥𝑦∥𝑒 1 =  2/3  , 𝑤=√  1 
   
𝑣2 =
 4/3   4/3  ∥𝑤∥ 30  2 
     

• overwrite 𝐴2:4,2:3 with product of 𝐼 − 2𝑣 2 𝑣𝑇2 and 𝐴2:4,2:3:

 2 4 2 
1 0 0 −2
  
−8 
𝐴 :=

𝐴 = 
0 𝐼 − 2𝑣 2 𝑣𝑇2 0 0 16/5 
0 0 12/5 




QR factorization 6.43
 Example
Third column of 𝑅

• compute reflector that maps 𝐴3:4,3 to multiple of 𝑒 1:

16/5 36/5 1 1 3
     
𝑦= , 𝑤 = 𝑦 + ∥𝑦∥𝑒 1 = , 𝑣3 = 𝑤=√
12/5 12/5 ∥𝑤∥ 10 1

• overwrite 𝐴3:4,3 with product of 𝐼 − 2𝑣 3 𝑣𝑇3 and 𝐴3:4,3:

 2 4 2 
0 0 −2 −8 
  
𝐼
𝐴 :=

𝐴 = 
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 0 −4 
0 0 0 




QR factorization 6.44
 Example

Final result

0 1 0
  
𝐼
𝐻3 𝐻2 𝐻1 𝐴 = (𝐼 − 2𝑣 1 𝑣𝑇1 ) 𝐴
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 𝐼 − 2𝑣 2 𝑣𝑇2
 2 4 2 
0 1 0 0 4/3 8/3 
  
𝐼 
=
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 𝐼 − 2𝑣 2 𝑣𝑇2 0 2/3 16/3 


0 4/3 20/3 



 2 4 2 
0 0 −2
  
𝐼  −8 
=
0 𝐼 − 2𝑣 3 𝑣𝑇3 0 0 16/5
 
 
0 0 12/5
 
 
 
 2 4 2 
0 −2 −8 


=
0 0 −4 


0 0 0 




QR factorization 6.45
 Complexity

Complexity in cycle 𝑘 (of algorithm on page 6.39): the dominant terms are

• (2(𝑚 − 𝑘 + 1) − 1)(𝑛 − 𝑘 + 1) flops for product 𝑣𝑇𝑘 ( 𝐴 𝑘:𝑚,𝑘:𝑛 )

• (𝑚 − 𝑘 + 1)(𝑛 − 𝑘 + 1) flops for outer product with 𝑣 𝑘
• (𝑚 − 𝑘 + 1)(𝑛 − 𝑘 + 1) flops for subtraction from 𝐴 𝑘:𝑚,𝑘:𝑛

sum is roughly 4(𝑚 − 𝑘 + 1) (𝑛 − 𝑘 + 1) flops

Total for computing 𝑅 and vectors 𝑣 1, . . . , 𝑣 𝑛 :

𝑛 ∫ 𝑛
4(𝑚 − 𝑘 + 1) (𝑛 − 𝑘 + 1) 4(𝑚 − 𝑡)(𝑛 − 𝑡)𝑑𝑡
∑︁
∼
𝑘=1 0
2 3 2
= 2𝑚𝑛 − 𝑛 flops
3

QR factorization 6.46
 Q-factor

the Householder algorithm returns the vectors 𝑣 1, . . . , 𝑣 𝑛 that define

𝑄˜ = 𝐻1 𝐻2 · · · 𝐻 𝑛
 
𝑄

• usually there is no need to compute the matrix [ 𝑄 𝑄˜ ] explicitly

• the vectors 𝑣 1, . . . , 𝑣 𝑛 are an economical representation of [ 𝑄 𝑄˜ ]
• products with [ 𝑄 𝑄˜ ] or its transpose can be computed as

𝑄˜ 𝑥 = 𝐻1 𝐻2 · · · 𝐻 𝑛 𝑥
 
𝑄

𝑄˜
𝑇
𝑦 = 𝐻𝑛 𝐻𝑛−1 · · · 𝐻1 𝑦

𝑄

QR factorization 6.47
 Multiplication with Q-factor

• the matrix–vector product 𝐻 𝑘 𝑥 is defined as

0
    
𝐼 𝑥1:𝑘−1 𝑥1:𝑘−1
𝐻𝑘 𝑥 = =
0 𝐼 − 2𝑣 𝑘 𝑣𝑇𝑘 𝑥 𝑘:𝑚 𝑥 𝑘:𝑚 − 2(𝑣𝑇𝑘 𝑥 𝑘:𝑚 )𝑣 𝑘

• complexity of multiplication 𝐻 𝑘 𝑥 is 4(𝑚 − 𝑘 + 1) flops:

• complexity of multiplication with 𝐻1 𝐻2 · · · 𝐻𝑛 or its transpose is

𝑛
4(𝑚 − 𝑘 + 1) ∼ 4𝑚𝑛 − 2𝑛2 flops
∑︁
𝑘=1

• roughly equal to matrix–vector product with 𝑚 × 𝑛 matrix (2𝑚𝑛 flops)

QR factorization 6.48