LECTURE 10

Householder Triangularization

Objective: The Householder algorithm is a process of

orthogonal triangularization, i.e., it reduces a matrix
to triangular form by a sequence of orthogonal matrix
operations. It is more stable than Gram-Schmidt, but it
cannot be used in an iterative fashion.

Householder and Gram-Schmidt

Recalling Lecture 8, the Gram-Schmidt iteration applies
a succession of elementary triangular matrices Rk to the
right of A so that the resulting matrix has orthonormal
columns:
AR1R2 . . . Rn = Q̂
where
R̂−1 = R1R2 . . . Rn.

Each Rk is upper-triangular and so is

R̂ = Rn−1Rn−1
−1
. . . R1−1.
Thus, A = Q̂R̂ is a reduced QR factorization of A.
In contrast, the Householder method applies a succession
of elementary orthogonal matrices Qk on the left of A, so
that the resulting matrix is upper-triangular:
QnQn−1 . . . Q1A = R
10-1
and defines
QT = QnQn−1 . . . Q1.

Because each Qk is orthogonal, so is

Q = QT1 QT2 . . . QTn .

Thus, A = QR is a full QR factorization of A.

So,

Gram-Schmidt = triangular orthogonalization

Householder = orthogonal triangularization

Triangularization by introducing zeros

The Householder method was first proposed by Alston
Householder in 1958.
It is an ingenious way to systematically design orthogonal
matrices Qk such that

QnQn−1 . . . Q1A

is upper-triangular.
IDEA: Each matrix Qk is designed to introduce zeros
below the diagonal in column k while preserving all zeros
previously introduced.

10-2
Here is a 5 × 3 example:
   
× × × × × ×
× × ×  0 × ×
  Q1  
× × ×  0 × ×
   
 × × ×  −→  0 × ×
× × × 0 × ×
   
× × × × × ×
 × ×  × ×
  Q2  
 × ×  0 ×
  −→  
 × ×  0 ×
× × 0 ×
   
× × × × × ×
 × ×  × ×
  Q3  
 ×  ×
  −→  
 ×  0 
× 0

In general, Qk operates on rows k to m.

At the beginning of step k, there is a block of zeros in the
first k − 1 columns of these rows.
Qk forms linear combinations of these rows, and thus ze-
ros are undisturbed (a linear combination of zeros is still
zero!).
After all n steps, all entries below the diagonal are zero.

10-3
Thus,
QnQn−1 . . . Q1A = R
is upper-triangular.

Householder reflectors
We now investigate the construction of the Qk .
Each Qk is chosen to be an orthogonal matrix of the form
' (
I 0
Qk =
0 F
where
I is (k − 1) × (k − 1)
F is (m − k + 1) × (m − k + 1) and orthogonal

Multiplication by F must introduce zeros in column k.

The Householder algorithm chooses F to be a House-
holder reflector.
Suppose at the beginning of step k, the (potentially)
nonzero entries to be zeroed are called x ∈ Rm−k+1.
The Householder reflector F performs the following op-
eration
 
  %x%
×  0 
× F  

x= .   Fx =  0 
 = %x%e1
. −→ 
 .. 
×
0
10-4
(F is orthogonal and cannot change the norm of x!)
Geometrically the idea is given in the following diagram:
!

F reflects x across the hyperplane H orthogonal to

v = %x%e1 − x

In general, a hyperplane can be characterized as the set

of points orthogonal to some nonzero vector v (see figure)
H maps every point on one side to its mirror image on
the other side.
In particular, x is mapped to %x%e1.
10-5
The orthogonal projection of y ∈ Rm onto H is given by
) *
vv T
Py = I − T y
v v
) T *
v y
= y−v .
vT v

To reflect y across H, we need to go twice as far!

) T
*
2vv
∴ Fy = I − T y
v v
) T *
v y
= y − 2v
vT v

Hence,
2vv T
F =I− T
v v
so that the only difference between the full rank and or-
thogonal F and the rank (m − 1) and orthogonal P is
the factor of 2.

The better of two reflectors

There is another possibility for a Householder reflector
that we have ignored so far. It corresponds to mapping
x to −%x%e1.
Mathematically, this is also an acceptable choice accord-
ing to our criteria for defining F .

10-6
!

Let us call these two mappings H +, H − according to

whether x is mapped to +%x%e1 or −%x%e1.
Although both mappings are satisfactory mathematically,
for a given x, one will have better numerical stability
properties then the other.
For numerical stability, we want that Hx not be too close
to x itself.
To achieve this, we can choose

v = −sgn(x1)%x%e1 − x

10-7
where x1 is the first component of x and
+
+1 if x1 ≥ 0
sgn(x1) =
−1 otherwise

To see why the choice of sign affects stability, consider

what happens if H + makes a small angles with respect
to e1. Then, v = %x%e1 − x is much smaller than either
%x%e1 or x.
In other words, there is a significant loss of precision when
computing v thanks to cancelation.
Our choice of v however ensures that %v% ≥ %x%.

The Householder algorithm

The following algorithm computes the factor R of a QR
factorization of an m × n matrix A with m ≥ n, over-
writing A with the result.
We also store n reflection vectors v1, v2, . . . , vn.

10-8
ALGORITHM 10.1: HOUSEHOLDER QR FACTOR-
IZATION
for k = 1 to n do
x = A(k : m, k)
vk = sgn(x1)%x%2e1 + x % This is −v, but
% that doesn’t matter
vk = %vvk%2
k
A(k : m, k : n) = A(k : m, k : n)
−2vk vkT A(k : m, k : n)
end for

Applying or forming Q
This algorithm reduces A to upper triangular form (the
“R” in the QR factorization).
The “Q” of this factorization (or even the Q̂) has not
been constructed.
The reason is that this takes additional work!
Quite often we do not need Q anyway, only its effect on
some vector
i.e., we only need the product Qx or QT x.
For this we note

QT = QnQn−1 . . . Q1

and
Q = Q1Q2 . . . Qn

10-9
(why?)
e.g. To solve Ax = b by QR factorization, we write
QRx = b, then Rx = QT b
→ so the only way we ever need Q is in QT b.
Recall A = QR =⇒ R = QT A
i.e., the same process that reduced A to R is equivalent
to multiplication by QT .
ALGORITHM 10.2: CALCULATION OF QT b
for k = 1 to n do
b(k : m) = b(k : m) − 2vk vkT b(k : m)
end for
Similarly, calculation of Qx can be achieved by the same
process reversed.
ALGORITHM 10.3: CALCULATION OF Qx
for k = n downto 1 do
x(k : m) = x(k : m) − 2vk vkT x(k : m)
end for

Operation count
The work involved in Algorithm 10.1 is dominated by the
inner most loop
A(k : m, j) − 2vk vkT A(k : m, j)

If the vector has length l = m − k − 1, we need l subtrac-

tions, l scalar multiplications l multiplications, and l − 1
10-10
additions for the dot product.

= 4l − 1 ≈ 4l (∼ 4 flops per entry operated on)

Schematically,
 here iswhat is going
 on for a 5 × 4 matrix:
× × × × ∗ ∗ ∗ ∗
× × × × 0 ∗ ∗ ∗
   
 × × × ×  −→  0 ∗ ∗ ∗ 
   
× × × × 0 ∗ ∗ ∗
× × × × 0 ∗ ∗ ∗
original matrix step 1
   
× × × × × × × ×
 ∗ ∗ ∗  × × ×
   
−→ 
 0 ∗  
∗  −→  ∗ ∗
 0 ∗ ∗  0 ∗
0 ∗ ∗ 0 ∗
step 2 step 3
 
× × × ×
 × × ×
 

−→  × × 
 ∗
0
step 4
At step k, rows 1 to k − 1 are unchanged and columns 1
to k are zero.
So, we do not operate on any of these elements.
10-11
So, at step k, we only operate on (m − k + 1)(n − k) + 1
elements.
∴ operation count

, n .
-
∼ (m − k + 1)(n − k) + 1
k=1
elements * 4 operations/elements
n
-
∼ 4 mn − (m + n + 1)k + k 2
'k=1
2 n(n + 1)
= 4 mn − (m + n + 1) +
2 (
n(n + 1)(2n + 1)
6
' 2 (
mn n3 2 2 3
∼ 4 − = 2mn − n
2 6 3

10-12