REDUCTION TO HESSENBERG AND TRIDIAGONAL FORMS 349

Exercise 5.4.55 Use MATLAB's help facility to find out about the commands schur
and rsf2csf . Use them to find the forms of Schur (Theorem 5.4.11) and Winter-
Murnaghan (Theorem 5.4.22) for the matrix

5.5 REDUCTION TO HESSENBERG AND TRIDIAGONAL FORMS

The results of the previous section encourage us to seek algorithms that reduce a
matrix to triangular form by similarity transformations, as a means of finding the
eigenvalues of the matrix. On theoretical grounds we know that there is no algorithm
that accomplishes this task in a finite number of steps; such an algorithm would
violate Abel's classical theorem, cited in Section 5.2. It turns out, however, that
there are finite algorithms, that is, direct methods, that bring a matrix very close to
upper-triangular form.
Recall that an n x n matrix A is called upper Hessenberg if a^ = 0 whenever
i > j + I. Thus an upper Hessenberg matrix has the form

In this section we will develop an algorithm that uses unitary similarity trans-
formations to transform a matrix to upper Hessenberg form in ^-n3 flops. This
algorithm does not of itself solve the eigenvalue problem, but it is extremely impor-
tant nevertheless because it reduces the problem to a form that can be manipulated
inexpensively. For example, you showed in Exercise 5.3.33 that both the LU and
QR decompositions of an upper Hessenberg matrix can be computed in O(n 2 ) flops.
Thus Rayleigh quotient iteration can be performed at a cost of only 0(n 2 ) flops per
iteration.
The reduction to Hessenberg form is especially helpful when the matrix is Her-
mitian. Since unitary similarity transformations preserve the Hermitian property, the
reduced matrix is not merely Hessenberg, it is tridiagonal. That is, it has the form
 350 EIGENVALUES AND EIGENVECTORS I

Tridiagonal matrices can be manipulated very inexpensively. Moreover, the symme-

try of the matrix can be exploited to reduce the cost the reduction to tridiagonal form
to a modest4/3n3flops.

Exercise 5.5.1 Let A ¤ C n x n be tridiagonal.

(a) Show that the cost of an LU decomposition of A (with or without pivoting) is

O(n) flops.

(b) Show that the cost of a QR decomposition of A is O(ri) flops, provided the
matrix Q is not assembled explicitly.

Reduction of General Matrices

The algorithm that reduces a matrix to upper Hessenberg form is quite similar to the
algorithm that performs a QR decomposition using reflectors. It accomplishes the
task in n — 2 steps. The first step introduces the desired zeros in the first column, the
second step takes care of the second column, and so on.
Let us proceed with the first step. Partition A as

Let Qi be a (real or complex) reflector such that

|2), and let
0T

Let

which has the desired zeros in the first column. This operation is just like the first
step of the QR decomposition algorithm, except that it is less ambitious. Instead of
transforming all but one entry to zero in the first column, it leaves two entries nonzero.
The reason for the diminished ambitiousness is that now we must complete a similarity
transformation by multiplying on the right by . Letting and
 REDUCTION TO HESSENBERG AND TRIDIAGONAL FORMS 351

recalling that , we have

Because of the form of Qi, this operation does not destroy the zeros in the first
column. If we had been more ambitious, we would have failed at this point.
The second step creates zeros in the second column of A\, that is, in the first
column of Â\. Thus we pick a reflector Qi G C^ n ~ 2 ^ x ( n ~ 2 ) in just the same way as
the first step, except that A is replaced by A\. Let

Then

We complete the similarity transformation by multiplying on the right by

Because the first two columns of Q% are equal to the first two columns of the identity
matrix, this operation does not alter the first two columns of A 3 / 2 . Thus

The third step creates zeros in the third column, and so on. After n — 2 steps
the reduction is complete. The result is an upper Hessenberg matrix that is unitarily
similar to A: B = Q*AQ, where

If A is real, then all operations are real, Q is real and orthogonal, and B is orthogonally
similar to A.
352 EIGENVALUES AND EIGENVECTORS I

A computer program to perform the reduction can be organized in exactly the

same way as a QR decomposition by reflectors is. For simplicity we will restrict our
attention to the real case. Most of the details do not require discussion because they
were already covered in Section 3.2. The one way in which the present algorithm is
significantly different from the QR decomposition is that it involves multiplication by
reflectors on the right as well as the left. As you might expect, the right multiplications
can be organized in essentially the same way as the left multiplications are. If we need
to calculate CQ, where Q = I—JUUT is a reflector, we can write CQ — C — jCuuT.
Thus we can compute v — C(ju) and then CQ = C — VUT. This is just a variant of
algorithm (3.2.40).

Real Householder Reduction to Upper Hessenberg Form

This algorithm takes as input an array that contains A G R n x n . It returns an

upper Hessenberg matrix B = QTAQ, whose nonzero elements are in their natural
positions in the array. The portion of the array below the subdiagonal is not set to zero.
It is used to store the vectors Uk, used to generate the reflectors
 REDUCTION TO HESSENBERG AND TRIDIAGONAL FORMS 353

where is the (n — A;) x (n — k) reflector used at step k. The leading 1 in u^ is

not stored. The scalar 7^ (k = 1 , . . . , n — 2) is stored in a separate array 7. Thus the
information needed to construct the orthogonal transforming matrix Q is available.
There are many ways to reorganize the computations in (5.5.2). In particular, there
are block variants suitable for parallel computation and efficient use of high-speed
cache memory. The Hessenberg reduction codes in LAPACK [1] use blocks.

Exercise 5.5.3 Count the flops in (5.5.2). Show that the left multiplication part costs about
|n3 flops, the right multiplication part costs about 2n3 flops, and the rest of the cost
of the reflector setups is O(n 2 ). Thus the total cost is about flops. Why do the
right multiplications require more flops than the left multiplications do? D

Suppose we have transformed A G E n x n to upper Hessenberg form B = QTAQ

using (5.5.2). Suppose further that we have found some eigenvectors of B and we
would now like to find the corresponding eigenvectors of A. For each eigenvector v
of B, Qv is an eigenvector of A. Since

we can easily calculate Qv by applying n - 2 reflectors in succession using (3.2.40)

repeatedly. In fact we can process all of the eigenvectors at once. If we have m of
them, we can build an n x m matrix V whose columns are the eigenvectors, then we
can compute QV — QiQ2 • • • Qn-2V, by applying (3.2.40) n — 2 times. The cost
is about 2n 2 m flops for m vectors.
If we wish to generate the transforming matrix Q itself, we can do so by computing
Q = QI = Q1Q2-- Qn-2I, by applying (3.2.40) n - 2 times.

The Symmetric Case

If A is Hermitian, then the matrix B produced by (5.5.2) is not merely Hessenberg, it
is tridiagonal. Furthermore it is possible to exploit the symmetry of A to reduce the
cost of the reduction to |n3 flops, less than half the cost of the non-symmetric case.
In the interest of simplicity we will restrict our attention to the real symmetric case.
We begin with the matrix
on bT
b A
In the first step of the reduction we transform A to A\ = QiAQi, where
an —TL 0 ••• 0

We save a little bit here by not performing the computation bTQi, which duplicates
the computation Qib.
354 EIGENVALUES AND EIGENVECTORS I

The bulk of the effort in this step is expended in the computation of the symmetric
submatrix . We must do this efficiently if we are to realize significant
2
savings. If symmetry is not exploited, it costs about 4n flops to calculate and
2
another 4n flops to calculate , as you can easily verify. Thus the entire
computation of A\ costs about 8n2 flops. It turns out that we can cut this figure in
half by carefully exploiting symmetry.
Qi is a reflector given in the form Thus

The terms in this expression admit considerable simplification if we introduce the

auxiliary vector v = —„Au. We have — „ÂuuT = VUT, —„UU T Â = UVT, and
"y2uuTÂuuT = —JUUTVUT. Introducing the scalar a = —^UTV, we can rewrite
this last term as 2auuT. Thus

The final manipulation is to split the last term into two pieces in order to combine
one piece with the term VUT and the other with uvT. In other words, let w = v + an.
Then

This equation translates into the code segment

which costs four flops per updated array entry. By symmetry we need only update
the main diagonal and lower triangle. Thus the total number of flops in this segment
i s 4 ( n — l)n/2 « 2n 2 . This does not include the cost of calculating w. First of all,
the computation v = —„Au costs about 2n2 flops. The computation a = — ^UTV
costs about In flops, as does the computation w = v + cm. Thus the total flop count
for the first step is about 4n2, as claimed.
The second step of the reduction is identical to the first step, except that it acts on
the submatrix AI . In particular, (unlike the nonsymmetric reduction) it has no effect
on the first row of AI . Thus the flop count for the second step is about 4(n — I)2 flops.
After n — 2 steps the reduction is complete. The total flop count is approximately
4(n2 + (n - I)2 + (n - 2)2 + •••)« f n3.
 REDUCTION TO HESSENBERG AND TRIDIAGONAL FORMS 355

Reduction of a Real Symmetric Matrix to Tridiagonal Form

This algorithm accesses only the main diagonal and lower triangle of A. It stores
the main-diagonal entries of the tridiagonal matrix B in a one-dimensional array d
(di = ba, i — 1,... ,n) and the off-diagonal entries in a one-dimensional array s
(Si = &i+i,i = &i,i+i, i = 1,. • • ,n — 1). The information about the reflectors used
in the similarity transformation is stored exactly as in (5.5.2).

Additional Exercises

Exercise 5.5.5 Write a Fortran subroutine that implements (5.5.2).

Exercise 5.5.6 Write a Fortran subroutine that implements (5.5.4).

Exercise 5.5.7 MATLAB's hess command transforms a matrix to upper Hessenberg form.
To get just the upper Hessenberg matrix, type B = hess (A) . To get both the
Hessenberg matrix and the transforming matrix (fully assembled), type [Q, B] =
hess(A).
(a) Using MATLAB, generate a random 100 x 100 (or larger) matrix and reduce
it to upper Hessenberg form:
n = 100
A = r a n d n ( n ) ; % or A = randn(n) + i * r a n d n ( n ) ;
[Q,B] = hess(A);

(b) Use Rayleigh quotient iteration to compute an eigenpair of B, starting from a

random complex vector. For example,
 356 EIGENVALUES AND EIGENVECTORS I

q = randn(n,l) + i*randn(n,1);
q = q/norm(q);
rayquo = q'*B*q;
qq = (B - rayquo*eye(n))\q;

and so on. Once you have an eigenvector qofB, transform it to an eigenvector

v = Qq of A. Your last Rayleigh quotient is your eigenvalue.

(c) Give evidence that your Rayleigh quotient iteration converged quadratically.

(d) Check that you really do have an eigenpair of A by computing the residual
norm \\Av — Xv\\2.
D

Exercise 5.5.8 The first step of the reduction to upper Hessenberg form introduces the needed
zeros into the vector b, where
r T

We used a reflector to do the task, but there are other types of transformations
that we could equally well have used. Sketch an algorithm that uses Gaussian
elimination with partial pivoting to introduce the zeros at each step. Don't forget that
each transformation has to be a similarity transformation. Thus we get a similarity
transformation to Hessenberg form that is, however, nonunitary. This is a good
algorithm that has gone out of favor. The old library EISPACK includes it, but it has
been left out of LAPACK. D

5.6 THE QR ALGORITHM

For many years now the most widely used algorithm for calculating the complete set of
eigenvalues of a matrix has been the QR algorithm of Francis [25] and Kublanovskaya
[45]. The present section is devoted to a description of the algorithm, and in the
section that follows we will show how to implement the algorithm effectively. The
explanation of why the algorithm works is largely postponed to Section 6.2. You can
read Sections 6.1 and 6.2 right now if you prefer.
Consider a matrix A G C n x n whose eigenvalues we would like to compute. For
now let us assume that A is nonsingular, a restriction that we will remove later. The
basic QR algorithm is very easy to describe. It starts with AQ — A and generates a
sequence of matrices ( Aj ) by the following prescription:

That is, Am-i is decomposed into factors Qm and Rm such that Qm is unitary and
Rm is upper triangular with positive entries on the main diagonal. These factors are
uniquely determined (Theorem 3.2.46). The factors are then multiplied back together
 THE QR ALGORITHM 357

in the reverse order to produce Am. You can easily verify that Am = Q*mAm-\Qm.
Thus Am is unitarily similar to Am-\.

Exercise 5.6.2
(a) Show that Am = Q^Am^Qm.
(b) Show that Am - RmAm-iR^.
Notice that part (a) is valid, regardless of whether or not A is singular. In contrast,
part (b) is valid if and only if A is nonsingular. (Why?) D

Since all matrices in the sequence (Aj) are similar, they all have the same eigenval-
ues. In Section 6.2 we will see that the QR algorithm is just a clever implementation
of a procedure known as simultaneous iteration, which is itself a natural, easily
understood extension of the power method. As a consequence, the sequence (Aj)
converges, under suitable conditions, to upper-triangular form

where the eigenvalues appear in order of decreasing magnitude on the main diagonal.
(As we shall see, this is a gross oversimplification of what usually happens, but there
is no point in discussing the details now.)
From now on it will be important to distinguish between the QR decomposition
and the QR algorithm. The QR algorithm is an iterative procedure for finding
eigenvalues. It is based on the QR decomposition, which is a direct procedure
related to the Gram-Schmidt process. A single iteration of the QR algorithm will be
called a QR step or QR iteration.
Each QR step performs a unitary similarity transformation. We noted in Sec-
tion 5.4 that numerous matrix properties are preserved under such transformations,
including the Hermitian property. Thus if A is Hermitian, then all iterates Aj will be
Hermitian, and the sequence (Aj) will converge to diagonal form.
If Am-i is real, then Qm, Rm, and Am are also real. Thus if A is real, the basic
QR algorithm (5.6.1) will remain within the real number system.

Example 5.6.3 Let us apply the basic QR algorithm to the real symmetric matrix

whose eigenvalues are easily seen to be AI = 9 and A2 = 4. Letting AQ = A, we

have AQ = QiRi, where