Lecture 1.

Matrix-Vector Multiplication

You already know the formula for matrix-vector multiplication. Nevertheless,

the purpose of this rst lecture is to describe a way of interpreting such prod-
ucts that may be less familiar. If b = Ax, then b is a linear combination of
the columns of A.

Familiar Denitions
Let x be an n-dimensional column vector and let A be an mn matrix (m rows,
n columns). Then the matrix-vector product b = Ax is an m-dimensional
column vector dened as follows:
n
X
bi = aij xj ; i = 1; : : : ; m. (1:1)
j =1
Here bi denotes the ith entry of b, aij denotes the i; j entry of A (ith row,
j th column), and xj denotes the j th entry of x. For simplicity, we assume in
all but a few lectures of this book that quantities such as these belong to C,
the eld of complex numbers. The space of m-vectors is Cm, and the space of
m  n matrices is Cmn.
The map x 7! Ax is linear , which means that, for any x; y 2 Cn and any
 2 C,
A(x + y) = Ax + Ay,
A(x) = Ax.
3
4 Part I Fundamentals
Conversely, every linear map from Cn to Cm can be expressed as multiplication
by an m  n matrix.

A Matrix Times a Vector

Let aj denote the j th column of A, an m-vector. Then (1.1) can be rewritten
n
X
b = Ax = xj aj . (1:2)
j =1
This equation can be displayed schematically as follows:
2 3 2 32 3 2 3 2 3 2 3
x1
66 7
7 6
6 77 6 x2 777 6 7 6 7 6 7
66 7 6 77 66 6
6 7
7 6
6 7
7 6
6 7
7
66 b 7
7 = 6
6 a1 a2


an 7 64 ... 75 = x1 6 a1 77 +
6 x26 a2 7 +


+ xn6 an 77 .
6 7 6
7 6 75 6
4 7 6
4 7 6
4 7
4 5 4 xn 5 5 5

In (1.2), b is expressed as a linear combination of the columns aj . Nothing

but a slight change of notation has occurred in going from (1.1) to (1.2). Yet
thinking of Ax in terms of the form (1.2) is essential for a proper understanding
of the algorithms of numerical linear algebra.
One way to summarize these dierent ways of viewing matrix-vector prod-
ucts is like this. As mathematicians, we are used to viewing the formula
Ax = b as a statement that A acts on x to produce b. The formula (1.2), by
contrast, suggests the interpretation that x acts on A to produce b.
Example 1.1. Fix a sequence of numbers fx1; : : : ; xmg. If p and q are polyno-
mials of degree < n and  is a scalar, then p + q and p are also polynomials
of degree < n. Moreover, the values of these polynomials at the points xi
satisfy the following linearity properties:
(p + q)(xi ) = p(xi) + q(xi ),
(p)(xi ) =  (p(xi )).
Thus the map from vectors of coecients of polynomials p of degree < n to
vectors ( p(x1 ); p(x2 ); : : : ; p(xm)) of sampled polynomial values is linear. Any
linear map can be expressed as multiplication by a matrix; this is an example.
In fact, it is expressed by an m  n Vandermonde matrix
2 3
6
1 x1 x21


xn1 1
7
6
6 1 x2 x22


xn2 1 7
7
A = 66 ... ... ... ... 7
7 .
6
4 7
5
1 xm x2m


xnm 1
Lecture 1 Matrix-Vector Multiplication 5
If c is the column vector of coecients of p,
2 3
6
c0 7
6
6 c1 7
7
c = 666 c2 7
7 , p(x) = c0 + c1x + c2 x2 +


+ cn 1xn 1,
6 ... 7
7
4 5
cn 1
then the product Ac gives the sampled polynomial values. That is, for each i
from 1 to m, we have
(Ac)i = c0 + c1xi + c2x2i +


+ cn 1xni 1 = p(xi). (1:3)
In this example, it is clear that the matrix-vector product Ac need not
be thought of as m distinct scalar summations, each giving a dierent linear
combination of the entries of c, as (1.1) might suggest. Instead, A can be
viewed as a matrix of columns, each giving sampled values of a monomial,
2 3
6
6 7
7
6 7
A = 66 1 x x2


xn 1 7
7 , (1:4)
6
4 7
5

and the product Ac should be understood as a single vector summation in the

form of (1.2) that at once gives a linear combination of these monomials,
Ac = c0 + c1x + c2 x2 +


+ cn 1xn 1 = p(x).
2
The remainder of this lecture will review some fundamental concepts in
linear algebra from the point of view of (1.2).

A Matrix Times a Matrix

For the matrix-matrix product B = AC , each column of B is a linear com-
bination of the columns of A. To derive this fact, we begin with the usual
formula for matrix products. If A is `  m and C is m  n, then B is `  n,
with entries dened by m
X
bij = aik ckj . (1:5)
k=1
Here bij , aik , and ckj are entries of B , A, and C , respectively. Written in terms
of columns, the product is
2 3 2 32 3
6 7 6 7 6 7
6
6 7
7 6
6 7
7 6 c1 c2


cn 75
6
6 b1 b2


bn 7 = 66
7 a1 a2


am 7 4
7 ,
6
4 7
5 6 4 7
5
6 Part I Fundamentals
and (1.5) becomes
m
X
bj = Acj = ckj ak . (1:6)
k=1
Thus bj is a linear combination of the columns ak with coecients ckj .
Example 1.2. A simple example of a matrix-matrix product is the outer
product . This is the product of an m-dimensional column vector u with an
n-dimensional row vector v; the result is an m  n matrix of rank 1. The outer
product can be written
2 3h i 2 3 2 3
v1 v2


vn v1 u1


vnu1
66 7
7 6
6 7
7 6
6 7
7
66 7 6 7 6 ... ... 7
66 u 7
7 = 66 v1 u v2 u


vn u 77 = 66 7
7 .
4 7
5 6
4 7
5 6 4 7
5
v1 um


vnum
The columns are all multiples of the same vector u, and similarly, the rows
are all multiples of the same vector v. 2
Example 1.3. As a second illustration, consider B = AR, where R is the
upper-triangular n  n matrix with entries rij = 1 for i  j and rij = 0 for
i > j . This product can be written
2 3 2 32 3
6 7 6 7 6
1


1 7
6
6 7
7 6
6 7
7 6
6 ... ... 7
7
6
6 b1


bn 77 = 6
6 a1


an 7 4
7 5.
6
4 7
5 6
4 7
5 1

The column formula (1.6) now gives

j
X
bj = Arj = ak . (1:7)
k=1
That is, the j th column of B is the sum of the rst j columns of A. The
matrix R is a discrete analogue of an indenite integral operator. 2
Range and Nullspace
The range of a matrix A, written range(A), is the set of vectors that can be
expressed as Ax for some x. The formula (1.2) leads naturally to the following
characterization of range(A):
Theorem 1.1. range(A) is the space spanned by the columns of A.
Lecture 1 Matrix-Vector Multiplication 7
Proof. By (1.2), any Ax is a linear combination of the columns of A. Con-
versely, any vector y in the space spanned by the
Pncolumns of A can be written
as a linear combination of the columns, y = j =1 xj aj . Forming a vector x
out of the coecients xj , we have y = Ax, and thus y is in the range of A. 2

In view of Theorem 1.1, the range of a matrix A is also called the column
space of A.
The nullspace of A 2 Cmn, written null(A), is the set of vectors x that
satisfy Ax = 0, where 0 is the 0-vector in Cm. The entries of each vector
x 2 null(A) give the coecients of an expansion of zero as a linear combination
of columns of A: 0 = x1a1 + x2a2 +


+ xn an.

Rank
The column rank of a matrix is the dimension of its column space. Similarly,
the row rank of a matrix is the dimension of the space spanned by its rows.
Row rank always equals column rank (among other proofs, this is a corollary
of the singular value decomposition, discussed in Lectures 4 and 5), so we refer
to this number simply as the rank of a matrix.
An m  n matrix of full rank is one that has the maximal possible rank (the
lesser of m and n). This means that a matrix of full rank with m  n must
have n linearly independent columns. Such a matrix can also be characterized
by the property that the map it denes is one-to-one:
Theorem 1.2. A matrix A 2 Cmn with m  n has full rank if and only if
it maps no two distinct vectors to the same vector.
Proof. (=)) If A is of full rank, its columns are linearly independent, so they
form a basis for range(A). This means that every b 2 range(A) has a unique
linear expansion in terms of the columns of A, and therefore, by (1.2), every
b 2 range(A) has a unique x such that b = Ax. ((=) Conversely, if A is
Pn aj are dependent, and there is a nontrivial linear
not of full rank, its columns
combination such that j =1 cj aj = 0. The nonzero vector c formed from the
coecients cj satises Ac = 0. But then A maps distinct vectors to the same
vector since, for any x, Ax = A(x + c). 2

Inverse
A nonsingular or invertible matrix is a square matrix of full rank. Note
that the m columns of a nonsingular m  m matrix A form a basis for the
whole space Cm . Therefore, we can uniquely express any vector as a linear
8 Part I Fundamentals
combination of them. In particular, the canonical unit vector with 1 in the
j th entry and zeros elsewhere, written ej , can be expanded:
m
X
ej = zij ai . (1:8)
i=1
Let Z be the matrix with entries zij , and let zj denote the j th column of
Z . Then (1.8) can be written ej = Azj . This equation has the form (1.6); it
can be written again, most concisely, as
2 3
6
6 7
7
6 7
6
6 e1


em 77 = I = AZ .
6
4 7
5

The matrix Z is the inverse of A. Any square nonsingular matrix A has a

unique inverse, written A 1, that satises AA 1 = A 1A = I .
The following theorem records a number of equivalent statements that hold
when a square matrix is nonsingular. These conditions appear in linear algebra
texts, and we shall not give a proof here. Concerning (f), see Lecture 5.
Theorem 1.3. For A 2 Cmm, the following conditions are equivalent:
(a) A has an inverse A 1,
(b) rank(A) = m,
(c) range(A) = Cm,
(d) null(A) = f0g,
(e) 0 is not an eigenvalue of A,
(f) 0 is not a singular value of A,
(g) det(A) 6= 0.
Concerning (g), we mention that the determinant, though a convenient notion
theoretically, rarely nds a useful role in numerical algorithms.

A Matrix Inverse Times a Vector

When writing the product x = A 1b, it is important not to let the inverse-
matrix notation obscure what is really going on! Rather than thinking of x as
the result of applying A 1 to b, we should understand it as the unique vector
that satises the equation Ax = b. By (1.2), this means that x is the vector
of coecients of the unique linear expansion of b in the basis of columns of A.
This point cannot be emphasized too much, so we repeat:
A 1b is the vector of coecients of the expansion of b
in the basis of columns of A.
 Lecture 1 Matrix-Vector Multiplication 9
Multiplication by A 1 is a change of basis operation:

Multiplication by A 1

b: A 1b:
coecients of coecients of
the expansion of b the expansion of b
in fe1; : : : ; emg in fa1 ; : : : ; am g
Multiplication by A

In this description we are being casual with terminology, using \b" in one
instance to denote an m-tuple of numbers and in another as a point in an
abstract vector space. The reader should think about these matters until he
or she is comfortable with the distinction.

A Note on m and n
Throughout numerical linear algebra, it is customary to take a rectangular
matrix to have dimensions m  n. We follow this convention in this book.
What if the matrix is square? The usual convention is to give it dimensions
n  n, but in this book we shall generally take the other choice, m  m. Many
of our algorithms require us to look at rectangular submatrices formed by
taking a subset of the columns of a square matrix. If the submatrix is to be
m  n, the original matrix had better be m  m.

Exercises
1. Let B be a 4  4 matrix to which we apply the following operations:
1. double column 1,
2. halve row 3,
3. add row 3 to row 1,
4. interchange columns 1 and 4,
5. subtract row 2 from each of the other rows,
6. replace column 4 by column 3,
7. delete column 1 (so that the column dimension is reduced by 1).
(a) Write the result as a product of eight matrices.
(b) Write it again as a product ABC (same B ) of three matrices.
2. Suppose masses m1 ; m2; m3; m4 are located at positions x1 ; x2; x3; x4 in a line
and connected by springs with spring constants k12; k23; k34 whose natural
 10 Part I Fundamentals
lengths of extension are `12; `23; `34. Let f1; f2; f3 ; f4 denote the rightward
forces on the masses, e.g., f1 = k12(x2 x1 `12).
(a) Write the 4  4 matrix equation relating the column vectors f and x. Let
K denote the matrix in this equation.
(b) What are the dimensions of the entries of K in the physics sense (e.g.,
mass times time, distance divided by mass, etc.)?
(c) What are the dimensions of det(K ), again in the physics sense?
(d) Suppose K is given numerical values based on the units meters, kilograms,
and seconds. Now the system is rewritten with a matrix K 0 based on centime-
ters, grams, and seconds. What is the relationship of K 0 to K ? What is the
relationship of det(K 0 ) to det(K )?
3. Generalizing Example 1.3, we say that a square or rectangular matrix R with
entries rij is upper-triangular if rij = 0 for i > j . By considering what space
is spanned by the rst n columns of R and using (1.8), show that if R is a
nonsingular m  m upper-triangular matrix, then R 1 is also upper-triangular.
(The analogous result also holds for lower-triangular matrices.)
4. Let f1 ; : : : ; f8 be a set of functions dened on the interval [1; 8] with the
property that for any numbers d1; : : : ; d8 , there exists a set of coecients
c1; : : : ; c8 such that
X
8
cj fj (i) = di; i = 1; : : : ; 8:
j =1

(a) Show by appealing to the theorems of this lecture that d1; : : : ; d8 determine
c1; : : : ; c8 uniquely.
(b) Let A be the 8  8 matrix representing the linear mapping from data
d1; : : : ; d8 to coecients c1; : : : ; c8. What is the i; j entry of A 1 ?