Explore the full ebook collection and download it now at textbookfull.

com

Matrix Algebra: Theory, Computations and

Applications in Statistics 3rd Edition James E.
Gentle

https://textbookfull.com/product/matrix-algebra-theory-
computations-and-applications-in-statistics-3rd-edition-
james-e-gentle/

OR CLICK HERE

DOWLOAD EBOOK

Browse and Get More Ebook Downloads Instantly at https://textbookfull.com

Click here to visit textbookfull.com and download textbook now
 Your digital treasures (PDF, ePub, MOBI) await
Download instantly and pick your perfect format...

Read anywhere, anytime, on any device!

Linear Algebra and Matrix Computations With MATLAB 1st

Edition Dingyü Xue

https://textbookfull.com/product/linear-algebra-and-matrix-
computations-with-matlab-1st-edition-dingyu-xue/

textbookfull.com

Matrix Differential Calculus with Applications in

Statistics and Econometrics 3rd Edition Jan R. Magnus

https://textbookfull.com/product/matrix-differential-calculus-with-
applications-in-statistics-and-econometrics-3rd-edition-jan-r-magnus/

textbookfull.com

Basic Matrix Theory Leonard E. Fuller

https://textbookfull.com/product/basic-matrix-theory-leonard-e-fuller/

textbookfull.com

Biota Grow 2C gather 2C cook Loucas

https://textbookfull.com/product/biota-grow-2c-gather-2c-cook-loucas/

textbookfull.com
Computations and Combinatorics in Commutative Algebra EACA
School Valladolid 2013 1st Edition Anna M. Bigatti

https://textbookfull.com/product/computations-and-combinatorics-in-
commutative-algebra-eaca-school-valladolid-2013-1st-edition-anna-m-
bigatti/
textbookfull.com

Inorganic Chemistry 3rd Edition James E. House

https://textbookfull.com/product/inorganic-chemistry-3rd-edition-
james-e-house/

textbookfull.com

Programming For Computations - Python: A Gentle

Introduction To Numerical Simulations With Python 3.6
Svein Linge
https://textbookfull.com/product/programming-for-computations-python-
a-gentle-introduction-to-numerical-simulations-with-python-3-6-svein-
linge/
textbookfull.com

Applied Linear Algebra and Matrix Analysis Second Edition

Thomas S. Shores

https://textbookfull.com/product/applied-linear-algebra-and-matrix-
analysis-second-edition-thomas-s-shores/

textbookfull.com

Matrix Methods: Applied Linear Algebra and Sabermetrics

4th Edition Richard Bronson

https://textbookfull.com/product/matrix-methods-applied-linear-
algebra-and-sabermetrics-4th-edition-richard-bronson/

textbookfull.com
Springer Texts in Statistics

James E. Gentle

Matrix
Algebra
Theory, Computations and Applications
in Statistics
Third Edition
Springer Texts in Statistics

Series Editors
G. Allen, Department of Statistics, Rice University, Houston, TX, USA
R. De Veaux, Department of Mathematics and Statistics, Williams College,
Williamstown, MA, USA
R. Nugent, Department of Statistics, Carnegie Mellon University, Pittsburgh,
PA, USA
Springer Texts in Statistics (STS) includes advanced textbooks from 3rd- to
4th-year undergraduate levels to 1st- to 2nd-year graduate levels. Exercise sets
should be included. The series editors are currently Genevera I. Allen, Richard
D. De Veaux, and Rebecca Nugent. Stephen Fienberg, George Casella, and
Ingram Olkin were editors of the series for many years.
James E. Gentle

Matrix Algebra
Theory, Computations and
Applications in Statistics

Third Edition
James E. Gentle
Fairfax, VA, USA

ISSN 1431-875X ISSN 2197-4136 (electronic)

Springer Texts in Statistics
ISBN 978-3-031-42143-3 ISBN 978-3-031-42144-0 (eBook)
https://doi.org/10.1007/978-3-031-42144-0
1st edition: © Springer Science+Business Media, LLC 2007
2nd edition: © Springer International Publishing AG 2017
3rd edition: © The Editor(s) (if applicable) and The Author(s), under exclusive license to
Springer Nature Switzerland AG 2024
This work is subject to copyright. All rights are solely and exclusively licensed by the
Publisher, whether the whole or part of the material is concerned, specifically the rights
of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on
microfilms or in any other physical way, and transmission or information storage and re-
trieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in
this publication does not imply, even in the absence of a specific statement, that such names
are exempt from the relevant protective laws and regulations and therefore free for general
use.
The publisher, the authors, and the editors are safe to assume that the advice and informa-
tion in this book are believed to be true and accurate at the date of publication. Neither the
publisher nor the authors or the editors give a warranty, expressed or implied, with respect
to the material contained herein or for any errors or omissions that may have been made.
The publisher remains neutral with regard to jurisdictional claims in published maps and
institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland
AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Paper in this product is recyclable.

To María
Preface to Third Edition

This book is different from the several other books on the general topic of
“matrix algebra and statistics” or “linear algebra and statistics” in its more
extensive coverage of the applications to statistical linear models (mostly in
Part II, especially Chap. 9) and the discussions of numerical computations
(mostly in Part III). This book also includes numerous examples of R in
matrix computations.
The lengths of the chapters vary; I emphasized unity of topics rather than
consistency of numbers of pages. Some topics receive attention at multiple
places in the book. The repetition is intentional because of different motiva-
tions or points of view. The book has extensive cross-references and a large
index to facilitate finding discussions in other parts of the book.
The book can serve different audiences. It can be used as a general reference
book for an applied mathematician, a self-learner, or someone who just needs
a refresher. The extensive index should be useful for such persons.
The book, especially Part I, the first seven chapters, can serve as a text for
a mathematically oriented course in linear algebra for advanced undergradu-
ates or graduates.
It could also serve as a textbook for a course in linear models. The primary
emphasis would be on Part II, with extensive cross-references to the relevant
theory in other chapters.
A course in statistical computing or numerical analysis could be based on
the nitty-gritty of computer arithmetic, the computational issues, and general
design of algorithms in Part III. The emphasis would be on numerical linear
algebra, and there would likely be extensive cross-references to the underlying
theory in other chapters.
There are separate sections in several chapters that discuss the R pro-
gramming system, starting from the basics and going through some of the
more advanced features of R. There are also many examples and numerous
exercises using R. This could serve as a quick course or a refresher course in
R. R is also used for illustration in various places in the text.
VIII Preface to Third Edition

As in the revisions for the second edition, in this third edition I have
corrected all known remaining typos and other errors; I have (it is hoped)
clarified certain passages; I have added some additional material; and I have
enhanced the Index. I have also added exercises in some of the chapters.
The overall organization of chapters has been preserved, but some sections
have been changed. The two chapters that have been changed most are the
original Chap. 4, which is now Chap. 7 and has more coverage of multivariate
probability distributions, and Chap. 9, with more material on linear models.
In this edition, I discuss the R software system much more frequently. It
is likely that most readers know R, but I give a brief introduction to R in
Chap. 1. The most commonly used objects in this book are of class matrix,
but I use data.frame for the linear models of Chap. 9. I do not use any of the
objects, functions, or operators in the set of Tidy packages, which are very
popular nowadays.
I require use of R in several exercises, and assume the reader/user has, or
will develop, at least a moderate level of competence in use of R. R, as any
software system, is learned through usage. Some of the exercises, especially in
Part III, also require competence in Fortran or C.
The notation and terms that I use are “standard”; that is, they are (I
think) among the most commonly used ones in discussions of matrices and
linear algebra, especially by statisticians. Before delving into the book, the
reader may want to take a quick look at Appendix A, and then refer to it
whenever it is necessary to refresh the recognition of a symbol or term.
In previous editions, I had included answers to selected exercises. In this
edition, I have moved the solutions online.

I thank the readers of the first and second editions who informed me
of errors, or who made suggestions for improvement. I particularly thank
Professor M. Yatov for extensive comments on notation and definitions, as
well as for noting several errata and gaps in logic. I also thank Shijia Jin
for many general comments and suggestions. Any remaining typos, omissions,
and so on are entirely my own responsibility.
I thank John Chambers, Robert Gentleman, and Ross Ihaka for their foun-
dational work on R. I thank the R Core Team and the many package devel-
opers and those who maintain the packages that make R more useful.
Again, I thank my wife, María, to whom this book is dedicated, for everything.
 Preface to Third Edition IX

I would appreciate receiving suggestions for improvement and notification of

errors. Notes on this book, hints and solutions to exercises, and errata are
available at
https://mason.gmu.edu/~jgentle/books/matbk/

Fairfax County, VA, USA James E. Gentle

June 30, 2023
Preface to Second Edition

In this second edition, I have corrected all known typos and other errors; I
have (it is hoped) clarified certain passages; I have added some additional
material; and I have enhanced the Index.
I have added a few more comments about vectors and matrices with com-
plex elements, although, as before, unless stated otherwise, all vectors and
matrices in this book are assumed to have real elements. I have begun to
use “.det(A)” rather than “.|A|” to represent the determinant of A, except in
a few cases. I have also expressed some derivatives as the transposes of the
expressions I used formerly.
I have put more conscious emphasis on “user-friendliness" in this edition.
In a book, user-friendliness is primarily a function of references, both internal
and external, and of the index. As an old software designer, I’ve always thought
that user-friendliness is very important. To the extent that internal references
were present in the first edition, the positive feedback I received from users of
that edition about the friendliness of those internal references (“I liked the fact
that you said ‘equation (x.xx) on page yy’, instead of just ‘equation (x.xx)’.
”) encouraged me to try to make the internal references even more useful.
It’s only when you’re “eating your own dogfood”, that you become aware of
where details matter, and in using the first edition, I realized that the choice of
entries in the Index was suboptimal. I have spent significant time in organizing
it, and I hope that the user will find the Index to this edition to be very useful.
I think that it has been vastly improved over the index in the first edition.
The overall organization of chapters has been preserved, but some sec-
tions have been changed. The two chapters that have been changed most are
Chaps. 3 and 12. Chapter 3, on the basics of matrices, got about 30 pages
longer. It is by far the longest chapter in the book, but I just didn’t see any
reasonable way to break it up. In Chap. 12 of the first edition, “Software for
Numerical Linear Algebra”, I discussed four software systems or languages:
C/C++, Fortran, MATLAB, and R, and did not express any preference for
one over another. In this edition, although I occasionally mention various
languages and systems, I now limit most of my discussion to Fortran and R.
XII Preface to Second Edition

There are many reasons for my preference for these two systems. R is ori-
ented toward statistical applications. It is open source and freely distributed.
As for Fortran versus C/C++, Python, or other programming languages, I
agree with the statement by Hanson and Hopkins (2013, page ix), “. . . Fortran
is currently the best computer language for numerical software.” Many people,
however, still think of Fortran as the language their elders (or they themselves)
used in the 1970s. (On a personal note, Richard Hanson, who passed away
recently, was a member of my team that designed the IMSL C Libraries in
the mid1980s. Not only was C much cooler than Fortran at the time, but the
ANSI committee working on updating the Fortran language was so fractured
by competing interests, that approval of the revision was repeatedly delayed.
Many numerical analysts who were not concerned with coolness turned to C
because it provided dynamic storage allocation and it allowed flexible argu-
ment lists, and the Fortran constructs could not be agreed upon.)
Language preferences are personal, of course, and there is a strong “cool-
ness factor” in choice of a language. Python is currently one of the coolest
languages, but I personally don’t like the language for most of the stuff I do.
Although this book has separate parts on applications in statistics and
computational issues as before, statistical applications have informed the
choices I made throughout the book, and computational considerations have
given direction to most discussions.
I thank the readers of the first edition who informed me of errors. Two
people in particular made several meaningful comments and suggestions. Clark
Fitzgerald not only identified several typos, he made several broad suggestions
about organization and coverage that resulted in an improved text (I think).
Andreas Eckner found, in addition to typos, some gaps in my logic, and also
suggested better lines of reasoning at some places. (Although I don’t follow
an itemized “theorem-proof” format, I try to give reasons for any non-obvious
statements I make.) I thank Clark and Andreas especially for their comments.
Any remaining typos, omissions, gaps in logic, and so on are entirely my
responsibility.
Again, I thank my wife, María, to whom this book is dedicated, for everything.

I used TEX via LATEX 2ε to write the book. I did all of the typing, program-
ming, etc., myself, so all misteaks (mistakes!) are mine. I would appreciate
receiving suggestions for improvement and notification of errors. Notes on
this book, including errata, are available at
https://mason.gmu.edu/~jgentle/books/matbk/

Fairfax County, VA, USA James E. Gentle

July 14, 2017
Preface to First Edition

I began this book as an update of Numerical Linear Algebra for Applications

in Statistics, published by Springer in 1998. There was a modest amount of
new material to add, but I also wanted to supply more of the reasoning behind
the facts about vectors and matrices. I had used material from that text in
some courses, and I had spent a considerable amount of class time proving
assertions made but not proved in that book. As I embarked on this project,
the character of the book began to change markedly. In the previous book,
I apologized for spending 30 pages on the theory and basic facts of linear
algebra before getting on to the main interest: numerical linear algebra. In
the present book, discussion of those basic facts takes up over half of the book.
The orientation and perspective of this book remains numerical linear al-
gebra for applications in statistics. Computational considerations inform the
narrative. There is an emphasis on the areas of matrix analysis that are im-
portant for statisticians, and the kinds of matrices encountered in statistical
applications receive special attention.
This book is divided into three parts plus a set of appendices. The three
parts correspond generally to the three areas of the book’s subtitle — theory,
computations, and applications — although the parts are in a different order,
and there is no firm separation of the topics.
Part I, consisting of Chaps. 1 through 6, covers most of the material in
linear algebra needed by statisticians. (The word “matrix” in the title of the
present book may suggest a somewhat more limited domain than “linear alge-
bra”; but I use the former term only because it seems to be more commonly
used by statisticians and is used more or less synonymously with the latter
term.)
The first four chapters cover the basics of vectors and matrices, concen-
trating on topics that are particularly relevant for statistical applications. In
Chap. 7, it is assumed that the reader is generally familiar with the basics of
partial differentiation of scalar functions. Chapters 4 through 6 begin to take
on more of an applications flavor, as well as beginning to give more consid-
eration to computational methods. Although the details of the computations
XIV Preface to First Edition

are not covered in those chapters, the topics addressed are oriented more to-
ward computational algorithms. Chapter 4 covers methods for decomposing
matrices into useful factors.
Chapter 5 addresses applications of matrices in setting up and solving
linear systems, including overdetermined systems. We should not confuse sta-
tistical inference with fitting equations to data, although the latter task is a
component of the former activity. In Chap. 5, we address the more mechanical
aspects of the problem of fitting equations to data. Applications in statistical
data analysis are discussed in Chap. 9. In those applications, we need to make
statements (that is, assumptions) about relevant probability distributions.
Chapter 6 discusses methods for extracting eigenvalues and eigenvectors.
There are many important details of algorithms for eigenanalysis, but they are
beyond the scope of this book. As with other chapters in Part I, Chap. 6 makes
some reference to statistical applications, but it focuses on the mathematical
and mechanical aspects of the problem.
Although the first part is on “theory”, the presentation is informal; neither
definitions nor facts are highlighted by such words as “Definition”, “Theorem”,
“Lemma”, and so forth. It is assumed that the reader follows the natural
development. Most of the facts have simple proofs, and most proofs are given
naturally in the text. No “Proof” and “Q.E.D.” or “ ” appear to indicate
beginning and end; again, it is assumed that the reader is engaged in the
development. For example, on page 378:
If A is nonsingular and symmetric, then .A−1 is also symmetric because
−1 T
.(A ) = (AT )−1 = A−1 .
The first part of that sentence could have been stated as a theorem and
given a number, and the last part of the sentence could have been introduced
as the proof, with reference to some previous theorem that the inverse and
transposition operations can be interchanged. (This had already been shown
before page 378 — in an unnumbered theorem of course!)
None of the proofs are original (at least, I don’t think they are), but in most
cases I do not know the original source, or even the source where I first saw
them. I would guess that many go back to C. F. Gauss. Most, whether they
are as old as Gauss or not, have appeared somewhere in the work of C. R. Rao.
Some lengthier proofs are only given in outline, but references are given for
the details. Very useful sources of details of the proofs are Harville (1997),
especially for facts relating to applications in linear models, and Horn and
Johnson (1991) for more general topics, especially those relating to stochastic
matrices. The older books by Gantmacher (1959) provide extensive coverage
and often rather novel proofs. These two volumes have been brought back into
print by the American Mathematical Society.
I also sometimes make simple assumptions without stating them explicitly.
For example, I may write “for all i” when i is used as an index to a vector.
I hope it is clear that “for all i” means only “for i that correspond to indices
of the vector”. Also, my use of an expression generally implies existence. For
 Preface to First Edition XV

example, if “AB” is used to represent a matrix product, it implies that “A

and B are conformable for the multiplication AB”. Occasionally I remind the
reader that I am taking such shortcuts.
The material in Part I, as in the entire book, was built up recursively. In the
first pass, I began with some definitions and followed those with some facts
that are useful in applications. In the second pass, I went back and added
definitions and additional facts that lead to the results stated in the first
pass. The supporting material was added as close to the point where it was
needed as practical and as necessary to form a logical flow. Facts motivated by
additional applications were also included in the second pass. In subsequent
passes, I continued to add supporting material as necessary and to address
the linear algebra for additional areas of application. I sought a bare-bones
presentation that gets across what I considered to be the theory necessary for
most applications in the data sciences. The material chosen for inclusion is
motivated by applications.
Throughout the book, some attention is given to numerical methods for
computing the various quantities discussed. This is in keeping with my be-
lief that statistical computing should be dispersed throughout the statistics
curriculum and statistical literature generally. Thus, unlike in other books on
matrix “theory”, I describe the “modified” Gram-Schmidt method, rather than
just the “classical” GS. (I put “modified” and “classical” in quotes because, to
me, GS is MGS. History is interesting, but in computational matters, I do
not care to dwell on the methods of the past.) Also, condition numbers of
matrices are introduced in the “theory” part of the book, rather than just
in the “computational” part. Condition numbers also relate to fundamental
properties of the model and the data.
The difference between an expression and a computing method is em-
phasized. For example, often we may write the solution to the linear system
−1
.Ax = b as .A b. Although this is the solution (so long as A is square and of
full rank), solving the linear system does not involve computing .A−1 . We may
write .A−1 b, but we know we can compute the solution without inverting the
matrix.
“This is an instance of a principle that we will encounter repeatedly:
the form of a mathematical expression and the way the expression
should be evaluated in actual practice may be quite different.”
(The statement in quotes appears word for word in several places in the book.)
Standard textbooks on “matrices for statistical applications” emphasize
their uses in the analysis of traditional linear models. This is a large and im-
portant field in which real matrices are of interest, and the important kinds of
real matrices include symmetric, positive definite, projection, and generalized
inverse matrices. This area of application also motivates much of the discussion
in this book. In other areas of statistics, however, there are different matrices
of interest, including similarity and dissimilarity matrices, stochastic matri-
ces, rotation matrices, and matrices arising from graph-theoretic approaches
XVI Preface to First Edition

to data analysis. These matrices have applications in clustering, data mining,

stochastic processes, and graphics; therefore, I describe these matrices and
their special properties. I also discuss the geometry of matrix algebra. This
provides a better intuition of the operations. Homogeneous coordinates and
special operations in .IR3 are covered because of their geometrical applications
in statistical graphics.
Part II addresses selected applications in data analysis. Applications are
referred to frequently in Part I, and of course, the choice of topics for coverage
was motivated by applications. The difference in Part II is in its orientation.
Only “selected” applications in data analysis are addressed; there are appli-
cations of matrix algebra in almost all areas of statistics, including the theory
of estimation, which is touched upon in Chap. 7 of Part I. Certain types of
matrices are more common in statistics, and Chap. 8 discusses in more detail
some of the important types of matrices that arise in data analysis and sta-
tistical modeling. Chapter 9 addresses selected applications in data analysis.
The material of Chap. 9 has no obvious definition that could be covered in a
single chapter (or a single part, or even a single book), so I have chosen to
discuss briefly a wide range of areas. Most of the sections and even subsections
of Chap. 9 are on topics to which entire books are devoted; however, I do not
believe that any single book addresses all of them.
Part III covers some of the important details of numerical computations,
with an emphasis on those for linear algebra. I believe these topics constitute
the most important material for an introductory course in numerical analysis
for statisticians and should be covered in every such course.
Except for specific computational techniques for optimization, random
number generation, and perhaps symbolic computation, Part III provides the
basic material for a course in statistical computing. All statisticians should
have a passing familiarity with the principles.
Chapter 10 provides some basic information on how data are stored and
manipulated in a computer. Some of this material is rather tedious, but it
is important to have a general understanding of computer arithmetic before
considering computations for linear algebra. Some readers may skip or just
skim Chap. 10, but the reader should be aware that the way the computer
stores numbers and performs computations has far-reaching consequences.
Computer arithmetic differs from ordinary arithmetic in many ways; for ex-
ample, computer arithmetic lacks associativity of addition and multiplication,
and series often converge even when they E∞are not supposed to. (On the com-
puter, a straightforward evaluation of . x=1 x converges!)
I emphasize the differences between the abstract number system .IR, called
the reals, and the computer number system .IF, the floating-point numbers
unfortunately also often called “real”. Table 10.3 on page 557 summarizes some
of these differences. All statisticians should be aware of the effects of these
differences. I also discuss the differences between .ZZ, the abstract number
system called the integers, and the computer number system .II, the fixed-
 Preface to First Edition XVII

point numbers. (Appendix A provides definitions for this and other notation
that I use.)
Chapter 10 also covers some of the fundamentals of algorithms, such as
iterations, recursion, and convergence. It also discusses software development.
Software issues are revisited in Chap. 12.
While Chap. 10 deals with general issues in numerical analysis, Chap. 11
addresses specific issues in numerical methods for computations in linear al-
gebra.
Chapter 12 provides a brief introduction to software available for com-
putations with linear systems. Some specific systems mentioned include the
IMSL™ libraries for Fortran and C, Octave or MATLAB® (or MATLAB®),
and R or S-PLUS® (or S-Plus®). All of these systems are easy to use, and
the best way to learn them is to begin using them for simple problems. I do
not use any particular software system in the book, but in some exercises, and
particularly in Part III, I do assume the ability to program in either Fortran or
C and the availability of either R or S-Plus, Octave or MATLAB, and Maple®
or Mathematica®. My own preferences for software systems are Fortran and
R, and occasionally these preferences manifest themselves in the text.
Appendix A collects the notation used in this book. It is generally “stan-
dard” notation, but one thing the reader must become accustomed to is the
lack of notational distinction between a vector and a scalar. All vectors are
“column” vectors, although I usually write them as horizontal lists of their
elements. (Whether vectors are “row” vectors or “column” vectors is generally
only relevant for how we write expressions involving vector/matrix multipli-
cation or partitions of matrices.)
I write algorithms in various ways, sometimes in a form that looks similar
to Fortran or C and sometimes as a list of numbered steps. I believe all of the
descriptions used are straightforward and unambiguous.
This book could serve as a basic reference either for courses in statistical
computing or for courses in linear models or multivariate analysis. When the
book is used as a reference, rather than looking for “Definition” or “Theo-
rem”, the user should look for items set off with bullets or look for numbered
equations, or else should use the Indexor Appendix A, beginning on page 653.
The prerequisites for this text are minimal. Obviously some background in
mathematics is necessary. Some background in statistics or data analysis and
some level of scientific computer literacy are also required. References to rather
advanced mathematical topics are made in a number of places in the text. To
some extent this is because many sections evolved from class notes that I
developed for various courses that I have taught. All of these courses were at
the graduate level in the computational and statistical sciences, but they have
had wide ranges in mathematical level. I have carefully reread the sections
that refer to groups, fields, measure theory, and so on, and am convinced that
if the reader does not know much about these topics, the material is still
understandable, but if the reader is familiar with these topics, the references
add to that reader’s appreciation of the material. In many places, I refer to
XVIII Preface to First Edition

computer programming, and some of the exercises require some programming.

A careful coverage of Part III requires background in numerical programming.
In regard to the use of the book as a text, most of the book evolved in one
way or another for my own use in the classroom. I must quickly admit, how-
ever, that I have never used this whole book as a text for any single course. I
have used Part III in the form of printed notes as the primary text for a course
in the “foundations of computational science” taken by graduate students in
the natural sciences (including a few statistics students, but dominated by
physics students). I have provided several sections from Parts I and II in online
PDF files as supplementary material for a two-semester course in mathemati-
cal statistics at the “baby measure theory” level (using Shao, 2003). Likewise,
for my courses in computational statistics and statistical visualization, I have
provided many sections, either as supplementary material or as the primary
text, in online PDF files or printed notes. I have not taught a regular “applied
statistics” course in almost 30 years, but if I did, I am sure that I would draw
heavily from Parts I and II for courses in regression or multivariate analy-
sis. If I ever taught a course in “matrices for statistics” (I don’t even know if
such courses exist), this book would be my primary text because I think it
covers most of the things statisticians need to know about matrix theory and
computations.
Some exercises are Monte Carlo studies. I do not discuss Monte Carlo
methods in this text, so the reader lacking background in that area may need
to consult another reference in order to work those exercises. The exercises
should be considered an integral part of the book. For some exercises, the re-
quired software can be obtained from netlib. Exercises in any of the chapters,
not just in Part III, may require computations or computer programming.
Penultimately, I must make some statement about the relationship of
this book to some other books on similar topics. Much important statisti-
cal theory and many methods make use of matrix theory, and many statis-
ticians have contributed to the advancement of matrix theory from its very
early days. Widely used books with derivatives of the words “statistics” and
“matrices/linear-algebra” in their titles include Basilevsky (1983), Graybill
(1983), Harville (1997), Schott (2004), and Searle (1982). All of these are use-
ful books. The computational orientation of this book is probably the main
difference between it and these other books. Also, some of these other books
only address topics of use in linear models, whereas this book also discusses
matrices useful in graph theory, stochastic processes, and other areas of appli-
cation. (If the applications are only in linear models, most matrices of interest
are symmetric, and all eigenvalues can be considered to be real.) Other dif-
ferences among all of these books, of course, involve the authors’ choices of
secondary topics and the ordering of the presentation.
 Preface to First Edition XIX

Acknowledgments

I thank John Kimmel of Springer for his encouragement and advice on this
book and other books on which he has worked with me. I especially thank
Ken Berk for his extensive and insightful comments on a draft of this book.
I thank my student Li Li for reading through various drafts of some of the
chapters and pointing out typos or making helpful suggestions. I thank the
anonymous reviewers of this edition for their comments and suggestions. I also
thank the many readers of my previous book on numerical linear algebra who
informed me of errors and who otherwise provided comments or suggestions
for improving the exposition. Whatever strengths this book may have can be
attributed in large part to these people, named or otherwise. The weaknesses
can only be attributed to my own ignorance or hardheadedness.
I thank my wife, María, to whom this book is dedicated, for everything.

I used TEX via LATEX 2ε to write the book. I did all of the typing, program-
ming, etc., myself, so all misteaks are mine. I would appreciate receiving sug-
gestions for improvement and notification of errors.

Fairfax County, VA, USA James E. Gentle

June 12, 2007
Contents

Preface to Third Edition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VII

Preface to Second Edition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XI

Preface to First Edition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XIII

1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Arrays. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Representation of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 What You Compute and What You Don’t. . . . . . . . . . . . . . . . . . . 7
1.6 R. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6.1 R Data Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.2 Program Control Statements. . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6.3 Packages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6.4 User Functions and Operators. . . . . . . . . . . . . . . . . . . . . . . 11
1.6.5 Generating Artificial Data. . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.6.6 Graphics Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.6.7 Special Data in R. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.6.8 Determining Properties of a Computer System in R. . . . . 14
1.6.9 Documentation: Finding R Functions and Packages. . . . . 15
1.6.10 Documentation: R Functions, Packages, and Other
Objects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6.11 The Design of the R Programming Language. . . . . . . . . . 16
1.6.12 Why I Use R in This Book. . . . . . . . . . . . . . . . . . . . . . . . . . 16
XXII Contents

Part I Linear Algebra

2 Vectors and Vector Spaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.1 Operations on Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.1.1 Linear Combinations and Linear Independence. . . . . . . . . 20
2.1.2 Vector Spaces and Spaces of Vectors. . . . . . . . . . . . . . . . . . 22
2.1.3 Basis Sets for Vector Spaces. . . . . . . . . . . . . . . . . . . . . . . . . 31
2.1.4 Inner Products. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.1.5 Norms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.1.6 Normalized Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.1.7 Metrics and Distances. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.1.8 Orthogonal Vectors and Orthogonal Vector Spaces. . . . . . 43
2.1.9 The “One Vector”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2 Cartesian Coordinates and Geometrical Properties of Vectors. . . 45
2.2.1 Cartesian Geometry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.2.2 Projections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.2.3 Angles Between Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.2.4 Orthogonalization Transformations: Gram–Schmidt. . . . . 48
2.2.5 Orthonormal Basis Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.2.6 Approximation of Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.2.7 Flats, Affine Spaces, and Hyperplanes. . . . . . . . . . . . . . . . . 53
2.2.8 Cones. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.2.9 Vector Cross Products in IR3 . . . . . . . . . . . . . . . . . . . . . . . . 56
2.3 Centered Vectors, and Variances and Covariances of Vectors. . . 58
2.3.1 The Mean and Centered Vectors. . . . . . . . . . . . . . . . . . . . . 59
2.3.2 The Standard Deviation, the Variance, and Scaled
Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.3.3 Covariances and Correlations Between Vectors. . . . . . . . . 61
Appendix: Vectors in R. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

3 Basic Properties of Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

3.1 Basic Definitions and Notation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.1.1 Multiplication of a Matrix by a Scalar. . . . . . . . . . . . . . . . 74
3.1.2 Symmetric and Hermitian Matrices. . . . . . . . . . . . . . . . . . . 74
3.1.3 Diagonal Elements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.1.4 Diagonally Dominant Matrices. . . . . . . . . . . . . . . . . . . . . . . 75
3.1.5 Diagonal and Hollow Matrices. . . . . . . . . . . . . . . . . . . . . . . 75
3.1.6 Matrices with Other Special Patterns of Zeroes. . . . . . . . . 75
3.1.7 Matrix Shaping Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.1.8 Partitioned Matrices and Submatrices. . . . . . . . . . . . . . . . . 80
3.1.9 Matrix Addition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.1.10 The Trace of a Square Matrix. . . . . . . . . . . . . . . . . . . . . . . . 84
 Contents XXIII

3.2 The Determinant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

3.2.1 Definition and Simple Properties. . . . . . . . . . . . . . . . . . . . . 85
3.2.2 Determinants of Various Types of Square Matrices. . . . . . 87
3.2.3 Minors, Cofactors, and Adjugate Matrices. . . . . . . . . . . . . 89
3.2.4 A Geometrical Perspective of the Determinant. . . . . . . . . 94
3.3 Multiplication of Matrices and Multiplication of
Vectors and Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.3.1 Matrix Multiplication (Cayley). . . . . . . . . . . . . . . . . . . . . . . 96
3.3.2 Cayley Multiplication of Matrices with Special
Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
3.3.3 Elementary Operations on Matrices. . . . . . . . . . . . . . . . . . . 102
3.3.4 The Trace of a Cayley Product that Is Square. . . . . . . . . . 110
3.3.5 The Determinant of a Cayley Product of Square
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
3.3.6 Outer Products of Vectors. . . . . . . . . . . . . . . . . . . . . . . . . . . 112
3.3.7 Bilinear and Quadratic Forms: Definiteness. . . . . . . . . . . . 112
3.3.8 Anisometric Spaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
3.3.9 The Hadamard Product. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
3.3.10 The Kronecker Product. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
3.3.11 The Inner Product of Matrices. . . . . . . . . . . . . . . . . . . . . . . 118
3.4 Matrix Rank and the Inverse of a Matrix. . . . . . . . . . . . . . . . . . . . 121
3.4.1 Row Rank and Column Rank. . . . . . . . . . . . . . . . . . . . . . . . 121
3.4.2 Full Rank Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
3.4.3 Rank of Elementary Operator Matrices and Matrix
Products Involving Them. . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
3.4.4 The Rank of Partitioned Matrices, Products of
Matrices, and Sums of Matrices. . . . . . . . . . . . . . . . . . . . . . 123
3.4.5 Full Rank Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
3.4.6 Full Rank Matrices and Matrix Inverses. . . . . . . . . . . . . . . 126
3.4.7 Matrix Inverses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
3.4.8 Full Rank Factorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
3.4.9 Multiplication by Full Rank Matrices. . . . . . . . . . . . . . . . . 132
3.4.10 Nonfull Rank and Equivalent Matrices. . . . . . . . . . . . . . . . 134
3.4.11 Gramian Matrices: Products of the Form AT A. . . . . . . . . 137
3.4.12 A Lower Bound on the Rank of a Matrix Product. . . . . . 138
3.4.13 Determinants of Inverses. . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
3.4.14 Inverses of Products and Sums of Nonsingular
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
3.4.15 Inverses of Matrices with Special Forms. . . . . . . . . . . . . . . 142
3.4.16 Determining the Rank of a Matrix. . . . . . . . . . . . . . . . . . . . 143
3.5 The Schur Complement in Partitioned Square Matrices. . . . . . . . 143
3.5.1 Inverses of Partitioned Matrices. . . . . . . . . . . . . . . . . . . . . . 144
3.5.2 Determinants of Partitioned Matrices. . . . . . . . . . . . . . . . . 144
3.6 Linear Systems of Equations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
3.6.1 Solutions of Linear Systems. . . . . . . . . . . . . . . . . . . . . . . . . . 145
XXIV Contents

3.6.2 Null Space: The Orthogonal Complement. . . . . . . . . . . . . . 147

3.6.3 Orthonormal Completion. . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
3.7 Generalized Inverses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
3.7.1 Immediate Properties of Generalized Inverses. . . . . . . . . . 148
3.7.2 The Moore–Penrose Inverse. . . . . . . . . . . . . . . . . . . . . . . . . . 151
3.7.3 Generalized Inverses of Products and Sums of
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
3.7.4 Generalized Inverses of Partitioned Matrices. . . . . . . . . . . 153
3.8 Orthogonality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
3.8.1 Orthogonal Matrices: Definition and Simple
Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
3.8.2 Unitary Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
3.8.3 Orthogonal and Orthonormal Columns. . . . . . . . . . . . . . . . 155
3.8.4 The Orthogonal Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
3.8.5 Conjugacy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
3.9 Eigenanalysis: Canonical Factorizations. . . . . . . . . . . . . . . . . . . . . 156
3.9.1 Eigenvalues and Eigenvectors Are Remarkable. . . . . . . . . 157
3.9.2 Basic Properties of Eigenvalues and Eigenvectors. . . . . . . 158
3.9.3 The Characteristic Polynomial. . . . . . . . . . . . . . . . . . . . . . . 160
3.9.4 The Spectrum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
3.9.5 Similarity Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . 168
3.9.6 Schur Factorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
3.9.7 Similar Canonical Factorization: Diagonalizable
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
3.9.8 Properties of Diagonalizable Matrices. . . . . . . . . . . . . . . . . 174
3.9.9 Eigenanalysis of Symmetric Matrices. . . . . . . . . . . . . . . . . . 176
3.9.10 Generalized Eigenvalues and Eigenvectors. . . . . . . . . . . . . 181
3.9.11 Singular Values and the Singular Value
Decomposition (SVD). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
3.10 Positive Definite and Nonnegative Definite Matrices. . . . . . . . . . . 185
3.10.1 Eigenvalues of Positive and Nonnegative Definite
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
3.10.2 Inverse of Positive Definite Matrices. . . . . . . . . . . . . . . . . . 186
3.10.3 Diagonalization of Positive Definite Matrices. . . . . . . . . . . 186
3.10.4 Square Roots of Positive and Nonnegative Definite
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
3.11 Matrix Norms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
3.11.1 Matrix Norms Induced from Vector Norms. . . . . . . . . . . . 188
3.11.2 The Frobenius Norm—The “Usual” Norm. . . . . . . . . . . . . 191
3.11.3 Other Matrix Norms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
3.11.4 Matrix Norm Inequalities. . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
3.11.5 The Spectral Radius. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
3.11.6 Convergence of a Matrix Power Series. . . . . . . . . . . . . . . . . 195
3.12 Approximation of Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
3.12.1 Measures of the Difference between Two Matrices. . . . . . 199
 Contents XXV

3.12.2 Best Approximation with a Matrix of Given Rank. . . . . . 199

Appendix: Matrices in R. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

4 Matrix Transformations and Factorizations. . . . . . . . . . . . . . . . . 217

4.1 Linear Geometric Transformations. . . . . . . . . . . . . . . . . . . . . . . . . . 219
4.1.1 Invariance Properties of Linear Transformations. . . . . . . . 219
4.1.2 Transformations by Orthogonal Matrices. . . . . . . . . . . . . . 221
4.1.3 Rotations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
4.1.4 Reflections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
4.1.5 Translations; Homogeneous Coordinates. . . . . . . . . . . . . . . 224
4.2 Householder Transformations (Reflections). . . . . . . . . . . . . . . . . . . 225
4.2.1 Zeroing All Elements but One in a Vector. . . . . . . . . . . . . 226
4.2.2 Computational Considerations. . . . . . . . . . . . . . . . . . . . . . . 227
4.3 Givens Transformations (Rotations). . . . . . . . . . . . . . . . . . . . . . . . 228
4.3.1 Zeroing One Element in a Vector. . . . . . . . . . . . . . . . . . . . . 229
4.3.2 Givens Rotations That Preserve Symmetry. . . . . . . . . . . . 230
4.3.3 Givens Rotations to Transform to Other Values. . . . . . . . 230
4.3.4 Fast Givens Rotations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
4.4 Factorization of Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
4.5 LU and LDU Factorizations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
4.5.1 Properties: Existence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
4.5.2 Pivoting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
4.5.3 Use of Inner Products. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
4.5.4 Properties: Uniqueness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
4.5.5 Properties of the LDU Factorization of a Square
Matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
4.6 QR Factorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
4.6.1 Related Matrix Factorizations. . . . . . . . . . . . . . . . . . . . . . . . 239
4.6.2 Matrices of Full Column Rank. . . . . . . . . . . . . . . . . . . . . . . 239
4.6.3 Nonfull Rank Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
4.6.4 Determining the Rank of a Matrix. . . . . . . . . . . . . . . . . . . . 241
4.6.5 Formation of the QR Factorization. . . . . . . . . . . . . . . . . . . 242
4.6.6 Householder Reflections to Form the QR Factorization. . .242
4.6.7 Givens Rotations to Form the QR Factorization. . . . . . . . 243
4.6.8 Gram-Schmidt Transformations to Form the
QR Factorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
4.7 Factorizations of Nonnegative Definite Matrices. . . . . . . . . . . . . . 244
4.7.1 Square Roots. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
4.7.2 Cholesky Factorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
4.7.3 Factorizations of a Gramian Matrix. . . . . . . . . . . . . . . . . . . 248
4.8 Approximate Matrix Factorization. . . . . . . . . . . . . . . . . . . . . . . . . . 249
4.8.1 Nonnegative Matrix Factorization. . . . . . . . . . . . . . . . . . . . 249
XXVI Contents

4.8.2 Incomplete Factorizations. . . . . . . . . . . . . . . . . . . . . . . . . . . 250

Appendix: R Functions for Matrix Computations and for Graphics. . .251
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254

5 Solution of Linear Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

5.1 Condition of Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
5.1.1 Condition Number. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
5.1.2 Improving the Condition Number. . . . . . . . . . . . . . . . . . . . 266
5.1.3 Numerical Accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
5.2 Direct Methods for Consistent Systems. . . . . . . . . . . . . . . . . . . . . . 268
5.2.1 Gaussian Elimination and Matrix Factorizations. . . . . . . . 268
5.2.2 Choice of Direct Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
5.3 Iterative Methods for Consistent Systems. . . . . . . . . . . . . . . . . . . . 273
5.3.1 The Gauss-Seidel Method with Successive
Overrelaxation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
5.3.2 Conjugate Gradient Methods for Symmetric
Positive Definite Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
5.3.3 Multigrid Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
5.4 Iterative Refinement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
5.5 Updating a Solution to a Consistent System. . . . . . . . . . . . . . . . . 280
5.6 Overdetermined Systems: Least Squares. . . . . . . . . . . . . . . . . . . . . 282
5.6.1 Least Squares Solution of an Overdetermined System. . . 284
5.6.2 Least Squares with a Full Rank Coefficient Matrix. . . . . . 286
5.6.3 Least Squares with a Coefficient Matrix Not of Full
Rank. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
5.6.4 Weighted Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
5.6.5 Updating a Least Squares Solution of an
Overdetermined System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
5.7 Other Solutions of Overdetermined Systems. . . . . . . . . . . . . . . . . . 290
5.7.1 Solutions That Minimize Other Norms of the
Residuals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
5.7.2 Regularized Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
5.7.3 Other Restrictions on the Solutions. . . . . . . . . . . . . . . . . . . 295
5.7.4 Minimizing Orthogonal Distances. . . . . . . . . . . . . . . . . . . . . 296
Appendix: R Functions for Solving Linear Systems. . . . . . . . . . . . . . . . 299
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300

6 Evaluation of Eigenvalues and Eigenvectors. . . . . . . . . . . . . . . . . 303

6.1 General Computational Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . 304
6.1.1 Numerical Condition of an Eigenvalue Problem. . . . . . . . . 304
6.1.2 Eigenvalues from Eigenvectors and Vice Versa. . . . . . . . . . 306
6.1.3 Deflation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
6.1.4 Preconditioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
6.1.5 Shifting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
6.2 Power Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
 Contents XXVII

6.3 Jacobi Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

6.4 QR Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
6.5 Krylov Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
6.6 Generalized Eigenvalues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
6.7 Singular Value Decomposition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320

7 Real Analysis and Probability Distributions of Vectors

and Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
7.1 Basics of Differentiation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
7.1.1 Continuity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
7.1.2 Notation and Properties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
7.1.3 Differentials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
7.1.4 Use of Differentiation in Optimization. . . . . . . . . . . . . . . . . 328
7.2 Types of Differentiation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
7.2.1 Differentiation with Respect to a Scalar. . . . . . . . . . . . . . . 329
7.2.2 Differentiation with Respect to a Vector. . . . . . . . . . . . . . . 330
7.2.3 Differentiation with Respect to a Matrix. . . . . . . . . . . . . . 335
7.3 Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
7.3.1 Multidimensional Integrals and Integrals Involving
Vectors and Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 338
7.3.2 Change of Variables; The Jacobian. . . . . . . . . . . . . . . . . . . 338
7.3.3 Integration Combined with Other Operations. . . . . . . . . . 339
7.4 Multivariate Probability Theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . 340
7.4.1 Random Variables and Probability Distributions. . . . . . . 340
7.4.2 Distributions of Transformations of Random Variables. . .344
7.4.3 The Multivariate Normal Distribution. . . . . . . . . . . . . . . . . 346
7.4.4 Distributions Derived from the Multivariate Normal. . . . 350
7.4.5 Chi-Squared Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . 350
7.4.6 Wishart Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
7.5 Multivariate Random Number Generation. . . . . . . . . . . . . . . . . . . 355
7.5.1 The Multivariate Normal Distribution. . . . . . . . . . . . . . . . . 355
7.5.2 Random Correlation Matrices. . . . . . . . . . . . . . . . . . . . . . . . 356
Appendix: R for Working with Probability Distributions and for
Simulating Random Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 360
XXVIII Contents

Part II Applications in Statistics and Data Science

8 Special Matrices and Operations Useful in Modeling and

Data Science. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
8.1 Data Matrices and Association Matrices. . . . . . . . . . . . . . . . . . . . . 368
8.1.1 Flat Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
8.1.2 Graphs and Other Data Structures. . . . . . . . . . . . . . . . . . . 369
8.1.3 Term-by-Document Matrices. . . . . . . . . . . . . . . . . . . . . . . . . 376
8.1.4 Sparse Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
8.1.5 Probability Distribution Models. . . . . . . . . . . . . . . . . . . . . . 377
8.1.6 Derived Association Matrices. . . . . . . . . . . . . . . . . . . . . . . . 377
8.2 Symmetric Matrices and Other Unitarily Diagonalizable
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378
8.2.1 Some Important Properties of Symmetric Matrices. . . . . 378
8.2.2 Approximation of Symmetric Matrices and an
Important Inequality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
8.2.3 Normal Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383
8.3 Nonnegative Definite Matrices: Cholesky Factorization. . . . . . . . 384
8.3.1 Eigenvalues of Nonnegative Definite Matrices. . . . . . . . . . 385
8.3.2 The Square Root and the Cholesky Factorization. . . . . . . 386
8.3.3 The Convex Cone of Nonnegative Definite Matrices. . . . . 386
8.4 Positive Definite Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
8.4.1 Leading Principal Submatrices of Positive Definite
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
8.4.2 The Convex Cone of Positive Definite Matrices. . . . . . . . . 389
8.4.3 Inequalities Involving Positive Definite Matrices. . . . . . . . 389
8.5 Idempotent and Projection Matrices. . . . . . . . . . . . . . . . . . . . . . . . 391
8.5.1 Idempotent Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
8.5.2 Projection Matrices: Symmetric Idempotent Matrices. . . 396
8.6 Special Matrices Occurring in Data Analysis. . . . . . . . . . . . . . . . . 397
8.6.1 Gramian Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398
8.6.2 Projection and Smoothing Matrices. . . . . . . . . . . . . . . . . . . 399
8.6.3 Centered Matrices and Variance-Covariance Matrices. . . 403
8.6.4 The Generalized Variance. . . . . . . . . . . . . . . . . . . . . . . . . . . 406
8.6.5 Similarity Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
8.6.6 Dissimilarity Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
8.7 Nonnegative and Positive Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . 409
8.7.1 Properties of Square Positive Matrices. . . . . . . . . . . . . . . . 411
8.7.2 Irreducible Square Nonnegative Matrices. . . . . . . . . . . . . . 412
8.7.3 Stochastic Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
8.7.4 Leslie Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
8.8 Other Matrices with Special Structures. . . . . . . . . . . . . . . . . . . . . . 418
8.8.1 Helmert Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
 Contents XXIX

8.8.2 Vandermonde Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419

8.8.3 Hadamard Matrices and Orthogonal Arrays. . . . . . . . . . . . 420
8.8.4 Toeplitz Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
8.8.5 Circulant Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
8.8.6 Fourier Matrices and the Discrete Fourier Transform. . . . 424
8.8.7 Hankel Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
8.8.8 Cauchy Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 428
8.8.9 Matrices Useful in Graph Theory. . . . . . . . . . . . . . . . . . . . . 429
8.8.10 Z-Matrices and M-Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . 432
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433

9 Selected Applications in Statistics . . . . . . . . . . . . . . . . . . . . . . . . . 437

9.1 Linear Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
9.1.1 Fitting the Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
9.1.2 Least Squares Fit of Full-Rank Models. . . . . . . . . . . . . . . . 443
9.1.3 Least Squares Fits of Nonfull-Rank Models. . . . . . . . . . . . 444
9.1.4 Computing the Solution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447
9.1.5 Properties of a Least Squares Fit. . . . . . . . . . . . . . . . . . . . . 449
9.1.6 Linear Least Squares Subject to Linear Equality
Constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451
9.1.7 Weighted Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
9.1.8 Updating Linear Regression Statistics. . . . . . . . . . . . . . . . . 453
9.1.9 Linear Smoothing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
9.1.10 Multivariate Linear Models. . . . . . . . . . . . . . . . . . . . . . . . . . 456
9.2 Statistical Inference in Linear Models. . . . . . . . . . . . . . . . . . . . . . . 458
9.2.1 The Probability Distribution of ε. . . . . . . . . . . . . . . . . . . . . 459
9.2.2 Estimability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
9.2.3 The Gauss-Markov Theorem. . . . . . . . . . . . . . . . . . . . . . . . . 465
9.2.4 Testing Linear Hypotheses. . . . . . . . . . . . . . . . . . . . . . . . . . . 466
9.2.5 Statistical Inference in Linear Models with
Heteroscedastic or Correlated Errors. . . . . . . . . . . . . . . . . . 467
9.2.6 Statistical Inference for Multivariate Linear Models. . . . . 468
9.3 Principal Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
9.3.1 Principal Components of a Random Vector. . . . . . . . . . . . 470
9.3.2 Principal Components of Data. . . . . . . . . . . . . . . . . . . . . . . 471
9.4 Condition of Models and Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
9.4.1 Ill-Conditioning in Statistical Applications. . . . . . . . . . . . . 474
9.4.2 Variable Selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
9.4.3 Principal Components Regression. . . . . . . . . . . . . . . . . . . . 476
9.4.4 Shrinkage Estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 477
9.4.5 Statistical Inference About the Rank of a Matrix. . . . . . . 479
9.4.6 Incomplete Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483
9.5 Stochastic Processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
9.5.1 Markov Chains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487
9.5.2 Markovian Population Models. . . . . . . . . . . . . . . . . . . . . . . . 489
9.5.3 Autoregressive Processes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
XXX Contents

9.6 Optimization of Scalar-Valued Functions. . . . . . . . . . . . . . . . . . . . 494

9.6.1 Stationary Points of Functions. . . . . . . . . . . . . . . . . . . . . . . 495
9.6.2 Newton’s Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
9.6.3 Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497
9.6.4 Maximum Likelihood. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
9.6.5 Optimization of Functions with Constraints. . . . . . . . . . . . 503
9.6.6 Optimization Without Differentiation. . . . . . . . . . . . . . . . . 509
Appendix: R for Applications in Statistics. . . . . . . . . . . . . . . . . . . . . . . . 510
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517

Part III Numerical Methods and Software

10 Numerical Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527

10.1 Software Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
10.1.1 Standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
10.1.2 Coding Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
10.1.3 Types of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
10.1.4 Missing Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 530
10.1.5 Data Structures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531
10.1.6 Computer Architectures and File Systems. . . . . . . . . . . . . 531
10.2 Digital Representation of Numeric Data. . . . . . . . . . . . . . . . . . . . . 532
10.2.1 The Fixed-Point Number System. . . . . . . . . . . . . . . . . . . . . 533
10.2.2 The Floating-Point Model for Real Numbers. . . . . . . . . . . 535
10.2.3 Language Constructs for Representing Numeric Data. . . 542
10.2.4 Other Variations in the Representation of Data:
Portability of Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
10.3 Computer Operations on Numeric Data. . . . . . . . . . . . . . . . . . . . . 549
10.3.1 Fixed-Point Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550
10.3.2 Floating-Point Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . 551
10.3.3 Language Constructs for Operations on Numeric Data. . .556
10.3.4 Software Methods for Extending the Precision. . . . . . . . . 558
10.3.5 Exact Computations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
10.4 Numerical Algorithms and Analysis. . . . . . . . . . . . . . . . . . . . . . . . . 561
10.4.1 Error in Numerical Computations. . . . . . . . . . . . . . . . . . . . 562
10.4.2 Efficiency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
10.4.3 Iterations and Convergence. . . . . . . . . . . . . . . . . . . . . . . . . . 576
10.4.4 Computations Without Storing Data. . . . . . . . . . . . . . . . . . 578
10.4.5 Other Computational Techniques. . . . . . . . . . . . . . . . . . . . . 579
Appendix: Numerical Computations in R. . . . . . . . . . . . . . . . . . . . . . . . 582
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583
 Contents XXXI

11 Numerical Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 591

11.1 Computer Storage of Vectors and Matrices. . . . . . . . . . . . . . . . . . . 591
11.1.1 Storage Modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
11.1.2 Strides. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
11.1.3 Sparsity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 592
11.2 General Computational Considerations for Vectors and
Matrices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593
11.2.1 Relative Magnitudes of Operands. . . . . . . . . . . . . . . . . . . . . 593
11.2.2 Iterative Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595
11.2.3 Assessing Computational Errors. . . . . . . . . . . . . . . . . . . . . . 596
11.3 Multiplication of Vectors and Matrices. . . . . . . . . . . . . . . . . . . . . . 597
11.3.1 Strassen’s Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599
11.3.2 Matrix Multiplication Using MapReduce. . . . . . . . . . . . . . 601
11.4 Other Matrix Computations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 601
11.4.1 Rank Determination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602
11.4.2 Computing the Determinant. . . . . . . . . . . . . . . . . . . . . . . . . 603
11.4.3 Computing the Condition Number. . . . . . . . . . . . . . . . . . . . 603
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 604

12 Software for Numerical Linear Algebra. . . . . . . . . . . . . . . . . . . . . 607

12.1 General Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 608
12.1.1 Software Development and Open-Source Software. . . . . . 608
12.1.2 Integrated Development, Collaborative Research,
and Version Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
12.1.3 Finding Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
12.1.4 Software Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 610
12.1.5 Software Development, Maintenance, and Testing. . . . . . . 619
12.1.6 Reproducible Research. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622
12.2 Software Libraries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624
12.2.1 BLAS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624
12.2.2 Level 2 and Level 3 BLAS, LAPACK, and Related
Libraries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 627
12.2.3 Libraries for High-Performance Computing. . . . . . . . . . . . 628
12.2.4 The IMSL Libraries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 632
12.3 General-Purpose Languages and Programming Systems. . . . . . . 634
12.3.1 Programming Considerations. . . . . . . . . . . . . . . . . . . . . . . . 636
12.3.2 Modern Fortran. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 638
12.3.3 C and C++. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 640
12.3.4 Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 641
12.3.5 MATLAB and Octave. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 642
Appendix: R Software for Numerical Linear Algebra. . . . . . . . . . . . . . . 644
Exercises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 647
Random documents with unrelated
content Scribd suggests to you:
bound to take it for granted that the King considered arms which
were unlawfully directed against his people as directed against his
own throne.
The remarks of Mr. Hallam on the bill of attainder, though, as
usual, weighty and acute, do not perfectly satisfy us. He defends the
principle, but objects to the severity of the punishment. That, on
great emergencies, the State may justifiably pass a retrospective act
against an offender, we have no doubt whatever. We are acquainted
with only one argument on the other side, which lias in it enough of
reason to hear an answer. Warning, it is said, is the end of
punishment. But a punishment inflicted, not by a general rule, but
by an arbitrary discretion, cannot serve the purpose of a warning. It
is therefore useless; and useless pain ought not to be inflicted. This
sophism has found its way into several books on penal legislation. It
admits, however, of a very simple refutation. In the first place,
punishments ex post facto are not altogether useless even as
warnings. They are warnings to a particular class which stand in
great need of warnings, to favourites and ministers. They remind
persons of this description that there may be a day of reckoning for
those who ruin and enslave their country in all the forms of law. But
this is not all. Warning is, in ordinary cases, the principal end of
punishment; but it is not the only end. To remove the offender, to
preserve society from those dangers which are to be apprehended
from his incorrigible depravity is often one of the ends. In the case
of such a knave as Wild, or such a ruffian as Thurtell, it is a very
important end. In the case of a powerful and wicked statesman, it is
infinitely more important; so important, as alone to justify the
utmost severity, even though it were certain that his fate would not
deter others from imitating his example. At present, indeed, we
should think it extremely pernicious to take such a course, even with
a worse minister than Strafford, if a worse could exist; for, at
present, Parliament has only to withhold its support from a Cabinet
to produce an immediate change of hands. The case was widely
different in the reign of Charles the First. That Prince had governed
during eleven years without any Parliament; and, even when
Parliament was sitting, had supported Buckingham against its most
violent remonstrances. Mr. Hallam is of opinion that a bill of pains
and penalties ought to have been passed; but he draws a distinction
less just, we think, than his distinctions usually are. His opinion, so
far as we can collect it, is this, that there are almost insurmountable
objections to retrospective laws for capital punishment, but that,
where the punishment stops short of death, the objections are
comparatively trifling. Now the practice of taking the severity of the
penalty into consideration, when the question is about the mode of
procedure and the rules of evidence, is no doubt sufficiently
common. We often see a man convicted of a simple larceny on
evidence on which he would not be convicted of a burglary. It
sometimes happens that a jury, when there is strong suspicion, but
not absolute demonstration, that an act, unquestionably amounting
to murder, was committed by the prisoner before them, will find him
guilty of manslaughter. But this is surely very irrational. The rules of
evidence no more depend on the magnitude of the interests at stake
than the rules of arithmetic. We might as well say that we have a
greater chance of throwing a size when we are playing for a penny
than when we are playing for a thousand pounds, as that a form of
trial which is sufficient for the purposes of justice, in a matter
affecting liberty and property, is insufficient in a matter affecting life.
Nay, if a mode of proceeding be too lax for capital cases, it is, à
fortiori, too lax for all others; for, in capital cases, the principles of
human nature will always afford considerable security. No judge is so
cruel as he who indemnifies himself for scrupulosity in cases of
blood, by license in affairs of smaller importance. The difference in
tale on the one side far more than makes up for the difference in
weight on the other. If there be any universal objection to
retrospective punishment, there is no more to be said. But such is
not the opinion of Mr. Hallam. He approves of the mode of
proceeding. He thinks that a punishment, not previously affixed by
law to the offences of Strafford, should have been inflicted; that
Strafford should have been, by act of Parliament, degraded from his
rank, and condemned to perpetual banishment. Our difficulty would
have been at the first step, and there only. Indeed we can scarcely
conceive that any case which does not call for capital punishment
can call for punishment by a retrospective act. We can scarcely
conceive a man so wicked and so dangerous that the whole course
of law must be disturbed in order to reach him, yet not so wicked as
to deserve the severest sentence, nor so dangerous as to require the
last and surest custody, that of the grave. If we had thought that
Strafford might be safely suffered to live in France, we should have
thought it better that he should continue to live in England, than
that he should be exiled by a special act. As to degradation, it was
not the Earl, but the general and the statesman, whom the people
had to fear. Essex said, on that occasion, with more truth than
elegance, “Stone dead hath no fellow.” And often during the civil
wars the Parliament had reason to rejoice that an irreversible law
and an impassable barrier protected them from the valour and
capacity of Wentworth.
It is remarkable that neither Hyde nor Falkland voted against the
bill of attainder. There is, indeed, reason to believe that Falkland
spoke in favour of it. In one respect, as Mr. Hallam has observed, the
proceeding was honourably distinguished from others of the same
kind. An act was passed to relieve the children of Strafford from the
forfeiture and corruption of blood which were the legal
consequences of the sentence. The Crown had never shown equal
generosity in a case of treason. The liberal conduct of the Commons
has been fully and most appropriately repaid. The House of
Wentworth has since that time been as much distinguished by public
spirit as by power and splendour, and may at the present moment
boast of members with whom Say and Hampden would have been
proud to act.
It is somewhat curious that the admirers of Strafford should also
be, without a single exception, the admirers of Charles; for,
whatever we may think of the conduct of the Parliament towards the
unhappy favourite, there can be no doubt that the treatment which
he received from his master was disgraceful. Faithless alike to his
people and to his tools, the King did not scruple to play the part of
the cowardly approver, who hangs his accomplice. It is good that
there should be such men as Charles in every league of villany. It is
for such men that the offer of pardon and reward which appears
after a murder is intended. They are indemnified, remunerated, and
despised. The very magistrate who avails himself of their assistance
looks on them as more contemptible than the criminal whom they
betray. Was Strafford innocent? Was he a meritorious servant of the
Crown? If so, what shall we think of the Prince, who having solemnly
promised him that not a hair of his head should be hurt, and
possessing an unquestioned constitutional right to save him, gave
him up to the vengeance of his enemies? There were some points
which we know that Charles would not concede, and for which he
was willing to risk the chances of civil war. Ought o not a King, who
will make a stand for any thing, to make a stand for the innocent
blood? Was Strafford guilty? Even on this supposition, it is difficult
not to feel disdain for the partner of his guilt, the tempter turned
punisher. If, indeed, from that time forth, the conduct of Charles had
been blameless, it might have been said that his eyes were at last
opened to the errors of his former conduct, and that, in sacrificing to
the wishes of his Parliament a minister whose crime had been a
devotion too zealous to the interests of his prerogative, he gave a
painful and deeply humiliating proof of the sincerity of his
repentance. We may describe the King’s behaviour on this occasion
in terms resembling those which Hume has employed when speaking
of the conduct of Churchill at the Revolution. It required ever after
the most rigid justice and sincerity in the dealings of Charles with his
people to vindicate his conduct towards his friend. His subsequent
dealings with his people, however, clearly showed, that it was not
from any respect for the Constitution, or from any sense of the deep
criminality of the plans in which Strafford and himself had been
engaged, that he gave up his minister to the axe. It became evident
that he had abandoned a servant who, deeply guilty as to all others,
was guiltless to him alone, solely in order to gain time for maturing
other schemes of tyranny, and purchasing the aid of other
Wentworths. He, who would not avail himself of the power which the
laws gave him to save an adherent to whom his honour was
pledged, soon showed that he did not scruple to break every law
and forfeit every pledge, in order to work the ruin of his opponents.
“Put not your trust in princes!” was the expression of the fallen
minister, when he heard that Charles had consented to his death.
The whole history of the times is a sermon on that bitter text. The
defence of the Long Parliament is comprised in the dying words of
its victim.
The early measures of that Parliament Mr. Hallam in general
approves. But he considers the proceedings which took place after
the recess in the summer of 1641 as mischievous and violent. He
thinks that, from that time, the demands of the Houses were not
warranted by any imminent danger to the Constitution, and that in
the war which ensued they were clearly the aggressors. As this is
one of the most interesting questions in our history, we will venture
to state, at some length, the reasons which have led us to form an
opinion on it contrary to that of a writer whose judgment we so
highly respect.
We will premise that we think worse of King Charles the First than
even Mr. Hallam appears to do. The fixed hatred of liberty which was
the principle of the King’s public conduct, the unscrupulousness with
which he adopted any means which might enable him to attain his
ends, the readiness with which he gave promises, the impudence
with which he broke them, the cruel indifference with which he
threw away his useless or damaged tools, made him, at least till his
character was fully exposed and his power shaken to its foundations,
a more dangerous enemy to the Constitution than a man of far
greater talents and resolution might have been. Such princes may
still be seen, the scandals of the southern thrones of Europe, princes
false alike to the accomplices who have served them and to the
opponents who have spared them, princes who, in the hour of
danger, concede every thing, swear every thing, hold out their
cheeks to every smiter, give up to punishment every instrument of
their tyranny, and await with meek and smiling implacability the
blessed day of perjury and revenge.
 We will pass by the instances of oppression and falsehood which
disgraced the early part of the reign of Charles. We will leave out of
the question the whole history of his third Parliament, the price
which he exacted for assenting to the Petition of Right, the perfidy
with which he violated his engagements, the death of Eliot, the
barbarous punishments inflicted by the Star-Chamber, the ship-
money, and all the measures now universally condemned, which
disgraced his administration from 1630 to 1640. We will admit that it
might be the duty of the Parliament, after punishing the most guilty
of his creatures, after abolishing the inquisitorial tribunals which had
been the instruments of his tyranny, after reversing the unjust
sentences of his victims, to pause in its course. The concessions
which had been made were great, the evils of civil war obvious, the
advantages even of victory doubtful. The former errors of the King
might be imputed to youth, to the pressure of circumstances, to the
influence of evil counsel, to the undefined state of the law. We firmly
believe that if, even at this eleventh hour, Charles had acted fairly
towards his people, if he had even acted fairly towards his own
partisans, the House of Commons would have given him a fair
chance of retrieving the public confidence. Such was the opinion of
Clarendon. He distinctly states that the fury of opposition had
abated, that a reaction had begun to take place, that the majority of
those who had taken part against the King were desirous of an
honourable and complete reconciliation, and that the more violent,
or, as it soon appeared, the more judicious members of the popular
party were fast declining in credit. The Remonstrance had been
carried with great difficulty. The uncompromising antagonists of the
court, such as Cromwell, had begun to talk of selling their estates
and leaving England. The event soon showed, that they were the
only men who really understood how much inhumanity and fraud lay
hid under the constitutional language and gracious demeanour of
the King.
The attempt to seize the five members was undoubtedly the real
cause of the war. From that moment, the loyal confidence with
which most of the popular party were beginning to regard the King
was turned into hatred and incurable suspicion. From that moment,
the Parliament was compelled to surround itself with defensive arms.
From that moment, the city assumed the appearance of a garrison.
From that moment, in the phrase of Clarendon, the carriage of
Hampden became fiercer, that he drew the sword and threw away
the scabbard. For, from that moment, it must have been evident to
every impartial observer that, in the midst of professions, oaths, and
smiles, the tyrant was constantly looking forward to an absolute
sway and to a bloody revenge.
The advocates of Charles have very dexterously contrived to
conceal from their readers the real nature of this transaction. By
making concessions apparently candid and ample, they elude the
great accusation. They allow that the measure was weak and even
frantic, an absurd caprice of Lord Digby, absurdly adopted by the
King. And thus they save their client from the full penalty of his
transgression, by entering a plea of guilty to the minor offence. To
us his conduct appears at this day as at the time it appeared to the
Parliament and the city. We think it by no means so foolish as it
pleases his friends to represent it, and far more wicked.
In the first place, the transaction was illegal from beginning to
end. The impeachment was illegal. The process was illegal. The
service was illegal. If Charles wished to prosecute the five members
for treason, a hill against them should have been sent to a grand
jury. That a commoner cannot be tried for high treason by the Lords,
at the suit of the Crown, is part of the very alphabet of our law. That
no man can be arrested by the King in person is equally clear. This
was an established maxim of our jurisprudence even in the time of
Edward the Fourth. “A subject,” said Chief Justice Markham to that
Prince, “may arrest for treason: the King cannot; for, if the arrest be
illegal, the party has no remedy against the King.”
The time at which Charles took this step also deserves
consideration. We have already said that the ardour which the
Parliament had displayed at the time of its first meeting had
considerably abated, that the leading opponents of the court were
desponding, and that their followers were in general inclined to
milder and more temper» measures than those which had hitherto
been pursued. In every country, and in none more than in England,
there is a disposition to take the part of those who are unmercifully
run down and who seem destitute of all means of defence. Every
man who has observed the ebb and flow of public feeling in our own
time will easily recall examples to illustrate this remark. An English
statesman ought to pay assiduous worship to Nemesis, to be most
apprehensive of ruin when he is at the height of power and
popularity, and to dread his enemy most when most completely
prostrated. The fate of the Coalition Ministry in 1784 is perhaps the
strongest instance in our history of the operation of this principle. A
few weeks turned the ablest and most extended Ministry that ever
existed into a feeble Opposition, and raised a King who was talking
of retiring to Hanover to a height of power which none of his
predecessors had enjoyed since the Revolution. A crisis of this
description was evidently approaching in 1642. At such a crisis, a
Prince of a really honest and generous nature, who had erred, who
had seen his error, who had regretted the lost affections of his
people, who rejoiced in the dawning hope of regaining them, would
be peculiarly careful to take no step which could give occasion of
offence, even to the unreasonable. On the other hand, a tyrant,
whose whole life was a lie, who hated the Constitution the more
because he had been compelled to feign respect for it, and to whom
his own honour and the love of his people were as nothing, would
select such a crisis for some appalling violation of law, for some
stroke which might remove the chiefs of an Opposition, and
intimidate the herd. This Charles attempted. He missed his blow; but
so narrowly, that it would have been mere madness in those at
whom it was aimed to trust him again.
It deserves to be remarked that the King had, a short time before,
promised the most respectable Royalists in the House of Commons,
Falkland, Colepepper, and Hyde, that he would take no measure in
which that House was concerned, without consulting them. On this
occasion he did not consult them. His conduct astonished them more
than any other members of the assembly. Clarendon says that they
were deeply hurt by this want of confidence, and the more hurt,
because, if they had been consulted, they would have done their
utmost to dissuade Charles from so improper a proceeding. Did it
never occur to Clarendon, will it not at least occur to men less
partial, that there was good reason for this? When the danger to the
throne seemed imminent, the King was ready to put himself for a
time into the hands of those who, though they disapproved of his
past conduct, thought that the remedies had now become worse
than the distempers. But we believe that In his heart he regarded
both the parties in the Parliament with feelings of aversion which
differed only in the degree of their intensity, and that the awful
warning which he proposed to give, by immolating the principal
supporters of the Remonstrance, was partly intended for the
instruction of those who had concurred in censuring the ship-money
and in abolishing the Star-Chamber.
The Commons informed the King that their members should be
forthcoming to answer any charge legally brought against them. The
Lords refused to assume the unconstitutional office with which he
attempted to invest them. And what was then his conduct? He went,
attended by hundreds of armed men, to seize the objects of his
hatred in the House itself. The party opposed to him more than
insinuated that his purpose was of the most atrocious kind. We will
not condemn him merely on their suspicions. We will not hold him
answerable for the sanguinary expressions of the loose brawlers who
composed his train. We will judge of his act by itself alone. And we
say, without hesitation, that it is impossible to acquit him of having
meditated violence, and violence which might probably end in blood.
He knew that the legality of his proceedings was denied. He must
have known that some of the accused members were men not likely
to submit peaceably to an illegal arrest. There was every îeason to
expect that he would find them in their places, that they would
refuse to obey his summons, and that the House would support
them in their refusal. What course would then have been left to him?
Unless we suppose that he went on this expedition for the sole
purpose of making himself ridiculous, we must believe that he would
have had recourse to force. There would have been a scuffle; and it
might not, under such circumstances, have been in his power, even
if it had been in his inclination, to prevent a scuffle from ending in a
massacre. Fortunately for his fame, unfortunately perhaps for what
he prized far more, the interests of his hatred and his ambition, the
affair ended differently. The birds, as he said, were flown, and his
plan was disconcerted. Posterity is not extreme to mark abortive
crimes; and thus the King’s advocates have found it easy to
represent a step which, but for a trivial accident, might have filled
England with mourning and dismay, as a mere error of judgment,
wild and foolish, but perfectly innocent. Such was not, however, at
the time, the opinion of any party. The most zealous Royalists were
so much disgusted and ashamed that they suspended their
opposition to the popular party, and, silently at least, concurred in
measures of precaution so strong as almost to amount to resistance.
From that day, whatever of confidence and loyal attachment had
survived the misrule of seventeen years was, in the great body of
the people, extinguished, and extinguished for ever. As soon as the
outrage had failed, the hypocrisy recommenced. Down to the very
eve of this flagitious attempt, Charles had been talking of his respect
for the privileges of Parliament and the liberties of his people. He
began again in the same style on the morrow; but it was too late. To
trust him now would have been, not moderation, but insanity. What
common security would suffice against a Prince who was evidently
watching his season with that cold and patient hatred which, in the
long run, tires out every other passion?
It is certainly from no admiration of Charles that Mr. Hallam
disapproves of the conduct of the Houses in resorting to arms. But
he thinks that any attempt on the part of that Prince to establish a
despotism would have been as strongly opposed by his adherents as
by his enemies, and that therefore the Constitution might be
considered as out of danger, or, at least, that it had more to
apprehend from the war than from the King. On this subject Mr.
Hallam dilates at length, and with conspicuous ability. We will offer a
few considerations which lead us to incline to a different opinion.
The Constitution of England was only one of a large family. In all
the monarchies of Western Europe, during the middle ages, there
existed restraints on the royal authority, fundamental laws, and
representative assemblies. In the fifteenth century, the government
of Castile seems to have been as free as that of our own country.
That of Arragon was beyond all question more so. In France, the
sovereign was more absolute. Yet, even in France, the States-
General alone could constitutionally impose taxes; and, at the very
time when the authority of those assemblies was beginning to
languish, the Parliament of Paris received such an accession of
strength as enabled it, in some measure, to perform the functions of
a legislative assembly. Sweden and Denmark had constitutions of a
similar description. Let us overleap two or three hundred years, and
contemplate Europe at the commencement of the eighteenth
century. Every free constitution, save one, had gone down. That of
England had weathered the danger, and was riding in full security. In
Denmark and Sweden, the kings had availed themselves of the
disputes which raged between the nobles and the commons, to unite
all the powers of government in their own hands. In France the
institution of the States was only mentioned by lawyers as a part of
the ancient theory of their government. It slept a deep sleep,
destined to be broken by a tremendous waking. No person
remembered the sittings of the three orders, or expected ever to see
them renewed. Louis the Fourteenth had imposed on his parliament
a patient silence of sixty years. His grandson, after the War of the
Spanish Succession, assimilated the constitution of Arragon to that
of Castile, and extinguished the last feeble remains of liberty in the
Peninsula. In England, on the other hand, the Parliament was
infinitely more powerful than it had ever been. Not only was its
legislative authority fully established; but its right to interfere, by
advice almost equivalent to command, in every department of the
executive government, was recognised. The appointment of
ministers, the relations with foreign powers, the conduct of a war or
a negotiation, depended less on the pleasure of the Prince than on
that of the two Houses.
What then made us to differ? Why was it that, in that epidemic
malady of constitutions, ours escaped the destroying influence; or
rather that, at the very crisis of the disease, a favourable turn took
place in England, and in England alone? It was not surely without a
cause that so many kindred systems of government, having
flourished together so long, languished and expired at almost the
same time.
It is the fashion to say, that the progress of civilisation is
favourable to liberty. The maxim, though in some sense true, must
be limited by many qualifications and exceptions. Wherever a poor
and rude nation, in which the form of government is a limited
monarchy, receives a great accession of wealth and knowledge, it is
in imminent danger of falling under arbitrary power.
In such a state of society as that which existed all over Europe
during the middle ages, very slight checks sufficed to keep the
sovereign in order. His means of corruption and intimidation were
very scanty. He had little money, little patronage, no military
establishment. His armies resembled juries. They were drawn out of
the mass of the people: they soon returned to it again: and the
character which was habitual, prevailed over that which was
occasional. A campaign of forty days was too short, the discipline of
a national militia too lax, to efface from their minds the feelings of
civil life. As they carried to the camp the sentiments and interests of
the farm and the shop, so they carried back to the farm and the
shop the military accomplishments which they had acquired in the
camp. At home the soldier learned how to value his rights, abroad
how to defend them.
Such a military force as this was a far stronger restraint on the
regal power than any legislative assembly. The army, now the most
formidable instrument of the executive power, was then the most
formidable check on that power. Resistance to an established
government, in modern times so difficult and perilous an enterprise,
was, in the fourteenth and fifteenth centuries, the simplest and
easiest matter in the world. Indeed, it was far too simple and easy.
An insurrection was got up then almost as easily as a petition is got
up now. In a popular cause, or even in an unpopular cause favoured
by a few great nobles, a force of ten thousand armed men was
raised in a week. If the King were, like our Edward the Second and
Richard the Second, generally odious, he could not procure a bow or
halbert. He fell at once and without an effort. In such times a
sovereign like Louis the Fifteenth or the Emperor Paul, would have
been pulled down before his misgovernment had lasted for a month.
We find that all the fame and influence of our Edward the Third
could not save his Madame de Pompadour from the effects of the
public hatred.
Hume and many other writers have hastily concluded that, in the
fifteenth century, the English Parliament was altogether servile,
because it recognised, without opposition, every successful usurper.
That it was not servile its conduct on many occasions of inferior
importance is sufficient to prove. But surely it was not strange that
the majority of the nobles, and of the deputies chosen by the
commons, should approve of revolutions which the nobles and
commons had effected. The Parliament did not blindly follow the
event of war, but participated in those changes of public sentiment
on which the event of war depended. The legal check was secondary
and auxiliary to that which the nation held in its own hands. There
have always been monarchies in Asia, in which the royal authority
has been tempered by fundamental laws, though no legislative body
exists to watch over them. The guarantee is the opinion of a
community of which every individual is a soldier. Thus, the king of
Cabul, as Mr. Elplunstone informs us, cannot augment the land
revenue, or interfere with the jurisdiction of the ordinary tribunals.
In the European kingdoms of this description there were
representative assemblies. But it was not necessary, that those
assemblies should meet very frequently, that they should interfere
with all the operations of the executive government, that they
should watch with jealousy, and resent with prompt indignation,
every violation of the laws which the sovereign might commit. They
were so strong that they might safely be careless. He was so feeble
that he might safely be suffered to encroach. If he ventured too far,
chastisement and ruin were at hand. In fact, the people generally
suffered more from his weakness than from his authority. The
tyranny of wealthy and powerful subjects was the characteristic evil
of the times. The royal prerogatives were not even sufficient for the
defence of property and the maintenance of police.
The progress of civilisation introduced a great change. War
became a science, and, as a necessary consequence, a trade. The
great body of the people grew every day more reluctant to undergo
the inconveniences of military service, and better able to pay others
for undergoing them. A new class of men, therefore, dependent on
the Crown alone, natural enemies of those popular rights which are
to them as the dew to the fleece of Gideon, slaves among freemen,
freemen among slaves, grew into importance. That physical force
which, in the dark ages, had belonged to the nobles and the
commons, and had, far more than any charter or any assembly,
been the safeguard of their privileges, was transferred entire to the
King. Monarchy gained in two ways. The sovereign was
strengthened, the subjects weakened. The great mass of the
population, destitute of all military discipline and organization,
ceased to exercise any influence by force on political transactions.
There have, indeed, during the last hundred and fifty years, been
many popular insurrections in Europe: but all have failed, except
those in which the regular army has been induced to join the
disaffected.
Those legal checks which, while the sovereign remained
dependent on his subjects, had been adequate to the purpose for
which they were designed, were now found wanting. The dikes
which had been sufficient while the waters were low were not high
enough to keep out the spring-tide. The deluge passed over them;
and, according to the exquisite illustration of Butler, the formal
boundaries which had excluded it, now held it in. The old
constitutions fared like the old shields and coats of mail. They were
the defences of a rude age: and they did well enough against the
weapons of a rude age. But new and more formidable means of
destruction were invented. The ancient panoply became useless;
and it was thrown aside to rust in lumber-rooms, or exhibited only
as part of an idle pageant.
 Thus absolute monarchy was established on the Continent.
England escaped; but she escaped very narrowly. Happily our insular
situation, and the pacific policy of James, rendered standing armies
unnecessary here, till they had been for some time kept up in the
neighbouring kingdoms. Our public men had therefore an
opportunity of watching the effects produced by this momentous
change on governments which bore a close analogy to that
established in England. Every where they saw the power of the
monarch increasing, the resistance of assemblies which were no
longer supported by a national force gradually becoming more and
more feeble, and at length altogether ceasing. The friends and the
enemies of liberty perceived with equal clearness the causes of this
general decay. It is the favourite theme of Strafford. He advises the
King to procure from the Judges a recognition of his right to raise an
army at his pleasure. “This place well fortified,” says he, “for ever
vindicates the monarchy at home from under the conditions and
restraints of subjects.” We firmly believe that he was in the right.
Nay; we believe that, even if no deliberate scheme of arbitrary
government had been formed by the sovereign and his ministers,
there was great reason to apprehend a natural extinction of the
Constitution. If, for example, Charles had played the part of
Gustavus Adolphus, if he had carried on a popular war for the
defence of the Protestant cause in Germany, if he had gratified the
national pride by a series of victories, if he had formed an army of
forty or fifty thousand devoted soldiers, we do not see what chance
the nation would have had of escaping from despotism. The Judges
would have given as strong a decision in favour of camp-money as
they gave in favour of ship-money. If they had been scrupulous, it
would have made little difference. An individual who resisted would
have been treated as Charles treated Eliot, and as Strafford wished
to treat Hampden. The Parliament might have been summoned once
in twenty years, to congratulate a King on his accession, or to give
solemnity to some great measure of state. Such had been the fate of
legislative assemblies as powerful, as much respected, as high-
spirited, as the English Lords and Commons. The two Houses,
surrounded by the ruins of so many free constitutions overthrown or
sapped by the new military system, were required to intrust the
command of an army and the conduct of the Irish war to a King who
had proposed to himself the destruction of liberty as the great end
of his policy. We are decidedly of opinion that it would have been
fatal to comply. Many of those who took the side of the King on this
question would have cursed their own loyalty, if they had seen him
return from war at the head of twenty thousand troops, accustomed
to carnage and free quarters in Ireland.
We think, with Mr. Hallam, that many of the Royalist nobility and
gentry were true friends to the Constitution, and that, but for the
solemn protestations by which the King bound himself to govern
according to the law for the future, they never would have joined his
standard. But surely they underrated the public danger. Falkland is
commonly selected as the most respectable specimen of this class.
He was indeed a man of great talents and of great virtues, but, we
apprehend, infinitely too fastidious for public life. He did not perceive
that, in such times as those on which his lot had fallen, the duty of a
statesman is to choose the better cause and to stand by it, in spite
of those excesses by which every cause, however good in itself, will
be disgraced. The present evil always seemed to him the worst. He
was always going backward and forward; but it should be
remembered to his honour that it was always from the stronger to
the weaker side that he deserted. While Charles was oppressing the
people, Falkland was a resolute champion of liberty. He attacked
Strafford. He even concurred in strong measures against Episcopacy.
But the violence of his party annoyed him, and drove him to the
other party, to be equally annoyed there. Dreading the success of
the cause which he had espoused, disgusted by the courtiers of
Oxford, as he had been disgusted by the patriots of Westminster, yet
bound by honour not to abandon the cause for which he was in
arms, he pined away, neglected his person, went about moaning for
peace, and at last rushed desperately on death, as the best refuge in
such miserable times. If he had lived through the scenes that
followed, we have little doubt that he would have condemned
himself to share the exile and beggary of the royal family; that he
would then have returned to oppose all their measures; that he
would have been sent to the Tower by the Commons as a stiller of
the Popish Plot, and by the King as an accomplice in the Rye-House
Plot; and that, if he had escaped being hanged, first by Scroggs, and
then by Jefferies, he would, after manfully opposing James the
Second through years of tyranny, have been seized with a fit of
compassion at the very moment of the Revolution, have voted for a
regency, and died a nonjuror.
We do not dispute that the royal party contained many excellent
men and excellent citizens. But this we say, that they did not discern
those times. The peculiar glory of the Houses of Parliament is that,
in the great plague and mortality of constitutions, they took their
stand between the living and the dead. At the very crisis of our
destiny, at the very moment when the fate which had passed on
every other nation was about to pass on England, they arrested the
danger.
Those who conceive that the parliamentary leaders were desirous
merely to maintain the old constitution, and those who represent
them as conspiring to subvert it, are equally in error. The old
constitution, as we have attempted to show, could not be
maintained. The progress of time, the increase of wealth, the
diffusion of knowledge, the great change in the European system of
war, rendered it impossible that any of the monarchies of the middle
ages should continue to exist on the old footing. The prerogative of
the crown was constantly advancing. If the privileges of the people
were to remain absolutely stationary, they would relatively
retrograde. The monarchical and democratical parts of the
government were placed in a situation not unlike that of the two
brothers in the Fairy Queen, one of whom saw the soil of his
inheritance daily washed away by the tide and joined to that of his
rival. The portions had at first been fairly meted out. By a natural
and constant transfer, the one had been extended; the other had
dwindled to nothing. A new partition, or a compensation, was
necessary to restore the original equality.
It was now, therefore, absolutely necessary to violate the formal
part of the constitution, in order to preserve its spirit. This might
have been done, as it was done at the Revolution, by expelling the
reigning family, and calling to the throne princes who, relying solely
on an elective title, would find it necessary to respect the privileges
and follow the advice of the assemblies to which they owed every
thing, to pass every bill which the Legislature strongly pressed upon
them, and to fill the offices of state with men in whom the
Legislature confided. But, as the two Houses did not choose to
change the dynasty, it was necessary that they should do directly
what at the Revolution was done indirectly. Nothing is more usual
than to hear it said that, if the Houses had contented themselves
with making such a reform in the government under Charles as was
afterwards made under William, they would have had the highest
claim to national gratitude: and that in their violence they overshot
the mark. But how was it possible to make such a settlement under
Charles? Charles was not, like William and the princes of the
Hanoverian line, bound by community of interests and dangers to
the Parliament. It was therefore necessary that he should be bound
by treaty and statute.
Mr. Hallam reprobates, in language which has a little surprised us,
the nineteen propositions into which the Parliament digested its
scheme. Is it possible to doubt that, if James the Second had
remained in the island, and had been suffered, as he probably would
in that case have been suffered, to keep his crown, conditions to the
full as hard would have been imposed on him? On the other hand,
we fully admit that, if the Long Parliament had pronounced the
departure of Charles from London an abdication, and had called
Essex or Northumberland to the throne, the new prince might have
safely been suffered to reign without such restrictions. His situation
would have been a sufficient guarantee.
 In the nineteen propositions we see very little to blame except the
articles against the Catholics. These, however, were in the spirit of
that age; and to some sturdy churchmen in our own, they may seem
to palliate even the good which the Long Parliament effected. The
regulation with respect to new creations of Peers is the only other
article about which we entertain any doubt. One of the propositions
is that the judges shall hold their offices during good behaviour. To
this surely no exception will be taken. The right of directing the
education and marriage of the princes was most properly claimed by
the Parliament, on the same ground on which, after the Revolution,
it was enacted, that no king, on pain of forfeiting his throne, should
espouse a Papist. Unless we condemn the statesmen of the
Revolution, who conceived that England could not safely be
governed by a sovereign married to a Catholic queen, we can
scarcely condemn the Long Parliament because, having a sovereign
so situated, they thought it necessary to place him under strict
restraints. The influence of Henrietta Maria had already been deeply
felt in political affairs. In the regulation of her family, in the
education and marriage of her children, it was still more likely to be
felt. There might be another Catholic queen; possibly, a Catholic
king. Little as we are disposed to join in the vulgar clamour on this
subject, we think that such an event ought to be, if possible,
averted; and this could only be done, if Charles was to be left on the
throne, by placing his domestic arrangements under the control of
Parliament.
A veto on the appointment of ministers was demanded. But this
veto Parliament has virtually possessed ever since the Revolution. It
is no doubt very far better that this power of the Legislature should
be exercised as it is now exercised, when any great occasion calls for
interference, than that at every change the Commons should have to
signify their approbation or disapprobation in form. But, unless a
new family had been placed on the throne, we do not see how this
power could have been exercised as it is now exercised. We again
repeat, that no restraints which could be imposed on the princes
who reigned after the Revolution could have added to the security
which their title afforded. They were compelled to court their
parliaments. But from Charles nothing was to be expected whicli was
not set down in the bond.
It was not stipulated that the King should give up his negative on
acts of Parliament. But the Commons had certainly shown a strong
disposition to exact this security also. “Such a doctrine,” says Mr.
Hallam, “was in this country as repugnant to the whole history of our
laws, as it was incompatible with the subsistence of the monarchy in
any thing more than a nominal preeminence.” Now this article has
been as completely carried into effect by the Revolution as if it had
been formally inserted in the Bill of Rights and the Act of Settlement.
We are surprised, we confess, that Mr. Hallam should attach so much
importance to a prerogative which has not been exercised for a
hundred and thirty years, which probably will never be exercised
again, and which can scarcely, in any conceivable case, be exercised
for a salutary purpose.
But the great security, the security without which every other
would have been insufficient, was the power of the sword. This both
parties thoroughly understood. The Parliament insisted on having
the command of the militia and the direction of the Irish war. “By
God, not for an hour!” exclaimed the King. “Keep the militia,” said
the Queen, after the defeat of the royal party: “Keep the militia; that
will bring back every thing.” That, by the old constitution, no military
authority was lodged in the Parliament, Mr. Hallam has clearly
shown. That it is a species of authority which ought not to be
permanently lodged in large and divided assemblies, must, we think,
in fairness be conceded. Opposition, publicity, long discussion,
frequent compromise; these are the characteristics of the
proceedings of such assemblies. Unity. secrecy, decision, are the
qualities which military arrangements require. There were, therefore,
serious objections to the proposition of the Houses on this subject.
But, on the other hand, to trust such a king, at such a crisis, with
the very weapon which, in hands less dangerous, had destroyed so
many free constitutions, would have been the extreme of rashness.
The jealousy with which the oligarchy of Venice and the States of
Holland regarded their generals and armies induced them
perpetually to interfere in matters of which they were incompetent
to judge. This policy secured them against military usurpation, but
placed them under great disadvantages in war. The uncontrolled
power which the King of France exercised over his troops enabled
him to conquer his enemies, but enabled him also to oppress his
people. Was there any intermediate course? None, we confess,
altogether free from objection. But on the whole, we conceive that
the best measure would have been that which the Parliament over
and over proposed, namely, that for a limited time the power of the
sword should be left to the two Houses, and that it should revert to
the Crown when the constitution should be firmly established, and
when the new securities of freedom should be so far strengthened
by prescription that it would be difficult to employ even a standing
army for the purpose of subverting them.
Mr. Hallam thinks that the dispute might easily have been
compromised, by enacting that the King should have no power to
keep a standing army on foot without the consent of Parliament. He
reasons as if the question had been merely theoretical, and as if at
that time no army had been wanted. “The kingdom,” he says, “might
have well dispensed, in that age, with any military organization.”
Now, we think that Mr. Hallam overlooks the most important
circumstance in the whole case. Ireland was actually in rebellion;
and a great expedition would obviously he necessary to reduce that
kingdom to obedience. The Houses had therefore to consider, not an
abstract question of law, but an urgent practical question, directly
involving the safety of the state. They had to consider the
expediency of immediately giving a great army to a King who was at
least as desirous to put down the Parliament of England as to
conquer the insurgents of Ireland.
Of course we do not mean to defend all the measures of the
Houses. Far from it. There never was a perfect man. It would,
therefore, be the height of absurdity to expect a perfect party or a
perfect assembly. For large bodies are far more likely to err than
individuals. The passions are inflamed by sympathy; the fear of
punishment and the sense of shame are diminished by partition.
Every day we see men do for their faction what they would die
rather than do for themselves.
Scarcely any private quarrel ever happens, in which the right and
wrong are so exquisitely divided that all the right lies on one side,
and all the wrong on the other. But here was a schism which
separated a great nation into two parties. Of these parties, each was
composed of many smaller parties. Each contained many members,
who differed far less from their moderate opponents than from their
violent allies. Each reckoned among its supporters many who were
determined in their choice by some accident of birth, of connection,
or of local situation. Each of them attracted to itself in multitudes
those fierce and turbid spirits, to whom the clouds and whirlwinds of
the political hurricane are the atmosphere of life. A party, like a
camp, has its sutlers and camp-followers, as well as its soldiers. In
its progress it collects round it a vast retinue, composed of people
who thrive by its custom or are amused by its display, who may be
sometimes reckoned, in an ostentatious enumeration, as forming a
part of it, but who give no aid to its operations, and take but a
languid interest in its success, who relax its discipline and dishonour
its flag by their irregularities, and who, after a disaster, are perfectly
ready to cut the throats and rifle the baggage of their companions.
Thus it is in every great division; and thus it was in our civil war.
On both sides there was, undoubtedly, enough of crime and enough
of error to disgust any man who did not reflect that the whole
history of the species is made up of little except crimes and errors.
Misanthropy is not the temper which qualifies a man to act in great
affairs, or to judge of them.
“Of the Parliament,” says Mr. Hallam, “it may be said, I think, with
not greater severity than truth, that scarce two or three public acts
of justice, humanity, or generosity, and very few of political wisdom
or courage, are recorded of them, from their quarrel with the King,
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com