Introduction to Matrix Analysisp—————« Introduction to Matrix Analysis &>_____{Second Edition} ——e#SIAM’s Classics in Applied Mathematics series consists of books that were previously allowed to go out of print. These books ate republished by SIAM as a professional service because they continue to be important resources for mathematical scientists. Editor-in-Chief Robert E. O'Malley, Je., University of Washington Editorial Board Richard A. Brualdi, University of Wisconsin- Madison Herbert B. Keller, California Institute of Technology Andrej Z. Manitius, George Mason University Ingram Olkin, Stanford University Stanley Richardson, University of Edinburgh Ferdinand Verhulst, Mathematisch Instituut, University of Utrecht Classi C.C. Lin and L. A. Segel, Mathematics Applied to Deterministic Problems in the Natural Sciences in Applied Mathematics Johan G. F. Belinfante and Bernard Kolman, A Survey of Lie Groups and Lie Algebras with Applications and Computational Methods James M. Ortega, Numerical Analysis: A Second Course Anthony V. Fiacco and Garth P, McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Techniques F. H. Clarke, Optimization and Nonsmooth Analysis George F. Carrier and Carl E. Pearson, Ordinary Differential Equations Leo Breiman, Probability R. Bellman and G. M. Wing, An Introduction to Invariant Imbedding Abraham Berman and Robert J. Plemmons, Nonnegative Matrices in the Mathematical Sciences Olvi L. Mangasarian, Nonlinear Programming *Carl Friedrich Gauss, Theory of the Combination of Observations Least Subject to Errors: Part One, Part Two, Supplement. Translated by G. W. Stewart Richard Bellman, Introduction to Matrix Analysis U. M. Ascher, R. M. M. Mattheij, and R. D. Russell, Numerical Solution of Boundary Value Problems for Ordinary Differential Equations K. E. Brenan, S. L. Campbell, and L. R. Petzold, Numerical Solution of Initial- Value Problems in Differential-Algebraic Equations Charles L. Lawson and Richard J. Hanson, Solving Least Squares Problems J. E. Dennis, Jr. and Robert B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations Richard E. Barlow and Frank Proschan, Mathematical Theory of ReliabilityClassics in Applied Mathematics (continued) Cornelius Lanczos, Linear Differential Operators Richard Bellman, Introduction to Matrix Analysis, Second Edition Beresford N. Parlete, The Symmetric Eigenvalue Problem Richard Haberman, Mathematical Models: Mechanical Vibrations, Population Dynamics, and Traffic Flow Peter W. M. John, Statistical Design and Analysis of Experiments Tamer Bagar and Geert Jan Ollsder, Dynamic Noncooperative Game Theory, Second Edition Emanuel Parz Petar Kokotovic, Hassan K. Khalil, and John O'Reilly, Singular Perturbation Methods in Control: Analysis and Design |, Stochastic Processes Jean Dickinson Gibbons, Ingram Olkin, and Milton Sobel, Selecting and Ordering Populations: A New Statistical Methodology James A. Murdock, Perturbations: Theory and Methods Ivar Ekeland and Roger Témam, Convex Analysis and Variational Problems Ivar Stakgold, Boundary Value Problems of Mathematical Physics, Volumes I and II J. M. Ortega and W. C. Rheinbolde, Iterative Solution of Nonlinear Equations in Several Variables David Kinderlehrer and Guido Stampacchia, An Introduction to Variational Inequalities and Their Applications F, Natterer, The Mathematics of Computerized Tomography Avinash C, Kak and Malcolm Slaney, Principles of Computerized Tomographic Imaging R. Wong, Asympototic Approximations of Integrals O, Axelsson and V. A. Barker, Finite Element Solution of Boundary Value Problems: Theory and Computation David R. Brillinger, Time Series: Data Analysis and TheoryThis page intentionally left blankIntroduction to Matrix Analysis {> __ rnd Edition} © Richard Bellman Society for Industrial and Applied Mathematics PhiladelphiaSociety for Industrial and Applied Mathematics 3600 University City Science Center, Philadelphia, PA 19104-2688. © 1997 by The RAND Corporation. 109876543 ‘This SLAM edition is an unabridged republication of the work first published by McGraw Hill Book Company, New York, New York, 1970. lll rights reserved. Printed in the United States of America. No part of this book may be reproduced, stored, or transmitted in any manner without the written permission of the RAND Corporation. For information, write RAND Book Program, 1700 Main Street, PO Box 2138, Santa Monica, CA 90407-2138. Library of Congress Catalog Card Number: 97-67651 ISBN 0-89871-399-4 SLAM. is a registered trademark,To NinaThis page intentionally left blankContents Chapter 2. Matrix Multiplication Vector by Ma Matrix Multiplication-Matrix by Matri BESRREEGSSE SommugQneneenr Quadratic Forms as Inner The Transpose Matrix Symmetric Matrices Hermitian Matrices Invariance of Distance~Orthogonal Matrices Unitary Matrices.» SaAPSosesChapter 3. Chapter 4, Chapter 5. Contents Diagonalization and Canonical Forms for Symmetric Matrices Recapitulation The Solution of Linear Homogeneous Equatior Characteristic Roots and Vectors wnnnmem Two Fundamental Properties of Symmetric M Matrices Reduction to Diagonal Form— Distinct Characteristic Roots ..u Reduction of Quadratic Forms to Canonical Positive Definite Quadratic Forms and Matrices Reduction of General Symmetric Matrices On the Positivity of the D, An Identity ou The Diagonalization of General Symmetric Matrices— Two-dimensional N-dimensional Case Characteristic Vectors Associated with Multiple Characteristic Roots. The Cayley Hamilton Theorem Simultaneous Reduction to Sum of Squares... Hermitian Matrices nrmamnnnsn The Original Maximization Problem Constrained Introduction Determinantal Criteria Representation as Sum of Squares .... Constrained Variation and Finsler’s Theorem The Case k= 1 A Minimization Problem..Chapter 6. Chapter 7. Chapter 8. Contents xi Composite Matrices... The Result for General k... Introduction The Rayleigh Quotient Variational Description of Characteristic ‘The Poincaré Separation Theorem A Representation Theorem. Approximate Techniques... Additive Inequalities from Multiplicative An Alternate Rout A Simpler Expression for Ay An.,xii Chapter 9. Chapter 10. Contents Arithmetic-Geometric Mean Inequality... Multiplicative Inequalities from Additive Dynamic Programming Introduction A Problem of Minimum Deviation .. Functional Equations. Recurrence Relations . A More Complicated Example Sturm-Liouville Problems Functional Equations. Jacobi Matrices . ‘Analytic Continuation Nonsymmetric Matrices... Complex A... Slightly Intertwined Systems ... Simplifications—1 Simplifications—II The Equation Ax = y Quadratic Deviati A Result of Stieltjes Matrices and Differential Equations Motivation .... Vector-matrix Notatior Norms of Vectors and Matrices Infinite Series of Vectors and Matrices ..... Existence and Uniqueness of Solutions of Linear Systems The Matrix Exponential.. Functional Equations-1 Functional EquationsII Functional Equations—III Nonsingularity of Solution .. Solution of Inhomogeneous Enuation- Constant Coefficients...... Inhomogeneous Equation-V: Inhomogeneous Equation—Adjoint Equation Perturbation Theory ... Non-negativity of Solutior Polya’s Functional Equation The Equation dX/dt = AX + XBChapter 11. Chapter 12. Contents xiii Explicit Solutions and Canonical Forms Introduction .. Euler's Method Construction of a Solution Nonsingularity of Second Method... The Vandermonde Explicit Solution of Linear Differential Equations— Diagonal Matrices ... Diagonalization of a Matrix Connection between Approaches Multiple Characteristic Roots. Jordan Canonical Form .. Multiple Characteristic Semidiagonal or Triangular Form- Normal Matrices An Approximation Theorem ... Another Approximation Theorem. The Cayley Hamilton Theorem. Alternate Proof of Cayley-Hamilton Theorem Linear Equations with Periodic Coefficients A Nonsingular Matrix Is an Exponenti An Alternate Proof ..... Some Interesting Transformations Symmetric Function, Kronecker Products and Circulant Introduction ... Powers of Characteristic Roots Kronecker Sum ...xiv Chapter 13, Chapter 14. Chapter 15. Chapter 16. Contents A Method of Lepann, Mean-square Deviatior Effective Tests for Stability A Necessary and Sufficient Condition for Stability Matrices... Differential Equations and Characteristic Values Effective Tests for Stability Matrices Markoff Matrices and Probability Theory Introduction A Simple Stochastic Process. Markoff Matrices and Probability Vectors Analytic Formulation of Discrete Markoff Processes Asymptotic Behavior .. First Proof... Second Proof of Independence of Initial State. Some Properties of Positive Markoff Matrices. Second Proof of Limiting Behavior.. General Markoff Matrices ... A Continuous Stochastic Process Positive Matrices, Perron’s Theorem, Introduction Some Sinole Growth Processes.Chapter 17, Contents xv Minkowski-Leontieff Matrices Positivity of [I - Al .. Strengthening of Theorem. li . An Economic Model 0. 02 Maintenance of Unstable Equilibrium. Discussion ..... Nonsingularity of X;(7) Analytic Form of Solutic Alternate Forms and Asymptotic Beha’ Representation of the Square Root of Riccati Differential Equation for R... Instability of Euler Equation... Proof that (x) Attains an Absolute Minimum Dynamic Programming Dynamic Programming Riccati Differential Equation Discussion nm More General Control Processes . Discrete Deterministic Control Processes. Discrete Stochastic Control Processes ...xvi Contents Potential Equation .. Discretized Criterion Function Chapter 18, Invariant Imbedding ... Chapter 19. The Renewal Equation Differential difference Equations .. Tychonov Regularization The Equation @(x) = A(x - ¢ x- 0) Obtaining an Initial Approxi \. Appendix A.‘Contents xvii Rank... Rank of a Quadratic Form. Law of Inertis Appendix C. A Method of Hermite... Appendix D. Moments and Quadratic Forms Introduction A Device of Stieltjes A Technique of E. Fischer Representation as Moment A Result of Herglotz IndexesThis page intentionally left blankForeword This book by Richard Bellman brought together a large number of results which were scattered throughout the literature. The book had enormous impact in such fields as numerical analysis, control theory, and statistics, It provided the vision and spirit for the SIAM Journal on Matrix Analysis and Applications, The reader should note that the problem sections contain many useful results that are not easily found elsewhere. We believe this new SIAM edition will be of continual benefit to researchers in the applied sciences where matrix analysis plays a vital role. Gene Golub Editor-in-Chief Classics in Applied Mathematics xix,This page intentionally left blankPreface to Second Edition Since the publication of the First Edition of this book in 1960, the field of matrix theory has expanded at a furious rate. In preparing a Second Edition, the question arises of how to take into account this vast proliferation of topics, methods, and results, Clearly, it would be impossible to retain any reasonable size and also to cover the developments in new and old areas in some meaningful fashion. A compromise is, therefore, essential. We decided after much soulsearching toadd some new results in areas already covered by including exercises at the ends. of chapters and by updating references. New areas, on the other hand, are tepresented by three additional chapters devoted to control theory, invariant imbedding, and numerical inversion of the Laplace transform. Asin the First Edition, we wish toemphasize the manner in which mathematical investigations of new scientific problems generate novel and interesting questions in matrix theory. Secondly, we hope to illustrate how the scientific background provides valuable clues as to the results to expect and even as to the methods of proof to employ. Many new areas have not been touched upon at all, or have been just barely mentioned: graph theory, scheduling theory, network theory, linear inequalities, and numerical analysis. We have preferred, for obvious reasons, to concentrate on topics to which a considerable amount of personal effort has been devoted. Matrix theory is replete with fascinating results and elegant techniques. It is a domain constantly stimulated by interactions with the outside world and it contributes to all areas of the physical and social sciences. It represents a healthy trend in modern mathematics. Richard Bellman University of Southern California xxiThis page intentionally left blankPreface Ouraim in this volume is to introduce the reader to the study of matrix theory, a field which with a great deal of justice may be called the arithmetic of higher mathematics, Although this is a rather sweeping claim, let us see if we can justify it. Surveying any of the classical domains of mathematics, we observe that the more interesting and significant parts are characterized by an interplay of factors. This interaction between individual elements manifests itself in the appearance of functions of several variables and correspondingly in the shape of variables which depend upon several functions. The analysis of these functions leads to transformations of multidimensional type. It soon becomes clear that the very problem of describing the problems that arise is itself of formidable nature. One has only to refer to various texts of one hundred years ago to be convinced that at the outset of any investigation there is a very real danger of being swamped by a seaof arithmetical and algebraical detail. And this is without regard of many conceptual and analytic difficulties that multidimensional analysis inevitably conjures up. It follows that at the very beginning a determined effort must be made to devise a useful, sensitive, and perceptive notation. Although it would certainly be rash to attempt to assign a numerical value to the dependence of successful research upon well-conceived notation, it is not difficult to cite numerous examples where the solutions become apparent when the questions are appropriately formulated. Conversely, a major effort and great ingenuity would be required were clumsy and unrevealing notation employed. Think, for instance, of how it would be to do arithmetic or algebra in terms of Roman numerals. A well-designed notation attempts to express the essence of the underlying mathematics without obscuring or distracting. With this as our introduction, we can now furnish a very simple syllogism. Matrices represent the most important of transformations, the linear transformations; transformations lie at the heart of mathematics, consequently, our first statement. This volume, the first of a series of volumes devoted to an exposition of the results and methods of modern matrix theory, is intended to acquaint the reader with the fundamental concepts of matrix theory. Subsequent volumes will expand the domain in various directions. Here we shall pay particular attention to the field of analysis, both from the standpoint of motivation and application. In consequence, the contents are specifically slanted toward the needs and aspirations of analysts, mathematical physicists, engineers of all shadings, and mathematical economists. Itturns out that the analytical theory of matrices, at the level at which we shall xxiiixxiv Preface treat it, falls rather neatly into three main categories: the theory of symmetric matrices, which invades all fields, matrices and differential equations, of particular concern to the engineer and physicist, and positive matrices, crucial in the areas of probability theory and mathematical economics. Although we have made no attempt to tie our exposition in with any actual applications, we have consistently tried to show the origin of the principal problems we consider. We begin with the question of determining the maximum or minimum of a function of several variables. Using the methods of calculus, we see that the determination of a local maximum or minimum leads to the corresponding question for functions of much simpler form, namely functions which are quadratic in all the variables, under reasonable assumptions concerning the existence of a sufficient number of partial derivatives. In this fashion, we are led to consider quadratic forms, and thus symmetric matrices, We first treat the case of functions of two variables where the usual notation suffices to derive all results of interest. Turning to the higher dimensional case, it becomes clear a better notation will prove of value. Nevertheless, a thorough understanding of the two-dimensional case is quite worthwhile, since all the methods used in the multidimensional case are contained therein. ‘We turn aside, then, from the multidimensional maximization problem to develop matrix notation. However, we systematically try to introduce ateach stage only those new symbols and ideas which are necessary for the problem at hand. It may surprise the reader, for example, to see how far into the theory of symmetric matrices one can penetrate without the concept of an inverse matrix. Consistent with these ideas, we have not followed the usual approach of deluging the novice with a flood of results concerning the solutions of linear systems of equations. Without for one moment attempting to minimize the importance of this study, it is still true that a significant number of interesting and important results can be presented without slogging down this long and somewhat wearisome road. The concept of linear independence is introduced in connection with the orthogonalization process, where it plays a vital role. In the appendix we present a proof of the fundamental result concerning the solutions of linear systems, and a discussion of some of the principal results concerning the rank of a matrix, This concept of much significance in many areas of matrix theory is not as important as might be thought in the regions we explore here. Too often, in many parts of mathematics, the reader is required to swallow on faith a large quantity of predigested material before being allowed to chew over any meaty questions, We have tried to avoid this. Once it has been seen that areal problem exists, then there is motivation for introducing more sophisticated concepts. This is the situation that the mathematician faces in actual research and in many applications. Although we have tried throughout to make the presentation logical, we havePreface XXV not belabored the point. Logic, after all, is a trick devised by the human mind to solve certain types of problems, But mathematics is more than logic, it is logic plus the creative process. How the logical devices that constitute the toolsof mathematics are to be combined to yield the desired results is not necessarily logical, no more than the writing of a symphony is a logical exercise, or the painting of a picture an exercise in syllogisms. Having introduced square matrices, the class of greatest importance for our work here, we turn to the problem of the canonical representation of real quadratic forms, or alternatively of areal symmetric matrix. The most important result of this analysis, and one that is basic for all subsequent development of the theory of symmetric matrices, is the equivalence, in a sense that will be made precise below, of every real symmetric matrix with a diagonal matrix. In other words, multidimensional transformations of this type to a very great extent can be regarded as a number of one-dimensional transformations performed simultaneously. The results of these preliminary chapters are instructive for several reasons. In the first place, they show what a great simplification in proof can be obtained by making an initial assumption concerning the simplicity of the characteristic roots. Secondly, they show that two methods can frequently be employed to circumvent the difficulties attendant upon multiplicity of roots. Both are potent tools of the analyst. The first is induction, the second is continuity. Of the two, continuity is the more delicate method, and requires for its Tigorous use, quite a bit more of sophistication than is required elsewhere in the book. Consequently, although we indicate the applicability of this technique wherever possible, we leave it to the ambitious reader to fill in the details. ‘Once having obtained the diagonal representation, we are ready to derive the min-max properties of the characteristic roots discovered by Courant and Fischer. The extension of these results to the more general operators arising from partial differential equations by Courant is a fundamental result in the domain of analysis. Having reached this point, it is now appropriate to introduce some other Properties of matrices. We turn then toa brief study of some of the important matrix functions. The question of defining a general function of a matrix is quite abit more complicated than might be imagined, and we discuss this only briefly. A number of references to the extensive literature on the subject are given. We now return to our original stimulus, the question of the range of values of a quadratic form. However, we complicate the problem to the extent of adding certain linear constraints. Not only is the problem of interest in itself, but it also supplies a good reason for introducing rectangular matrices. Having gone this far, it also turns out to be expedient to discuss matrices whose elements are themselves. matrices. This further refinement of the matrix notation is often exceedingly useful. Following this, we consider a number of interesting inequalities relating to characteristic roots and various functions of the characteristic roots. This chapterXXVi Preface is rather more specialized than any of the others in the volume and reflects perhaps the personal taste of the author more than the needs of the reader. The last chapter in the part devoted to symmetric matrices deals with the functional equation technique of dynamic programming. A number of problems related to maximization and minimization of quadratic forms and the solution of linear systems are treated in this fashion. The analytic results are interesting in their dependence upon parameters which are usually taken to be constant, while the recurrence relations obtained in this way yield algorithms which are sometimes of computational value. In the second third of the volume, we turn our attention to the application of matrix theory to the study of linear systems of differential equations. No previous knowledge of differential equations is assumed or required. The requisite existence and uniqueness theorems for linear systems will be demonstrated in the course of the discussion, The first important concept that enters in the study of linear systems with constant coefficients is that of the matrix exponential. In terms of this matrix function, we have an explicit solution of the differential equation. The case of variable coefficients does not permit resolution in this easy fashion. To obtain an analogous expression, it is necessary to introduce the product integral, a concept we shall not enter into here. The product integral plays an important rolein modern quantum mechanics, Although the construction of the exponential matrix solves in a very elegant fashion the problem of constructing an explicit solution of the linear equation with constant coefficients, it does not yield a useful representation for the individual ‘components of the vector solution. For this purpose, we employ a method due to Euler for finding particular solutions of exponential type. In this way we are once again led tothe problem of determining characteristic roots and vectors of a matrix. Since the matrix is in general no longer symmetric, the problem is very much more complicated than before. Although there are again a number of canonical forms, none of these are as convenient as that obtained for the case of the symmetric or hermitian matrix. The representation of the solution as a sum of exponentials, and limiting cases, permits us to state a necessary and sufficient condition that all solutions of a homogeneous system tend to the zero vector as the time becomes arbitrarily large. This leads to a discussion of stability and the problem of determining in a simple fashion when a given system is stable. The general problem is quite complicated. Having obtained a variety of results for general, not necessarily symmetric, matrices, we turn to what appears to be a problem of rather specialized interest. Given a matrix A, how do we determine a matrix whose characteristic roots are specified functions of the characteristic roots of A? If we ask that these functions be certain symmetric functions of the characteristic roots of A, then in a very natural fashion we are led to one of the important concepts of the algebraic side of matrix theory, the Kronecker product of two matrices. As we shall see, however,Preface XXvii in the concluding part of the book, this function of two matrices arises also in the study of stochastic matrices. The final part of the volume is devoted to the study of matrices all of whose elements are non-negative. Matrices of this apparently specialized type arise in two important ways, First in the study of Markoff processes, and secondly in the study of various economic processes. ‘A consideration of the physical origin of these matrices makes intuitively clear a number of quite important and interesting limit theorems associated with the names of Markoff, Perron, and Frobenius. In particular, a variational representation due to Wielandt plays a fundamental role in much of the theory of positive matrices. A brief chapteris devoted to the study of stochastic matrices. This area dealing with the multiplicative, and hence noncommutative, aspect of stochastic processes, rather than the additive, and hence commutative, aspect, is almost completely unexplored. The results we present are quite elementary and introductory. Finally, in a series of appendices, we have added some results which were either tangential to the main exposition, or else of quite specialized interest. The applications of Selberg, Hermite, and Fischer of the theory of symmetric matrices are, however, of such singular elegance that we felt that it was absolutely imperative that they be included. Now a few words to the reader first entering this fascinating field. As stated above, this volume is designed to be an introduction to the theory of matrix analysis. Although all the chapters are thus introductory in this sense, to paraphrase Orwell, some are more introductory than others. Consequently, it is not intended that the chapters be read consecutively. The beginner is urged to read Chaps. | through 5 for a general introduction to operations with matrices, and for the rudiments of the theory of symmetric matrices. At this point, it would be appropriate to jump to the study of general square matrices, using Chaps. 10 and 11 for this purpose. Finally, Chaps. 14 and 16 should be studied in order to understand the origins of Markoff matrices and non-negative matrices in general. Together with the working out of a number of the exercises, this should cover a semester course at an undergraduate senior, or first-year graduate level. ‘The reader who is studying matrix theory on his own can follow the same program. Having attained this level, the next important topic is that of the minimum- maximum characterization of the characteristic roots. Following this, we would suggest Kronecker products. From here, Chap. 6 seems appropriate. At this point, the remaining chapters can be taken in any order. Now a few comments concerning the exercises. Since the purpose of mathematics is to solve problems, it is impossible to judge one’s progress without breaking a lance on a few problems from stage to stage. What we have attempted to do is to provide problems of all levels of difficulty, starting from those whichxxviii Preface merely illustrate the text and ending with those which are of considerable difficulty. These latter have usually been lifted from current research papers. Generally, the problems immediately following the sections are routine and included for drill purpose. A few are on ahigher level, containing results which we shall use at a subsequent time. Since they can be established without too much effort we felt that it would be better to use them as exercises than to expand the text unduly by furnishing the proofs. In any case, the repetition of the same technique would make for tedious reading. On the other hand, the problems at the end of the chapter in the Miscellaneous Exercises are usually of a higher level, with some of greater complexity than others. Although the temptation is to star those problems which we consider more difficult, a little thought shows that this is bad pedagogy. After all, the purpose of a text such as this is to prepare the student to solve problems on his own, as they appear in research in the fields of analysis, mathematical physics, engineering, mathematical economics, and so forth. It is very seldom the case that a problem arising in this fashion is neatly stated with a star attached. Furthermore, it is important for the student to observe that the complication of a problem can practically never be gauged by its formulation. One very quickly learns that some very simple statements can conceal major difficulties. Taking this all into account, we have mixed problems of all levels together in a fairly random fashion and with ‘no warning as to their complexity. It follows that in attacking these problems, one should do those that can be done, struggle for a while with the more obdurate ones, and then return to these from timeto time, as maturity and analytic ability increase. Next, a word about the general plan of the volume. Not only do we wish to present a number of the fundamental results, but far more importantly, we wish to present a variety of fundamental methods. In order to do this, we have at several places presented alternative proofs of theorems, or indicated alternative proofs in the exercises. It is essential to realize that there are many different approaches to matrix theory, as indeed to all parts of mathematics. The importance of having all approaches available lies in the fact that different extensions may require different approaches. Indeed, some extensions may be routine with one approach and formidable, or impossible, following other routes. In this direction, although it is quite elegant and useful to derive many of the results of matrix theory from results valid for general operators, it is also important to point out various special techniques and devices which are particularly valuable in dealing with finite-dimensional transformations. It is for this reason that we have tried to rescue from oblivion a number of simple and powerful approaches which were used fifty to seventy-five years ago in the days of halcyon innocence. The human mind being what it is, repetition and cross-sections from different angles are powerful pedagogical devices. In this connection, it is appropriate to quote Lewis Carroll in the Hunting of the Snark, Fit the First—“I have said it thrice: What I tell you three times is true.”—The Bellman. Let us now briefly indicate some of the many fundamental aspects of matrixPreface XxiX theory which have reluctantly not been discussed in this volume. Inthe first place, we have omitted any discussion of the theory of computational treatment of matrices. This theory has vastly expanded in recent years, stimulated by the extraordinary abilities of modern computers and those that are planned for the immediate future, and by the extraordinary demands of the current physical and economic scenes. The necessity for the development of new techniques in the numerical treatment of matrices lies in the fact that the problem of solving systems of linear equations involving a large number of variables, or that of finding the characteristic Toots and vectors of a matrix of high dimension, cannot be treated in any routine fashion. Any successful treatment of these problems requires the use of new and subtle techniques. So much work has been done in this field that we thought it wise todevote a separate volume of this series to its recital. This volume will be written by George Forsythe. Inadifferent direction, the combinatorial theory of matrices has mushroomed in recent years. The mathematical theory of games of Borel and von Neumann, the theory of linear programming, and the mathematical study of scheduling theory have all combined to create virtually a new field of matrix theory. Not only do novel and significant analytic and algebraic problems arise directly in this way, but the pressure of obtaining computational results also has resulted in the development of still further techniques and concepts. A volume devoted to this field will be written by Alan Hoffman. Classically, topological aspects of matrix theory entered by way of the study of electrical networks. Here we encounter a beautiful blend of analytic, algebraic, and geometric theory. A volume on this important field is to be written by Louis Weinberg. We have, of course, barely penetrated the domain of matrix theory in the foregoing enumeration of topics. ‘On the distant horizon, we foresee a volume on the advanced theory of matrix analysis. This would contain, among other results, various aspects of the theory of functions of matrices, the Loewner theory, the Siegel theory of modular functions of matrices, and the R matrices of Wigner. In the more general theory of func- tionals of matrices, the Baker-Campbell-Hausdorff theory leads to the study of product integrals. These theories have assumed dominant roles in many parts of modern mathematical physics. Another vast field of matrix analysis has developed in connection with the study of multivariate analysis in mathematical statistics. Again we have felt that results in this domain are best presented with the background and motivation of statistics. In a common ground between analysis and algebra, we meet the theory of group representation with its many beautiful applications to algebra, analysis, and mathematical physics. This subject also requires a separate volume. Inthe purely algebraic domain, there is the treatment of ideal theory by wayXXX Preface of matrices due to Poincaré. We have not mentioned this because of the necessity for the introduction of rather sophisticated concepts. In the exercises, however, there are constant reminders of the interconnection between complex numbers, quaternions, and matrices, and hints of more general connections. Closely related is the study of matrices with integer elements, a number theory of matrices. Despite the attractiveness of these topics, we felt that it would be best to have them discussed in detail in a separate volume, It is important to note that not even the foregoing enumeration exhausts the many roles played by matrices in modern mathematics and its applications. Wehaveincludedin the discussions at theends of the chapters, andoccasionally in the exercises, a large number of references to original papers, current research papers, and various books on matrix theory. The reader who becomes particularly interested in a topic may thus pursue it thoroughly. Nonetheless, we make no pretension to an exhaustive bibliography, and many significant papers are not mentioned, Finally, I wish to express my heartfelt appreciation of the efforts of a number of friends who devotedly read through several drafts of the manuscript. Through their comments and criticisms, a number of significant improvements resulted in content, in style, and in the form of many interesting exercises. To Paul Brock, Ky Fan, and Olga Taussky, thanks. My sincere thanks are also due to Albert Madansky and Ingram Olkin who read several chapters and furnished a number of interesting exercises. I would like especially to express my gratitude to The RAND Corporation for its research policies which have permitted generous support of my work in the basic field of matrix theory. This has been merely one aspect of the freedom offered by RAND to pursue those endeavors which simultaneously advance science and serve the national interest. Atthe last, a vote of thanks to my secretary, Jeanette Hiebert, who unflinchingly typed hundreds of pages of equations, uncomplainingly made revision after revision, and devotedly helped with the proofreading. Richard Bellman1 Mazimization, Minimization, and Motivation 1, Introduction. The purpose of this opening chapter is to show how the question of ascertaining the range of values of a homogeneous quad- ratic function of two variables enters in @ very natural way in connection with the problem of determining the maximum or minimum of a general function of two variables. We shall treat the problem of determining the extreme values of a quadratic function of two variables, or as we shall say, a quadratic form in two variables, in great detail. There are several important reasons for doing so. In the first place, the three different techniques we employ, algebraic, analytic, and geometric, can all, suitably interpreted, be gen- eralized to apply to the multidimensional cases we consider subsequently. Even more important from the pedagogical point of view is the fact that the algebraic and analytic detail, to some extent threatening in the two- dimensional case but truly formidable in the N-dimensional case, rather pointedly impels us to devise a new notation. A detailed examination of this case thus furnishes excellent motivation for the introduction of new concepts. 2. Maximization of Functions of One Variable. Let f(z) be a real function of the real variable z for z in the closed interval (a,b], and let us suppose that it possesses a convergent Taylor series of the form fo=fo+@- Ot eZ rot w around each point in the open interval (a,b). Let cbe a stationary point of f(z), which is to say a point where f’(z) = 0, and let it be required to determine whether c is @ point at which f(z) is a relative maximum, a relative minimum, or a stationary point of more complicated nature.2 Introduction to Matrix Analysis If c is ® stationary point, the expansion appearing in (1) takes the simpler form fee) = 4 + EFM jue + - @ If f"(¢) = 0, we must consider further terms in the expansion. If, however, f”(c) » 0, its sign tells the story. When fc) > 0, f(2) has a relative minimum at 2 = c; when f"(c) < 0, f(z) has a relative maximum atzr=c EXERCISE Iff"(©) = 0, what are sufficient conditions that c furnish a relative maximum? 8, Maximization of Functions of Two Variables. Let us now pursue the same question for a function of two variables, f(z,y), defined over the rectangle a; < z < bi, a: < y < by, and possessing a convergent Taylor series around each point (¢1,¢:) inside this region. Thus, for |x — cil and |y — ¢y| sufficiently small, we have x Leen @- pu oe Seu) = flener) + (@ — er) 3c: fy Ww - 4) 5. 3e,4 + @ — enty — on 52 ase. a) Here eee nee @) oY asne yaa and 80 on. Let (ci,¢2) be a stationary point of f(x,y) which means that we have the equations Yan He @) Oey 3c, Then, as in the foregoing section, the nature of f(z,y) in the immediate neighborhood of (c,,¢s) depends upon the behavior of the quadratic terms appearing in the expansion in (1), namely, Qa(zy) = ale — 01)? + 2b — er)(y — 2) + ely —cx)* (4) where to simplify the notation we have set of a1 of 2 3 a0t ~ Seide 6)Maximization, Minimization, and Motivation 8 To simplify the algebra still further, let us set z-a=u y-a=v ® and consider the homogeneous quadratic expression Q(u,v) = au* + 2buv + cot ”%) An expression of this type will be called a quadratic form, specifically a quadratic form in the two variables u and ». Although we are interested only in the behavior of Q(u,») in the vicinity of u =v = 0, the fact that Q(u,v) is homogeneous permits us to examine, if we wish, the range of values of Q(u,v) as u and v take all real values, or, if it is more convenient, the set of values assumed on u? + v? = 1, The fact that Q(ku,kv) = k*Q(u,v) for any value of k shows that the set of values assumed on the circle u* + vt = k* is related in a very simple way to the values assumed on u? + »? = 1, Tf Q(u,») > 0 for all u and » distinct from u = v = 0, f(x,y) will have a relative minimum at z = ¢1, y = ¢2; if Q(u,v) < 0, there will be a relative maximum. If Q(u,v) can assume both negative and positive values, we face a stationary point of more complicated type—a saddle point. Although number of quite interesting algebraic and geometric ques- tions arise in connection with saddle points, we shall not be concerned with these matters in this volume. If Q(u,v) is identically zero, the problem is, of course, even more com- plicated, but one, fortunately, of no particular importance. EXERCISE 2. Can the study of the positivity or negativity of a homogeneous form of the fourth degree, Q(u,v) = aou‘ + aiuy + aqutv? + ayuv® + a, be reduced to the study of the corresponding problem for quadratic forms? 4. Algebraic Approach. Let us now see if we can obtain some simple relations connecting the coefficients a, b, and ¢ which will tell us which of the three situations described above actually occurs for any given quadratic form, au? + 2buv + cv?, with real coefficients. In order to determine the sign of Q(u,v), we complete the aquare in the expression au? + 2buy and write Q(u,v) in the following form: aun) = a (ut p+ (eB) x (1) provided that a ¥ 0. If a = 0, but ¢ ~ 0, we carry out the same type of transformation, reversing the roles of u and v. If a=c = 0, then Q(u,v) reduces to4 Introduction to Matrix Analysis Qu. If b ¥ 0, it is clear that Q(u,v) can assume both negative and positive values. If b = 0, the quadratic form disappears. Let us then henceforth assume that a + 0, since otherwise the problem is readily resolved. From (1) it follows that Q(u,v) > 0 for all nontrivial u and v (i.e., for all u and v distinct from the pair (0,0). We shall employ this expression frequently below), provided that Bo a>0 ¢-=>0 (2) Similarly, Q(u,») < 0 for all nontrivial u and v, provided that we have the inequalities a<0 c-—~<0 (3) Conversely, if Q is positive for all nontrivial u and v, then the two inequalities in (2) must hold, with » similar result holding for the case where Q is negative for all nontrivial u and », ‘We have thus proved the following theorem. Theorem 1. A set of necessary and sufficient conditions that Q(u,v) be- positive for all nontrivial u and v ts that ab a>0 2 bl50 (4) Observe that we say a set of necessary and sufficient conditions, since there may be, and actually are, a number of alternative, but, of course, equivalent, sets of necessary and sufficient conditions, We usually try to obtain as many alternative sets as possible, since some are more con- venient to apply than others in various situations. EXERCISE 1, Show that a set of necessary and sufficient conditions that Q(u,v) be positive is that ¢ > 0, ac —B* > 0. 6. Analytic Approach. As noted above, to determine the range of values of Q(u,v) it is sufficient to examine the set of values assumed by Q(u,v) on the circle u* + v* = 1. If Q is to be positive for all nontrivial values of u and v, we must have min Q(u,v) >0 qd) wtetel while the condition max Q(u,v) <0 (2) whpotetMaximization, Minimization, and Motivation 5 is the required condition that Q(u,v) be negative for all u and v on the unit circumference. To treat these variational problems in a symmetric fashion, we employ a Lagrange multiplier. Consider the problem of determining the stationary points of the new quadratic expression R(uv) = aut + 2buv + cvt — r(u? + v4) (3) The conditions aR/du = 8R/év = 0 yield the two linear expressions au + bv — u=0 bu +. — = 0 i) Eliminating u and v from these two equations, we see that ) satisfies the determinantal equation | | b o- 1 =0 6) or MA (ator+ac— dt =0 Oy Since the discriminant is (a +0)" — 4(ac — b8) = (a — 0) + 408 Oy) clearly non-negative, we see that the roots of (6) which we shall call >; and 2s are always real. Unless a = c and b = 0, these roots are dis- tinct. Let us consider the case of distinct roots in detail. If 6 = 0, the roots of the quadratic in (6) are A1 =a, A2=c. In the first case, 4: = a, the equations in (4) are (a—d)u=0 (Av =0 8) which leaves u arbitrary and v = 0, if a ¥c. Since we are considering only the case of distinct roots, this must be so. If b + 0, we obtain the nontrivial solutions of (4) by using one of the equations and discarding the other. Thus u and v are connected by the relation. (a — du = —bv (9) In order to talk about a particular solution, let us add the requirement that u? +»? = 1. This is called a normalization. The values of u and » determined in this way are ty = —0/(0* + (a — d)2)* (10) v1 = (a — &v)/(8 + (a — &))* with another set (u:,v2) determined in a similar fashion when ). is used in place of d1.6 Introduction to Matrix Analysis 6. Analytic Approach—II. Once , and A; have been determined by way of (5.6),! 1; and »; are determined by the formulas of (5.10). ‘These values, when substituted, yield the required minimum and maximum values of au? + 2buv + ce? on u? + vt = 1, It turns out, however, that we can proceed in @ very much more adroit fashion, Returning to the linear equations in (6.4), and multiplying the first by u and the second by », we obtain aut + Qbuy + cot — (ut +o!) = 0 q) This result is not unexpected; it is special case of Euler's theorem con- cerning homogeneous functions, i.e., 1% 4 B29 @ if Q(u,v) is homogeneous of degree 2. Since uj? + v = I, fori = 1, 2, we see that Dr = aus? + 2buys + ov? (3) Ya = aug? + Qbuae + cvs? Hence one solution of the quadratic equation in (6.6) is the required maximum, with the other the required minimum. We observe then the remarkable fact that the maximum and minimum values of Q(u,v) can be obtained without any explicit calculation of the points at which they are obtained. Nonetheless, as we shall see, these points have important features of their own, Let us now derive an important property of the points (uj,%), still without using their explicit values. As we have seen in the foregoing sections, these points are determined by the sets of equations au; + by, — Ati = 0 atta + bos — ate = 0 bus + ov: — Dats = 0 @) ut tots) Considering the first set, we have, upon multiplying the first equation by ‘tus, the second by v2, and adding, tus(ates + boy — dates) + va(bur + cv: — dani) = atytls + b(uas + uve) + cos — Ar(uits + v2) = 0 (5) Similarly, the second set yields auyus + bus + uss) + cows — da(usa + 0102) = 0 (3) ‘Double numbers in parentheses refer to equations in another section of the chapter.Maximization, Minimization, and Motivation 7 Subtracting, we have Qs — As) (urtle + v2) = 0 (7) Since A, » Xz (by assumption), we obtain the result that ut, + yr = 0 @) The geometric significance of this relation will be discussed below, Let us also note that the quantity av: — uss is nonzero. For assume that it were zero. Then, together with (8), we would have the two linear equations in u, and 9, uit, + vive = 0 uv: — vu = 0 @ Since u; and v, are not both zero, a consequence of the normalization conditions in (4), we must have the determinantal relation Ur on Ye. us 0 (10) or us? + v2? = 0, contradicting the last relation in (4). EXERCISE 1. Show that for any two sets of values (u:,us) and (v1,0:) we have the relation (unt + ort) (ust + vat) = Caius + vans)? + (ties — uni), and thus again that uv: — ust » 0 if the u; and v; are as above. 7. A Simplifying Transformation. Armed with a knowledge of the properties of the (u,v;) contained in (6.4) and (6.8), let us see what hap- pens if we make the change of variable u= uu + uv! v= ou! + vw! 0 This is 2 one-to-one transformation since the determinant uiv2 — ui is nonzero, as noted at the end of Sec. 6. In the first place, we see that ut vt = (us? + vita? + (us? + v2t)v™ + 2(urue + viv2)u'o’ = ul? $v? (2) It follows that theset of values assumed by Q(u,v) onthecircleu? + v? = 1 is the same as the set of values assumed by Q(uru’ + us’, vu’ + vw’) on the circle u’? + v’* = 1. Let us now see what the expression for Q looks like in terms of the new variables, We have, upon collecting terms,8 Introduction to Matrix Analysis Q(usu! + uw’, vu! + ve’) = (aus? + 2buyw, + cvt)u’? + (aus? + Buss + cvst)o’? + 2(aurus + b(uiva + wars) + cors)u’r’ (3) Referring to (6.3), (6.6), and (6.8), we see that this reduces to au’? + Aw’? « The effect of the change of variable has been to eliminate the cross- product term 2buv. This is quite convenient for various algebraic, analytic and geometric purposes, as we shall have occasion to observe in subsequent chapters where @ similar transformation will be applied in the multidimensional case. As a matter of fact, the principal part of the theory of quadratic forms rests upon the fact that an analogous result holds for quadratic forms in any number of variables. EXERCISE 1, Referring to (4), determine the conic section described by the equation Q(u,») = 1 in the following cases: @) 1 >0,%>0 () > 0, <0 (©) Mar > 0 @ 1 =0,%>0 8. Another Necessary and Sufficient Condition. Using the preceding representation, we see that we can make the following statement, Theorem 2. A necessary and sufficient condition that Q(u,v) be positive for all nontrivial u and » is that the roots of the determinantal equation a-X 6b bc a be positive. EXERCISE 1, Show directly that the condition stated above is equivalent to that given in Theorem 1. 9, Definite and Indefinite Forms. Let us now introduce some termi- nology. If Q(u,0) = au? + 2buv + co > 0 for all nontrivial u and v, we shall say that Q(u,v) is positive definite, If Q(u,v) <0 for these values of u and o, we shall call Q(u,v) negative definite. If Q(u,v) can be of either sign, we shall say that it is indefinle. If we merely have the inequality Q(u,») > 0 for all nontrivial u and v, we say that Q is non-negative definite, with non-positive definite defined analogously, Oocasionally, the term positive indefinite is used in place of non-negative definite.Maximization, Minimization, and Motivation 9 EXERCISES 1, Show that if aut + 2biuw +c and au? + 2dsuw + cw" are both positive definite, then aya.u* + 2bibuv + cia" is positive definite, &. Under what conditions is (a,u; + asus)? + (biu: + dius)? positive definite? 8. In terms of the foregoing notation, how can one tell whether aut + 2buo + cot = 1 representa an ellipse, a hyperbola, or # parabola? 10. Geometric Approach. Let us now consider a variant of the fore- going method which will yield a valuable insight into the geometrical significance of the roots, \ and Xs, and the values (1,0,). Assume that the equation aut + 2buy + oot = a) represents an ellipse, as pictured: " tay) ‘The quantity r = (u* + v*)* denotes the length of the radius vector from the origin to a point (u,v) on the ellipse. Let us use the fact that the problem of determining the maximum of Q(u,v) on u? + vt = 1 is equivalent to that of determining the minimum of u® + v? on the curve Q(u,v) = 1. The Lagrange multiplier formalism as before leads to the equations u— d(au + bv) =0 » — Ou + cv) = 0 @ These yield the equation 1-ay dd -n 1-a/7° @) or a-} 8 1/=0 ® db e-x x If 4; is @ root of this equation and (u,v,) the corresponding extremum point, we see as above that Ut + oF = de (6)10 Introduction to Matrix Analysis From this we conclude that one root of (4) is the square of the minimum distance from the origin to the ellipse and that the other is the square of the maximum distance. We observe then that the variational problem we have posed yields in the course of its solution the lengths of the major and minor axes of the ellipse. The condition of (6.8) we now recognize as the well-known perpendicularity or orthogonality of the principal axes of an ellipse. The linear transformation of (7.1) is clearly a rotation, since it pre- serves both the origin and distance from the origin. We see that it is precisely the rotation which aligns the coordinate axes and the axes of the ellipse. EXERCISES 1. From the foregoing facts, conclude that the area of the ellipse is given by /(ac — 6), 2, Following the algebraic approach, show that necessary and sufficient conditions that the form Q = au? + 2buv + cot + 2duw + ewt + 2fow be positive definite are that and bes dfe >0 ab a>0 le t]>o 8. Similarly, following the analytic approach, show that a necessary and sufficient condition that @ be positive definite is that all the roots of the determinantal equation =k db a bo c-d ff [mo qd fe be positive. ‘4. If Q is positive definite, show that the equation Q(u,v,w) = 1 represents an ellipsoid and determine its volume. 11. Discussion. We have pursued the details of the two-dimensional case, details which are elementary but whose origins are perhaps obscure, in order that the reader will more readily understand the need for a better notation and appreciate its advantages, The results which seem 0 providential here, and which have essentially been verified by direct calculation, will be derived quite naturally in the general case. The basic ideas, however, and the basic devices are all contained in the preceding discussions. MISCELLANEOUS EXERCISES 1. For what values of 2 and 2: is the quadratic form (1121 + aut: — bs)* + (an.2 + a2:¢2 — bs) a minimum, and what is the minimum value? 2 Show that (a? + y.*)(z:* + y:*) can be written in the form Gayaite + asziys + artayr + aayrye)* + (bitite + detrys + ditrys + days)®Maximization, Minimization, and Motivation oy for values of a; and b; which are independent of 2; and ys, and determine all such values. 3. Show that there existe no corresponding result for (et tut tant) eat + yet + 224) 4. If zu? + 2zyuv + 202 and yu! + 2y.uv + yo! are positive definite, then ah ain > a 2| |x | ‘Taya Tas t. tal [ya Ye 5. Utilize this result to treat Exercise 1 of Seo. 9. Establish the validity of the Lagrange multiplier formalism by considering the maximum and minimum values of (au? + 2buv + ev!)/(u! + 0), 7. What linear transformations leave the quadratic form Q(z,,23) = Mz? + 23!) + (1\- (1 + 2)? invariant?” HereO mY @) i The following properties of the inner product are derived directly from the definition: (zy) = (yz) (2a) (a + y, 2 + w) = (x2) + (ww) + (2) + (yw) (2b) (exz,y) = ex(2,y) (2c) ‘This is one way of “multiplying” two vectors. However, there are other ways, which we shall not use here. The importance of the inner product lies in the fact that (2,2) can he considered to represent the square of the “length” of the real vector z. We thus possess a method for evaluating these non-numerical quantities, EXERCISES A. If z is real, show that (z,z) > 0 unless z = 0. 2 Show that (uz + oy, uz + vy) = ut(z,z) + 2uv(z,y) + *(y,y) is @ non-negative definite quadratie form in the ecalar variables u and v if z and y are real and hence that @u)* S @z)(yv) for any two real vectors 2 and y (Cauchy’s inequality),Vectors and Matrices 15 8. What is the geometric interpretation of this result? 4. Using this result, show that (ety, 2 + ys S (a2) + (yy) for any two real vectors. 5. Why is this inequality called the “triangle inequality”? 8. Show that for any two complex vectors z and y we have lawl $ @DOn 6. Orthogonality. If two real vectors are connected by the relation (z,y) = 0 () they are said to be orthogonal, The importance of this concept derives to a great extent from the following easily established result. Theorem 1, Let {2'} be a set of real vectors which are mutually orthogo- nal; that is, (z',2!) = 0, ¢ » j, and normalized by the condition that @r) =1 Then if a particular vector x has the representation aM z= ) cat (2) 2 we have the following values for the coefficients c= (az) (3) and, in addition, M (oa) = Y ct “ & EXERCISES 1. Why in the proper concept of orthogonality for complex vectors the relation @p) = 0 and the proper normalization the condition (2,2) = 1? 2. Show that (2,9) = Ty)- 8. In N-dimensional Euclidean space, consider the coordinate vectors 1 0 0 0 1 o 0 0 0 ve vale wm16 Introduction to Matrix Analysis Show that the y‘are mutually orthogonal, and are, in addition, normalised. A sat of this type is called orthonormal, If z = d cay’, determine the values of the ci, and i=1 discuss the geometric meaning of the result, 7. Matrices. Let us now introduce the concept of a mairéz, An array of complex numbers written in the form an an + -* aw ay as + Gay A=) w On, Gwe *'* Gyn, will be called a square matriz, Singe these are the only types of matrices we shall consider to any extent, we shall reserve the term “matrix” for these entities. When other types of matrices are introduced subse- quently, we shall use an appropriate modifying adjective. The quantities aj are called the elements of A and the integer N is the dimension. The quantities aj, ai, . - . , au are said to constitute the ith row of A, sind the quantities ay, ¢xy, . . . , ay; are said to constitute the jth column, Throughout, matrices will be denoted by upper-case letters, X, Y, Zand A, B,C, From time to time we shall use the short- hand notation 4 = (ay) (2) The determinant sssooiated with the array in (1) will be written JA] or Jay]. The simplest relation between matrices is that of equality ; two matrices are called equal if and only if their elements are equal. Following the same path as for vectors, we next define the operation of addition. The sum of two matrices A and B is written A +B and defined by the notation A+B = (ay + bs) 3) Multiplication of a matrix A by a scalar c; is defined by the relation aA = Ac = (c1ay) (4) Finally, by A we shall mean the matrix whose elements are the com- plex conjugates of A. When the elements of A are real, we shall call A a real matrix. EXERCISES 1. Show that 0, the null matrix defined by the condition that all its elements are sero, is uniquely determined by the condition that 4 +0 = A for all 4.Vectors and Matrices td 4. Show that matrix addition is commutative and associative. 8. Show that Ai +4: + --- + Aw ia unambiguously defined. 8. Matrix Multiplication—Vector by Matrix. In order to make our algebra more interesting, we shall define some multiplicative operations. In order to render these concepts reasonable, we return to linear trane- formations of the form x ot, 1 1,2,...,N w a1 where the coefficients a, are complex quantities, Since, after all, the whole purpose of our introduction of vector-matrix notation is to facilitate the study of these transformations, it is only fair that in deriving the fundamental properties of vectors and matrices we turn occasionally to the defining equations for guidance. ‘The point we wish to make is that the definitions of addition and multi- plication of vectors and matrices are not arbitrary, but, on the contrary, are essentially forced upon us by the analytic and geometric properties of the entities we are studying. Given two vectors 2 and y related as in (1), we write y= Ax (2) This relation defines multiplication of a vector z by a matrix A. Observe carefully the order in which the product is written, EXERCISES 1, Show that (4 + B)(t +y) = Az + Ay + Bx + By. 2. Consider the identity matriz I defined by the tableau Explicitly, J = (8s), where 8; is the Kronecker delta symbol defined by the relation &e m0, ty =i sj Show that = by =) Biadey oy18 Introduction to Matrix Analysis 8. Show that Zz = zor all x, and that this relation uniquely determines I. 4. Bhow that (4,42) = 5 ( 5 ouei)* me 9. Matrix Multiplication—Matrix by Matrix. Now let us see how to define the product of a matrix by a matrix. Consider a second linear transformation 2= By @ which converts the components of y into the components of z. In order to express the components of z in terms of the components of z, where, a8 above, y = Az, we write x w= Sone $00(S an) wer Ni = | CG bias) 25 @) If we now introduce a new matrix C = (c,) defined by the relations x ey =) buay 4f=1,2...,0 @) m we may write z= Cz (4) Since, formally, 2 = By = B(Az) = (BA)z © we are led to define the product of A by B, C=BA @) where C is determined by (3). Once again, note carefully the order in which the product is written. EXERCISES 1. Show that (A + B)(C + D) = AC + AD + BC + BD, a. If cos 8 — 7 = (se 6 cos “ show that T(6:)T (6s) = T(6s)T (01) = T(6 + 01) % Let A be a matrix with the property that as; = 0, if j »%, diagonal matrix, and let B be a matrix of similar type. Show that AB is again a diagonal matrix, and that AB = BA,Vectors and Matrices 19 4. Let A be a matrix with the property that a; = 0, j > i, a triangular or semi diagonal matrix, and let B be a matrix of similar type. Show that AB is again a triangular matrix, but that AB BA, in general. 6. Let a ay by by od a ed Show that AB is again a matrix of the same type, and that AB = BA. (As we shall subsequently see, these matrices are equivalent to complex numbers of the form a; + iay if a and a3 are real.) 6, Use the relation |AB| = |A| |Bj to show that (ait + a*)(b:* + ba?) = (@ib, — asbs)* + (axbr + aibs)* 1. Let af tt ios a: + iae 4 [3 tia a - il and B be a matrix of similar type. Show that ABis a matrix of similar type, but that AB x BA in genet 8 Use the relation |AB| = |A| |B| to express (a;* + as! + as? + a4)(b:" + bs? + bst + be) as a eum of four squares. 9. Let, 4 a a-l7 a, -a ay 0, a a of —aq a) -q ay and B be a matrix of similar type. Show that 4B is a matrix of similar type, but that, AB # BAingeneral. Evaluate |A|. (These matrices are equivalent to quaternions,) 10, Consider the linear fractional transformation a similar transformation, with coefficients a1, bi, cx, d replaced by as, bs, ¢2, da, show that T,(T2(e)) and 72(7(z)) are again transformations of the same type. 11. If the expression aid; — bc; » 0, show that 71-1(2) is a transformation of the same type, If aid — bic, = 0, what type of a transformation is Te)? 12. Consider a correspondence between T(z) and the matrix a by Arn [: a] written A, ~ T,(). Show thatif Ai ~ T,(z) and As ~ T(z), then Aids ~ Ti(Ta(e)). What then is the condition that 7y(T2(z)) = T1(T1(z)) for all 2? 18. How can the foregoing results be used to obtain a representation for the iterates of Te)? [of 14. Show that (N37 a + 10. If X = (ou), where 24 = (—"-# (TZ) (the binomial coefficient), then Diet20 Introduction to Matrix Analysis 16. From the fact that we can establish a correspondence 10 01 1~[ Hl =~ al deduce that arithmetic can be carried out without the aid of negative numbers. 10. Noncommutativity, What makes the study of matrices so fasci- nating, albeit occasionally thorny, is the fact that multiplication is not commutative. In other words, in general, AB#BA (1) A simple example is furnished by the 2 X 2 matrices 12 21 a-[} 2] 2-23] @ and more interesting examples by the matrices appearing in Exercises 7 and 9 of Sec. 9. Then, [$2] ~-[88] If AB = BA, we shall say that A and B commute. The theory of matrices yields « very natural transition from the tame domain of ecalars and their amenable algebra to the more interesting and realistic world where many different species of algebras abound, each with its own singular and yet compensating properties. EXERCISES 1, Show that AJ = [A = A forall A, and that this relation uniquely determines the identity matrix, 2. Bhow that AO = 0A = 0 for all A, and that this relation uniquely determines the null matrix. 8. Let the rows of A be considered to consist, respectively, of the components of the vectors ai, a, . .., a%, and the columns of B to consist of the components of bb... BN. Then we may write AB = [(atb5)} 6. If AX = XA for all X, then A is 8 scala: multiple of I. 11. Associativity. Fortunately, although commutativity does not hold, associativity of multiplication is preserved in this new algebra. In other words, for all A, B, and C we have (AB)C = A(BC) (1) This means that the product ABC is unambiguously defined without theVectors and Matrices 21 aid of parentheses. ‘To establish this result most simply, let us employ the “dummy index” convention, which asserts that any index which is repeated is to be summed over all of its admissible values. The ith element in AB may then be written Gabry @) Employing this convention and the definition of multiplication given above, we see that (AB)C [(aabseu] 30 abui)eu (3) [aix(bssey)] which establishes the equality of (AB)C and A(BC). EXERCISES 1. Show that ABCD = A(BCD) = (AB)(CD) = (ABC)D, and yenerally that Ay, «+» Ay hea a unique significance. 2. Show that A" = 4 - A (n times) A is unambiguously defined, and that A» = AmAn mn = 1,2,.... 8. Show that 4 and B* commute if A and B commute. 4. Write xem [2G noo) meex= [2 2] a given matrix. Using the relation X**! = XX*, derive recurrence relations for the z\(n) and thus derive the analytic form of the 2:(n). 5. Use theae relations for the case where x= [2 2] sane or x -|3 7] ,, 24 complex 6. Use these relations to find explicit representations for the elements of X* where A 10 x= [3 i] X=foxt oo% 1, Find all 2 X 2 matrix solutions of X* = X. 8. Find all 2 X 2 matrix solutions of X* = X, where n is a positive integer. 12, Invariant Vectors. Proceeding as in Seo. 5 of Chap. 1, we see that the problem of determining the maximum and minimum values of N x Q =} axtsa; for 2; satisfying the relation ) 2¢ = 1 can be reduced to wet22 Introduction to Matrix Analysis the problem of determining the values of d for which the linear homoge- neous equations x Y, aves = das =1,2,..., q) - possess nontrivial solutions. In vector-matrix terms, we can write these equations in the form of a single equation Az = dz (2) Written in this way, the equation has a very simple significance. We are looking for those vectors x which are transformed into scalar multiples of themselves by the matrix A. Thinking of x as representing a direction indicated by the N direction numbers 21, 22, . . . , zw, we are searching for invariant directions, We shall pursue this investigation vigorously in the following chapters. In the meantime, we wish to introduce a small amount of further nota- tion which will be useful to us in what follows. 18. Quadratic Forms as Inner Products. Let us now present another justification of the notation we have introduced. The quadratic form Q(uv) = aut + 2buv + co? can be written in the form u(au + bv) + v(bu + ev) a) Henee, if the vector z and the matrix A are defined by u _fao e-[] 4-6 1] ® we see that Q(u,v) = (2,Az) (3) Similarly, given the N-dimensional quadratic form OG) = oe exerts @ aren where without loss of generality we may take ay = ag, we can write § x x Q@) = x [ ), wai] +a D> ayes] + +++ tan [) anes] dat = @,A2) (6) where z has the components 2 and A = (a4).Vectors and Matrices Ps) EXERCISES 1. Does (2,42) = (2,Bz) for all z imply that A = B? 2. Under what conditions docs (z,Az) = 0 for all z? 14. The Transpose Matrix. Let us now define a most important matrix function of A, the transpose matrix, by means of the relation A’ = (ay) @ ‘The rows of A’ are the columns of A and the rows of A are the columns of A’. We are led to consider this new matrix in the following fashion. Con- sider a set of vectors {x} and another set {y}, and form all inner products (z,y) composed of one vector from one set and one from the other. Suppose now that we transform the set {2} by means of matrix multi- plication of A, obtaining the new set {Az}. Forming inner products a8 before, we obtain the set of values (Az,y). Observe that (Aay) = 1 ip) ayes] + wf S Ce mb) ovat] (2) aaa jaa amt or, rearranging, (Azy) = a [3 aay} +2 [3 anys] tot eS anys] 1 it ist = (2,A'y) (3) In other words, as far as the inner product is concerned, the effect of the transformation A on the set of 2’s is equivalent to the transformation A‘ on the set of y’s. We can then regard A’ as an induced transformation or adjoint transformation. This simple, but powerful, idea pervades much of classical and modern analysis. 15. Symmetric Matrices. From the foregoing discussion, itis plausible that matrices satisfying the condition A=A! () should enjoy special properties and play an important role in the study of quadratic forms. This is indeed the case. ‘These matrices are called symmetric, and are characterized by the condition that ay; = Oye (2) The first part of this volume will be devoted to a study of the basic properties of this class of matrices for the case where the ay are real.4 Introduction to Matrix Analysis Henceforth, we shall use the term “symmetric” to denote real sym- metric. When there is any danger of confusion, we shall say real sym- metric or complex symmetric, depending upon the type of matrix we are considering. EXERCISES 1, Show that (A’)’ = A. 2. Show that (A + B)’ = AsAY, (AN! = (A, 8 Show that AB is not necessarily symmetric if A and B are. 4. Show that A’BA is symmetric if B is symmetric. 8. Show that (Az,By) = (z,4’By). 6. Show that [A] = |A’|. ' + BY, (ABY = BYA’, (AAs... An) AL... ” 1. Show that when we write Qlz) =) uzun therein no loss of generality in aget assuming that ay = ay. 16. Hermitian Matrices. As we have mentioned above, the important scalar function for complex vectors turns out not to be the usual inner product, but the expression (z,9). If we note that (Az,g) = (22) 1) where z = Ay, we see that the induced transformation is now 4’, the complex conjugate of the transform of A. Matrices for which A= (2) are called Hermitian, after the great French mathematician Charles Hermite. We shall write A* in place of A’ for simplicity of notation. As we shall see, all the vital properties of symmetric matrices have immediate analogues for Hermitian matrices. Furthermore, if we had wished, we could have introduced a notation {zu] = @9) @) in terms of which the properties of both types of matrices can be derived simultaneously. There are advantages to both procedures, and the reader can take his choice, once he has absorbed the basic techniques. EXERCISES 4. A real Hermitian matrix is symmetric. 2. (A*)* = A, (AB)* = BAA, (AsAL AARVectors and Matrices 25 17. Invariance of Distance—Orthogonal Matrices. Taking as our guide the Euclidean measure of distance, we introduced the quantity (z,z) as a measure of the magnitude of the real vector 7. It is a matter of some curiosity, and importance too as we shall see, to determine the linear transformations y= Tz which leave (z,z) unchanged, In other words, we wish to determine 7’ so that the equation (x,2) = (Tz,72) q) is satisfied for all x. Since (Tz,T2) = (2,T'Tz) @) and 1°T is symmetric, we see that (1) yields the relation TT =1 @) A real matrix 7’ possessing this property is called orthogonal, EXERCISES 1, Show that 7” is orthogonal whenever T is. 2. Show that every 2 X 2 orthogonal matrix with determinant +1 can be written in the form [ 6 — sine ine cme What is the geometrical significance of this reault? 8. Show that the columns of 7 are orthogonal vectors, 4. Show that the product of two orthogonal matrices is again an orthogonal matrix 5. Show that the determinant of an orthogonal matrix is +1. 6. Let Tw be an orthogonal matrix of dimension N, and form the (N + 1)-dimen- sional matrix 10+ 0 a Teo] oy oO Show that T'y,: is orthogonal. 7. Show that if T is orthogonal, z = Ty implies that y = T’z. 8. If AB = BA, then TAT’ and TBT’ commute if T is orthogonal, 18. Unitary Matrices. Since the appropriate measure for a complex vector is (2,2), we see that the analogue of the invariance condition of (17.3) is TT =! oy Matrices possessing this property are called unitary, and play the same role in the treatment of Hermitian matrices that orthogonal matrices enjoy in the theory of symmetric matrices.26 Introduction to Matrix Analysis EXERCISES 1. Show that 7° is unitary if T is. 2. Show that the product of two unitary matrices is again unitary. 3. Show that the determinant of a unitary matrix has absolute value 1. 4. Show that if 7 is unitary, x = Ty implies that y = T*z. 6. Obtain a result corresponding to that given in Exercise 2 of Sec. 17 for the ele- ments of a 2 X 2 unitary matrix. (Analogous, but more complicated, results hold for the representation of the elements of 3 X 3 orthogonal matrices in terms of elliptic functions, See F. Caspary, Zur Theorie der Thetafunktionen mit zwei Argu- menten, Kronecker J., XCIV, pp. 74-86; and F. Cespary, Sur les systémes ortho- gonaux, formés par les fonctions théte, Comptes Rendus de Paria, CIV, pp. 490-493.) 6. Is a real unitary matrix orthogonal? 7. Is a complex orthogonal matrix unitary? 8. Every 3 X 3 orthogonal matrix can be represented as the product of cosa sina O|[cosb 0 -sind 1 o 0 —sina cosa 0 0 i 0 {JO cose sine o 0 ijlsind’ 0 cos b ine cose. What is the geometric origin of this result? MISCELLANEOUS EXERCISES 1. Prove that an ay +++ aw x an an sss ay as = Glatt (0 alee + +) Ov, Gwe +++ GNN 2 a m +++ gy 0 where afi is the cofaetor of ai; in fass|. 2. Let 1;; denote the matrix obtained by interchanging the ith and jth rows of the unit matrix I. Prove that Tal Talesdye = lay 8. Show that I;;A is a matrix identical to 4 except that the ith and jth rows have been interchanged, while AJ,, yields the interchange of the ith and jth columns. 4. Let Hi; denote the matrix whose ijth element is h, and whose other elements are zero, Show that (I + Hy)A yields a matrix identical with A except that the ith row hes been replaced by the ith row plus A times the jth row, while A(I + Hii) has a similar effect upon columns, 5. Let H, denote the matrix equal to I except for the fact that the element one in the rr position is equal to k. What are H,A and AH, equal to? 6. If A ig real and AA’ =0, then A = 0. 7. It AA* = 0, then A = 0. 8. Show that if 7 is orthogonal, its elements are uniformly bounded, Similarly, if U is unitary, its elements are uniformly bounded in absolute value. 9 Let d: denote the determinant of the aystem of linear homogeneous equations derived from the relationsVectors and Matrices 27 [San] [Fae] =o 4k=1,%..5,N yet vat regarding the N(N + 1)/2 quantities ziz,, i,j = 1,2, ...,N,asunknowns. Then dy = [ag|¥O-O4 (Scha fi). 10. Show lim 4*, lim Be may exist ss n> ©, without lim (AB)* existing, It is sufficient to take A and B two-dimensional. x 11, Suppose that we are given the system of linear equations ay2; = by mm £=1,2,...,N, If it is possible to obtain the equations 2; = cy, 2 = 01, ..- zy = ew, by forming linear combinations of the given equations, then these equations yield a solution of the given equations, and the only solution (R. M. Robinson). 12, Introduce the Jacobi bracket symbol {4,B] = AB — BA, the commutator of A and B, Show, by direct calculation, that {4,{B,Cll + {B,{C,All + (,{4, Bl] = 0 18, Let ri = et"! be an irreducible root of unity, and let ry = et, k = 1, 2, +++)m— 1, Consider the matrix bdo d Lon tee ret Ts Dore tt teat naa n= Show that T/nMis unitary. Hence, ifs =) ety, then ys ) ertesingy, So not n=l and > Ina an > lval?. ‘This transformation is called a finite Fourier transform, =o ao 24, Suppose that jhe sy ¥ 1 wien = So os{ mor {sa} with bi = dig = 1for all, Then lai] = [] cx. See J. L. Burchnall.t be Consider the Gauss transform B = (bj) of the matrix A = (aij), bn = dna, be = an ana — anon), &>1 Let An = (ac), 4, § = 2,...,N. Show that I = Bl = au {alal = Anil — |Z - Al) (D. M. Kotelyanskis) + J. L, Burchnall, Proc. Edinburgh Math, Soc., (2), vol. 9, pp. 100-104, 1954.28 Introduction to Matrix Analysis 16. Show that the matrices Lia b 1 a, -b) A=]01 0 Bemjo 1 0 oot oo 1 satisfy the relation AB = I. 17, Show that an al am wo By = toys + ender + ays + wes) (e + tyr + wa) aa tat tant — Saye where w isa cube root of unity, 18. Hence, show that if Q(z) = 21° + 7:' + 23 — 3zizsts, then Q(@)Q(V) = Q(e), where ¢ is » bilinear form in the 2 and yi, that is, «: = Zaiyst;ys, where the coefficients ‘css ate independent of the 2 and y;, 19. Show that 1 om om TRO OR TR Blea tas tart pee ee ee) and thus that Get pat past tad tat +t + yd)a at tat tat ted, where the 2 are bilinear forms in the x and y:. It was shown by Hurwits that a produet of N squares multiplied by product of N equares is a product of N aquares in the above sense, only when N = 1, 2, 4, 8; see A. Hurwitz.'' For an exposition of the theory of reproducing forms, see C. C, MacDutfee.*.* 20. Prove that a matrix A whose elements are given by the relations au Megat tej =(-pe inj =0 t>j satisfies the relation At = I. Here () is the binomial coefficient, nl/kl(n — k)1. 21, Let ys = ys(ats, . . . ,2w) be a set of N functions of the xi, i = 1, 2, aN. ‘The mateix J = J(y,2) = (@y:/4z;) is called the Jacobian matrix and ite determinant the Jacobian of the transformation, Show that Jew ug) = Se2) 22. Consider the relation between the N* variables ys; and the N* variables iy given by ¥ = AXB, where A and B are constant matrices, Show that |J(¥,X)| = [ABI 1 A, Hurwitz, Uber die Komposition der quadratiachen Formen von beliebig vielen iablen, Math. Werke, bd It, Basel, 1933, pp. 565-571, C. C, MacDuffee, On the Composition of Algebraic Forms of Higher Degree, Bull, Amer, Math, Soc., vol, 1 (1945), pp. 198-211, +J, Radon, Lineare Scharen orthogonaler Mat 1 (1921), pp. 1-14, vi , ABA. Math. Sem, Hamb., vol.Vectors and Matrices 29 ” 29. If ¥ = XX", where X is triangular, then J(¥,X)| = 2 [] 2u*” int 24. Show that the problem of determining the maximum of the funetion (z, Az) — 2(2,b) leads to the vector equation Az = b, 26. Similarly, show that the problem of minimizing (z,Bz) + 2(2,Ay) + (y,By) — 2(a,2) — 2(,y) over ali z and y leads to the simultaneous equations Bz + Ay =a, A's + By =b. 26. Let f(z) be a function of the variable which assumes only two values, = 0, z= 1. Show that f(a) may be written in the form a+ bz, where a = (0), b= 10) — $0). 27. Let g(z) be a function of the same type, which itself assumes only two values 0 of 1. Then f(g(2)) = a: + bz. Show that a; and 6; are linear combinations of cand 6, and thus that the effect of replacing = by g(z) is equivalent to a matrix trans- formation of the vector whose components are a and b, 28. Let f(21,21) be a function of the two variables z; and 2 each of which assumes only the values Gand 1, Show that we may write f(z1,21) = a1 + ast + ass + anit 29. Let 9x(21,21), (21,21) be functions of the same type, each of which itself assumé only two values, 0 or 1, Then if we write Songs) = 0) + ayes + ayaa + aziz we have a matrix transformation +, ay a ay] ag | 4 ay a a For the ease where g1(#1,21) = 2121, g2(71,22) = 21(1 — #1), evaluate M. 80. Generalize the foregoing results to the case where we have a function of N variables, (zis, . . . ,tw), and to the case where the z; can assume any of a given finite act of values, (The foregoing results are useful in the study of various types of logical nets; see, for example, R. Bellman, J. Holland, and R. Kalaba, Dynamic Programming and the Synthesis of Logical Nets, J. Aesoc. Comp. Mach., 1959.) 81. If A is a given 2 X 2 matrix and X an unknown 2 X 2 matrix, show that the equation AX — XA = I’has no solution. 382. Show that the same result holds for the case where A and X are N x N matrices. 88. Consider the relation between the N(N + 1)/2 variables yi; andthe N(N + 1)/2 variables 2; given by Y = AXA’, where X and Y are symmetric. Show that \(Y,X)| = 1A/¥*+1, Two semiexpository papers discussing matters of this nature have been written by W. L. Deemer and I. Olkin.! $4. Construct a 4X4 aymmetric orthogonal matrix whose elements are £1, ‘This question, and extensions, is of interest in connection with Hadamard’s inequality, see Sec. 7 of Chap. 8, and (amazingly!), in connection with the design of experimenta. See R. E, A. C. Paley? and R, L. Plackett and J, P. Burman,* 1 W. L. Deemer and I. Olkin, Jacobians of Matrix Transformations Useful in Multi- variate Analysis, Biometrika, vol. 38, pp. 345-367, 1951. I. Olkin, Note on “Jacobians of Matrix Transformations Useful in Multivariate Analysis,” Biometrika, vol. 40, pp. 43-46, 1953. +R, E. A.C, Paley, On Orthogonal Matrices, J. Math. and Physics, vol. 12, pp. 311-320, 1933, *R. L, Plackett and J. P. Burman, The Design of Optimum Multifactorial Experi- ments, Biometrika, vol. 33, pp. 305-325, 1948.30 Introduction to Matrix Analysis Bibliography The reader who is interested in the origins of matrix theory is urged to read the monograph by MacDuffee C. C. MacDuffee, The Theory of Matrices, Chelsea Publishing Co.. New York, 1946. Although we shall occasionally refer to various theorems by the names of their discoverers, we have made no serious attempt to restore every result, to its rightful owner. As in many other parts of mathematics, there is a certain amount of erroneous attribution. When aware of it, we have attempted to rectify matters. §8. It should be pointed out that other physical situations may very well introduce other types of algebras with q different “addition” and “multiplication.” As I. Olkin points out, in the study of socio- metrics, a multiplication of the form A+B = (ayby) is quite useful. Oddly enough, as will be seen subsequently, this Schur product crops up in the theory of partial differential equations. $10. For an extensive discussion of commutativity, see 0. Taussky, Commutativity in Finite Matrices, Amer. Math. Monthly, vol. 64, pp. 229-235, 1957. For an interesting extension of commutativity, see B. Friedman, n-commutative Matrices, Math. Annalen, vol. 136, 1958, pp. 343-347. For ® discussion of the solution of equations in X of the form ‘A = AX, XA = A’X, see H. 0. Foukes, J. London Math. Soc., vol. 17, pp. 70-80, 1942, $11. Other algebras can be defined in which commutativity and associa tivity of multiplication both fail. A particularly interesting example is the Cayley algebra. For a discussion of these matters, see A.A. Albert, Modern Higher Algebra, University of Chicago Press, Chicago, 1937. G. A. Birkhoff and 8. Maclane, Survey of Modern Algebra, The Macmillan Company, New York, 1958.Vectors and Matrices 31 814. In the case of more general transformations or operators, A” is often called the adjoint transformation or operator. ‘The reason for its importance resides in the fact that occasionally the induced transforma- tion may be simpler to study than the original transformation. Further- more, in many cases, certain properties of A are only simply expressed when stated in terms of A’. In our discussion of Markoff matrices in Chap. 14, we shall see an example of this. $16. The notation H* for ” appears to be due to Ostrowski: A. Ostrowski, Uber die Existenz einer endlichen Basis bei gewissen Funktionensystemen, Math. Ann., vol. 78, pp. 94-119, 1917. For an interesting geometric interpretation of the Jacobi bracket identity of Exercise 12 in the Miscellaneous Exercises, see W. A. Hoffman, Q. Appl. Math., 1968. Some remarkable results have been obtained recently concerning the number of multiplications required for matrix multiplication. See 8. Winograd, The Number of Multiplications Involved in Computing Certain Functions, Proc. IFIP Congress, 1968. For some extensions of scalar number theory, see F. A. Ficken, Rosser’s Generalization of the Euclidean Algorithm, Duke Math, J., vol. 10, pp. 355-379, 1943. A paper of importance is M. R. Hestones, A Ternary Algebra with Applications to Matrices and Linear Transformations, Archive Rat, Mech, Anal,, vol. 11, pp. 138-194, 1962.3 Diagonalization and Canonical Forms for Symmetric Matrices 1, Recapitulation. Our discussion of the problem of determining the x stationary values of the quadratic form Q(z) = _). ayaa; on the sphere aged ay? + 2y!-+ ++ + + 2yt = 1 led us to the problem of finding nontrivial solutions of the linear homogeneous equations x Y ayes oe 821,206.50 @ a) We interrupted our story at this point to introduce vector-matrix notation, stating in extenuation of our excursion that this tool would permit us to treat this and related questions in a simple and elegant fashion. Observe that the condition of symmetry ay = aj is automatically satis- fied in (1) in view of the origins of these equations. This simple but significant property will permit us to deduce a great deal of information concerning the nature of the solutions. On the basis of this knowledge, we shall transform Q(z) into a simpler form which plays » paramount role in the higher theory of matrices and quadratic forms. Now to resume our story! 2, The Solution of Linear Homogeneous Equations. We require the following fundamental result. Lemma. A necessary and sufficient condition that the linear system x Y bua mA possess a nontrivial solution is that we have the determinantal relation [bis] = 0 @) 32 0 t-12. =.= (tyDiagonalization and Canonical Forms 38 As usual, by “nontrivial” we mean that at least one 2; is nonzero. This result is actually a special case of a more general result concerning linear systems in which the number of equations is not necessarily equal to the number of variables, a matter which is discussed in Appendix A. Here, however, we shall give a simple inductive proof which establishes all that we need. Proof of Lemma, The necessity is clear. If |by| * 0, we can solve by Cramer's rule, obtaining thereby the unique solution 2; =z: = -- = 0. Let us concentrate then on the sufficiency. It is clear that the result is true for N = 1. Let us then see if we can establish its truth for N, assuming its validity for N — 1. Since at least one of the b,; is nonzero, or else the result is trivially true, assume that one of the elements in the first row is nonzero, and then, without loss of generality, that this element is bu. Turning to the linear system in (1), let us eliminate 2, between the first and second equations, the first and third, and so on. The resulting system has the form (- (es - byte) Beet (60 - yea) aw =0 at + (rue — SPH) on = 0 m2 (3) Let us obtain a relation between the determinant of this system and the original N X N determinant, (b,j|, in the following way. Subtracting b1:/b1: times the first row of |bj;| from the second, bs:/bi: times the first row from the third row and a0 on, we obtain the relation bu ba bw] [bn = bu ba ++ by |=|0 (bus ~ Pe (ou - npr) : : = ber bys ++ byw] 0 ( _ (bee - a) (4) Hence, the determinant of (N — 1)-dimensional system in (3) is zero, since by assumption bi, # O and |bs| = 0. From our inductive hypothe-34 Introduction to Matrix Analysis sia it follows that there exists a nontrivial solution of (3), 22,23, . . . , tw. Setting x a= byai/bu 6) & we obtain, thereby, the desired nontrivial solution of (1), 21,2, . . « , Zw. EXERCISE 1, Show that if A is real, the equation Az = 0 always possesses a real nontrivial solution if it possesses any nontrivial solution. 8. Characteristic Roots and Vectors. Setting x equal to the vector whose components are 2; and A = (aj), we may write (1.1) in the form Ag = dt (qt) Referring to the lemma of Sec, 2, we know that a necessary and suf- ficient condition that there exist a nontrivial vector z satisfying this equation is that \ be a root of the determinantal equation lag — Ads = 0 @) or, as we shall usually write, jA -aI} =0 8) This equation is called the characteristic equation of A. As a poly- nomial equation in }, it possesses N roots, distinct or not, which are called the characteristic roots or characteristic values. If the roots are distinct, we shall occasionally use the term simple, as opposed to multiple. The hybrid word eigenvalue appears with great frequency in the literature, a bilingual compromise between the German word ‘“Eigen- werte”’ and the English expression given above. Despite its ugliness, it seems to be too firmly entrenched to dislodge. Associated with each distinct characteristic value 4, there is a character- istic vector, determined up toa scalar multiple. This characteristic vector may be found via the inductive route sketched in Sec. 2, or following the path traced in Appendix A. Neither of these is particularly attractive for large values of N, since they involve a large number of arithmetic operations. In actuality, there are no easy methods for obtaining the characteristic roots and characteristic vectors of matrices of large dimension. As stated in the Preface, we have deliberately avoided in this volume any references to computational techniques which can be employed toDiagonalization and Canonical Forms 35 determine numerical values for characteristic roots and vectors, If) is multiple root, there may or may not be an equal number of associated characteristic vectors if A is an arbitrary square matrix. ‘These matters will be discussed in the second part of the book, devoted to the study of general, not necessarily symmetric, matrices. For the case of symmetric matrices, multiple roots cause a certain amount of inconvenience, but nothing of any moment. We will show that a real symmetric matrix of order N has N distinct characteristic vectors. EXERCISES 1. A and A’ have the same characteristic values, &. T’AT — MM = T(A — NI)T if T is orthogonal. Hence, A and T’AT have the same characteristic values if 7 is orthogonal. 8. A and T*AT have the same characteristic values if 7 is unitary. 6. Show by direct calculation for A and B, 2 X 2 matrices, that AB and BA have the same characteristic equation. 6. Does the result hold generally? 7, Show that any scalar multiple apart from zero of a characteristic vector is aleo ‘a characteristic vector. Hence, show that we can always choose a characteristic vector x 90 that (2,2) = 1. 8. Show, by considering 2 X 2 matrices, that the characteristic roots of A +B cannot be obtained in general as sums of characteristic roots of A and of B. 9, Show that a similar comment is true for the characteristic roots of AB. 10. For the 2 Xx 2 case, obtain the relation between the characteristic roots of A and those of A’, 14, Does a corresponding relation hold for the characteristic roots of A and A* forn = 3,4,... 4. Two Fundamental Properties of Symmetric Matrices. Let us now give the simple proofs of the two fundamental results upon which the entire analysis of real symmetric matrices hinges. Although we are interested only in symmetric matrices whose elements are real, we shall insert the word “real” here and there in order to emphasize this fact and prevent any possible confusion. Theorem 1. The characteristic roots of a real symmetric matrix are real. Proof. Assume the contrary. Since A is a real matrix, it followa from the characteristic equation |A — \J| = 0 that the conjugate of any com- plex characteristic root ) is also a characteristic root. We obtain this result and further information from the fact that if the equation Az = dt (a) holds, then the relation AE = KE 2)36 Introduction to Matrix Analysis is also valid. From these equations, we obtain the further relations (Az) = 2) (2,42) = K(a2) @ Since A is symmetric, which implies that (2,42) = (AZ,z) = (z,A3), the foregoing relations yield O=(-H@), (4) whence \ = X, a contradiction. This means that the characteristic vectors of a real symmetric matrix A can always be taken to be real, and we shall consistently do this. ‘The second result is: Theorem 2, Characteristic vectors associated with distinct character- istic roots of a real symmetric matrix A are orthogonal. Proof. From Az = de Ay = wy ©) d # u, we obtain (y,Az) = My,2) 6) (2, Ay) = w(zy) Since (2,Ay) = (Az,y) = (y,Az), subtraction yields 0 = A — aay) @ whence (z,y) = 0. ‘This result is of basic importance. Its generalization to more general operators is one of the cornerstones of classical analysis. EXERCISES 1. A characteristic vector cannot be associated with two distinct characteristic values, 2. Show by means of a 2 X 2 matrix, however, that two distinct vectors can be associated with the same characteristic root. 8. Show by means of an example that there exist 2 x 2 aymmetric matrices A and B with the property that |A — dB] is identically zero, Hence, under what conditions on B can we assert that all roots of |A — AB] = O are real? 4. Show that if A and B are real aymmetric matrices, and if B is positive definite, the roots of [A — AB] = O are all real. 5. Show that the characteristic roots of a Hermitian matrix are real and that the characteristic vectors corresponding to distinct characteristic roots are orthogonal using the generalized inner product (7,9). 6. Let the elements as, of A depend upon a parameter t. Show that the derivatives of |A| with respect to ¢ can be written as the sum of N determinants, where the kth determinant is obtained by differentiating the elements of the kth row and leaving the others unaltered.Diagonalization and Canonical Forms 37 » 1, Show that the derivative of |A — AIj with respect to dis equal to — y l4y - afl, et where Apis the (N — 1) x (N — 1) mateix obtained from A by striking out the kth row and column. 8. From this, conclude that if \ is a simple root of A, then at least one of the deter- minante {Ax — AJ] is nonzero. 9. Use this result to show that if ) is a simple root of A, characteristic vector associated with X can always be taken to be a vector whose components are poly- nomials in \ and the elements of A. 5. Reduction to Diagonal Form—Distinct Characteristic Roots. We can obtain an important result quite painlessly at this juncture, if we agree to make the simplifying assumption that A has distinct character- istic roots 1, Xs, ..., Av. Let 2!, 2%, .. . , 2% be an associated set of characteristic vectors, normalized by the condition that @z)=1 i=1,2,...,N a) Consider the matrix T' formed upon using the vectors z‘ as columns. Schematically, T = (a',2%,... 2%) (2) Then 7” is the matrix obtained using the 2! as rows, ‘a at : (3) a” Since I'T = ((#,2*)) = (84) 4) (in view of the orthogonality of the z‘ as characteristic vectors associated with distinct characteristic roots of the aymmetric matrix A), we see that T is an orthogonal matrix. We now assert that the product AT has the simple form AT = (Aa!,et®, . . . Awa) (6) by which we mean that the matrix AT has as its ith column the vector aati, It follows that TAT = (Aaiz)) = Qsby) @)38 Introduction to Matrix Analysis (recalling once again the orthogonality of the 2), ‘The matrix on the right-hand side has as its main diagonal the charac- teristic values A,, Az, . . . , Av, and zeros every place else. A matrix of this type is, as earlier noted, called a diagonal matrix. Multiplying on the right by 7” and on the left by 7, and using the fact that TT’ = I, we obtain the important result Aw. This process is called reduction to diagonal form. As we shall see, this representation plays a fundamental role in the theory of symmetric matrices. Let us use the notation dy, EXERCISES 1, Show that At = (4/45,,), and that At = TA‘T", fork = 1,2,.... 2. Show that if A has distinct characteristic roots, then A satisfies its own charac- teristic equation. This is a particular case of a more general result we shall establish later on. 8. If A has distinct characteristic roots, obtain the set of characteristic vectors associated with the characteristic roots of At, k = 2,3,....Diagonalization and Canonical Forms 39 6. Reduction of Quadratic Forms to Canonical Form. Let us now show that this matrix transformation leads to an important transforma- tion of Q(z). Setting z = Ty, where T is as defined in (2) of Sec. 5, we have (z,Az) = (Ty,ATy) = (y,T'ATy) = (w,Au) qd) or the fundamental relation x ayer; = (2) i Since T is orthogonal, we see that x = Ty implics that Tr = T'Ty =y (3) Hence to each value of x corresponds precisely one value of y and conversely. We thus obtain the exceedingly useful result that the set of values assumed by Q(z) on the sphere (z,2) = 1 is identical with the set of values assumed by (y,Ay) on the sphere (y,y) = 1. So far we have only established this for the case where the ), are all distinct. As we shall see in Chap. 4, it is true in general, constituting the foundation stone of the theory of quadratic forms. EXERCISES 1. Let A have distinct characteristic roote which are all positive. Use the preceding result. to compute the volume of the N-dimensional ellipsoid (z,4z) = 1. 2 Prove along the preceding lines that the characteristic roots of Hermitian ‘matrices are real and that characteristic vectors associated with distinct character- istic roots are orthogonal in terms of the notation {z,y] of Sec. 16 of Chap. 2. 3. Show that if the characteristic roots of a Hermitian matrix A are distinct, we can find a unitary matrix T auch that A = TAT*. ‘This again is a particular case of a more general result we shall prove in the following chapter. 4 Let A be a real matrix with the property that A’ =—A, a skew-symmetric matrix. Show that the characteristic roots are either zero or pure imaginary. 5. Let Tbe an orthogonal matrix. Show that all characteristic roots have absolute value one. 6. Let T be a unitary matrix. Show that all characteristic roots have absolute value one. 1. Suppose that we attempt to obtain the representation of (5.2) under no restric- tion on the characteristic roots of the symmetric matrix A in the following way. To begin with, we assert that we can alwaye find a symmetric matrix B, with elements arbitrarily emall, possessing the property that A + B has simple characteristic roots. We do not dwell upon this point since the proof is a bit more complicated than might be suspected; see Soe. 16 of Chap. 11. Let {u1] be the characteristic roots of A + B ‘Then, as we know, there exists an orthogonal matrix $ such thatrT Introduction to Matrix Analysis i 0 A+B=8 : s By, Sinee 8 is an orthogonal matrix, ita elements are uniformly bounded. Let {By} be ‘a sequence of matrices approaching 0 such that the corresponding sequence of orthog- onal matrices (S,} converges. The limit matrix must then be an orthogonal matrix, aay T, Since lim wy = 2s, we have _e lim py A = lim (A +8,) = lim 8, : lim s, — Bonen . lim py Since this proof relies upon a number of analytic concepts which we do not wish to introduce until much later, we have not put it into the text. It is an illustration of a quite useful metamathematical principle that reaulte valid for general real symmetric matrices can always be established by first considering matrices with distinct charac- tetiatie roote and then passing to the limit. 7. Positive Definite Quadratic Forms and Matrices. In Sec. 9 of Chap. 1, we introduced the concept of a positive definite quadratic form in two variables, and the allied concepts of positive indefinite or non-negative definite. Let us now extend this to N-dimensional quad- ratic forms. If A = (a) is a real symmetric matrix, and x Q(z) = Yo ayeay > 0 ayer for all real nontrivial 2;, we shall say that Qy(z) is positive definite and that A is positive definite. If Qu(z) > 0, we shall say that Qu(z) and A are positive indefinite. ¥ Similarly, if H is a Hermitian form and Py(z) = > hg; > 0 for i all complex nontrivial 2, we shall say that Py(x) and H are positive definite. EXERCISES 1. If A is a symmetric matrix with distinct characteristic roots, obtain a set of necessary and sufficient conditions that A be positive definite. 2. There exists a scalar c: such that A + cI is positive definite, given any sym- metric matrix A.Diagonalization and Canonical Forms 4t 8. Show that we can write A in the form x Am Yn where the E; are non-negative definite matrices, Then x Ate > AEs att for k = 1, 2, 4. If pd) is @ polynomial in d with scalar coefficients, p(\) = > tah, let p(A) Fo x denote the matrix p(4) =) cid Show that p() = > paoks % A MISCELLANEOUS EXERCISES 1, Let A and B be two symmetric matrices, Then the roots of |A — \B| = 0 are all real if Bis positive definite. What can be said of the vectors satisfying the relations Az = Bat %. Every matrix is uniquely expressible in the form A = H +S, where H is Hermitian and S is ekew-Hermitian, that is, 8* = —S. 8. As an extension of Theorem 1, show that if ) is a characteristic root of a real matrix A, then [Im (4)| < d(N(N — 1)/2)¥, where d= max lay —ajl/2 (I. Bendizson) Asis 4. Generally, let A be complex; then if dy = max lai) ds = max lay + bl id dy = max Jai — 54/2 [2 we have DiS Nd [Re Q)| fault cee x B Y means Y leu tear oS fy x N Dm aor s Yo Kou ~ e/a UL Sehur) & a t Far more sophisticated results are available. See the papers by A. Brauer in Duke Math. J., 1946, 1947, 1948, where further references are given. See also W. V. Parker, Characteristic Roots and Ficlds of Value of a Matrix, Bull. Am, Math. Soc. vol. 57, pp. 103-108, 1951.42 Introduction to Matrix Analysis 6. So far, we have not dismissed the possibility that a characteristic root may have several associated charecteristic vectors, not all multiples of a particular character- istic vector. As we shall sec, thie can happen if A has multiple characteristic roots. For the case of distinct characteristic roots, this cannot occur. Although the sim- plest proof uses concepts of the succeeding chapter, consider a proof along the fol- lowing lines: (a) Let z! and y be two characteristic vectors associated with A: and suppose that v xf exe! for any scalar ¢:. Then 2! and ¢ = y — 2%(24,y)/(z'2!) are characteristic ‘vectors and z! and z are orthogonal. (@) Let 2" be the normalized multiple of 2. Then ato at 2 s- is an orthogonal transformation. (c) A = SDS', where ™ M o De= Ma o dw, (@) It follows that the transformation z = Sy changes (z,Az) into (y,Dy). (@) Assume that A is positive definite (if not, consider A + 11), then, on one hand, the volume of the ellipsoid (z,Az) = 1 is equal to the volume of the ellipsoid Dat Pdayt bos baw =, and, on the other hand, from what has just preceded, is equal to the volume of dat bd? boos sb Aaywt = 1 ‘This is a contradiction if ds y 4. Bibliography $2. This proof is taken from the book by L. Mirsky, L. Mirsky, Introduction to Linear Algebra, Oxford University Press, New York, 1955;Diagonalization and Canonical Forms 43 88. The term “latent root” for characteristic value is due to Sylvester. For the reason, and full quotation, see N. Dunford and J. T. Schwartz, Linear Operators, part I, Inter- science Publishers, New York, 1958, pp. 606-607. The term “spectrum” for set of characteristic values is due to Hilbert. §5. Throughout the volume we shall use this device of examining the case of distinct characteristic roots before treating the general case. In many cases, we can employ continuity techniques to deduce the general case from the special case, as in Exercise 7, Sec. 6. Apart from the fact that the method must be used with care, since occasionally there is a vast difference between the behaviors of the two types of matrices, we have not emphasized the method because of its occasional dependence upon quite sophisticated analysis. It is, however, a most powerful technique, and one that is well worth acquiring.4 Reduction of General Symmetric Matrices to Diagonal Form 1. Introduction. In this chapter, we wish to demonstrate that the results obtained in Chap. 3 under the assumption of simple characteristic roots are actually valid for general symmetric matrices. The proof we will present will afford us excellent motivation for discussing the useful concept of linear dependence and for demonstrating the Gram-Schmidt orthogonalization technique. Along the way we will have opportunities to discuss some other inter- esting techniques, and finally, to illustrate the inductive method for deal- ing with matrices of arbitrary order. 2. Linear Dependence. Let z', 2, . . , 2 be a set of k N-dimen- sional vectors. If a set of scalars, ci, cz, . . . , Cs, at least one of which is nonzero, exists with the property that ew! + eet + ++ + ek = 0 (a) where 0 represents the null vector, we say that the vectors are linearly dependent. If no such set of scalars exist, we say that the vectors are linearly independent. Referring to the results concerning linear systems established in Appendix A, we see that this concept is only of interest if k < N, since any k vectors, where k > N, are related by a relation of the type given in (1). EXERCISES 1, Show that any set of mutually orthogonal nontrivial vectors is linearly inde- pendent. 2. Given any nontrivial vector in N-dimensional space, we can always find N — 1 vectors which together with the given N-dimensional vector constitute » linearly independent set. 8, Gram-Schmidt Orthogonalization. Let x', 2’, . . , 2% bea set of N real linearly independent N-dimensional vectors. We wish to show “Reduction of General Symmetric Matrices 45 that we can form suitable linear combinations of these base vectors which will constitute a set of mutually orthogonal vectors. ‘The procedure we follow is inductive. We begin by defining two new vectors as follows. yen ya tage w where a1; is @ scalar to be determined by the condition that y' and y? are orthogonal. The relation (a, 2? + aus!) = 0 (2) (yy?) yields the value au = —(2',2*)/(a',2") @) Since the set of z‘ is by assumption linearly independent, we cannot have z' equal to the null vector, and thus (z',z') is not equal to zero. Let us next set y? = a? + aga! + aay? (4) where now the two scalar coeflicients a1, and azz are to be determined by the conditions of orthogonality Gy) =0 (yy?) = 0 (6) ‘These conditions are easier to employ if we note that (1) shows that the equation in (5) is equivalent to the relations Ge) =O (yizt) = ©) These equations reduce to the simultaneous equations (etx!) + an(z'z") + an(2%z') = 0 : (eta) + an(z'2") + an(atz') = 0 2) which we hope determine the unknown coefficients a2: and ax. We can solve for these quantities, using Cramer’s rule, provided that the determinant (ez!) (2%) De Vela) (eta) ®) is not equal to zero. To show that Dz is not equal to zero, we can proceed as follows. If D, = 0, we know from the lemma established in Sec. 2 of Chap. 3 there are tivo scalars r, and 4, not both equal to zero, such that the linear equations n(v'z') + si(zt2) = 0 my ri(ata*) + ar(2?.z") = 06 Introduction to Matrix Analysis are satisfied. These equations may be written in the form G!, rw! + ae") = 0 Gt, ra! +e) =0 (10) Multiplying the first equation by r; and the second by s; and adding, we obtain the resultant equation ay) ‘This equation, however, in view of our assumption of reality, can hold only if riz" + s2* = 0, contradicting the assumed linear independence of the x, Hence D; # 0, and there is a unique solution for the quantities Gy; and Gay, At the next step, we introduce the vector (rit! + 812%, riz! + 822) = y= 2' + ays! + ayct + ase? (12) As above, the conditions of mutual orthogonality yield the equations (tz) = (62) = 42h) = 0 (13) which lead to the simultaneous equations (e429) + an(ztz!) + aro(2*,2') + an(c4z) = 0 #=1,2,3 (14) We can solve for the coefficients as:, dsz, and ass, provided that the determinant De=(@7) 4 5=1,2,3 (15) is nonzero. The proof that D, is nonzero is precisely analogous to that given above in the two-dimensional case. Henoe, we may continue this procedure, step by step, until we have obtained a complete set of vectors {y‘} which are mutually orthogonal. These vectors can then be normalized by the condition (y‘y‘) = 1. We then say they form an orthonormal set. The determinants D, are called Gramiane, EXERCISES 1. Consider the interval {—1,1] and define the inner product of two real functions J) and g(t) in the following way: Gio) = fsa a Let Pa(d) = 34, and define the other elements of the sequence of real polynomials (P,(0)) by the condition that P,(t) is of degree n and (Ps,Pm) = 0, m x! n,(Py,Ps) = 1. Prove that we have (P./m) = 0, for 0 0 for all nontrivial sets of values of the u; Consequently, the result we43 Introduction to Matrix Analysis wish to establish can be made a corollary of the more general result that the determinant D = lay @) of any positive definite quadratic form x Q= > tial 8) sige is positive. We shall show that this positivity is a simple consequence of the fact that it is nonzero, All this, as we shall see, can easily be derived from results we shall obtain further on, It is, however, interesting and profitable to see the depth of various results, and to note how far one can go by means of fairly simple reasoning. Let us begin by observing that D is never zero, For, if D = 0, there is a nontrivial solution of the system of linear equations x Y avy = 0 f=1,2,...,N (4) im From these equations, we conclude that x oN a= J u(Y eww) = 0 6) @ contradiction. This is essentially the same proof we used in Sec. 3. The novelty arises in the proof that D is positive. Consider the family of quadratic forms defined by the relation x PQ) = 2+ (1-9) Fw © a where is a scalar parameter ranging over the interval (0,1). For all ) in this interval, it is clear that P(A) is positive for all nontrivial w. Con- sequently, the determinant of the quadratic form P(A) is never zero. For \ = 0, the determinant has the simple formReduction of General Symmetric Matrices 49 clearly positive. Since the determinant of P(X) is continuous in ) for 0 <2 <1, and never zero in this range, it follows that positivity at = 1, the desired result. 5, An Identity. An alternative method of establishing the positivity of the D, when the vectors 2* are linearly independent depends upon the following identity, which we shall call upon again in a later chapter devoted to inequalities. Theorem 1, Let 2',i = 1,2, . . . ,k, be a set of N-dimensional vectors, N>k. Then Tl Bylo ss waht Eh , 1 : WV osetcdook = Fy > : @ ay. a where the sum is over all sets of integers (i1} with 1 By > yt > yes] = 3 > ae a (3) yt = “ iskdms¥ Dae 5 ve ye Fy ot km iat im ms Let us begin with the result te tml [att tat tan? tay tt eam Tite Tet I nem Ye We Ym) =| tas taiyttmym ya ty? tym? yeertyzrtynem| (4) a tt em Mee tae tnt Yee ye TY mem za $217 tent which we obtain from the rule for multiplying determinants. The last determinant may be written as the sum of three determinants, of which the first is typical,50 Introduction to Matrix Analysis tt Dee + Ty + Sum Tete + Lier + Twew ways vat tue + ym? uate + yite + Yemen Tike Yata + yet t+ mem ee? + 21? + em? (5) Subtracting y./z, times the first column from the second column, and 24/2» times the first column from the third column, we are left with the determinant TP TY + mlm Tet + mem Tye Yt ym? Yet + mem Tete iti t Ymem 21 + eq? Writing this determinant as a sum of two determinants, of which the following is typical, we are left with ® BP Liye Te + Lem Tye YP yeti + Yotm Bite yir 2 + em? @ and subtracting z;/y: times the second column from the third column, we obtain the determinant BP Lis Letm Taye YI Yen Tite Yuet Em Summing over all k, J, and m, we obtain the left side of (3). Observing that the above procedures reduce us to a determinant of the form given in (8) in 3 X 2 ways, we see that 9 factor of 1/3! is required. 6. The Diagonalization of General Symmetric Matrices—Two Dimen- sional. In order to show that a general real symmetric matrix can be reduced to diagonal form by means of an orthogonal transformation, we shall proceed inductively, considering the 2 X 2 case first. Let fan az] _ fat ao[ts ele [2] ® be a symmetric matrix, and let 4, and 2! be an associated characteristic root and characteristic vector. This statement is equivalent to the relations (8) Ag'= det oor tz!) = Ait (@8z!) = tae (2) where 21 and 212 are the components of 2!, which we take to be normal- ized by the condition (z!,z') = 1. As we know, we can form a 2 X 2 orthogonal matrix T, one of whose columns is 2. Let the other column be designated by z*. We wish to show that TAT, = [ | @)Reduction of General Symmetric Matrices 51 where As and As are the two characteristic values of A, which need not be distinct. This, of course, constitutes the difficulty of the goncral case. We already have a very simple proof for the euse where the roots are distinct. Let us show frst that " di bie mar, =[% be @ where by: and bry are a yet unknown parameters, ‘The significant fact is that the element below the main diagonal is zero. We have " — op] Mtn (a',2*) mar, = m [Meu (ou) © upon referring to the equations in (2). Since 7's = 7, we obtain, upon carrying through the multiplication, Paz (os) Ma bis [ye (42) [0 de ® whore bis and bys are parameters whose value wo shall determine below. Let us begin with bis, and show that it has the value zero. This follows from the fact that 7,47; is a symmetric matrix, for (T,AT))' = TAT)! = TiA'T, = TAT (7) Finally, let us show that bz: must be the other characteristic root of A. This follows from the fact already noted that the characteristic roots of A are identical with the characteristic roots of T,A7. Hence bi: = dz. This completes the proof of the two-dimensional case. Not only is this case essential for our induction, but it is valuable because it contains in its treatment precisely the same ingredients we shall use for the general case. 7. N-dimensional Case. Let us proceed inductively. Assume that for each k, k = 1, 2, .. . , N, we can determine an orthogonal matrix T, which reduces a real symmetric matrix A = (ai),1, = 1,2,... ky to diagonal form, m TAT: = . w de, The cloments of the main diagonal 4; must then be characteristic roots of A. Under this assumption, which we know to be valid for N = 2, let us show that the same reduction can be carried out for N + 1.32 Introduction to Matrix Analysis ‘a! at Avg = (ai) =| | @) aN and let A, and z! be, respectively, an associated characteristic value and normalized characteristic vector of Aw41. Proceeding as in the two-dimensional case, we form an orthogonal matrix 7; whose first column is z!. Let the other columns be designated as a,c, ... ,2*+1, so that T; has the form Ty = (iat, . 2. ehh) (3) Then as before, the first step consists of showing that the matrix T,AwsiT has the representation Me bn bu bw 0 TMAwaTs = dy ® 0 a, The elements of the first column are all zero except for the first which is \; b. The quantities bis, bis, . . . , div will be (6) determined below The matrix Aw is an N X N matrix Carrying out the multiplication, we have (atc) (at,zty (at,2"+1) (aa!) (ahat) +++ (ate) Anal) = : : ye Mtn aut) + (aleX tt) ® Mite (a4zt) + (aa) Ditiwer (GM gH) (ght 41)Reduction of General Symmetric Matrices 53 Since 7’; is an orthogonal matrix, we see that De Bin tt Daag TiAvaT = @” An 0 To determine the values of the quantities by, we use the fact that T,AnsT1 is symmetric. This shows that these quantities are zero and simultaneously that the matrix Aw must be symmetric. ‘The result of all this is to establish the fact that there is an orthogonal matrix 7, with the property that A 0 TiAwals = 6) , . . 0 with Ay a symmetric matrix. Finally, let us note that the characteristic roots of the matrix Ay must be Xs, As, - - - , Avsu, the remaining N characteristic roots of the matrix Ayy1. This follows from what we have already observed concerning the identity of the characteristic roots of Ay, and T;Aw,:71, and the fact that the characteristic equation of the matrix appearing on the right- hand side of (7) has the form A: — { |4w — AZ| = 0. Let us now employ our inductive hypothesis. Let Tw be an orthogonal matrix which reduces Aw to diagonal form, Form the (N + 1)-dimen- sional matrix Loe 0 Swa =|) Ty (9) 0 which, as we know, is also orthogonal. It is readily verified that ., 1 Ae Srg(TAnnT )Se41 = _ (10) Dae54 Introduction to Matrix Analysis Since we may write Siyi(TiAweiTi)Su41 = (TiSw4)'Anga(T Sx) (yy we see that T,Sy4: is the required diagonalizing orthogonal matrix for Any We have thus proved the following basic result. Theorem 2 Let A be a real symmetric matriz, Then it may be trans- formed into diagonal form by means of an orthogonal transformation, which is to say, there is an orthogonal matrix T’ such that mh 0 ds TAT = (a2) 0 > hs, where d; are the characteristic roots of A. Equivalently, . @A2) = ) ru? (13) a where y = T's. EXERCISE 1. What can be said if A is a complex symmetric matrix? 8. A Necessary and Sufficient Condition for Positive Definiteness. The previous result immediately yields Theorem 3. Theorem 8. A necessary and sufficient condition that A be positive definite is that all the characteristic roots of A be positive. Similarly, we see that a necessary and sufficient condition that A be Positive indefinite, or non-negative definite, is that all characteristic roots of A be non-negative. EXERCISES 1. A = BB? is positive definite if B is real and |B| x 0. 2. H = CC* is positive definite if |C| + 0. 8. If A is aymmetric, then 1 + eA is positive definite if « is sufficiently amall. 9. Characteristic Vectors Associated with Multiple Characteristic Roots. If the characteristic roots of A are distinct, it follows from the orthogonality of the associated characteristic vectors that these vectors are linearly independent. Let us now examine the general case. Suppose that d: is a root ofReduction of General Symmetric Matrices 55 multiplicity &. Is it true that there exist & linearly independent charac- teristic vectors with d1 as the associated characteristic root? If so, is it true that every characteristic vector associated with \, is a linear com- bination of these & vectors? To answer the question, let us refer to the representation in (7.12), and suppose that i = A: = © + * = As, but that A; x As for ktl...,N t Retracing our steps and writing AL 0 Bt AT=T () 0 dy, it follows that the jth column of 7’ is a characteristic vector of A with the characteristic value ¥;. Since T’ is orthogonal, its columns are linearly independent. Hence, if \. is a root of multiplicity &, it possesses & linearly independent characteristic vectors. It remains to show that any other characteristic vector y associated with d, is a linear combination of these & vectors. Let y be written as a linear combination of the N columns of T, x > eat @) The coefficients c; can be determined by Cramer’s rule since the deter- minant is nonzero as a consequence of the linear independence of the 2‘. Since characteristic vectors associated with distinct characteristic Toots are orthogonal, we see that c; = 0 unless 2* is a characteristic vector associated with \i. This shows that y is a linear combination of the characteristic vectors associated with \,, obtained from the columns of T. 10. The Cayley-Hamilton Theorem for Symmetric Matrices. From (7.12), we see that for any polynomial p(d) we have the relation pm) 0 P(d2) p(A) = T rT qa) PQw),56 Introduction to Matrix Analysis If, in particular, we choose p(\) = |A — XI, the characteristic poly- nomial of A, we see that p(A) = 0. This furnishes a proof of the follow- ing special case of a famous result of Cayley and Hamilton. Theorem 4. Every symmetric matrix satisfies ite characteristic equation. As we shall see subsequently, this result can be extended to arbitrary square matrices. EXERCISE 1, Use the method of continuity to derive the Cayley-Hamilton theorem for gen- eral symmetric matrices from the result for aymmetric matrices for simple roots. 11, Simultaneous Reduction to Diagonal Form. Having seen that we can reduce a real symmetric matrix to diagonal form by means of an orthogonal transformation, it is natural to ask whether or not we can simultaneously reduce two real symmetric matrices to diagonal form. ‘The answer is given by the following result. Theorem 5. A necessary and sufficient condition that there exist an orthogonal matrix T with the property that M mL Ae ae TAT = . T'BT = . () aw. Ay, ts that A and B commute. Proof. The proof of the sufficiency is quite simple if either A or B has distinct characteristic roots. Assume that A has distinct character- istic roots, Then from Az= dt (2) we obtain A(Bz) = B(Az) = B(x) = \(Bz) @) From this we see that Bz is a characteristic vector associated with A whenever = is, Since the characteristic roots are assumed distinct, any two characteristic vectors associated with the same characteristic root must be proportional. Hence we have Bi=pet ¢=1,2,...,N @) where the 4; are scalars—which must then be the characteristic roots of the matrix B. We see then that A and B have the same characteristic vectors, 1, a%,..., 2”.Reduction of General Symmetric Matrices 57 T can be taken to be T = (a2%,... 2) (5) Consider now the general case where \, is a characteristic root of multiplicity % with associated characteristic vectors z!, a,..., aA Then from (3) we can only conclude that A B= Joa i=1,2,...,% ©) ro Let us, however, see whether we can form suitable linear combinations of the 2‘ which will be characteristic vectors of B. We note, to begin with, that the matrix C’ = (cy) is symmetric for the orthonormality of the 2! yields Br) = ci; = (Bai,z') = og ) A Consider the linear combination )' ai. We have Hence if the a; are chosen so that Y eva = ra; G=1,2...,k (9) mn B (3 xt) = ry (s es) (0) x which means that r1 is a characteristic root of B and ) ais an associ- a we will have ated characteristic vector. ‘The relation in (9) shows that r; is a characteristic root of C and the a; components of an associated characteristic vector. Thus, if T, is a k-dimensional orthogonal transformation which reduces C to diagonal form, the set of vectors 2‘ furnished by the relation =T%l. (11)58 Introduction to Matrix Analysis will be an orthonormal set which are simultaneously characteristic vectors of A and B. Performing similar transformations for the characteristic vectors associated with each multiple characteristic root, we obtain the desired matrix T. The necessity of the condition follows from the fact that two matrices of the form AL a Ms be A=T : Y Bel . T’ (12) dy. aN, always commute if T is orthogonal. 12, Simultaneous Reduction to Sum of Squares. As was pointed out in the previous section, the simultaneous reduction of two symmetric matrices A and B to diagonal form by means of an orthogonal transforma- tion is possible if and only if A and B commute. For many purposes, however, it is sufficient to reduce A and B simultaneously to diagonal form by means of a nonsingular matrix. We wish to demonstrate Theorem 6. Theorem 6, Given two real symmetric matrices, A and B, with A positive definite, there exists a nonsingular matriz T such that TAT =I ) “1 T'BT = oy, Proof. Let S be an orthogonal matrix such thatReduction of General Symmetric Matrices 59 Then, if z = Sy, we have » Aa) = Y rot (3) (=,B2) = (W,S'BSy) Now perform a “stretching” transformation 4 - = TH t=1,2,...,N (4) that is y=Se Then Ny 2, du? = (22) 6) (y,S’BSy) = (2,5,S'BSSs2) Denote by C the matrix S,S’BSS2, and let Ss be an orthogonal matrix which reduces C to diagonal form, Then, if we set z = Sjw, we have x (Ce) = (w,S'0Sw) = J wont 3) while (2,2) = (w,w). It follows that T = SSiSs ) is the desired nonsingular matrix. EXERCISES 1, What is the corresponding result in case A ia non-negative definite? 2. Let the notation A > B for two symmetric matrices denote the fact that A — B is positive definite, Use the foregoing result, to show that A > B > 0 implies that Bt> A 18, Hermitian Matrices. It is clear that precisely the same techniques as used in establishing Theorem 2 enable one to establish Theorem 7. Theorem 7. If H is a Hermitian matriz, there exists a unitary matriz U such that60 Introduction to Matrix Analysis 14. The Original Meximization Problem. We are now in a position to resolve the problem we used to motivate this study of the positivity and negativity of quadratic forms, namely, that of deciding when a sta- tionary point of a function f(zi,z2, . . . ,tw) is actually a local maximum or 8 local minimum, Let c = (cies, . . . sev) be @ stationary point, and suppose that f possesses continuous mixed derivatives of the second order. It is conse- quently sufficient to consider the nature of the quadratic form a= (2h) = ba o where, as before, (2) evaluated at the point 1 = ¢1, 21 = ¢1, . . «tw = ew. We see then that a sufficient condition that c be a local minimum is that Q be positive definite, and a necessary condition that ¢ be a local minimum is that Q be positive indefinite. ‘The criterion given in Sec. 8 furnishes a method for determining whether Q is positive or negative definite. If Q vanishes identically, higher order terms must be examined, If W is large, it is, however, not a useful method. Subsequently, in Chap. 6, we shall derive the most useful criterion for the positive definite- ness of a matrix. EXERCISE 1. Show that a set of sufficient conditions for f(z:,z2) to have a local maximum is >o Sass Su fon <0 lf he 15. Perturbation Theory—I. We can now discuss a problem of great theoretical and practical interest. Let A be a symmetric matrix possess- ing the known characteristic values 1, \2, . . . , Aw and corresponding characteristic vectors z', z*,..., 2%, What can we say about the characteristic roots and vectors of A +B, where B is a symmetric matrix and ¢ is a “small” quantity? How small e has to be in order to be called so will not be discussed here, since we are interested only in the formal theory. If A and B commute, both may be reduced to diagonal form by the same orthogonal transformation. Consequently, with a suitable reorder- ing of the characteristic roots of B, the characteristic roots of A + «B will be \: + eu, i= 1,2, ... , N, while the characteristic vectors will be as before. Let us then consider the interesting case where AB ~ BA. For the sake of simplicity, we shall consider only the case where the character- istic roots are distinct.Reduction of General Symmetric Matrices 61 It is to be expected that the characteristic roots of A + ¢B will be distinct, for e sufficiently small, and that they will be close to the charae- teristic roots of A. It will follow from this, that the characteristic veo- tors of A + ¢B will be close to those of A. One way to proceed is the following. In place of the character- istic equation |A — J| = 0, we now have the characteristic equation |A + eB — 1] =0, The problem thus reduces to finding approximate expressions for the roots of this equation, given the roots of the original equation and the fact that ¢ is small. To do this, let 7 be an orthogonal matrix reducing A to diagonal form, Then the determinantal equation may be written Mt ecu — A er _— cow cn Metece — A ew . : =0 ecu nee Aw + ecww — ® where T’BT = C= (cy). We now look for a set of solutions of this equation of the form N= At due + due? +--+ We leave it as a set of exercises for the reader versed in determinantal expansions to obtain the coefficients di,, dx, and so on, in terms of the elements cy. We do not wish to pursue it in any detail since the method we present in the following section is more powerful, and, in addition, can be used to treat perturbation problems arising from more general operators. 16. Perturbation Theory—II. Let us now suppose not only that the characteristic roots and vectors of A + ¢B are close to those of A, but that we actually have power series in ¢ for these new quantities. Write u;, y‘ as an associated pair of characteristic root and character- istic vector of A + eB and Xj, 2‘ a8 a similar set for A. Then we set mie =e Hae + a te w yottet erty ss To determine the unknown coefficients Ay, 4x, . . . and the unknown vectors z‘!, 2*2, . . . , we substitute these expressions in the equation (A + B)y = ny! (2) and equate coefficients,62 Introduction to Matrix Analysis We thus obtain (A + Bat + ext Feat $d = Ota teat Mote test +++) @) and the series of relations Aa! = dat Az + Bat = da + daat Az + Batt = dealt + dail + deat (4) Examining these equations, we see that the first is identically satisfied, but that the second equation introduces two unknown quantities, the scalar dj and the vector 2‘, Similarly, the third equation introduces two further unknowns, the scalar 2 and the vector 2". At first sight this appears to invalidate the method. However, examin- ing the second equation, we see that it has the form (4 — Ada = (Wal - B)st 6) where the coefficient matrix A — \,/ is singular, Consequently, there will be a solution of this equation only if the right-hand-side vector Qal — B)z* possesses special properties. ‘When we express the fact that (Au — B)z‘ possesses these properties, we will determine the unknown scalar dy. Let us simplify the notation, writing wlay al — Bt =2 © ‘Then we wish to determine under what conditions upon z the equation (A - ADy =2 7) has a solution, when ), is a characteristic root of A, and what this solu- tion is. Let z', z*,... , 2% be the normalized characteristic vectors of A, and consider the expressions for y and z as linear combinations of these vectora, bat (8) g the iseReduction of General Symmetric Matrices 6 where b= U2) g=@) f=1,2,...,N (9) Substituting in (7), we obtain the equation x (A — xd) > bai = > (10) ist is or, since Azé = d2i,j = 1,2,...,N, x x Y a0) - ro =) gat (1) ms ro Equating the coefficients of 2’, we obtain the relations KOs - m= fF =12%...,N (12) We see that these equations are consistent if and only if a=0 (13) Then 0; is arbitrary, but the remaining b; are given by j=o/W—-M) 7 =1,2...,N G80 (14) What the condition in (13) asserts is that (7) has a solution if and only if z is orthogonal to 2‘, the characteristic vector associated with 2,. If so, there is one-parameter family of solutions of (7), having the form y = bat + > ee (18) im where 0, is arbitrary. Returning to (6), we see that we can take b, =0, since @ different choice of b merely affects the normalization of the new characteristic vector yi, The orthogonality condition yields the relation (2, al — B)z’) = 0 (16) or, the simple result, da = (Ba) a7) EXERCISES 1. Find the value of di. 2. Consider the case of multiple roots, first for the 2 X 2 case, and then, in general. 8. Let denote the characteristic roots of A and Xj(z) those of A + 2B. Show(7 Introduction to Matrix Analysis MCA +48) = Yammy ay where (=) = (cam ( y BS MBS». . BS, bibs haem I 20 8) = -B, The Z, are defined by the relation A = Saws and §; = > Fx/Qu — 4) ors (T. Kato). 4, Let A and B be 10 X 10 matrices of the form O1.+-O O01--- 0 ou : ou : Ae op Be Lo * a oa 0 o 10-1 0. A and B are identical except for the element in the (10,1) place, which is 10-1" in the case of B, Show that |\ — Al = 0! and that [A — B] =X! — 10-1, Hence, the characteristic roots of Bare 10-1, 10-'w, . . . , 10-40%, where w is an irreducible tenth root of unity. What is the explanation of this phenomenon? (Forsythe.) MISCELLANEOUS EXERCISES 1. Show that a real symmetric matrix may be written in the form x Am Yagi & where the Ei are non-negative definite matrices satisfying the conditions BBy=0, ij BO = Be is called the spectral decomposition and the 2, are the characteristic roots of A. Thi of A. @, Let A be a real skew-symmetric matrix; ai; = —a,z. Then the characteristic roots are pure imaginary or zero, Let z + iy be a characteristic vector associated with ia, where « is a real nonzero quantity and where z and y are real. Show that wand y are orthogonal. 8. Referring to the above problem, let 7; be an orthogonal matrix whose firat two columns are z and y, Ty = (ayetzt, 2%) Show that On) 4 TAT =| \-n 0, 0 Ayes where Ay-s is again skew-symmetric,Reduction of General Symmetric Matrices 65 4, Prove inductively that if A is a real skew-symmetric matrix of even dimension we can find an orthogonal matrix T such that 0 —m 0 oO Ow —m 0 TAT = : 0 . ( 0 ar) —aw 0 where some of the «: may be zero. 5. If A is of odd dimension, show that the canonical form is ow «5 0 TAT = 0 oy’ —uw 0 0 0, where again some of the a; may be 2e1 8. Show that the determinant of a skew-symmetric matrix of odd dimension is zero. T. The determinant of a skew-symmetric matrix of even dimension is the square of a polynomial in the elements of the matrix. 8. Let A be an orthogonal matrix, and let 4 be a characteristic root of absolute value 1 but not equal to +1, with z + iy an associated characteristic vector, z and y real, Show that 2 and y are orthogonal. 9. Proceeding inductively as before, show that every orthogonal matrix A can be reduced to the form ‘cosh: — gin Ay (& M cos 9) ° cos hy = — ain Da’ aq? Gi cons 7 #1 6 . +1 10, Prove that TAT’ is a positive definite matrix whenever T is an orthogonal matrix and A is a diagonal matrix with positive elements down the main diagonal,66 Introduction to Matrix Analysis 11, Prove Theorem 5 by means of an inductive argument, along the lines of the proof given in See. 6. 12. Establish the analogue of the result in Exercise 9 for unitary matrices, 18. Write down a sufficient condition that the function f(z1,21, . . . ,2w) possess a local maximum at z= c,%=1,2,...5 14, Given that 4 positive definite matrix, find all solutions of XX’ = A, 16, Define the Gramian of N real functions fy, fs, . . . , fu over (a,b) by means of the expression Gtuty «=| freon | Prove that if . ° 2 Qeyty . «tw = a) — ) aft] dt Blon- Y se] then in Q = Ueinfnr »« afm) “oo. Ff For some further results, see J. Geronimus.' For many further results, see G, Szego.' 16. If we consider the (N — 1)-dimensional matrix 2-1 -1 2 -1 -1 1, where all undesignated elements are sero, show that w-1 i - HI = T] (0-2-2 c00 ay Ey ant and determine the characteristic vectors. 17, Hence, show that if z; = 0 and the z; are real, then at x > Gee — sad! Ale soa ie Determine the case of equality, 18, Show that if the z; are all real and zo = zy,1 = 0 that x x » (= aaa 4 int Soy i“ = 4 1J, Geronimus, On Some Peraymmetric Determinants Formed by the Polynomials of M. Appeli, J. London Math, Soc., vol. 6, pp. 55-58, 1931. *Q. Szego, Orthogonal Polynomials, Am. Math, Soc. Collog. Publ., vol. 23, 1939,Reduction of General Symmetric Matrices 67 x and that if 21 = tye y = = 0, then oH x x y (2 — 2eu)' > Aint ) zt d & (Ky Fan, 0. Taussky, and J. Todd) 19. Find the characteristic roots and vectors of the matrix associated with the x ‘ quadratic form Y (x — 2))% Fn 20. Let A and B be real symmetric matrices auch that A is non-negative definite, Then if |A + 4B| = 0, there existe nontrivial real vector such that (A + 4B)z = 0 (Peremans-Duparo-Lekkerkerker). 41, If A and B are symmetric, the characteristic roots of AB are real, provided that at least one is non-negative definite. 9, If A and B are symmetric, the characteristic roots of AB — BA are pure complex. 8, Reduce Qw(z) = 2121 + 15 + «+ + + 2y-12y to diagonal form, determining the characteristic vectors and characteristic values. Similarly, reduce Py(z) = ity fret f+ + + ayattw + aN. ‘24. If A is complex square matrix, then A is HU, where H is non-negative definite Hermitian and U is unitary. Further results and references to earlier results of Autonne, Wintner, and Murna- ghan may be found in J. Williamson.? 46, If A, B, and C are symmetric and positive definite, the roots of A + AB +0] =0 have negative real parts (Parodi). 26. Let A be a square matrix. Set |A| = |A|, — |A|-, where ||, denotes the sum of the terma which are given a positive sign in the determinantal expansion. Show that if A is a positive definite Hermitian matrix, then |A|_ > 0 (Schur-Leng). 27. Lot A be a real matrix with the property that there exists a positive definite matrix M such that A’M = MA. A matrix A with this property is called sym- ‘metrizable. Show that this matrix A possesses the following properties: (a) All characteristic roots are real, () In terms of a generalized inner product [2,y] = (2,My), characteristic vec- tors associated with distinct characteristic roote are orthogonal. (€) Let 7 be called “orthogonal” if it is real and its columns are orthogonal in the extended sense. Then there exists an “‘orthogonal”” matrix 7’ such that TAT" is diagonal. ‘28. Extend in a similar fashion the concept of Hermitian matrix. 29. Call a matrix A idempotent if A* = A. Show that a necessary and aufficient condition that a symmetric matrix 4 of rank k be idempotent is that k of the charac- teristic roots are equal to one and the remaining N — & are equal to zero. (The notion of rank is defined in Appendix A.) 80. If A is a symmetric idempotent matrix, then the rank of A is equal to the trace of A. (te trace of A is equal to ») au.) cot 'J, Williamson, A Generalization of the Polar Representation of Nonsingular Matrices, Bull. Am. Math. Soc., vol. 48, pp. 856-863, 1942, 5 See A. Kolmogoroff, Math. Ann., vol. 112, pp. 155-160, 1936.68 Introduction to Matrix Analysis ‘81. The only nonsingular symmetric idempotent matrix is the identity matrix. 88. If A is an N X N symmetric idempotent matrix of rank k, then A is positive definite if k = N, positive indefinite if k < N. 83. If A is a symmetric idempotent matrix with aq = 0, for @ particular value of t, then every element in the ith row and ith column is equal to zero. $4. Ifeach A; is symmetric, and y A; = I, then the three following conditions a are equivalent: (a) Each A; is idempotent, @) Ads = 0 ig. © > nu = N, where n: the rank of A; and N the dimension of the A; 85, Let Aybe a collection of N x W symmetric matrices where the rank of A; is ps. let A= ya have rank p. Consider the four conditions ¢ C1, Each A; is idempotent. C2, Aaj =O tej. C3, A is idempotent. ch. p= Yn cov ‘Then 1. Any two of the three conditions C1, C2, C3 imply all four of the conditions C1, C2, 63, C4. 2. C3 and C4 imply C1 and C2, For a proof of the foregoing and some applications, see F. A. Graybill and G. Mar- saglia,? D. J. Djokovié.! 86. Using the fact that 22,23 < 2: + 21 for any two real quantities 2; and 23, show that the quadratic form s x koN ay bse + bat + Y > oyna; + Yo anay iar iektl pel ie kt aged in positive definite if by > O and 2 ie sufficiently large. 81. Show that) axzcy is positive definite if Yous.’ + > lulz is positive fh i a 88. A matrix A = (aj,), i, j = 1, 2, 3, 4, is called a Lorentz matrix if the transfor- mation x = Ay leaves the quadratic form Q(z) = 2! — 21 — m1 — z¢ unchanged; that is, Q(z) = Q(y). Show that the product of two Lorentz matrices is again a Lorents matrix. 89. Show that i = (ti + 622)/(1 — Bis ya = Gar + 2)/(1 — 6) wnt we a where 0 < 6 <1 is a Lorentz transformation. ‘F. A. Graybill and G. Marsaglia, Idempotent Matrices and Quadratic Forms in the General Linear Hypothesis, Ann. Math, Stat., vol. 28, pp. 678-686, 1957, *D, J. Djokovié, Note on Two Problems on Matrices, Amer, Math. Monthly, to appear,Reduction of General Symmetric Matrices 6 40. Show that any Lorentz transformation is a combination of an orthogonal transformation of the variables 23, zs, 24 which leaves 2; fixed, a transformation of the type appearing above, and ® possible change of sign of one of the variables (a reflection).! Physical implications of Lorents transformations may be found in J. L, Synget 41, Let H be a non-negative Hermitian matrix. Show that given any | interval {a,6] we can find a sequence of complex functions {fi(0)], i= 1,2, ... , auch that han [Pnon@a 4 5=1,2, 42, Let A = (a;;) bean N X N symmetric matrix, and Ay: the symmetric matrix obtained by taking the (NV — 1) x (N — 1) matrix whose elements are aij i, § = 1,2,...,N—1, By means of an orthogonal transformation, reduce A to the form mn a uw 0 Es A= 0 Byes No ao 2y-1 own where ms are the characteristic roots of Avi. Using this representation, determine the relations, if any, between multiple characteristic roots of A and the characteristic roots of Ax-is 43. Using these results, show that the rank of a symmetric matrix can be defined as the order, N, minus the number of zero characteristic roots. 44. If H is & positive indefinite Hermitian matrix, then H = TT*, where T is triangular and has real and non-negative diagonal elements. See H.C. Lee. 46. Necessary and sufficient conditions for the numbers di, ds, . . . , dy to be the diagonal elements of a proper orthogonal matrix are di] <1 j = 1,2,» - » Ny and [de] Sn 2-22 min |d;|, whore \ = 1 if the number of negative di is even 1sisN mh and 0 otherwise (L. Mirsky, Amer. Math. Monthly, vol. 68, pp. 19-22, 1959). 46, If A is © symmetric matrix with no characteristic root in the interval {a,b}, then (A — a/)(4 — b1) is positive definite (Kato's lemma), Can we use this result to obtain estimates for the location of intervals which are free of characteristic root of At 47, Show that min [ e-(1 + axt + tee baat dt = 1/0 +0) inf (Lat ees pagth)dt = 1/(n +1) (L. J. Mordeli, Bquationes Mathematicae, to appear) 1Cf. I. G. Petrovsky, Lectures on Partial Differential Equations, Interscience Pub- lishers, Inc,, New York, 1954. L, Synge, Relativity, the Special Theory, Interscience Publishers, Inc., New York, 1956. tI. Schur, Math. Z., vol. 1, p. 206, 1918, + H.C, Lee, Canonical Factorization of Non-negative Hermitian Matrices, J. London Math. Soc., vol. 23, pp. 100-110, 1948,1 Introduction to Matrix Analysis 48. Let s be an N-dimensional vector such that (s,2) = 1. Determine the mini- mum of (# — 6, A( —8)) over this #region ascuming that A is Hermitian. See G. E. Forsythe and G. H, Golub, On the Stationary Values of a Second Degree Poly- nomial on the Unit Sphere, J. Soc. Indus. Appl. Math., vol. 13, pp. 1050-1088, 1965. 49. Let, my i= 1, 2... , N, be respectively the characteristic roota of A and B. Set M = max (|a¢s),|besl), @ = ) Ibcy — ass|/M. To each root d: there is a c Foot wy such that [Ar — uj] < (N+ 2)Md!™, Furthermore, the Xv and a; can be put, into one-to-one correspondence in such a way that [As — yl $2(V + 1)!MduN (A. Ostrowski, Mathematical Miscellanea XXVI: On the Continuity of Characteristic Roots in Their Dependence on the Matriz Elements, Stanford University, 1059). 80. Let {4x} be » bounded, monotone-increasing sequence of positive definite matrices in the sense that there exists a positive definite matrix B such that B — A, is non-negative definite for any n and such that A, — Ay-1 is non-negative definite for any n, Show that A, converges toa matrix A as n—> co (Riese). Bibliography and Discussion 82. We suppose that the reader has been exposed to the rudiments of the theory of linear systems of equations. For the occasional few who may have missed this or wish a review of some of the basic results, we have collected in Appendix A a statement and proof of the results required for the discussion in this chapter. The reader who wishes may accept on faith the few results needed and at his leisure, at some subsequent time, fill in the proofs. §5. The result in this section is a particular example of a fundamental principle of analysis which states that whenever a quantity is positive, there exists a formula for this quantity which makes this positivity appar- ent. Many times, it is not a trivial matter to find formulas of this type, nor to prove that they exist. See the discussion in G. H. Hardy, J. E. Littlewood, and G. Polya, Inequalities, Cam- bridge University Press, New York, pp. 57-60, 1934. and a related comment at the end of Chap. 5. 8. The concept of a positive definite quadratic form, as natural extension of the positivity of a scalar, is one of the most powerful and fruitful in all of mathematics. The paper by Ky Fan indicates a few of the many ways this concept can be used in analysis. Ky Fan, On Positive Definite Sequences, Ann. Math., vol. 47, pp. 593-607, 1946.Reduction of General Symmetric Matrices 2 For an elegant and detailed discussion of other applications, see the monograph Ky Fan, Les Fonclions définies-positives et les Fonctions complatement monotones, fascicule CXIV, Mem. sci. math., 1950. In the appendices at the end of the volume, we also indicate some of the ingenious ways in which quadratic forms may be used in various parts of analysis. A discussion of the diagonalization of complex non-Hermitian sym- metric matrices may be found in C. L. Dolph, J. E. McLaughlin, and I. Marx, Symmetric Linear Transformations and Complex Quadratic Forms, Comm. Pure and Appl. Math., vol. 7, 621-632, 1954. Questions of this nature arise as special eases of more general problems dealing with the theory of characteristic values and functions of Sturm- Liouville equations with complex coefficients. 15. For a further discussion of these problems, and additional refer- ences, soe R. D. Brown and I. M. Bassett, A Method for Calculating the First Order Perturbation of an Eigenvalue of a Finite Matrix, Proc. Phys. Sov., vol. 71, pp. 724-732, 1958. Y. W. Chen, On Series Expansion of a Determinant and Solution of the Secular Equation, J. Math. Phys., vol. 7, pp. 27-84, 1966. H.S. Green, Matriz Mechanics, Erven P. Noordhoff, Ltd., Groningen, Netherlands, 1965. The book by Green contains a discussion of the matrix version of the factorization techniques of Infeld-Hull. For some interesting extensions of the concept of positive definitenoss, see M. G. Krein, On an Application of the Fixed Point Principle in the Theory of Linear Transformations of Spaces with an Indefinite Metric, Am. Math. Soc. Transl., (2), vol. 1, pp. 27-35, 1955, where further references may be found. 816. In a paper devoted to physical applications, M. Lax, Localized Perturbations, Phys. Rev., vol. 94, p. 1391, 1954, the problem of obtaining the solution of (A + B)z = dx when ouly a few elements of B are nonzero is discussed.72 Introduction to Matrix Analysis For some interesting results concerning skew-symmetric matrices, see M. P. Drazin, A Note on Skew-symmetrie Matrices, Math. Gaz, vol. XXXVI, pp. 253-255, 1952. N, Jacobson, Bull. Amer. Math, Soc., vol. 45, pp. 745-748, 1939. With regard to Exercise 24, see also H. Stenzel, Uber die Datstellbarkeit einer Matrix als Product . Math. Zeit., vol. 15, pp. 1-25. Finally, for some interesting connections between orthogonality and quadratic forms, see W. Groebner, Uber die Konstruktion von Systemen orthogonaler Polynome in ein- und zwei-dimensionaler Bereich, Monatsh. Math., vol. 52, pp. 38-54, 1948. H. Larcher, Proc, Amer. Math. Soc., vol. 10, pp. 417-423, 1959. For an extensive generalization of the results of the two foregoing chapters, see F. V. Atkinson, Multiparametric Spectral Theory, Bull. Am. Math. Soc., vol. 74, pp. 1-27, 1968.5 Constrained Maxima 1, Introduction. In the previous chapters, we have discussed the problem of determining the set of values assumed by (z,Az) as z ranged over the region (z,z) = 1. In particular, we were interested in deter- mining whether (z,Az) could assume both positive and negative values. In this chapter we shall investigate this question under the condition that in addition to the relation (z,z) = 1, z satisfies certain additional linear constraints of the form (,b) =e 1,2,...,% a) In geometric terms, we were examining the set of values assumed by (z,Az) as x roamed over the unit sphere. We now add the condition that z simultaneously lie on a set of planes, or alternatively, is a point on a given N — & dimensional plane. Constraints of this nature arise very naturally in various algebraic, analytic, and geometric investigations. As a first step in this direction, we shall carry out the generalization of the algebraic method used in Chap. 1, obtaining in this way a more useful set of necessary and sufficient conditions for positive definiteness. These results, in turn, will be extended in the course of the chapter. 2. Determinantal Criteria for Positive Definiteness. In Chap. 4, it was demonstrated that a necessary and sufficient condition that a real symmetric matrix be positive definite is that all of its characteristic roots be positive, Although this is a result of theoretical value, it is relatively difficult to verify. For analytic and computational purposes, it is impor- tant to derive more usable criteria. The reason why a criterion in terms of characteristic roots is not useful in applications is that the numerical determination of the characteristic roots of a matrix of large dimension is a very difficult matter. Any direct attempt based upon a straightforward expansion of the determi- nant |A — \J| is surely destined for failure because of the extraordinarily large number of terms appearing in the expansion of a determinant. A determinant of order N has N! terms in its complete expansion. Since 3 a4 Introduction to Matrix Analysis 10! = 3,628,000 and 20! & 2,433 X 10°, it is clear that direct methods cannot be applied, even with the most powerful computers at one’s disposal. The numerical determination of the characteristic roots and vectors constitutes one of the most significant domains of matrix theory. As we have previously indicated, we will not discuss any aspects of the problem here. For the form in two variables, Qait2) = aut? + Zatz + arte? (1) we obtained the representation 2 @= onl + tan) + (0 - ae) a @) under the assumption that a1, % 0. From this, we concluded that the relations 1 aie ay >0 " M4 an >o ) were necessary and sufficient for A to be positive defin Let us now continue inductively from this point. Consider the form in three variables, Q(e1,22,02) = anti? + Qareiee + ares? + Qantas + Bauzies + anes? (4) Since au > 0 is clearly a necessary condition for positive definiteness, ‘as we see upon setting t2 = 22 = 0, we can write Q@,22,2) = an (« + Tey suey + (on = 2!) a on an +2 (os ~ Satu) tata + (0 If Q(21,22,2a) is to be positive definite, we see upon taking ny + (ute t+ aust) _ 9 aun that the quadratic form in zs and 24 must be positive definite. It follows, upon applying the result for 2X 2 matrices given in (3), that a set of "At the rate of one operation per microsecond, 20! operations would require over 50,000 years!Constrained Maxima 75 necessary and sufficient conditions that Q(x1,r2,22) be positive definite are ayy? Aiea aut oan an an>0 an->0 >0 6) au ay — U0 g,, — at 6 an au The first two conditions we recognize. Let us now see if we can per- suade the third to assume a more tractable appearance. Consider the determinant an as au az as] |g ay — Dy= {an ay des) = ”) ax az as] |g gy, — The last equation follows upon subtracting the first row multiplied by ay/ay from the second, and the first row multiplied by ais/an from the third (a method previously applied in Sec. 2 of Chap. 3). It follows then that the conditions in (6) may be written in the sug- geative form : On a2 M43} >0 Jan an an|/>0 ) Gy Gay as | ay di a2 aa an >0 Consequently we have all the ingredients of an inductive proof of Theorem 1. A necessary and sufficient set of conditions that A be posi- tive definite is that the following relations hold: D>O k=1,2,...,N (9) ‘where Dealal 45=1,2...,8 (10) We leave the details of the complete proof as an exercise for the reader. 3. Representation as Sum of Squares. Pursuing the analysis a bit further, we see that we can state the following result. Theorem 2. Provided that nv Dz is equal to zero, we may write x Qlentn «x)= ) (Di/Deadnt Do = 1 io) & where x went Yaa ba12...,N-1 ® in ww = 20 The ci are rational functions of the ary. This is the general form of the representation given in (2.5).16 Introduction to Matrix Analysis EXERCISE 1. What happens if one or more of the Ds are sero? 4. Constrained Variation and Finsler’s Theorem. Let us now con- sider the problem of determining the positivity or negativity of the quad- ratic form Q(z1,22, . . . ,tn) when the variables are allowed to vary over all z; satisfying the constraints x > bity = mn §=1,2...,hh 0 whenever (2,Bx) = 0, where B is a positive indefinite matriz, then there exists a scalar constant d such that (z,Az) + X(z,Bz) is positive definite. Proof. Writex = Ty, where T is an orthogonal matrix chosen so that k (Br) = Y wy? we > 15k SN (2) o& Since B is by assumption positive indefinite, we know that (z,Bz) can be put in this form. Under this transformation, we haye " (Az) = > Cys (3) cea 7 k If (2, Az) is to be positive whenever (2,B2) = ) uy? = 0, we must have & the relation x > cus > 0 @ ssshen for all nontrivial yus1, Yara, - + 5 YeConstrained Maxima 7 In the (N — k)-dimensional space (ys+1,Ysin - + - x), make an orthogonal transformation which converts the quadratic form in (4) into a sum of squares. Let the variables y1, ys, . . - , ye femain unchanged. In N-dimensional y space, write y = Sw to represent this transformation. In the w, variables, we have (2,Az) = Q(w) = y cen - y wan? ao fn x & ke +2 YY dsnwy + J dutoany (6) ens ce Our problem then reduces to demonstrating that a quadratic form of the type k d ), use + Q(w) ) a is positive definite whenever »; >0,i=1,2,..., N, and 2 is suf- ficiently large. Our proof is complete upon using the result indicated in Exercise 36 at the end of Chap. 4. Returning to the original variational problem, we now make the obser- vation that the k equations in (1) can be combined into one equation, 3 (Saey'=0 a ois Regarding 5 (s bias)? as @ positive indefinite form, we see that ‘Theorem 3 ‘yields the following result: A necessary and sufficient condition that Q(tuts, . . . tn) be positive for all nontrivial values satisfying the linear equations in (1) is that the ‘quadratic form Plouty, «au = Qlenty am 0S (3 bias)? be positive definite for all sufficiently large positive }. EXERCISE 4. Establish the foregoing result by considering the maximum over all 2; of the function Glen «st orp) ca f@) = - (1. Herstein)8 Introduction to Matrix Analysis 5, TheCa: 1. Let us begin the task of obtaining a usable criterion from the foregoing result with a case of frequent occurrence, k = 1. The quadratic form P is then x P= (ay + dbubudey a agat For this to be positive definite for all large positive X, we must have, in accord with Theorem 1, the determinantal inequalities lay + Abby] >0 4, 7=1,2,...,% @) fork =1,2,...,N. In order to obtain simpler equivalents of these inequalities, we employ the following matrix relation. ay aie 7 * Oy Ady PL 0 Ox Gs *** Gm dba 1 oo Gut Oz *** Gee Ab, [] 0 10 by by ts by HE Lb be oh L air + Ady? aye + Abbe + + + ay + Adib, Adi Qn + Abid; oer + Abe? + + + xe + Abrby Ade = (3) x, + Abbi daa + Abab2 + + ea + AO? Ady 0 0 see 0 -1 Taking determinants of both sides, we see that positivity of the determinant |a; + Abubyl, 7,7 = 1,2, ... , k, for all large \ is equiva- lent to negativity of the determinant ay a ay Ndi Qn G22 ++ * Gey Abe (4) Oy. Ga + * Gy Abe a! | for large positive } and k = 1,2,...,N.Constrained Maxima 9 Hence, we must have the relation Or ae °° * On by On ay st On Gn der ++ Oe be Qt. Ga + Gat » - <0 6) Cn CT a) byobe tt be OF} for all large positive \ and k = 1, 2, iN. Consequently, we see that a sufficient condition that @ be positive for all nontrivial z; satisfying the linear relation in bits + bate + 0 + + btw = is that the bordered determinants satisfy the relations ay di ts * au by Qn Ge *** aw be <0 (6) Ge, Gee ++ Oe be by bp ot be 0 fork = 1,2,...,N. Itis clear that the conditions fan az +++ an by] (dn dae + xe be) so (7) Oy ez + Oke Oe by be sss bh 0 N are necessary, but that some of these determi- nants ean be zero without disturbing the positivity property of Q. The simplest way in which these can be ze1 for some of the b; to be zero, Referring to (5), we see that if some element of the sequence of bor- dered determinants is zero, say the kth, then a necessary condition for for k =80 Introduction to Matrix Analysis positivity is that the relation Quoc au Oy + * + Ou >0 (8) Mot Oe be valid. A further discussion of special cases is contained in the exercises following. EXERCISES 1, Assuming that by » 0, obtain the foregoing results by solving for zx, zy = = (bits + bata + + + + + byaitn-)/bw, and considering Q(z) as a quadratic form in 2 ty zw-1 Does this yield an inductive approach? 2. If by = 0, the condition in (7) yields —ai.b* $0. Hence, if bs » 0, we must have an > 0. What form do the relations in (6) and (7) take if by = by = +++ = b= O1Sr aijeen; = O and ‘ the plane 2 twa = 0. Prove that a necessary and sufficient condition that the im plane be tangent to the surface is that! a, as ay wy Oy ay us a3 ay us| =0 Gis aay ty uw uw 0 ua 4. Determine the minimum distance from the quadrie surface (2,42) = 0 to the plane (u,z) = 0. 6. Prove that a necessary and sufficient condition that the line determined by the planes (u,z) = (v,2) = 0 be either tangent to (z,Az) = 0 or a generator, is that the bordered determinant woo us Ue A us vs we 4 ur us us um 0 0 M1 M2 mm 0 0 =0 + Cf, M. Bocher, Introduction to Higher Algebra, Chap. 12, The Macmillan Company, New York, 1947.Constrained Maxima 81 6. Determine the minimum distance from the quadric surface (z,Az) = 0 to the line determined by the planes (u,z) = (v,z) = 0. 6 A Minimization Problem. Closely related to the foregoing is the problem of determining the minimum of (2,2) over alll z satisfying the constraints vik a) (wa!) = As usual, the vectors and scalars appearing are real. Without loss of generality, we can assume that the vectors a! are linearly independent. For any z we know that the inequality (2,2) (z,a) (z,a') @a') (@at) +++ (a',a) : 20 Q) (z,a*) (a',at) +++ (aya) is valid. This is equivalent to the relation O bb bb... by by (oa) 6, (8) 2 aT ® since the determinant |(a‘,a’)| is positive in view of the linear independ- ence of the a‘. The right side in (3) is the actual minimum vatue, since equality is attained in (2) if z is chosen to be a linear combination of the a’, z= 5 gal (4) A In order to satisfy (1), the c; are taken to be the solutions of the equations 4 Ce (5) gun Since |(a‘,a')| 0, there is @ unique solution, which yields the unique minimiaing vector x.82 Introduction to Matrix Analysis EXERCISES 1, Deduce from this result that the quadratic form Om a a Q(z) = tN A zn ia negative lefinite whenever A is positive definite. 2. Give an independent proof of this result by showing that the associated quadratic form P(e) = 22,(zita + tots + ++ + + tvewss) + (2,42) ‘cannot be positive definite, or even positive indefinite if A is positive definite. % Assuming that (z,Az) possesses a positive minimum on the intersection of the planes (z,a°) = be, ¢ = 1, 2,, . . , k determine this minimum and thus derive the results of the preceding section in an alternate fashion. 7. General Values of k. Let us now return to the problem posed in Sec. 4. It is not difficult to utilize the same techniques in the treatment of the general problem. Since, however, the notation becomes quite cumbersome, it is worthwhile at this point to introduce the concept of a rectangular matrix and to show how this new notation greatly facilitates the handling of the general problem. Once having introduced rectangular matrices, which it would be well to distinguish by a name such as “array,” as Cayley himself wished to do, we are led to discuss matrices whose elements are themselves matrices, We have hinted at this in some of our previous notation, and we shall meet it again in the study of Kronecker products. 8. Rectangular Arrays. A set of complex quantities aj arranged in the form Aq| . @ ax, ane will be called a rectangular matriz. There is now no restriction that M=N. Weill call this array an M x N matrix. Observe the order of M and N, There is no difficulty in adding two M x N matrices, but the concept of multiplication is perhaps not so clear, We shall allow multiplication of a K X M matrix B by an M X N matrix A. This arises as beforeConstrained Maxima 83 from the iteration of linear transformations, Thus AB = C = (cu) (2) Schematically, M K K A|N | B\|M=|C!N a K XN matrix, where Mu cy =) aabys @) im The transpose A’ of an M X N matrix is an N X M matrix obtained by interchanging rows and columns. In many cases, these rectangular matrices can be used to unify a presentation in the sense that the distinction between vectors and matrices disappears. However, since there is a great conceptual differ- ence between vectors and square matrices, we feel that particularly in analysis this blending is not always desirable. Consequently, rectangular matrices, although of great significance in many domains of mathematics, will play a small role in the parts of matrix theory of interest to us here. Hence, although in the following section we wish to give an example of the simplification that occasionally ensues when this notation is used, we urge the reader to proceed with caution. Since the underlying mathe- matical ideas are the important quantities, no notation should be adhered to slavishly, It is all a question of who is master. EXERCISES 1. Let z and y be N-dimensional column vectors. Show that ay =a) 4G =1,% 0.00 2. If z is an N-dimensional column vector, then {MI = zz"| = a¥ = (2,04 9. Composite Matrices. We have previously defined matrices whose elements were complex quantities. Let us now define matrices whose elements are complex matrices, A= (Ag) @a Introduction to Matrix Analysis This notation will be particularly useful if it turns out that, for example, a4 X 4 matrix may be written in the form Qn Gi Ou Ow On G2 an ax -[4 ax] @) Gs, yz Gas One An Aas, On On Gu Ou, with the A;; defined by an = [2 eT An = [28 ax] on on = ® Ms eee) eee cera or, alternatively, with the definitions a an as au An=|an an a2 Au = | an @) an on an, au. An = (aa aa aa) An = (au) For various purposes, as we shall see, this new notation possesses certain advantages. Naturally, we wish to preserve the customary rules for addition and multiplication. It is easy to verify that addition is carried out in the expected fashion, and we leave it as an exercise for the reader to verify that the product of (Ag) and (By) may safely be defined as follows: (Au)(Bu) = (Cu) 6) where x Cus) AaBy © provided that all the products AaB, are defined. Furthermore, the associativity of multiplication is also preserved. EXERCISES 1. From the relation A BYE D -#] ‘AD — BC BA — AB c¢ DIL- A} |cD- Dc DA -cB. deduce that lé ze = |AD — BC) if BA = AB or CD = DC, and the required products exist.Constrained Maxima 85 2 Let M = A +4B, A and B real, and let mo [5 A] Show that |M, —M| = |M —AI| [Mf -XI| — (Szaraki and Wazewski) 8. Hence, show that |M,| = |4f] |1|, and that the characteristic roote of My are those of M and Mf. 4. Show that if M is Hermitian, then M, is symmetric. 10. The Result for General k. Let us now discuss the general problem posed in Sec. 4. As Theorem 3 asserts, the quadratic form r§ coon (ay + 3 babes) 2st ) a must be positive definite for large 2. Hence we must have the determinantal relations Jas +r 3 bd4| > 0 tj=12%. 8 (2) A forn = 1,2,..., N andall sufficiently large positive . In order to simplify this relation, we use the same device as employed in Sec. 5, together with our new notation, Let By denote the rectangular array bn ba ts bar biz bee bee By=|_ : (3) Diy baw ++ bay, Consider the following product of (N + k)-dimensional matrices, Av By] [In ¢|- An +ByBy By “@ By —-h)|By Te 0 -h where J, is the k-dimensional unit matrix and Ay = (as) 4 9=1,2,...,N Taking determinants of both sides, we have (a | Be MI = [da + ABB 6)86 Introduction to Matrix Analysis Hence, the conditions of (2) are replaced by (-)* >0 (6) ay Bw i for all large 2. I¢ follows as before that sufficient conditions for positivity are readily obtained, ‘The necessary conditions require a more detailed discussion because of the possible occurrence of zero values, both on the part of the coefficients by and the bordered determinants. MISCELLANEOUS EXERCISES 1. If (4;) is an arbitrary act of r X r matrices, there are unitary matrices U and V of order r X r, such that UAV = Dj, D; diagonal and real, if and only if AvAy = AjA; and AjAi = A;A; for all i and j (N. A. Wiegmann). x 2. Let) sii; be a positive definite form. Show that is a positive definite form in the N — 1 variables 21,24, . . +) 2-1 Zn ++) 2H What are the characteristic roote and vectors of the matrix of this quadratic form? Generalize this result. 8. Introduce the following correspondence between complex numbers and matrices: raativ~z=[ 3 | Show that @ ZW = WZ @) 202! ()2~Z, w~W implies that ew ~ZW 4. Use this correspondence and de Moivr Bforn= 1,22... 6. From Exercise 3, we ace that we have the correspondence amfot] é-[-t a] e- [2p Ste] theorem to determine the form of Using this in the matrix [ESE 3]Constrained Maxima 87 or, dropping parenthesee, om om om -m on -m on -t om ot) mt -m ot Want a= Consider the equivalence Q~Q,. Show that Q~Q,, W~ W, results in QW ~ Qu Hence, determine |Q,|, and the characteristic values of Q,. 6. How does one determine the maximum value of (z,4z) subject to the constraints (2) = 1, Geof) = 0,5 = 1,2,..., 87 ‘1. Write, for two symmetric matrices, A and B, A > B to indicate that A — B is non-negative definite. Show that A > B does not necessarily imply that A* > B*. 8. Let g(t) > 0 and consider the quadratic form ante) = foto (3 aut) a Show that Qw(z) is positive indefinite, and, hence, derive various determinantal inequalities for the quantities m =f," (et at (Stet). 9. Let g(t) > 0 and consider the Hermitian form Hvta) = fo 5 nettle vo a Show that Hy(z) ia positive definite, and, hence, derive determinantal inequalities 1 for the quantitics m, = i, (tye de (0. Toeplite). 10. Show that > a 0 for x > 0 is treated in J. W. Gaddum, Linear Inequalities and Quadratic Forms, Pacific J. Math., vol. 8, pp. 411-414, 1958,6 Functions of Matrices 1, Introduction, In this chapter, we wish to concentrate on the con- cept of a function of a matrix. We shall first discuss two important matrix functions, the inverse function, defined generally, and the square root, of particular interest in connection with positive definite matrices. Following this, we shall consider the most important scalar functions of a matrix, the coefficients in the characteristic polynomial, ‘The problem of defining matrix functions of general matrices is a rather more difficult problem than it might seem at first glance, and we shall in consequence postpone any detailed discussion of various methods that have been proposed until a later chapter, Chap. 11. For the case of symmetric matrices, the existence of the diagonal canonical form removes most of the difficulty. This diagonal representation, established in Chap. 4, will also be used to obtain a parametric representation for the elements of symmetric matrices which is often useful in demonstrating various results, As an example of this we shall prove an interesting result due to Schur con- cerning the composition of two positive definite matrices. Finally, we shall derive an important relation between the determi- nant of a positive definite matrix and the associated quadratic form which will be made a cornerstone of a subsequent chapter devoted to inequalities, and obtain also an analogous result for Hermitian matrices. 2. Functions of Symmetric Matrices. As we have seen, every real symmetric matrix can be represented in the form dw, 90Functions of Matrices o1 where Tis an orthogonal matrix. From this it follows that dat at 0 Ata . T" @) dwt. for any integer k. Hence, for any analytic function f(z) we can define f(A) to be FQa) £01) 0 f(A) =T - T (3) SOx), whenever the scalar functions f(d,) are well defined. 3. The Inverse Matrix. Following the path pursued above, we are led to define an inverse matrix as follows. ar Art 0 T (1) dw whenever the }, are all nonzero. It is clear that A~' has the appropriate properties of an inverse, namely, AA“ = AA = 1 (2) 4. Uniqueness of Inverse. It is natural to inquire whether or not the inverse matrix is unique, if it exists. In the first place, let us observe that the condition that the d, are all nonzero is equivalent to the con- dition that |A| #0. For if \ = 0 is a root of |A — AJ| = 0, we see that [A| = 0, and if |A| = 0, then \ = 0 is a root. If A is 8 matrix for which |A| = 0, it will be called singular. If |A| # 0, A is called nonsingular, It is immediately seen that a singular matrix cannot have an inverse, For, from AB = 1, we have |AB| = 1 or |A| |B] = 1. No such equa- tion can hold if |A| = 0. Let us now show, however, that any nonsingular matrix, symmetric or not, possesses a unique inverse, For the case where A is symmetric, the matrix found in this new way must coincide with that given by (3.1).92 Introduction to Matrix Analysis ‘The equation AB = / is equivalent to the N* equations x D ows = 65 F=1,2,...,N w et Fixing the value of j, we obtain N linear equations for the quantities byy, k=1,2,...,N. The determinant of this set of equations ie |A| = |as{, regardless of the value of j. It follows then that if A is nonsingular, there is a unique solution of the equations in (1), Furthermore, we see that the solution is given by At = (as/|A]) (2) where a, is the cofactor of aj in the expansion of the determinant of A. EXERCISES 1, Show that A~, if it exists, is aymmetric if A is symmetric in three way (a) Directly from (3.1) (b) Using the expreasion in (4.2) (©) Using the relations AA~! = A-!A = J and the uniqueness of the inverse 2. If A is singular, show that we can find a matrix B whose elements are arbitrarily small and such that A + B is nonsingular, 3. Let [AZ — Al =A" + c:A¥-! + + +» + ey, the characteristic polynomial of A. Show that A-! = —(A¥-! + c,A¥-2 + ++ + + ew-i)/em, whenever cy x 0. 4. If As, As, By, Bs are nonsingular square matrices of the same order, then ‘At ay" < (o — BiBr'Ay)"" (Ai — Bebra) Ar By (By = AvAyBs)"? (Bs — AaArB,)~* 5, Let A be an M X N matrix and consider the eet of inconsistent linear equations z= Ay, where 2 is an N-dimensional vector and y an M-dimensional vector. To obtain an approximate solution to these equations, we determine y as the minimum of the quadratic function (x — Ay, z — Ay). Show that, under the above assumption of inconsistency, there is a unique solution given by y = (A’A)~1A’z, and determine the minimum value of (z — Ay, 2 — Ay). 6. Let A > B, for two symmetric matrices A and B, denote as previously the fact that A — B is non-negative definite. Show that A > B implies that B~! > A-', provided that A~t and B-! exist, 7. If S ia a real skew-symmetric matrix, then J + 8 is nonsingular, 8. If Sis. real skew symmetric, then T= (1st +s> is orthogonal (the Cayley transform), 8. If A ig an orthogonal matrix such that A + / is nonsingular, then we can write A=(I-8)(1 +8) where S ia a real skew-symmetric matrix, 10. Given a matrix A, we can find a matrix J, having +18 slong the main diagonal and zeroes elsewhere, such that JA + / is nonsingular. 11. Using this result, show that every orthogonal matrix A can be written in theFunctions of Matrices 93 form Ass = Syl + sy where J is as above. 12. Show that (@) A: > Bs, As > By implies Ay + Ar > Bi + Be (b) Ai > Bs, B: 2 Cr, implies Ay > Cy (©) Ai > Bs, As > Bs, does not necessarily imply A141 > ByBs, even if ArAs and B,By are both symmetric. Thus, A > B does not necessarily imply that A? > B}, (4) Ar > By implies that T/A,7 > P'B,T 18. Show that A-! can be defined by the relation (z,A~'z) = max [2(z,y) — (yAy)] ¥ if A is positive definite. 14. From this, show that B-! > A~'if A and B are aymmetric and A > B > 0. 16. If A is a matrix whose elements are rational integers, and if |A| = +1, then A-! is a matrix of the same type. 16. If A is a matrix whose elements are complex integers, ie., numbers of the form 2) + iy; where x, and y, are rational integers, and if |A| = +1 or +i, then 4-' isa matrix of the same type. 17. What is a necessary and sufficient condition that the solutions of Az = y be integers whenever the components of y are integers, given that the elements of A are integers? 18. Show that |A + iB)? = |A[}|I + A~\BA~'B| if H = A + éB is Hermitian, and A is nonsingular. 19. Show that (2,H2) = (zz) + (y,Ay) + 2(Bz,y) if H = A + iB is Hermitian ands =z + iy. 20. Show that (z,Az) = (z,A.z) where A, = (A + A")/2. 21. If A and B are alike except for one column, show how to deduce the elements of A“! from those of B-', 22, If A has the form 0 nm 0 ym 0m 0 O yO xm a-[° 0 wy then, provided that y: x 0, in A~' the element bys is given by by =O kod want = ( =H, keven Yael where zo = 1. (Clement) 28. From this deduce the form of A~!, 6, Square Roots. Since a positive definite matrix represents a natural generalization of a positive number, it is interesting to inquire whether or not a positive definite matrix possesses a positive definite square root.cry Introduction to Matrix Analysis Proceeding as in Sec. 2, we can define A¥ by means of the relation rtd ri 0 AM=T - r a obtaining in this way a matrix satisfying the relation B* = A, which is positive definite if A is positive definite, There still remains the question of uniqueness. To settle it, we can proceed as follows. Since B is symmetric, it possesses a representation ay, where Sis orthogonal. It follows that Band B* = A commute. Hence, both can be reduced to diagonal form by the same orthogonal matrix 7. The relation Bt = A shows that B has the form given in (1) where the positive square roots are taken. EXERCISES 1, How many symmetric square roots does an N X N positive definite matrix possess? 2. Can a aymmetric matrix possess nonsymmetrie square roots? 3. If Bis Hermitian and B > 0, and if A? = Band (A + A*) > 0, then A is the unique non-negative Hermitian square root of B (Putnam, On Square Roots and Logarithms of Operators, Purdue University, PRF-1421, 1958). 4. If e4 = B, B positive definite, and A < (2 log 2)f, then A ia the unique positive definite logarithm of B (Putnam). 6. Parametric Representation. The representation aw,Functions of Matrices 95 furnishes us a parametric representation for the elements of a symmetric matrix in terms of the elements t;; of an orthogonal matrix 7, namely, x oy =) Aina 2) eet which can be used as we shall see in a moment to derive certain properties of the ay, EXERCISE 1. Obtain a parametric representation in terms of cos @ and sin 6 for the elements of a general 2 X 2 symmetric matrix. 7. A Result of I, Schur. Using this representation, we can readily prove Theorem 1. Theorem 1. If A = (a,), B = (bi) are positive definite, then = (aby) is positive definite. Proof. We have N » aybiaans = : bans (3 Atle) fa hs =yu (3 mataszn) M>0 a) a N\A y As a quadratic form in the variables zulu the expression) biadutitn is ihe positive unless the quantities xitu are all zero. Since Yatut = Sat Sut = Yat @ & i we see that this cannot be true for all i, unless all the 2; are zero, This establishes the required positive definite character. EXERCISE 1, Use Exercise 2 of the miscellaneous exercises of Chap. 5 to obtain an inductive proof of this result, starting with the easily established result for N = 2. 8. The Fundamental Scalar Functions. Let us now consider some properties of the scalar functions of A determined by the characteristic polynomial, IAT — Al = % — @i(A)ANH! + 2( AY? + + (= 1)%@n(A) (1)96 Introduction to Matrix Analysis In this section the matrices which ocour are general square matrices, not necessarily symmetric. It follows from the relations between the coefficients and the roots of polynomial equation that dx (A) = Aide + dw @) On the other hand, writing out the term in \¥-' of the characteristic polynomial as it arises from the expansion of the determinant |A — AJ|, we see that oA) = au + Get +> + ww 3) while setting } = 0 yields x(A) = |A] @ The linear function ¢,(A) is of paramount importance in matrix theory. It is called the trace of A, and usually written tr (A). It follows from (3) that tr (A + B) = tr (A) + tr (B) 6) and a little calculation shows that tr (AB) = tr (BA) (6) for any two matrices A and B. These relations hold despite the fact that there is no simple relation connecting the characteristic roots of A, Band A + Bor AB. This last result is a special case of Theorem 2. Theorem 2, For any two matrices A and B, we have g(AB) = $(BA) k=1,2,...,N @” The proof will depend upon a fact that we already know, namely, that the result is valid for k = N, that is, |AB| = |BA| for any two matrices. If A is nonsingular, we have the relations [AT = AB] = |AQA- ~ B)| = |QA-! — B)A] = PT — BAL 8) which yield the desired result, If A is singular, we can obtain (8) by way of a limiting relation, starting with A + el, where is small and A + ef is nonsingular.Functions of Matrices 7 In the exercises immediately below, we indicate another approach which does not use any nonalgebraic results. EXERCISES 1. Show that |J +A] = 1+¢tr(A) + + 2. Show that #x(7AT-) = $i(A); os(PAT) = = (4) if T is orthogonal. 8. Show that any polynomial p(au,e1s, . . . aw) in the elements of A which has the property that p(AB) = p(BA) for all A and B must be a polynomial in the fune- tions $(A). 4. Show that tr (aay = = tr ((BA)*) fork = 1,2, . . . ,andhence that ¢.(AB) = $x(BA) for k = 1, 2, 5. Show that u (AX) - 0 for all X implies that A = 0. 6. Exercise 2 and Theorem 2 above are both specia! cases of the result that for any m matrices Ay, Ar, . . . , Am, of order N, the quantity ¢i(A142 + +.- Aq) remains unchanged after a cyclic permutation of the factors Ay, As, .. . » Am 9. The Infinite Integral {“ e49 dz. One of the most important integrals in analysis is the following: tem [oe [Teen de wo where dz = dzydz,--- dry. Although the indefinite integral cannot be evaluated in finite terms, it turns out that the definite integral has a quite simple value. Theorem 3. If A is positive definite, we have or te = fp (2) We shall give two proofs. First Proof. Let T be an orthogonal matrix reducing A to diagonal form, and make the change of variable z = Ty. Then (2, Az) = (y,Ay) (3) ” Furthermore, [] dz; = [] dys, since the Jacobian of the transformation int in z = Ty is |T|, which can be taken tobe +1. Since z = Ty is a one-to- one transformation, it follows that . [ ent aa x : Hf een dys = hy @ Iy= dwn? dy98 Introduction to Matrix Analysis y From the fact that [] \, = [A], and the known evaluation [Letae =o we obtain the result stated in (2). Let us now give a second proof. In Sec. 3 of Chap. 5, it was shown that we may write » (Aa) = Dor nt Dow 6 & where x ment Yo dy; k=1,2,...,N © jen provided that A is positive definite. Let us then make a change of varia- ble from the zs to the ys. The Jacobian is again equal to 1, and the transformation is one-to-one. It follows that r=([ron) [core LL evennives dyn) - 3a Tae (2) EXERCISES 1. Show that if A is positive definite and B symmetric, we have . = ecm we i, : ve [Teens 2 de = a where the principal value of |A + ‘B)* is taken, in the following steps: (a) Set z = Ty, whore T’AT = A, a diagonal matrix (b) Make the further change of variable, yx = 24/ds%* (c) The integrand is now x - Y wten @ fet Reduce C to diagonal form by means of an orthogonal transformation z = Sw, (@) Evaluate the remaining integral, ‘2. Evaluate integrals of the form tee fon 1m, n positive integers or zoro, in the following fashion, Write ante nia as® day dea tee Gran — anFunctions of Matrices 99 Two = ( 0 for all positive definite B. & Show that CMe tea + nal = ett and hence, obtain relations connecting ¢4(A) and tr (A). 9. Show that AB + Band BA + B have the same determinant by showing that tr ((AB + B)") = tr (BA + B)*) forn = 1,2, .. 10, Let By = 1, By = ABi-1 + kil, whore ky = tr (A Bj..)/i; then (A — Mp mt thet + Leverrier)Functions of Matrices 10! 11, Let 4, B,..., be commutative matrices, and let f(e,zs,...) be any rational function. The characteristic roots @;, ds, . . . , dn of A, bi, by, - . - , dn of B, . . . , can be ordered in such a way that the characteristic roots of f(A,B, . . .) are f(ay,bs, « . .)s f(@ajbs, . . »), and 80 on. 12, Show that every nonsingular matrix may be reprosented as a product of two not necessarily real symmetric matrices in an infinite number of ways (Voss). 18. If His non-negative definite Hermitian, there existe a triangular matrix 7’ such that H = TT* (Toeplitz), 14, For every A there is a triangular matrix such that TA is unitary (E. Schmidt), 15. Let P, Q, R, X be matrices of the second order. Then every characteristic root. of a solution X of PX? + QX + R = 0 isa root of [Px + Qd + Rl = 0 (Sylvester). 16. If A, B, C are positive definite, thon the roots of |4d? + Bd + C] = 0 have negative real parts, 17. Consider matrices A and B such that AB = r.BA, where r1 is a qth root of unity. Show that the characteristic roots of A and B, Xs and i, may be arranged so that those of A +B are (Mt + wit)" and those of AB are r-"\iyi. Different branches of the gth root may have to be chosen for different values of ¢ (Potter). 18. Using the representation anit apa ~ eeae-eotn TT des and expanding hoth sides in powers of «, obtain representations for the coefficients in the expansion of |A + ¢B] in powers of «, and thus of the fundainental functions on(A). 19. Let A and B be Hermitian matrices with the property that 14 + c2B has the characteristic roots ¢:\; + Cz for al} ecalars ¢, and ¢x, where Xi, ui are, respectively, the characteristic roots of A and B. Then AB = BA (Motzkin). 20. Show that A is nonsingular if Jax) >) lou, i=1,2,.. 1. The characteristic vectors of A are characteristic vectors of p(A), for any poly nomial p, but not necessarily conversely. 22. If A and B are symmetric matrices such that [P= NAL = wBl = 0 = 4 = 4B] for all hand y, then AB = 0 (Craig-Hotelling). 28. Let AB = 0, and let p(4,B) be a polynomial having no constant term. Then IMF = p(A,B)| = X-¥M — p(A,0)] [AT — p(B,0)| (H. Schneider), 24. Generalize this result to p(A,,Ax,..- An), where A;A; =0, 1 0, zur — i* > 0, In other words, this is the region where X is positive definite, ‘Then ATO 34) JY) = for Re (s) >34 (Ingham-Siegel). (See the discussion of Sec. 9 for further information concerning results of this nature.) 28, Let A be a complex matrix of order N, A = (an) = (ba + itr). Let B be the real matrix of order 2N obtained by roplacing br + ic. by its matrix equivalent bre <] ba tien ~[ br So Show that A-! can be obtained from B-! by reversing thia process after B-! has been obtained. 29. If A is aymmetric and || = 0, then the bordered determinant, an an aN aw ty Be) = am, am ayy zw mw 0 multiplied by the leading second minor of A is the square of a linear function of the =; (Burnside-Panton). 380. Consequently, if A is aymmetric and the leading first minor vanishes, the determinant and its leading second minor have opposite signs. 81, Let X and A be 2X 2 matrices. Find all solutions of the equation X* = A which have the form X = ci + csA, where cy and cs are scalars, 82, Similarly, treat the case where X and A are 3 X 3 matrices, with X = cif + aA + c4A%, 88. Let f(0) be » polynomial of dogree leas than or equal to N — 1, and let di, De + +) Aw be the M distinct characteristic roots of A. Then va = Soo Mh [ASM] 1 sign dnt (Sylvester interpolation formula) What is the right-hand side equal to when f(A) is a polynomial of degree greater than NP ‘$4. Is the formula valid for f(A) = A“? ‘36. Show that Nl Ale Jocaner 8 80° ten = oTFunctions of Matrices 103 86. If the elements ai; of the N x N matrix A are real and satisfy the conditions jel ax: > J) laut = 1,2, . . ., M, thom the absolute term inthe Laurent expansion of Set von Ken see) = TE (one /s) jet 4 on [sil = [esl =» - » = [ew| is equal to Al“. Alternatively, ¥ 1 ii fT fate [4] $ § ’ [ =| ; where the contour of integration is |z;| = 1. ‘This result remains valid if A is complex and |ais| > y |ai;|. A number of further int reults are given in P, Whittle. ‘The original result was first given by Jacobi and applications of it may be found in H. Weyl? and in A. R. Forayth.* ve definite, then Dots > 0, and thus a ‘87. Show that if A and B are non-nej establish Schur’s theorem, given in Sec. 7 (Fejer). 88, Let {As} be a set of N XN symmetric matrices, and introduce the inner product (AgAs) = tr (Asa) Given a set of M = N(N +1)/2 linearly independent symmetric matrices (Act, construct an orthonormal linearly independent set (Y;} using the foregoing inner product and show that every N x N eymmetrie matrix X ean be written in the form u X=) eat, where ci = (X,Y). & 89. For the case of 2 X 2 matrices, take w-foo] &-[o i] #-[0 al What is the corresponding orthonormal set? 40. Similarly, in the general case, take for the A, the matrices obtained in the analogous fashion, and construct the Yi. 41, The Cayley-Hamilton theorem asscrts that A satisfies a scalar polynomial of degree N. Call the minimum polynomial associated with A the scalar polynomial of minimum degree, q(x), with the property that 9(4) = 0. Show that 9(x) divides |A - nl. 1P. Whittle, Some Combinatorial Results for Matrix Powers, Quart. J. Math., vol. 7, pp. 316-320, 1956. 1H, Weyl, The Classical Groups . . . , Princeton University Pross, Princeton, N.J., 1946, 4A. R. Forsyth, Lectures Introductory to the Theory of Functions of Two Complex Variables, Cambridge University Prose, New York, 1914.104 Introduction to Matrix Analysis 42. Find the Jacobian of the transformation Y = X-1,* 43. Evaluate the matrix function A [foe ~ Aytae under various assumptions concerning A and the contour C, i ‘The fit use of a contour integral =~ [ QL = T)-Y0) dd to represent f(T) is i Jo found in Poincaré (H. Poincaré, Sur lee groupes continus, Trans, Cambridge Phil. Soc., vol. 18, pp. 220-225, 1899, and Ocurres, vol. 3, pp. 173-212). 44, If Az = b has a solution for all b, thon A! exists, 45. For any two positive constants k: and kz, there exist matrices A and Bauch that every element of AB — I is less than ky in absolute value and such that there exist, elements of BA — I greater than ky (Householder). If Aris nonsingular, then Ar At taille = deter As Ae 47. If BC = CB, and 714: -B oT 0 +. Cc -B OT Oe. Aw] 0 c -B I o then 10: 1 Dt 0 1. 4°") dD, DT 0 eee Ds Dy Dy T 0 . Determine recurrence relations for the Di. 48. If A is an N X M matrix of rank r, r 0, then F(A) is positive definite whenever A is nwo positive definite. 52. If F(A) is positive definite for every A of the form (ai.;), then F(2) = y cna" So with > 0(W, Rudin), An earlier result is due to Schoenberg, Duke Math. J, 1942. A = (aj) is called a Jacobi matrix if aij = 0, for ji — jl 22. IfA Jacobi matrix, to what extent are the elements az; determined, if the ie root are given? This is a finite version of a problem of great impor- tance in mathematical physics, namely that of determining the coefficient function ‘o(z), given the spectrum of u” + \4(z)u = 0,aubject to various boundary eonditions.* G4. If A has the form am 1 0 eee A=|1 wi oo: Oia t oo... how does one determine the a; given the characteristic roots? 65, Show that for any A, there is a generalized inverse X such that AXA = A, XAX = X, XA = (XA), AX = (AX)’ (Penrose), 66. X is called a generalized inverse for A if and only if AXA = A. Show that X is a gonoralized inverse if and only if thero exist matrices U, V, and W such that In U x-@ ly wl P 0 Pag = [¢ a and In is the identity matrix of dimension n. 87. X is called a reflexive generalized inverse for A if and only if AXA = A and XAX = X. Show that x=a[f u |’ (C. A. Rhode) where vv 68. Show that if Az = y is consistent, the gencral solution is given by z= Aly + (I AtA)e where A! is the Penrose-Moore inverse and z is arbitrary (Penrose). For compu tational aspects see Urquehart.? 1G. Borg, Eine Umkehrung der Sturm-Liouvilleschen Eigenwerte Aufgabe, Acta Math., vol. 78, pp. 1-96, 1946, I. M. Gelfand and B. M. Leviton, On the Determina- tion of a Differential Equation from Its Spectral Function, Trans. Am. Math. Soc, vol. 2, pp, 253-304, 1955. * See the reference in the Bibliography and Discussion, Sec. 3. "N.S. Urquehart, Computation of Generalized Inverse Matrices which Satisfy Specified Conditions, SIAM Review, vol. 10, pp. 216-218, 1968.106 Introduction to Matrix Analysis See also Langenhop,’ Radhakrishna Rao,t and Schwerdtfeger.? 89. If A~! exists and if more than N of the elements of A are positive with the others non-negative, then A~* has at least one negative clement, A has the form PD with P a permutation matrix and D a diagonal matrix with positive elements if and only if A and A~! both have non-negative entries (Spira). 60, Consider the matrix A + «B where A is positive definite, B is real and 9 metric and ¢ is small. Show that we can write (A + eB)" = AnH(1 + 68)-14 where $ is aymmotric, and thus obtain a eymmetric perturbation formule, (A + eBYt = Amt AMSAH Ee, 61. Let f(z) be analytic for |e] <1, f(e) = aet+aw+ +++. Write f(t = Go, + aye + amet + +--+, k= 1, 2,..,. Can one evaluate layl, ¢, j = 0, 1, « ..,N, in simple terms? 62, Write f(e) = bie + dae? + + + +, and f(e) = buz + buxzt + + ++, wheres”) in the kth iterate, that ia, f = f(f), ...,f® = sf), .... Can one evaluate [es 4, § = 0,1, ..., N, in simple terms? Bibliography and Discussion §1. There is an extensive discussion of functions of matrices in the book by MacDuiee, C, C, MacDuffee, The Theory of Matrices, Ergebnisse der Mathe- matik, Reprint, Chelsea Publishing Co., New York, 1946. More recent papers are R. F, Rinehart, The Derivative of a Matrix Function, Proc. Am. Math. Soc., vol. 7, pp. 2-5, 1956. H. Richter, Uber Matrixfunktionen, Math. Ann., vol. 122, pp. 16-34, 1950. 8. N. Afriat, Analytic Functions of Finite Dimensional Linear Transformations, Proc. Cambridge Phil. Soc., vol. 55, pp. 51-56, 1959. and an excellent survey of the subject is given in R. F. Rinehart, The Equivalence of Definitions of a Matric Fune- tion, Am, Math, Monthly, vol. 62, pp. 395-414, 1955. 1, E, Langenhop, On Generalized Inverses of Matrices, SIAM J. Appl. Math., vol. 15, pp. 1239-1246, 1967. *C, Radhakrishna Rao, Calculus of Generalized Invorses of Matrices, I: General Theory, Sankhya, sor. A, vol. 29, pp. 317-342, 1967. 4H. Schwerdtfeger, Remarks ou the Gencralized Inverse of a Matrix, Lin, Algebra Appl., vol. 1, pp. 325-328, 1988,Functions of Matrices 107 See also P. §. Dwyer and M. 8. Macphail, Symbolic Matrix Derivatives, Ann, Math, Stat., vol. 19, pp. 517-534, 1948, The subject is part of the broader field of functions of hypercomplex quantities. For the case of quaternions, which is to say matrices having the special form mt tom —m om tu ot @ ha a Zi 22 there is an extensive theory quite analogous to the usual theory of func- tions of a complex variable (which may, after all, be considered to be the theory of matrices of the special form alee 1: with z and y real); see R. Fueter, Functions of a Hyper Complex Variable, University of Zurich, 1948-1949, reprinted by Argonne National Laboratory, 1959. R. Fueter, Commentarit Math. Helveticii, vol. 20, pp. 419-20, where many other references can be found. R. F. Rinehart, Elements of a Theory of Intrinsic Functions on Algebras, Duke Math. J., vol. 27, pp. 1-20, 1960. Finally, let us mention a paper by W. E. Roth, W. E. Roth, A Solution of the Matrix Equation P(X) = A, Trans. Am, Math, Soc., vol. 30, pp. 597-599, 1928. which contains a thorough discussion of the early results of Sylvester, Cayley, and Frobenius on the problem of solving polynomial equations of this form. Further references to works dealing with functions of matrices will be given at the end of Chap. 10. 8. The concept of a generalized inverse for singular matrices was dis- cussed first by E. H. Moore, General Analysis, Part I, Mem. Am. Phil. Soc., vol. 1, p. 197, 1935. and then independently discovered by R. Penrose, A Generalized Inverse for Matrices, Proc. Cambridge Phil. Soc., vol. 51, pp. 406-413, 1955.108 Introduction to Matrix Analysis ‘These and other matters are discussed in M. R. Hestenes, Inversion of Matrices by Biorthogonalization, J. Soc. Indust. Appl. Math., vol. 6, pp. 51-90, 1958, An exposition of the computational aspects of matrix inversion which contains a detailed account of many of the associated analytic questions may be found in J. von Neumann and H. Goldstine, Numerical Inverting of Matrices of High Order, Bull. Am, Math, Soc., vol. 53, pp. 1021-1099, 1947. 4. An excellent account of various methods of matrix inversion, together with many references, is given in D. Greenspan, Methods of Matrix Inversion, Am. Math. Monthly, vol. 62, pp. 303-319, 1955. The problem of deducing feasible, as distinguished from theoretical, methods of matrix inversion is one of the fundamental problems of numerical analysis. It will be extensively discussed in a succeeding volume by G. Forsythe. The problem of determining when a matrix is not singular is one of great importance, and usually one that cannot be answered by any direct calculation. It is consequently quite convenient to have various simple tests that can be applied readily to guarantee the nonvanishing of the determinant of a matrix. Perhaps the most useful is the following (Iixercise 20 of Miscellaneous Exercises) : If A is 6 real matrix and asl > lax] ¢=1,2,...,, then a Al #0. “ For the history of this result and a very elegant discussion of exten- sions, see ©. Taussky, A Recurring Theorem on Determinants, Am. Math. Monthly, vol. 56, pp. 672-676, 1949. This topic is intimately related to the problem of determining simple estimates for the location of the characteristic values of A in terms of the elements of A, a topic we shall encounter in part in Chap. 16. Gener- ally, any extensive discussion will be postponed until a later volume. A useful bibliography of works in this field is 0, Taussky, Bibliography on Bounds for Characteristic Roots of Finite Matrices, National Bureau of Standards Report, September, 1951. It should be noted, however, that the foregoing result immediately yields the useful result of Gersgorin that the characteristic roots of A mustFunctions of Matrices 109 lie inside the circles of center au and radius ) |ay|,for? = 1,2, .. . ,N. ih The original result is contained in 8. Gersgorin, Uber die Abgrenzung der Kigenwerte einer Matrix, I, Akad. Nauk SSSR, vol. 7, pp. 749-754, 1931. 6. As mentioned previously, » parametric representation of different type can be obtained for 3 x 3 matrices in terms of elliptic functions; see F, Caspary, Zur Theorie der Thetafunktionen mit zwei Argumenten, Kronecker J., pp. 74-86, vol. 94. F, Caspary, Sur les systémes orthogonaux formés par les fonctions théta, Comptes Rendus de Paris, pp. 490-493, vol. 104. as well aa a number of other papers by the same author over the same period, 87. See I. Schur, Bemerkungen zur Theorie der beschrainkten Bilinearformen mit unendlich vielen Vertinderlichen, J. Math., vol. 140, pp. 1-28, 1911. See also L. Fejer, Uber die Eindeutigkeit der Losung der linearen partiellen Differentialgleichung zweiter Ordnung, Math. Z., vol. 1, pp. 70-79, 1918. where there is reference to an earlier work by T. Moutard. See also H. Lewy, Composition of Solutions of Linear Partial Differential Equations in Two Independent Variables, J. Math. and Mech., vol. 8, pp. 185-192, 1959. The inner product (A,B) = Zayb, ia used by R. Oldenburger for another purpose in R. Oldenburger, Expansions of Quadratic Forms, Bull. Am, Math. Soc., vol. 49, pp. 136-141, 1943. §9. This infinite integral plays a fundamental role in many areas of analysis. We shall base our discussion in the chapter on inequalities upon it, and an extension. ‘An extensive generalization of this representation may be obtained from integrals first introduced by Ingham and Siegel. See, for the original results,110 Introduction to Matrix Analysis A. E. Ingham, An Integral which Occurs in Statistics, Proc. Cam- bridge Phil. Soc., vol. 29, pp. 271-276, 1933. ©. L. Siegel, Uber die analytische Theorie der quadratischen Formen, Ann, Math,, vol. 36, pp. 527-606, 1935. For extensions and related results, see R. Bellman, A Generalization of Some Integral Identities Due to Ingham and Siegel, Duke Math, J., vol. 24, pp. 571-578, 1956. ©. 8. Herz, Bessel Functions of Matrix Argument, Ann. Math., vol. 61, pp. 474-523, 1955, I. Olkin, A Class of Integral Identities with Matrix Argument, Duke Math, J., vol. 26, pp. 207-213, 1959. 8, Bochner, “Group Invariance of Cauchy’s Formula in Several Variables,” Ann. Math., vol. 45, pp. 686-707, 1944. R. Bellman, Generalized Eisenstein Series and Nonanalytic Auto- morphic Functions, Proc. Natl. Acad. Sci, U.S., vol. 36, pp. 356-350, 1950. H. Maass, Zur Theorie der Kugelfunktionen einer Matrixvariablen, Math, Z., vol. 135, pp. 391-416, 1958. H. 8. A. Potter, The Volume of Certain Matrix Domain, Duke Math, J., vol. 18, pp. 391-397, 1951. Integrals of this type, analogues and extensions, arise naturally in the field of statistics in the domain of multivariate analysis. See, for example, T. W. Anderson and M. A. Girshick, Some Extensions of the Wishart Distribution, Ann. Math. Siat., vol. 15, pp. 345-357, 1944. G. Rasch, A Functional Equation for Wishart’s Distribution, Ann. Math, Stat., vol. 19, pp. 262-266, 1948. R. Sitgreaves, On the Distribution of Two Random Matrices Used in Classification Procedures, Ann. Math. Stat., vol. 23, pp. 263-270, 1952. In connection with the concepts of functions of matrices, it is appropri- ate to mention the researches of Loewner, which have remarkable con- nections with various parts of mathematical physics and applied mathe- matics, As we have noted in various exercises, A > B, in the sense that A, B are symmetric and A — B is non-negative definite, does not necessarily imply that A? > B, although it is true that B-! > A if A2B>O0. The general question was first discussed in ©. Loewner, Math. Z., vol. 38, pp. 177-216, 1934.Functions of Matrices Ml See also R. Dobsch, Math. Z., vol. 43, pp. 353-388, 1937. In his paper Loewner studies the problem of determining functions f(z) for which A > B implies that f(A) > f(B). This leads to a class of functions called positive real, possessing the property that Re [f(z)] > 0 whenever Re (z) > 0. Not only are these functions of great importance in modern network theory, cf. L, Weinberg and P. Slepian, Positive Real Matrices, Hughes Research Laboratories, Culver City, Calif., 1958. F. H. Effertz, On the Synthesis of Networks Containing Two Kinds of Elements, Proc. Symposium on Modern Network Synthesis, Poly- technic Institute of Brooklyn, 1955, where an excellent survey of the field is given, and R. J. Duffin, Elementary Operations Which Generate Network Matrices, Proc, Am. Math. Soc., vol. 6, pp. 335-339, 1955; but they also play a paramount role in certain parts of modern physics; see the expository papers ‘A.M. Lane and R. G. Thomas, R-matrix Thoory of Nuclear Reac- tions, Revs. Mod. Physics, vol. 30, pp. 257-352, 1958. E, Wigner, On » Class of Analytic Functions from the Quantum ‘Theory of Collisions, Ann, Math., vol. 53, pp. 36-67, 1951. Further discussion may be found in a more recent paper, C. Loewner, Some Classes of Functions Defined by Difference or Differential Inequalities, Bull. Am. Math. Soc., vol. 56, pp. 308-319, 1950. and in J. Bendat and 8, Sherman, Monotone and Convex Operator Func- tions, Trans. Am. Math. Soc., vol. 79, pp. 58-71, 1955. The concept of convex matrix functions is also discussed here, treated first by F. Kraus, Math, Z., vol. 41, pp. 18-42, 1936. 810, This result is contained in R. Bellman, Representation Theorems and Inequalities for Hermitian Matrices, Duke Math. J., 1959.7 Variational Description of Characteristic Roots 1. Introduction. In earlier chapters, we saw that both the largest and smallest characteristic roots had a quite simple geometric significance which led to variational problems for their determination. As we shall see, this same geometric setting leads to corresponding variational problems for the other characteristic values. However, this variational formulation is far surpassed in usefulness and elegance by another approach due to R. Courant and E. Fischer. Both depend upon the canonical form derived in Chap. 4. We shall then obtain some interesting consequences of this second representation, which is, in many ways, the link between the elementary and advanced analytic theory of symmetric matrices. 2. The Rayleigh Quotient. As we know, the set of values assumed by the quadratic form (z,Az) on the sphere (2,2) = 1 is precisely the same set taken by the quadratic form (y,Ay) = As? + Awa? + °° > + Awyn? on (yy) = 1,4 = T’AT, y = Tz, with T orthogonal. Let us henceforth suppose that M>MD6+ Daw q) Since, with this convention, (y,Ay) 2 Aw(ys? + wa? + + > + yw?) 2) (yAy) SQ? + we? +o + yw?) we readily obtain the representations = max WY) — any (ert) “— ma) Te Ga) @) = min YAY = nin GAD Mee mina)" Ga) The quotient 42) (2,Az 2) = Gay (4) is often called the Rayleigh quotient, uaVariational Description of Characteristic Roots 118 From the relation in (3), we see that for all z we have (2,Az) M2 tea) Relations of this type are of great importance in applications where one is interested in obtaining quick estimates of the characteristic values. Remarkably, rudimentary choices of the 2; often yield quite accurate approximations for \: and Av. Much of the success of theoretical physics depends upon this phenomenon. ‘The reason for this interest in the characteristic roots, and not so much the characteristic vectors, lies in the fact that in many physical appli- cations the d; represent characteristic frequencies. Generally speaking, in applications the ); are more often the observables which allow a com- parison of theory with experiment. Matters of this type will be discussed in some slight detail in Sec, 12 below. 8. Variational Description of Characteristic Roots. Let us begin our investigation of variational properties by proving Theorem 1. Theorem 1. Let x‘ be a set of N characteristic vectors associated with the di. Then, for k = 1, 2, > lw () iN, = max (z,Az)/(@,a) (1) where Rx is the region of x space determined by the orthogonality relations (@ez)=-0 i=1,2,...,k-1240 (2) Geometrically, the result is clear. To determine, for example, the second largest semiaxis of an ellipsoid, we determine the maximum dis- tance from the origin in a plane perpendicular to the largest somiaxis. To demonstrate the result analytically, write z= ) wat @) Then x @,Az) = Y rant O os o=\w 2) d ‘The result for A: is precisely that given in Sec. 2, Consider the expres- sion for Xz. The condition (z,2') = 0 is the same as the assertion that14 Introduction to Matrix Analysis x w= 0. It is clear then that the maximum of ) duu subject to ¥ ea tu? = 1 is equal to As ro In exactly the same fashion, we see that \y has the stated value for the remaining values of k. 4 Discussion. It is known from physical considerations that stif- fening of a bar or plate results in an increase in all the natural frequencies. The analytic equivalent of this fact is the statement that the character- istic roots of A + B are uniformly larger than those of A if B is positive definite. To demonstrate the validity of this statement for \; or Aw is easy. We have, using an obvious notation, {2,(A + B)z] M(A + B) = max ay max | 42) 4. (2,82) =max|“a) + Ga) | (z,Az) , . (a, Bz) 2 max “Tay min“) 2 (A) + Aw(B) (1) Since by assumption dy(B) > 0, we see that d1(A + B) > d4(A) if Bis positive definite. The proof that \x(A + B) > dx(A) proceeds similarly, If, however, we attempt to carry through a proof of this type for the roots Xs, As) « « . , An-a, We are baffled by the fact that the variational description of Theorem 1 is inductive. The formula for \s depends upon the characteristic vector x1, and so on. Furthermore, as noted above, in general, in numerous investigations of engineering and physical origin, we are primarily concerned with the characteristic values which represent resonant frequencies. The charac- teristic vectors are of only secondary interest. We wish then to derive @ representation of 4, which is independent of the other characteristic values and their associated characteristic vectors. EXERCISES 1. Show that Av(A + B) > dAw(A) + Aw(B) if A and B are positive definite. 2. Suppose that B is merely positive indefinite. Must dy(A + B) actually be greater than Ay(A)? 5. Geometric Preliminary. In order to see how to do this, let us take an ellipsoid of the form z*/a* + y*/b* + 2#/c? = 1, and pass a plane through the origin of coordinates. The cross section will be an ellipseVariational Description of Characteristic Roots 115 which will have a major semiaxis and a minor semiaxis. If we now rotate this cross-sectional plane until the major xis assumes its smallest value, we will have determined the semiaxis of the ellipsoid of second greatest length. The analytic transliteration of this observation will furnish us the desired characterization. Like many results in analysis which can be demonstrated quite easily, the great merit of this contribution of Courant and Fischer lies in its discovery. 6. The Courant-Fischer min-max Theorem. Let us now demonstrate Theorem 2. Theorem 2 The characteristic roots s, i= 1, 2,..., N, may be defined as follows: Ai = max (2,A2)/(2,2) = min max, (z,Az)/(z,2) Get Gad Me = min max (z,Azx)/(z,2) Wyy=} p20 (a) Equivalently, dw = min (@,Ax)/(x,2) dy = max min (@, Ax) /(@,2) (2) (ua =t (ea) = and s0 on. Proof. Let us consider the characteristic root Xs. Define the quantity wu: = min max (z,Azx)/(z,2) @) (a) 2 ea) 0 We shall begin by making an orthogonal transformation which converts A into diagonal form, z = Tz. Then ® x w= min max {Y met/ Sat} waar (teay~o | Ly mh = min max {-- +} 0) (ey) 1 Gy) =O Setting T’y = y', this becomes max { at eyyn0 (6)6 Introduction to Matrix Analysis Since (Ty',Ty') = (y',y'), we may write u:= min max {[-- +} (6) (ua) =1 (a0) 90 In other words, it suffices to assume that A is a diagonal matrix. In place of maximizing over all z, let us maximize over the subregion defined by Si aa +++ ey = Oand @y) =0 0) Since this is a subregion of the z region defined by (z,y) = 0, we have vy > min max Pert + Aet/at + a] ® wn Since (zi? + dstza*)/(e1? + 22%) > de for all 2: and 2s, it follows that we have demonstrated that us 2s. Now let us show that uz: < s Todo this, in the region defined by (y,y) = 1, consider the set consisting of the single value y: = 1, This choice of the vector y forces z to satisfy the condition that its first component z, must be zero. Since the minimum over a subset must always be greater than or equal to the minimum over the larger set, we see that as max (Quaza® + doze? +o + + Anew?) /@a? + 2x? + + + 2nd} s max (Qraee? +o + + Awew?)/G@a? + ++ + + ent) = De (9) It follows that wr = ds. Now let us pursue the same techniques to establish the result for general k, Consider the quantity m= min max (2,Az)/(z,2) das Gn=0 = (10) Consider maximization over the subregion defined by S: ty = ty. = + = av =0 (11) It follows that m > min max (+ -} > (12) apt 6 since 5 rat / . 2? > ds for all z m1 Similarly, to show that 4, < 4s, we consider minimization over the subregion of (y‘,y') = 1,7 = 1,2, .. . ,k — 1, consisting of the (k — 1)Variational Description of Characteristic Roots 7 veetors 1 0 0 1 pel 0 . vel. see yet] (13) 0. The orthogonality conditions (y‘z) = 0 are then equivalent to ee ae) (14) As before, we see that u: < A, and thus that u, = A. 7. Monotone Behavior of 4.(A). The representation for 4, given in Theorem 2 permits us to conclude the following result. Theorem 3, Let A and B be symmetric matrices, with B non-negative definite, Then M(A + B)>M(A) k= 1,2,...,N qa) If Bis positive definite, then (A +B)>M(A) k=1,2,...,N (2) EXERCISE 1, Obtain a lower bound for the difference (A + B) — d«(4). 8 A Sturmian Separation Theorem. Theorem 2 also permits us to demonstrate Theorem 4, Theorem 4, Consider the sequence of symmelric matrices A,= (a) 47 =1,2 0.057 (w forr =1,2,...,N. Let \4(A,), k = 1, 2,... , 7, denote the kth characteristic root of An, where, consistent with the previous notation, (Ae) 2 A(Ar) 2B vo + BAC AY) (2) Then Anea(Assa) SMC Ai) S Ae(Ace) (3) Proof. Let us merely sketch the proof, leaving it as a useful exercise for the reader to fill in the details. Let us prove 2(Aunt) S 14s) S Ai Aaa) (4)118 Introduction to Matrix Analysis We have wu(Ad) = max (2,Ae2)/(0,2) Aide) = max (2, Asys2)/(@.2) 7 where the 2 appearing in the first expression is i-dimensional, and the x appearing in the second expression is (i + 1)-dimensional. Considering variation over the set of (i + 1)-dimensional vectors 2 with (¢ + I)st component zero, we see that 41(Ai) < di(Asgi)> To obtain the inequality \2(Ai1) < A1(A,), we use the definition of the \, in terms of max-min. Thus (A) = max min (2,Az)/(2,2) PS Aneel ry (z,y§) =O 1 6 Ae(dea) = max min (2, Asest)/(2,2) @ mt aR SET The vectors appearing in the expression for (A) are all i-dimensional, while those appearing in the expression for \2(A,41) are all (¢ + 1)-dimen- sional, In this second expression, consider only vectors x for which the (i + 1)st component is zero. Then it is clear that for vectors of this type we maximize over the y* by taking vectors y* whose (i + 1)st com- Ponents are zero. It follows that \a(As1) < A( Ai). 9. A Necessary and Sufficient Condition that A be Positive Definite. Using the foregoing result, we can obtain another proof of the fact that a necessary and sufficient condition that A be positive definite is that Ai] > 0,4 =1,2,...,M. As we know, a necessary and sufficient condition that A be positive definite is that 4,(A) > 0,4 = 1,2,...,N. If A is positive definite, and thus \4(4) > 0, we must have, by virtue of the above separation theorem, (4) > 0, = 1,2,...,¢, and thus [4d = [] 44(4) > 0. bet On the other hand, if the determinants | A,| are positive for all k, we must have as the particular case k = 1, ,(A1) > 0. Since Ya(Aa) 2B x(Aa) 2 Aa( Ar) q) the condition |A2| > O ensures that X2(A:) > 0. Proceeding inductively, we establish that \,(A;) > 0, k = 1, 2, .,¢, for? =1,2,...,N, and thus obtain the desired result. ‘Again the details of the “complete proof are left as an exercise for the reader. 10, The Poincaré Separation Theorem. Let us now establish the fol- lowing result which is useful for analytic and computational purposes.Variational Description of Characteristic Roots 119 Theorem 5. Let {y+}, b= 1,2... , K, be a aet of K orthonormal vectors and set =) uny', 20 that i x (Ax) =) unuly', Ay? w) of Set Be=(yjAy) kj l=1,2,...,K (2) Then (Br) S (A) As(Bx) = dw-s(A) .,K-1 @) The results follow immediately from Theorem 4. 11, A Representation Theorem. Let us introduce the notation JAls = AwAwaa + + Avot a We wish to demonstrate Theorem 6. /f A is positive definite, tt apo , eae dVy @) where the integration is over a k-dimensional linear subspace of N-dimen- sional space R, whose volume element is dV, and the maximization is over all R. Proof. It is easy to see that it is sufficient to take (,Az) in the form Ati? + Aste? + ++ + + Awtn®. Hence, we must show that i — <= Ouest berth oo pane ow ya Tmax |, e + dV, (8) Denote by Va(e) the volume of the region defined by Ait? + dame? +o + + Anew? Sp (a) =0 1=1,2,...,N—k (4) where the a! are N — k linearly independent vectors. It is clear that Vale) = e*?Va(1) (5) [= foggeornroonma o erp! dp Then120 Tntroduction to Matrix Analysis To complete the proof, we must show that the maximum of V.(1) is attained when the relations (z,a') = 0 are z, = 0,21 =0,... 21 =0. This, however, ig a consequence of the formula for the volume of tho ellipsoid, 1 damit, and the min-max characterization of the tent characteristic roots given in Theorem 2. 12. Approximate Techniques. The problem of determining the mini- 1 stu . mum of h u’® dt over all functions satisfying the constraints fiawa=t (1a) u(0) = u(t) =0 (1b) is one that can be treated by means of the calculus of variations. Using standard variational techniques, we aro led to consider the Sturm- Liouville problem of determining values of X which yield nontrivial solutions of the equation wu + rau =0 — u(0) = u(l) =0 ) Since, in general, the differential equation cannot be solved in terms of the elementary transcendents, various approximate techniques must be employed to resolve this problem. In place of obtaining the exact variational equation of (2) and using an approximate method to solve it, we can always replace the original vari- ational problem by an approzimate variational problem, and then use an exact method to solve this new problem. One way of doing this is the following. Let {u(t)} be a sequence of linearly independent functions over (0,1] satisfying the conditions of (10), that is, u(0) = w(1) = 0. We sttempt to find an approximate solution to the original variational problem having the form x um Y sant @) & The problem that confronts us now is finite dimensional, involving only the unknowns 2, t2, ... , tv. We wish to minimize the quad- ratic form é 1 DY, aay fy uous at ( isan subject to the constraint " Dae, f.' aealQu(d dt = 1 © agenVariational Description of Characteristic Roots 121 It is clear that important simplifications result if we choose the sequence {u(0)} 80 that either 1 , Jo w@ujo amas orf ulduty t= ay 6) The first condition is usually easier to arrango, if ¢(¢) is not a function of particularly simple type. * Thus, for example, we may take uj(t) = sin kt (0) properly normalized, or wi) = Pa) @) the kth Legendre polynomial, again suitably normalized. ‘This last would be an appropriate choice if q(t) were a polynomial in t, since integrals of the form f thu;(t)us(t) dt are readily evaluated if the w(t) are given by (8). This procedure leads to a large number of interesting and significant problems. In the first place, wo are concerned with the question of con- vergence of the solution of the finite-dimensional problem to the solution of the original problem. Second, we wish to know something about the rate of convergence, a matter of great practical importance, and the mode of convergence, monotonic, oscillatory, and so forth. We shall, however, not pursue this path any further here. EXERCISE 1, Let 40, aa, problem posed in (4) necting Xs) and A¥—OP . , Av) denote characteristic values associated with the d (5), for N = 2, 3,.... What inequalities exist con- MISCELLANEOUS EXERCISES Let A and B be Hermitian matrices with respective characteristic values 4: > MS + 22 > ++, and let the characteristic values of A + B be », > Doves, then ds + my > vist for d+ SN +1 (Weyl). 2. By considering A as already reduced to diagonal form, construet an inductive proof of the Poincaré separation theorem, 8. What are necessary and sufficient conditions that an.2;" + 2aiei21 + ante? > 0 for all x1, 21 > OP 4. What ore necessary and sulficiont conditions that) auzizy 2 0 for 21 24 Aa 220? x 5. Can one obtain corresponding resulta for > aiseen,? Sn 6. Prove that aut ta” a0 = oe oe is positive definite for r > 0 if A = (a;;) is positive definite and a; > 0.122 Introduction to Matrix Analysis 7. Does a sorresponding result hold for 3 X 3 matrices? 8. Prove that (ais) is positive definite if a;: > 0 and (ja:,)) is positive definite. 9. Show that the characteristic roots of A}4BA)s are leas than those of A if B trix all of whose roots are between 0 and 1 and A is positive definite. orthogonal, z real, is (Tz,Az) < (z,Az) for all z if A is positive definite? 11, Define rank for a symmetric matrix as the order of A minus the number characteristic roots. Show, using the results of this chapter, that this definition is equivalent to the definition given in Appendix A. 12, Suppose that we have a set of real symmetric matrices depending upon a parameter, scalar or vector, q,{4(q)]. We wish to determine the maximum char- acteristic root of each matrix, a quantity we shall call f(q), and then to determine the imum over q of this function, Let us proceed in the following way, Choose an initial value of g, Say go, and let 2° be a characteristic vector associated with the char- acteristic root f(g). To determine our next choice g1, we maximize the expression (z*,A@z*)/(2",z") over all q values. Cali one of the values yielding a maximum q, and let z' be a characteristic vector associated with the characteristic root f(q:). We continue in this way, determining a sequence of values f(qo), f(q:), . . . . Show that I) SF) Sf) < ++ 18. Show, however, by means of an example, that we may not reach the value max f(g) in this way. “44. Under what conditions on A(q) ean we guarantee reaching the absolute maxi- mum in this way? 15, Let the apectral radius r(A), of a square matrix A be defined to be the greatest of the absolute values of its characteristic roots. Let H be a positive definite Her- mitian matrix and let 9(A,H) = max [(Az,HAz) /(z, Hz) Show that r(A) = min 9(4,#). 16. If A iso real matrix, show that r(A) = min g(4,8), where now the minimum is taken over all real syininetric matrices (H. Osborn, The Existence of Conservation Laws, Annals of Math., vol. 69, pp. 105-118, 1959). Bibliography and Discussion §1. The extension by R. Courant of the min-max characterization from the finite case to the operators defined by partial differential equa- tions is one of the foundations of the modern theory of the characteristic values of operators. For a discussion of these matters, see R. Courant and O. Hilbert, Methoden der mathematischen Physik, Berlin, 1931, reprinted by Interscience Publishers, Inc., New York. §2. The Rayleigh quotiont affords a simple means for obtaining upper bounds for the smallest characteristic value. The problem of obtaining lower bounds is much more difficult. For the case of more general operVariational Description of Characteristic Roots 123 ators a method developed by A. Weinstein and systematically extended by N. Aronszajn and others is quite powerful. For an exposition and further references, see J. B, Diaz, Upper and Lower Bounds for Eigenvalues, Calculus of Variations and Its Applications, L, M. Graves (od.), McGraw-Hill Book Company, Inc., New York, 1958. See also G. Temple and W. G. Bickley, Rayleigh’s Principle and Its Appli- cation to Engineering, Oxford, 1933. ‘An important problem is that of determining intervals within which characteristic values can or cannot lie. See T. Kato, On the Upper and Lower Bounds of Figenvalues, J. Phys. Soc. Japan, vol. 4, pp. 334-339, 1949. H. Bateman, Trans. Cambridge Phil. Soc., vol. 20, pp. 371-382, 1908. G. Temple, An Elementary Proof of Kato’s Lemma, Matematika, vol. 2, pp. 39-41, 1955. §6. See the book by Courant and Hilbert cited above and E. Fischer, Uber quadratische Formen mit reellen Kooffizienten, Monatsh. Math. u. Physik, vol. 16, pp. 234-249, 1905. For # discussion, and some applications, see G. Polya, Estimates for Eigenvalues, Studies in Mathematics and Mechanics, presented to R. Von Mises, Academic Press, Inc., New York, 1954. There are extensive generalizations of this result. Some are given in Chap. 7, Secs, 10, 11, and 13, results due to Ky Fan. See Ky Fan, On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations, I, Proc. Natl. Acad. Sci. U.S., vol. 35, pp. 652-655, 1949, An extension of this is due to Wielandt, H. Wielandt, An Extremum Property of Sums of Figenvalues, Proc. Am. Math. Soc., vol. 6, pp. 106-110, 1955. See also Ali R. Amir-Mooz, Extrome Properties of Eigenvalues of a Hermitian Transformation and Singular Values of the Sum and Product of a Linear Transformation, Duke Math, J., vol. 23, pp. 463-477, 1956.124 Introduction to Matrix Analysis A further extension is due to M. Marcus and R. Thompson, A Note on Symmetric Functions of Eigenvalues, Duke Math. J., vol. 24, pp. 43-46, 1957. M. Marcus, Convex Functions of Quadratic Forms, Duke Math. J., vol. 24, pp. 321-325, 1957. 98. Results of this type may be established by means of the classical techniques of Sturm. See W. 8. Burnside and A. W. Panton, Theory of Equations, vol. II, Longmans, Green & Co,, Inc., New York, 1928. $10. H. Poincaré, Sur les équations aux dérivées partielles de la physique mathématique, Am. J. Math., vol. 12, pp. 211-294, 1890. §11. R. Bellman, I. Glicksberg, and 0. Gross, Notes on Matrix Theory —VI, Am. Math. Monthly, vol. 62, pp. 571-572, 1955. 12. An extensive discussion of computational techniques of this type is contained in L. Collatz, Higenwertproblemen und ihre numerische Behandlung, New York, 1948. Finally, let us mention the monograph M. Parodi, Sur quelques propriétés dea valeurs charactéristiques des matrices carréea, fascicule CXVII, Mém. sci. mathé., 1952. The problem of obtaining corresponding variational expressions for the characteristic values of complex matrices, which arise in the study of linear dissipative systema, is one of great difficulty. For some steps in this direction, see R. J. Duffin, A Minimax Theory for Overdamped Networks, J. Rat, Mech. Analysis, vol. 4, pp. 221-233, 1958, and the results of Dolph et al., previously referred to. R. J. Duffin, The Rayleigh-Ritz Method for Dissipative or Gyro- scopic Systems, Q. Appl. Math., vol. 18, pp. 215-222, 1960. R. J. Duffin and A. Schild, On the Change of Natural Frequencies Induced by Small Constraints, J. Rat. Mech. Anal., vol. 6, pp. 731~ 758, 1987. A problem of great interest in many parts of physics is that of deter- mining g(t) given the characteristic values corresponding to variousVariational Description of Characteristic Roots 125 boundary conditions. See G. Borg, An Inversion of the Sturm-Liouville Eigenvalue Problem. Determination of the Differential Equation from the Eigenvalues, Acta Math., vol. 78, pp. 1-96, 1946. I. M. Gelfand and B. M. Levitan, On the Determiriation of a Differ- ential Equation from Its Spectral Function, Jz. Akad. Nauk SSSR, Ser. Math., vol. 15, pp. 309-360, 1951. B. M. Levitan and M. G. Gasymov, Determination of a Differential Equation from Two Spectra, Uspekhi Mat. Nauk, vol. 19, pp. 3-64, 1964, R. Bellman, A Note on an Inverse Problem in Mathematical Physics, Q. Appl. Math., vol. 19, pp. 269-271, 1961. L. E. Anderson, Summary of Some Results Concerning the Deter- mination of the Operator from Given Spectral Data in the Case of a Difference Equation Corresponding to a Sturm-Liouville Differential Equation, to appear. See also R. Yarlagadda, An Application of Tridiagonal Matrices to Network Synthesis, STAM J. Appl. Math., vol. 16, pp. 1146-1162, 1968.8 Inequalities 1, Introduction, In this chapter, we shall establish a number of inter- esting inequalities concerning characteristic values and determinants of symmetric matrices. Our fundamental tools will be the integral identity im - f. a f. eee) de @ valid if A is positive definite, its extension given in Theorem 6 of Chap. 7, and some extensions of min-max characterizations of Courant-Fischer, due to Ky Fan. Before deriving some inequalities pertaining to matrix theory, we shall establish the standard inequalities of Cauchy-Schwarz and Hélder. Sub- sequently, we shall also prove the arithmetic-geometric mean inequality, since we shall require it for one of the results of the chapter. 2. The Cauchy-Schwarz Inequality. The first result we obtain has already been noted in the exercises. However, it is well worth restating. Theorem 1, For any two real vectors x and y we have (zy)? S (@2)(y,9) q) Proof. Consider the quadratic form in u and v, Q(u,v) = (us + vy, ux + vy) = ur(z,r) + 2uv(z,y) + v*(y,y) (2) Since Q(u,») is clearly non-negative definite, we must have the relation in (1), Wesee that (1) is a special, but most important case, of the non- negativity of the Gramian determinant established in Sec. 4 of Chap. 4. 3. Integral Version. In exactly the same way, we can establish the integral version of the preceding inequality. Theorem 2, Let f(x), o(z) be functions of z defined over some region R. MT haat sere (at)Inequalities 127 Proof. Since (f — g)* 2 0, we have 2p Sit tot @) Hence, fg ia integrable if j? and g* are. Now consider the integral fq fu + mytav @) which by virtue of the above inequality exists. As quadratic form in u and », the expression in (3) is non-negative definite. Consequently, as before, we see that (1) must be satisfied. EXERCISES 1. We may also proceed as follows: ead(nad — ean? = (wa) [ (ea) - 2 GUE 4. Gav] = = y FY, 2 — y SH) tuo (= ~ 9 = — WG ” x Ny 2. Prove that ( > aun) < ( Y=") (2 wt/ns) if the ds are positive. a fh =n 8. Hence, show that (z,Az)(y,4~'y) > (z,y)* if A is positive definite, Establish the result without employing the diagonal form. 4. Holder Inequality. As an extension of (1), let us prove Theorem 3. Let p > 1, and q = p/(p — 1); then x N x > am < Q a)” (> ws)" a) ah a We if ts, ys > 0. Proof. Consider the curve veut (2) where p > 1. Pe | gy ie128 Introduction to Matrix Analysis It is clear that the area of the rectangle OvRu is less than or equal to the sum of the areas OPu and OQ», wos fur du + foe do 8) with equality only if » = w?-!. Hence, if u, » > 0, p > 1, we have <@ye 4) wss + 7 (4) Now set successively wn n/(S 20)" y (6) o=m/(> we)" ea &=1,2,...,N, and sum over k. The result is x x x & ina 0 and the integrals on the right exist. 6. Concavity of [4]. Let us now derive some consequences of the integral identity of (1.1). The fir Theorem 4 If A and B are positive definite, then )hA + (1 — XB) > [AP BIO a) 1 f@ 1__g(z)t 5 fer _ 41s". Pf, fe) avg Ie g(2)e dV forrO0 |A| if A is positive definite and B is real symmetric. 2. Show that Az + (1 — A)y > Py" for z,y >0,0 aww Since A is positive definite, the matrix (a), i,j = 2, . . . , N, is positive definite; hence the quadratic form in a1z, ais, . . . , dw appearing as the second term on the right in (2) is negative definite. Thus Qe + '* daw Gs = Ow [Al San} | (3) Jaw, +> > aww whence the result follows inductively.130 Introduction to Matrix Analysis Second Proof. Consider the integral in (1.1) for the case N = 3, Replacing x: by —z, and adding, we have ip ef) pe ee en ste ‘) de dzdz (4) where Be ea (5) Since z + #~? > 2 for all z > 0, we have (ipa (| eomtemmenem desde) ([" ewde,) a 7. Toul [aa ® where As = (ay), #,j = 2,3. Hence, lans)4] 4a) > |A)* @ whence the result for N = 3 follows inductively. The inequality for general N is derived similarly. EXERCISE 1 Let Di = lal 4 f= 4, 2%. ~~, my Dim leu § famti..., my De lal ij = math. Then lA] $ Dis - + «Dy 7. Hadamard’s Inequality. The most famous determinantal inequal- ity is due to Hadamard. ‘Theorem @ Let B be an arbitrary nonsingular real square matrix. rs Il ) bat) a set WS Proof. Apply Theorem 5 to the positive definite square matrix “a concavity of \ax-i © == As. Now define the matrix function JAly = Awana oo Me qy Then we may state Theorem 7, Theorem 7, If A and B are positive definite, we have IMA + (1 —2)Bh > [ALB @ for0 0, then ms y Ny T] eas > [] apes (Ky Fen) int iat 4. Establish Hadamard’s inequality for Hermitian matrices. 6. If A is a positive definite N X N matrix and Py denotes the product of all principal k-rowed minors of A, then PL > PACH) > pVCF) > py'/G) > py (Geass) See L. Mirsky," €. Let A, denote the principal submatrix of A formed by the first p rows and p columns, and let B, and C, have similar meanings. Then (Je) = (La (IBY yp 9, Additive Inequalities from Multiplicative. Let us now give an example of a metamathematical principle which asserts that every multi- plicative inequality has an associated additive analogue. Consider Theorem 7 applied to the matrices A=I+X B=I+e¥ e>0 (1) where X and Y are now merely restricted to be symmetric, If ¢ is suf- ficiently small, A and B will be positive definite. Letzi > 21> - ++ Daw, y12 422° + * > ywhe the characteristic roots of X and ¥, respectively, and 2: > #: > -- - > ew the character- 1L, Mirsky, On a Generalization of Hadamard’s Determinantal Inequality Due to Ssasz, Arch. Math., vol. VIII, pp. 274-275, 1957,132 Introduction to Matrix Analysis istic roots of XX + (lL — A)¥. Then Theorem 7 yields the result (1 + e@w)(L + eens) + + + (L + eas) > [1 + exw)(1 + exya) © > > (1 + ex) {+ eyw)(L + yw) + (L + ey} (2) To first-order terms in ¢, this relation yields 1+ ew tent ++ +a) 21+ delew tavit +++ +m) + (l= dNe(yw + wit ++ ty) +0) (3) Letting «— 0, we obtain wwtanat- ++ tad(ewt +++ +m) + (yw t+ ywat ++ +y) (4) Let us state this result, due to Ky Fan, in the following form, Theorem 8, Let us define the matriz function Si(A) = dw Faw too $M (8) for a symmetric matrix A. Then Si[MA + (1 — 2)B] > dSi(A) + (1 — 0)8K(B) © forOSX<1,k=1,2,...,™. From this follows, upon replacing A by — A, Theorem 9. If TMA) = tet oo s th @ where A is symmetric, then TAA + (1 — d)B] SATA) + (1 — THB) (8) for0SX Dale) aon x Ddwdnar (ung = wn)? ) fra Hence x x | x N u? Us we Ui ek Xe 2 dra] y ¥ LL Di(x,y) > dwdw-r| “y x (6) Udy vi UD, ve or Did & Da(z,y) 2 dwAnn @ This proves the result for k = 1. To derive the general result, we use the identity for the Gramian given in Sec, 5 of Chap. 3, rather than the Lagrangian identity used above. EXERCISE 1, Use Theorem 9, together with Theorem 4, to establish Theorem 7, 11. A Simpler Expression for AwAy-1 ‘°° Xs. Since determinants are relatively complicated functions, let us obtain a simpler repre- sentation than that given in Theorem 10. We wish to prove Theo- rem 11.184 Introduction to Matrix Analysis Theorem 11. If A is positive definite, Avda ss + Aa = min (24,A2!)(2%,Az4) + + + (eH, Aa Ht) (1) where R is as defined in (10.2). Proof. We know that (2',Az/), 7,7 = 1,2,...,N —k +1, is posi- tive definite if A is positive definite. Hence, Theorem 5 yields yom \@sae)| < TL @az) @) jet Thus weit min |(¢',Az/)| < min [] (@Az/) @) R R dat Choosing 2‘ = 2,4 = N,N —1,...,N —k +1, we see that watt min J] (#429 < dwkwaa ++ Ms C) fet This result combined with Theorem 9 yields Theorem 10. EXERCISES 1. Prove Theorem 7, using the foregoing result. 2. Consider the case k = N and specialize z' so as to show that Aw S min ag 3. Considering the general case, show that Dwdwer s+ + Me S Qwwawauwas © + + Oak 4 Show that min tr (AB)/N = |A|‘, where the minimum is taken over all B 3 satisfying the conditions B > 0, |B| = 1, and A is positive definite, 5. Hence, show that |A + Bit” > |A|*” + |B|'"" if A and B are positive definite (A. Minkowski). 12, Arithmetic-Geometric Mean Inequality. For our subsequent pur- poses, we require the fundamental arithmetic-geometric mean inequality. Theorem 12. If x; > 0, then x YD aN > (erte + + ayy (a) fm Proof. Although the result. can easily be established via calculus, the following proof is more along the lines we have been pursuing.t Starting t This proof is due to Cauchy.Inequalities 135 with the relation (a,% — a,4)? > 0, or $8 > (yay @ the result for N = 2, we set a= i+ b)/2 a= (bs + b0/2 @) The result is bit bet bat bey (bi tba) (ba + bs 4 = 2 2 > Gibby @ Continuing in this fashion, we see that (1) is valid for N a power of 2. To complete the proof, we use a backward induction, namely, we show that the result is valid for N — 1 if it holds for N, To do this, set Wi ye N= tH we BERL es (6) and substitute in (1). The result is the same inequality for N — 1. Examining the steps of the proof, we see that there is strict inequality unless 2) = 2; = ++ = ay. 13. Multiplicative Inequalities from Additive. In a preceding section, we showed how to deduce additive inequalities from multiplicative results. Let us now indicate how the converse deduction can be made. Theorem 18, The inequality » x > MS > (2, zt) () mS k=1,2,... ,N, valid for any set of orthonormal vectors {x‘}, yields the inequality Ny x Tha < Tl @4z (2) ina ink Proof. Consider the sum cade + Cugdess +o = bewhw = exe tea $+ Aw) + (Cost — C4)(Meut Foo + Aw) $s + ew — cwa)aw (8) If we impose the restriction that OSauS etn S+** Sew 4) we have, upon using (1), the inequality Cada + Chysdegs + +7 + ewAw < ca(zt, Act) + Cegr(ettt,Agttl) + +++ + ew(eX,AzX) (5) for any set of orthonormal vectors {z*} and scalar constants satisfying (2).136 Introduction to Matrix Analysis We deduce from (5) that min (s eh) < min (3, ex(a,Aa') | © where R is the region in ¢ space defined by (4). Now it follows from Theorem 12 that y — ye > (etagn © © en)MOEED ODay AQ) MOREE (7) with equality only if all Asc; are equal, whence Ce = (Aedega 8 + Aw) HED Ag (8) Since ts > Ana S * ++ B dy, it follows that 0 |Hi, with equality if and only if B = 0 (0. Taussky). 6. If Hy is a positive definite Hermitian matrix and H, is a Hermitian matrix, then Hy + He is positive definite. if and only if the characteristic values of Hi~'Hy are all greater than —1 (Ky Fan-O. Taussky). 6. Let K, be a positive definite matrix and K; such that KiK,is Hermitian, Then KK, is positive definite if and only if all characteristic values of Ky are real and posi- tive (Ky Fan-0, Taussky). 7. If A and Bare symmetric matrices, the characteristic roots of AB — BA are pure complex. Hence, show that tr ((AB)!) < tr (A1B2). 8. If A, B are two matrices, the square of the absolute value of any characteristic root of AB is greater than or equal to the product of the minimum characteristic root of AA’ by the minimum characteristic root of BB’. ®, Let H = A + iB be positive definite Hermitian; then |A| > |B] (Robertson). 10. If A = B+C, where B is positive definite and C is skew-symmetric, then [Al 2 |B] (0. Taussky). 11. If 4 is symmetric, A and J ~ A are non-negative definite, and 0 is orthogonal, then |J — AO| > |I — Al (0. Taussky). 12. Establish Schur’s inequality! x Das x last coil 18. Let A be syminctric and k be the number of zero characteristic roots. Then fae fuel im mt where the summation is now over nonzero roots. From the Cauchy inequality (Fs (Sm) (tr A)t < (N = k) tr (Ad) deduce that whence N tr (A¥) — (tr A)? Br at) 14. From the foregoing, conclude that the rank of a syinmetric matrix A is greater than or equal to (tr A)#/tr (A). 16. Obtain similar relations in terns of tr (A¥) for k = 1, 2, 3, (The foregoing trick was introduced by Schnirelman in connection with the problein of representing every integer as a suin of at inost a fixed numnber of primes.) 16. Let Ay be the smallest characteristic value and A: the largest characteristic value of the positive definite matrix A. Then (a2)? < (Az2)(A~te,z) 5 OP? + dy) Daw 41, Schur, Math, Ann., vol. 66, pp. 488-510, 1909. (z,2)*138 Introduction to Matrix Analysis This is a special case of more general results given by W. Greub and W. Rheinboldt.? 17. If X is positive definite, then X + X-! > 21. 18. If X ia positive definite, there is unique ¥ which minimizes tr (XY~) subject to the conditions that Y be positive definite and have prescribed diagonal elements, 19. This matrix Y satisfies an equation of the form X = YAY, where Aisa diagonal matrix of positive elements 2. 7 20. The minimum value of tr (XY~*) ny didi (P. Whittle). it 21. If Band C are both positive definite, is BC + CB > 2B4CBKT 22. If A, B, and C are positive definite, is (A + B)4C(A + B)K < AMCAM + BYCBY? 8. Let A and B be N X N complex matrices with the property that I — A*A and I —B*B are both positive semidefinite. Then |J — A*B|* > |I — A*A| II = BB). (Hua, Acta Math. Sinica, vol. 5, pp. 463-470, 1955; see Math, Rev., vol. 17, p. 703, 1956.) For extensions, using the representation of Sec, 10 of Chap. 6, see R. Bellman, Representation Theorems and Inequalities for Hermitian Matrices, Duke Math, J., 1959, and, for others using » different method, M. Marcus, On a Determinantal Inequality, Amer. Math, Monthly, vol. 85, pp. 266-268, 1958, 24. Let A, B>0, Then tr ((AB)™*) < tr ((A*B4)™), n= 0, 1; « . . (Golden). 25. Let X+ denote the Penrose-Moore generalized inverse. Define the parallel aum of two non-negative definite matrices A and B by the expression A:B = A(A + B)*B Then A:B = B:A, (A:B)C = A:(B:C). 26. If Az = dz, Bz = ua, then (A:B)z = (Atu)z. 47. Tag be > 0, then Q au): 0) a) y (av:bo) and if A, B > 0, then tr (A:B) (te A)s(te B), with equality if and only if A= csB, e:a scalar. 88. If A, B > 0, then |A:B] < 14]:/BI. 29. Can one derive the result of Exercise 27 from the result of Exercise 28 and conversely? For the results in Exercises 25-28, and many further results, see W. N. Anderson, Jr., and R. J. Duffin, Series and Parallel Addition of Matrices, J. Math. Anal. Appl., to appear. 80. Let Ay, Ar > 0 and M(41,43) denote the convex hull of the set of matrices C such that C > Ay and C > As. Similarly define M(AuA14s) with Ay >0. Ie it true that M(A1,A,43) = M(M(A4,As),4s), with the obvious definition of the right- hand side? 81. A matrix is said to be doubly etochaatic if its elements are nonnegative and the row and column sums are equal to one. A conjecture of Van der Waerden then states that per(A) > NI/N* with equality only if as; = 1/N, N the dimension of A. Here, per(A) ig the expression obtained from |4| by changing all minus signs into plus signs. Establish, the conjecture for N = 2,3. See Marcus and Minc.? For an application of this result to some interesting problems in statistical mechanics, see Hammersley # 1W. Greub and W. Rheinboldt, On a Generalization of an Inequality of L. V. Kantorovich, Proc. Am. Math. Soc., 1959, *M. Marcus and H. Mine, Am. Math. Monthly, vol. 72, pp. 677-501, 1965. + J, M. Hammersley, An Improved Lower Bound for the Multidimensional Dimer Problem, Proc. Cambridge Phil. Soc., vol. 64, pp. 455-463, 1968.Inequalities 139 Bibliography and Discussion §1. The modern theory of inequalities, and the great continuing inter- est in this field, stems from the classic volume G. H. Hardy, J. E. Littlewood, and G. Polya, Inequalities, Cam- bridge University Press, New York, 1934. An account of more recent results, together with work in different direc- tions, will be found in E.F. Beckenbach and R. Bellman, Inequalities, Springer-Verlag, 1961. A much more extensive account of inequalities pertaining to matrices and characteristic root will be found therein. ‘The principal results of this chapter have been motivated by papers by Ky Fan, Ky Fan, Some Inequalities Concerning Positive-definite Matrices, Proc. Cambridge Phil. Soc., vol. 51, pp. 414~421, 1955. Ky Fan, On a Theorem of Weyl Concerning the Figenvalues of Linear Transformations, I, Proc. Natl. Acad. Sci. U.S., vol. 35, pp. 652-655, 1949. Ky Fan, On a Theorem of Weyl Concerning the Eigenvalues of Linear Transformations, II, Proc. Natl. Acad. Sci. U.S., vol. 36, pp. 31-35, 1950. Ky Fan, Maximum Properties and Inequalities for the Eigenvalues of Completely Continuous Operators, Proc. Natl. Acad. Sci. U.S., vol. 37, pp. 760-766, 1951. Ky Fan, Problems 4429 and 4430, Am. Math. Monthly, vol. 58, p. 194, 1951; Solutions in vol. 60, pp. 48-50, 1953. (Theorems 7 and 11 are to be found in these pages.) ‘The methods employed are, in the main, quite different. Our aim has been to show the utility of representation theorems in the derivation of inequalities. 82 to 84. An intensive discussion of these classical inequalities will be found in the book by Hardy, Littlewood, and Polya cited above. 5. The result appears to be due to Ky Fan; of. the third and fifth references cited above; of. also the papers by Oppenheim.140 Introduction to Matrix Analysis ‘A. Oppenheim, Inequalities Connected with Definite Hermitian Forms, J. London Math. Soc., vol. 5, pp. 114-119, 1930. A. Oppenheim, Inequalities Connected with Definite Hermitian Forms, II, Am. Math, Monthly, vol. 61, pp. 463-466, 1954. See also further references given there. Applications of a similar nature, using the representation of Sec. 10 of Chap. 6, may be found in R. Bellman, Hermitian Matrices and Representation Theorems, Duke Math, J., 1959. where some results due to Hus are generalized; see L. K. Hua, Inequalities Involving Determinants, Acta Math. Sinica, vol. 5, pp. 463-470, 1955; Math. Reva., vol. 17, p. 703, 1956. Hua uses a different type of integral representation, derived from the general theory of group representations. 88. See E. F. Beckenbach, An Inequality for Definite Hermitian Determi- nants, Bull. Am. Math. Soc., pp. 325-329, 1929. §7. Hadamard’s inequality is one of the most proved results in analysis with well over one hundred proofs in the literature. The proof here follows: R. Bellman, Notes on Matrix Theory—II, Am. Math. Monthly, vol. LX, pp. 174-175, 1953. Hadamard has a quite interesting remark concerning his inequality in his fascinating monograph, J. Hadamard, The Psychology of Invention in the Mathematical Field, Princeton University Press, Princeton, N.J., 1949. For the original result, see J. Hadamard, Résolution d’une question relative aux déterminants, Bull. sci. math., vol. 2, pp. 240-248, 1893. For one extension, see E, Fischer, Uber den Hadamardschen Determinantensatz, Arch. Math. u. Phys., (8), vol. 13, pp. 32-40, 1908. and for a far-reaching extension, seeInequalities 41 I. Schur, Uber endliche Gruppen und Hermitesche Formen, Math, Z., vol. 1, pp. 184-207, 1918. See also J. Williamson, Note on Hadamard’s Determinant Theorem, Bull. Am. Math. Soc., vol. 58, pp. 608-613, 1947. H. 8. Wilf, Hadamard Determinants, Mobius Functions and the Chromatic Number of a Graph, Bull. Am. Math. Soc., vol. 74, pp. 960-964, 1968. §8. For this result and the related Theorem 11, see the fifth reference of Ky Fan given above. The proof here follows R. Bellman, I. Glicksberg, and O. Gross, Notes on Matrix Theory— VI, Am. Math. Monthly, vol. 62, pp. 571-572, 1955. The result of Exercise 2 is due to Bergstrdm, H. Bergstrgm, A Triangle Inequality for Matrices, Den 11e Skandi- naviske Matematiker-kongressen, Trondheim, 1949, Oslo, pp. 264-267, 1952. while the method of proof follows R. Bellman, Notes on Matrix Theory—IV: An Inequality Due to Bergstrgm, Am, Math. Monthly, vol. 62, pp. 172-173, 1955. The result of Exercise 3 is given in R. Bellman, Notes on Matrix Theory—IX, Am. Math. Monthly, vol. 64, pp. 189-191, 1957. where two methods are used, one depending upon the generalization of the Ingham-Siegel representation referred to above and the other upon Bergstrém’s inequality. §9. This section illustrates an important technique for obtaining two results from one. Occasionally, the multiplicative results are easier to obtain ab initio. The result follows R. Bellman, Notes on Matrix Theory—VII, Am. Math. Monthly, vol. 62, pp. 647-648, 1955. $12. The book by Hardy, Littlewood, and Polya contains a history of the arithmetic-geometric mean inequality together with a large number of proofs. The one given here is our favorite because of its dependence upon backward induction.142 Introduction to Matrix Analysis §13. We have omitted the proof of inequality (1) which is given in Ky Fan’s second paper. The present derivation of (2) from (1) is given in R. Bellman, Notes on Matrix Theory—XV: Multiplicative Proper- ties from Additive Properties, Am. Math. Monthly, vol. 65, pp. 693- 694, 1958. The topic of inequalities for various functions of characteristic values has been thoroughly explored by a number of authors; cf. the following articles where a large number of additional references will be found. A. Ostrowski, Sur quelques applications des fonctions convexes et coneaves au sens de I. Schur, J. math. pures et appl., vol. 31, no. 9, pp. 253-292, 1952. M. D. Marcus and L, Lopes, Inequalities for Symmetric Functions and Hermitian Matrices, Can. J. Math., vol. 8, pp. 524-531, 1956, M. D. Marcus and J. L. McGregor, Extremal Properties of Her- mitian Matrices, Can. J. Math., vol. 8, 1956. M. Marous and B. N. Moyls, Extreme Value Properties of Hermitian Matrices, Department of Mathematics, University of British Colum- bia, Vancouver, Canada, 1956. M. Marcus, B. N. Moyls, and R. Westwick, Some Extreme Value Results for Indefinite Hermitian Metrics II, Illinois J. Math., vol. 2, pp. 408-414, 1958. We have omitted all discussion of the interesting and important ques- tion of inequalities for the characteristic roots in terms of the elementsof A. Thorough discussions of this field may be found in the papers by A. Brauer, Limits for the Characteristic Roots of a Matrix, IV, Duke Math. J., vol. 19, pp. 75-91, 1952. E, T. Browne, Limits to the Characteristic Roots of Matrices, Am. Math. Monthly, vol. 46, pp. 252-265, 1939. W. V. Parker, The Characteristic Roots of Matrices, Duke Math. J., vol. 12, pp. 519-626, 1945. Next, det us mention the paper N.G, DeBruijn and D. Van Dantzig, Inequalities Concerning Minors and Eigenvalues, Nieww. Arch. Wisk., vol. 4, no. 3, pp. 18-35, 1956. where a systematic derivation of a large number of determinantal inequalities will be found.Inequalities 143 Next, a number of interesting matrix inequalities are given in the papers by Masani and Wiener and by Helson and Lowdeuslager referred to at the end of Chap, 11, and in Ky Fan and A. J. Hoffman, Some Metric Inequalities in the Space of Matrices, Proc. Amer. Math. Soc., vol. 6, pp. 111-116, 1955. Finally, see 0, Teussky, Bibliography on Bounds for Characteristic Roots of Finite Matrices, National Bureau of Standards Rept, 1162, September, 1951. O. Taussky, Positive-Definite Matrices and Their Role in the Study of the Characteristic Roots of General Matrices, Advances in Mathe- matics, vol. 2, pp. 175-186, Academic Press Inc., New York, 1968. A recent paper containing inequalities quite different from those described above is L. Mirsky, Inequalities for Normal and Hermitian Matrices, Duke Math, J., vol. 24, pp. 591-599, 1957. See also E. V. Haynsworth, Note on Bounds for Certain Determinants, Duke Math. J., vol. 24, pp. 313-320, 1957. M. Marcus and H, Mino, Extensions of Classical Matrix Inequalities, Lin, Algebra Appl., vol. 1, pp. 421-444, 1968. H. J. Ryser, Inequalities of Compound and Induced Matrices with Applications to Combinatorial Analysis, Illinois J. Math., vol. 2, pp. 240-253, 1958. H. J. Ryser, Compound and Induced Matrices in Combinatorial Analysis, Proc. Symp. Appl. Math., Combin. Anal., pp. 149-167, 1960. R, C. Thompson, Principal Submatrices of Normal and Hermitian Matrices, [Ulinois J. Math., vol. 10, pp. 296-308, 1966. R. C. Thompson and P. McEnteggert, Principal Submatrices, II: The Upper and Lower Quadratic Inequalities, Lin. Algebra Appl., vol. 1, pp. 211-243, 1968.9 Dynamic Programming 1, Introduction. In Chap. 1, we encountered quadratic forms in con- nection with maximization and minimization problems, and observed that systems of linear equations arose in connection with the determi- nation of the extrema of quadratic functions. Although simple conceptually, the solution of linear equations by means of determinants is not feasible computationally for reasons we have discussed in previous pages. Consequently, we must devise other types of algorithms to compute the solution. This suggests that it might be well to develop algorithms connected directly with the original maxi- mization problem without considering the intermediate linear equations at all, In this chapter, we shall study a number of problems in which this can be done. Our basic too! will be the functional equation technique of dynamic programming. 2. A Problem of Minimum Deviation. Given a sequence of real numbers {a:},k = 1,2, . . . ,N, we wish to determine another sequence {za}, k= 1,2,..., N, “close” to the ax and “close” to each other. Specifically, we wish to minimize the quadratic expression x x Qe) = Y eal — ta)? + ) daley - ay)? ay om where c, and di are prescribed positive quantities, and zo is a given constant. Proceeding in the usual fashion, taking partial derivatives, we obtain the system of linear equations ex(t1 — to) — ex(ts — 21) + a(t - a) = 0 x(tu — a1) — Cya(Teps — 2x) + dx(ve — ax) = 0 (2) en(tw — ty-1) + dy(tw — aw) = 0 146Dynamic Programming M5 If N is large, the solution of this system requires some care. In place of using any of a number of existing techniques to resolve this system of equations, we shall pursue an entirely different course. 3. Functional Equations. Let us consider the sequence of minimization problems: Minimize over all x x x Q(t) =) exfes = aus)? +) ales — an)? a ie & with z,_1 = u, a given quantity. It is clear that the minimum value depends upon u, and, of course, upon r. Let us then introduce the sequence of functions {f,(u)), where —«© =ai-t) O< 2 Use this representation to obtain anote: approximating matrix problem of the form x wer SD kuping t21 Rs. 2 uy 7. Functional Equations. Since A, as given by (6.4), is real and sym- metric, we know that the characteristic values can be determined from the problem:149 Dynamic Programming Minimize x y (ua = us—1)* (1) eh subject to the conditions ty = uy = 0 (2a) Not duu? (2b) et In place of this problem, consider the problem of determining the minimum of x > (uy — we-a)* (3) kee subject to the conditions yaa = 0 (4a) uy =0 (4b) <1 > dat = 1 (40) ie .,N-1, where -« - (er + ax, tax, + - ~~ +0%-'2y)* subject to attat poss paytad (e) at + (a + on)* + [ar + at, + (a + db) +++ + fer + “ ; (a+ bry +++" + [a+ (N~Dblew|* subject to at bast + = ats150 Introduction to Matrix Analysis For a discussion of these questions by other means, see A, M. Ostrowski, On the Bounds for a One-parameter Family of Matrices, J. fur Math., vol, 200, pp. 190-200, 1958, 8. Jacobi Matrices. By a Jacobi matriz, we shall mean a matrix with the property that the only nonzero elements appear along the main diagonal and the two contiguous diagonals. Thus aj=0 fi-j1 22 qa) Let us now consider the question of solving the system of equations Az =e (2) where A is a positive definite Jacobi matrix. As we know, this problem is equivalent to that of minimizing the inhomogeneous form Q(z) = (#,Az) — 2(c,z) 3) Writing this out, we see that we wish to minimize Q(z) = aur? + 2arerite + arate? + Zarseets + * + 2ay—awtw-rtw + Onwan? — Qe, — Qeate— +++ — 2ewew (4) Consider then the problem of minimizing Qe(z,z) = ait? + Qartiee + aget2® + Zagetata + ++ + $ Qos etn ite + Ouern? — Qert — Bests — ++ * — Dery (5) fork = 1,2,..., with Qu(t,2) = auest — 2er, © Define Sfi(2) = min Qi(z,2z) @ Then it is easy to see that fez) = min (aure? — ere + fi-r(oa — On—aate)] (8) for & = 2,3,.... Once again, we observe that each function fi(2) is a quadratic in 2, file) = tte wet k=1,2,.., @) * with fi@) = = an Substituting in (8), we obtain recurrence relations connecting us, , and w with ws, t1, and ws, We leave the derivation of these as exercises for the reader.Dynamic Programming 151 EXERCISES 1. Generalize the procedure above in the case where A is a matrix with the property that a; = 0, |j — i] > 3. 2. Obtain relations corresponding to those given in Sec. 7 for the largest and smallest values of a aymmetric matrix A with a; = 0, |j — i] > 2; and aj; = 0, |j - i] > 3. 9. Analytic Continuation. We were able to apply variational tech- niques to the problem of solving the equation Az = ¢ at the expense of requiring that A be symmetric and positive definite. It is tempting to assert that the explicit solutions we have found are valid first of all for symmetric matrices which are not necessarily positive definite, provided only that none of the denominators which occur is zero, and then suita- bly interpreted for not necessarily symmetric matrices. The argument would proceed along the following lines. The expres- sions for the 2, are linear functions of the ¢; and rational functions of the a. An equality of the form x > aay =e @ i valid in the domain of a, space where A is positive definite should cer- tainly be valid for all a, which satisfy the symmetry condition, It is clear, however, that the rigorous presentation of an argument of this type would require a formidable background of the theory of func- tions of several complex variables. Consequently, we shall abandon this path, promising as it is, and pursue another which requires only the rudimentary facts concerning analytic continuation for functions of one complex variable. Consider in place of the symmetric matrix A the matrix zI + A, where zisa-scalar. If z is chosen to be a sufficiently large positive number, this matrix will be positive definite, and will be a Jacobi matrix if A is a Jacobi matrix, ‘We can now apply the principle of analytic continuation. The 2, a8 given by the formulas derived from the variational technique, are rational functions of z, analytic for the real part of z sufficiently large. Hence, identities valid in this domain must hold for z = 0, provided none of the functions appearing has singularities at z = 0. These singularities can arise only from zeroes of the denominators. In other words, the formulas are valid whenever they make sense. 10. Nonsymmetric Matrices. In order to apply variational tech- niques to nonsymmetric matrices, we must use a different device. Consider the expression Say) = (@,Bx) + 2(2,Ay) + (y,By) — 2(@,2) — 2(6,y) qa)152 Introduction to Matrix Analysis where we shall assume that B is symmetric and positive definite and that A is merely real. Minimizing over z and y, we obtain the variational equations Be + Ay Alt + By a t @) In order to ensure that f(2,y) is positive definite in x and y, we must impose some further conditions upon B. Since we also wish to use ana- lytic continuation, perhaps the simplest way to attain our ends is to choose B = 21, where z is a sufficiently large positive quantity. The functional equation technique can now be invoked as before. We leave it to the reader to work through the details and to carry out the analytic continuation. 11. Complex A. Let us now see what can be done in case A is sym- metric, but complex, A = B + iC, Band Creal. The equation Az = ¢ takes the form (B+ iC) + ty) =a + ib qa) or, equating real and complex parts, Br - Cy = Cx + By Observe that these relations establish a natural correspondence between the N-dimensional complex matrix B + iC and the 2N-dimensional matrix [é “5 @) a relationship previously noted in the exercises. In order to relate this matrix, and the system of equations in (2) to a variational problem, we consider the quadratic form S(e,y) = (2,Cx) + 2(z,By) — (y,Cy) — 2(6,2) — 2(a,y) (4) If we impose the condition that C is positive definite, it follows that f(e,y) is convex in x and concave in y and hence, as a consequence of general theorems, which we shall mention again in Chap. 16, that 3 @ min max f(z,y) = max min f(z,y) (5) 2 4 v8 Since f(z,y) is a quadratic form, there is no need to invoke these general results since both sides can be calculated explicitly and seen to be equal. We leave this as an exercise for the reader. Before applying the functional equation technique, we need the further result that the minimum over the z; and the maximum over the y; can beDynamic Programming 153 taken in any order. In particular, min max = minmax min max 6) fousy acest) (Watts WM) ah (eh eon eth) (Wn one dD This we also leave to the enterprising reader. Problems of this nature are fundamental in the theory of games, which we shall discuss briefly in Chap. 16. Combining the foregoing results, we can treat the case of complex A. If necessary, we consider A + iz/, where z is a sufficiently large positive quantity, once again introduced for the purposes of analytic continuation. 12, Slightly Intertwined Systems. Let us now consider the problem of resolving @ set of linear equations of the form Outs + yete + Gites = C1 G21 + Grate + Onats = C2 Gyits + Gyxt2 + ats + bite = Cy bits + aute + Gusts + Geers = Ce Geate + Osets + asets = Cs Agate + Geuts + Agere + dats = Cy by—s@an—a + Gay —2aw—2tan—2 + Gan—2.n—12an—1 + Gan—2.sn Ton sy -1,aN—27an—2 + Gay —1,aw—1an—1 H Osw-1.sN Tay Gan,sn—2tan-2 + Osw,sw-itw-a + Osw,swTew ay If the coefficients b; were all zero, this problem would separate into N simple three-dimensional problems. It is reasonable to assume that there is some way of utilizing the near-diagonal structure. We shall call matrices of the type formed by the coefficients above slightly intertwined. They arise in a variety of physical, engineering, and economic analyses of multicomponent systems in which there is ‘‘weak coupling" between different parts of the system. To simplify the notation, let us introduce the matrices j= 1,2,3 (2) An = (itor—ssat—2) ty Zak Cue es | ra c= | cur (3) re ce k=12.... We shall assume initially that the matrix of coefficients in (1) is a positive definite. Hence, the solution of the linear system is equivalent and the vectors154 Introduction to Matrix Analysis to determining the minimum of the inhomogeneous quadratic form (e142) + (eta!) + +++ + (0%,AaX) = 2chat) — 2(e%n%) — ++ + — 2(e%,2%) + Qbiwsey + Berets + + ++ + by—vtaw—stew-2 (4) This we shall attack by means of functional equation techniques. Introduce the sequence of functions {fy(2)], 2 <2< =, N=1, 2,... , defined by the relation Sul) = x x wet [YX Gay -2 Y Gx) 42D deren +2erw] (6) & & & Proceeding in the usual fashion, we obtain the recurrence relation ful) = ~ (@*,Ane™) + eran — 2(0%,2") + fa(bv—stav—2)] (6) where Ry is three-dimensional region ~ © < ayy, tant, Bana < ©. 18. Simplifications—I. We can write (12.6) in the form fw(2) = min { min [(2%,Aye®) + 2zaaw — 2(C%,2%)] + fv—t(bv—1ean—a)] wan Bw () Introduce the sequence of functions ox(@y) = min [(e%,A%z¥) + 2ztaw — 2(0%,2%)] (2) sane where zx-2 = y. Then fu(e2) = min [gw(2,u) + frr(by—wy)) (3) y a simple one-dimensional recurrence relation. 14, Simplifications—II. We can, as in the section on Jacobi matrices, go even further, if we observe that f(z) is for each N a quadratic in z, fu() = uy + 2vne + wye? ) where uy, vx, wy are independent of 2. Using this relation in (3), we readily obtain recurrence relations for {ux,vv,tov}. EXERCISES 1, Obtain recurrence relations which enable us to compute the largest and smallest characteristic roots of slightly intertwined matrices. 2. Extend the foregoing technique to the case where the Ay are not all of the same dimension,Dynamic Programming 155 15. The Equation Ax = y, Let us now pursue a simple theme. If A is a positive definite matrix, we know that the solution of Ar=y () given by z = A-'y can also be obtained as the solution of the problem of minimizing the quadratic form Q(z) = (z,Az) — 2(z,y) @) The minimum has the value —(y,A~"y). Comparing the two approaches to the problem of solving (1), we can obtain some interesting identities. Introduce the function of N variables x Solvay «<< sux) = min [ 5 agra; - 2 ) rays] (3) a inl aged Write x x , ayes — 2 y Bye ona mt wot aya vat = ayytyt + t age ity — 2 y 2(yi— Gintw) — 2tnyw (4) A From this expression, we obtain the functional equation Fugues + « « w) = min [awwty® — Qewyw t+ fv-iyr — QivtnsY2 — Gavtn, . - - Yn—1— Ov-intw)] (5) In order to use this relation in some constructive form, we recall that Fu(yisyr, +» - sw) = —(¥,A~'y). Hence, write » Soya, © + uw) = > oN uu (6) 1 Returning to (5), we obtain the relation x > eN yyy = min [awnen? — 2evyw 1 = ory Nat + D. cs(N = (ye = awww (vy — aywzn) | (2) a or, collecting terms, ” wat > ex(N)uayy = min [ 20% {ayer + > anayncy(N — »} jan By wpa Nn1 + ty {-200 ~2 Sane -v}+ J atv - uw] © fs iat156 Introduction to Matrix Analysis The minimization can now be readily performed, Nol ww + J. amyiey(V - 1) ty = =} @) anw + 2 auveajyey(N = 1) age while the minimum value itself is given by oF eu( — Yuu) (ann + py auvayuoy(N — 1)) = (uw + 5 seine, (N — »)" Nt seh ann + > ainazney(N — 1) agel x Equating coefficients of yy; in (10) and 2 ca(N)yay, we obtain recur- ft rence relations connecting the sequences {c4(N)} and fey(N — 1)). 16. Quadratic Deviation. The problem of minimizing over all zx the quadratic form Qw(e) = I," (M0 - ¥ noo) )+ dt a wt is, in principle, quite easily resolvable. Let {h(l)} denote the ortho- normal sequence formed from {g(t)} by means of the Gram-Schmidt orthogonalization procedure, where, without loss of generality, the g(t) are taken to be linearly independent. Then min Qx(e) = min f,” (s) - 5 aha(o) at . 4 mH N = fora (fy mana)? @) This result may be written . [vra- i Ja’ Feds Oke 4) ds at @) where y ke(o,t) =) da(syhat ® ShDynamic Programming 157 Let us now obtain a recurrence relation connecting the members of the sequence {kw(s,é)}. Introduce the quadratic functional, a function of the function f(), y déx(f) = min fr ( - ? zm)! at (6) = kel Employing the functional equation technique, we have for N = 2, du(f) = min dxa(f — angn) ® and rr d3(f) = min f," (f — 2g) at ® In order to use (6), we employ the technique used above repeatedly— we take advantage of the quadratic character of ¢x(f). Hence, on(s) = min [f" (F — anon) dt =f V's) — axa GO — zxon(D)kw(e,) ds a] = ain {fr ae fo" f." kuvls,ogleys() ae at = Ben | f.” fow at — f." f,” ow(erstorkn-sX 6,0) de dt] tant [J owtdt — f° J." kw-ale.0an(s)on(o deat} (8) Obtaining the minimum value of zy, and determining the explicit value of the minimum, we obtain the recurrence relation ew(ot) = ke-a(ost) + 9¢@0H( guledon) _ 2gn(s) daserilen te) ds +E Lf ia feito tact (tat) das dts (9) where die =f" onto) ae — ff bx-vstonisanto ase (10) 1%. A Result of Stieltjes. Since we have been emphasizing in the preceding sections the connection between the solution of Az = b and the minimization of (z,Az) — 2(b,z) when A is positive definite, let us use the same idea to establish an interesting result of Stieltjes. We shall obtain @ generalization in Chap, 16. Theorem 1. If A is a positive definile matrix with the property that ay < 0 for i # j, then A~' has all positive elements.158 Introduction to Matrix Analysis Proof. Consider the problem of minimizing Q(z) = (2,42) - 2(b,2), where the components of 6 are all positive. Assume that 21, t2,... , ty <0, e431, Tay, . + , Zy > 0, at the minimum point. Writing out (2,Az) — 2(6,2) in the form 2 oN autit + data? + aynay? + > ageiay + > 2 asrety agen =F NoN x + x + > aueiay + aaa 2 bax — 2 > ba; (1) ehh asehen = eke we see, in view of the negativity of a,; for 7 # j, that we can obtain a smaller value of Q(z) by replacing x; by —2, fori = 1,2,... , k, and leaving the other values unchanged, provided at least one of the 2;, i=k+41,...,N, is positive. In any case, we see that all the 2; can be taken to be non-negative. To show that they are actually all positive, if the b; are positive, we observe that one at least must be positive. For if Sms sts Sty =0 at the minimum point, then zy determined as the value which minimizes zy" — 2byay is equal to by and thus positive. Since Az = b at the minimum point, we have aut; = — Y ayy ¢=1,2,...,N—1 (2) ims which shows that 2; > 0. We see then that A-' is a vector with positive components whenever bis likewise. This establishes the non-negativity of the elements of A-'. To show that A~ actually has all positive elements, we must show that. A-'b has positive components whenever b has non-negative components with at least one positive. Turning to (2), we see that the condition a,; < 0,1 # j, establishes this. MISCELLANEOUS EXERCISES 1, Given two sequences {as} and [bs], k = 0, 1, 2,..., N, we often wish to determine a finite sequence [z+], k = 0, 1, . .. , M which expresses bs most closely Mu in the form by = > ae. i= To estimate the closeness of fit, we use the sum x a mate) = d (m- Yvan)? v>ared =Dynamic Programming 159 Consider the quadratic form in yo, 1, . » . » yw defined by Sore lyoury + ~ » syn) = min [(yo — xoao)® + (ys ~ omy — 2rGo)? + + + (yw — toy — BON — + + = 2Mtw—m)*) Show that Iroldoinn «= 4) = ( 5 w) (s at) - (3 anys)" So NE a and that, generally, SwaeYory + + + yw) = min [(yo ~ zoa0)* $F Sireuatouly = aby Ya — etsy «6 Y= Bey) 2. Write ” Soatinns «+ ye) = Yo caves Fo and use the preceding recurrence relation to obtain the ci(N,M) in terms of the e(N = 1, M = 1). 8. Write fu.m(voyy - + yx) = (y,Amny) and obtain the recurrence relation in terms of matrices. For an alternate discussion of this problem which is paramount in the Kolmogorov- Wiener theory of prediction, see N. Wiener, The Extrapolation, Interpolation and Smoothing of Stationary Time Series and Engineering Applications, John Wiley & Sons, Inc., 1942, and particularly the Appendix by N. Levinson. 4. Let A and B be two positive definite matrices of order N with AB # BA and a given N-dimensional vector. Consider the vector 24 = ZwZw-1 * - » ZsZic, where each matrix Z; is either an A or a B. Suppose that the Z, 7 = 1,2,..., N,are to be chosen so as to maximize the inner product (2w,b) where & is a fixed N-dimen- sional veetor. Define the funetion Sule) = max (zx,b) tel forN =1,2,...,andalle. Then file) = max ((Ae,b),(Beb))» Ino) = max (Jy-(A0,fu-(BO) N= 2,3, . 5. Does there exist a scalar d such that fir(c) ~ Mg(c) as VN 2? 6. What is the answer to this question if AB = BA? 7. Suppose that (is a given positive definite matrix, and the Z: are to be chosen so that ZyZv-1 ++ ZZ,C has the maximum maximum characteristic root. Let gx(C) denote this maximum maximorum, ‘Then 9(C) = max [¢(AC),4(BC)], gw(C) = max [gy1(AC),gv-1(BC)) NN = 2,3, . where @(X) is used to denote the maximum characteristic root of X. 8. Does there exist a scalar such that gw(C) ~ WA(c) as N+ =? See C. Bohm, Sulla minimizzazione di una funzione del prodotto di enti non commutati, Lincei- Rend, Sei. fis. Mat. e Nat., vol. 23, pp. 386-388, 1957.160 Introduction to Matrix Analysis Bibliography §1. The theory of dynamic programming in actuality is the atudy of multistage decision processes. It turns out that many variational prob- Jems can be interpreted in these terms. This interpretation yields a new approach which is useful for both analytic and computational purposes. In this chapter, pursuant to our general program, we are interested only in analytic aspects. For a detailed discussion of the general theory, the reader is referred to R. Bellman, Dynamic Programming, Princeton University Press, Princeton, N.J., 1987. 82. The problem treated here is a simple example of what is called a “gmoothing” problem; ef. R. Bellman, Dynamic Programming and Its Application to Vari- ational Problems in Mathematical Economics, Caleulue of Varia~ tions; Proceedings of Symposia in Applied Mathematics, vol. VIII, pp. 115-139, McGraw-Hill Book Company, Inc., New York, 1958. For a discussion of smoothing problems of a different type, see the following: I. J. Schoenberg, On Smoothing Operations and Their Generating Functions, Bull. Am. Math. Soc., vol. 59, pp. 199-230, 1953. N, Wiener, Cybernetics, John Wiley & Sons, Inc., New York, 1951. §3. The derivation of this recurrence relation is » particular appli- cation of the principle of optimality; see the book referred to above in §1. The results discussed in §3 to §5 were first presented in R. Bellman, On a Class of Variational Problems, Quart. Appl. Math., vol. XIV, pp. 353-359, 1957. 87. These results were presented in R, Bellman, Eigenvalues and Functional Equations, Proc. Am. Math. Soc., vol. 8, pp. 68-72, 1957. §8. This discussion follows R. Bellman, On a Class of Variational Problems, Quart. Appl. Math, vol. XIV, pp. 353-359, 1957. For a number of results dependent upon the fact that Jacobi matrices arise in the treatment of random walk processes, seeDynamic Programming 161 8. Karlin and J. McGregor, Coincident Properties of Birth and Death Processes, Technical Rept. 9, Stanford University, 1958, P, Dean, The Spectral Distribution of a Jacobian Matrix, Proc. Cambridge Phil, Soc., vol. 52, pp. 752-755, 1956. 99. This discussion, given for the case of finite matric extended to cover more general operators. An appli nique is contained in R, Bellman and 8. Lehman, Functional Equations in the Theory of Dynamic Programming—X: Resolvents, Characteristic Functions and Values, Proc. Nail. Acad. Sci. U.S., vol. 44, pp. 905-907, 1958, , can be readily tion of this tech- 10-811. The contents of these sections are taken from a more general setting in R. Bellman and S. Lehman, Functional Equations in the Theory of Dynamic Programming—IX: Variational Analysis, Analytic Con- tinuation and Imbedding of Operators, Duke Math. J., 1960. For an extensive discussion of matters of this type, see C. L. Dolph, A Saddle Point Characterization of the Schwinger Stationary Points in Exterior Scattering Problems, J. Soc. Ind. Appl. Math., vol. 5, pp. 89-104, 1957. ©. L. Dolph, J. E. McLaughlin, and I, Marx, Symmetric Linear Transformations and Complex Quadratic Forms, Comm, Pure and Appl. Math., vol. 7, pp. 621-632, 1954. $12 to §14. The results are taken from R. Bellman, On Some Applications of Dynamic Programming to Matrix Theory, Illinois J. Math., vol. 1, pp. 207-301, 1957. 815 to §16. The results follow R. Bellman, Dynamic Programming and Mean Square Deviation, The RAND Corporation, Paper P-1147, September 13, 1957. 817. This result was given by T. J. Stieltjes, Sur les racines de X, = 0, Acta Math., vol. 9, 1886- 1887. A generalization of this result will be given in Chap. 16. For another application of this device, see R. Bellman, On the Non-negativity of Green’s Functions, Boll. Unione Matematica, vol. 12, pp. 411-413, 1957.162 Introduction to Matrix Analysis For some further applications of functional equation techniques to quadratic forms, see R. E. Kalman and R. W. Koepcke, Optimal Synthesis of Linear Sampling Control Systems Using Generalized Performance Indexes, Am. Soc. Mech. Eng., Instrument and Regulators Conferences, Newark, April 2-4, 1958, R, Bellman and J. M, Richardson, On the Application of Dynamic Programming to a Class of Implicit Variational Problems, Quart. Appl. Math., 1959. M. Freimer, Dynamic Programming and Adaptive Control Processes, Lincoln Laboratory Report, 1959. H. A. Simon, Dynamic Programming under Uncertainty with » Quadratic Criterion Function, Zeonometrica, vol. 24, pp. 74-81, 1956. See Chap. 17 for many further results and references. The study of routing problems introduces a new type of matrix composition, Write C = M(A,B) if ey = min (au + by). See R. Bellman, K. L. Cooke, and Jo Ann Lockett, Algorithms, Graphs, and Computers, Academic Press Inc., New York, 1970. C. Berge, The Theory of Graphs and Its Applications, John Wiley & Sons, New York, 1962. J. F, Shapiro, Shortest Route Methods for Finite State Deterministic Dynamic Programming Problems, S[AM J. Appl. Math., vol. 16, pp. 1232-1250, 1968,10 Matrices and Differential Equations 1, Motivation. In this chapter which begins the second main portion of the book, we shall discuss the application of matrix theory to the solu- tion of linear systems of differential equations of the form dx, x E ay, 20)=a i=1,2,...,N (yy it where the a; are constants. In order to understand the pivotal positions that equations of this apparently special type occupy, let us explain a bit of the scientific back- ground. Consider a physical system S whose state at any time ¢ is assumed to be completely described by means of the N functions 7,(t), z(t), ..., tw(t). Now make the further assumption that the rate of change of all these functions at any time ¢ depends only upon the values of these functions at this time. This is always an approximation to the actual state of affairs, but a very convenient and useful one. The analytic transliteration of this statement is a set of differential equations of the form $s = plete, .. - stn) i=1,2...," @) with an associated set of initial conditions 2,(0) = 6 1,2,...,N 3) The vector c = (¢i,¢x, . . . Cv) represents the initial state of the system. Sets of constants, {¢;}, for which Serer, +. tv) =O ¢=1,2,...,N (4) play a particularly important role. They are obviously equilibrium states since S cannot depart from them without the intervention of external forces. 163164 Introduction to Matrix Analysis Whenever such states exist, it is of great interest to study the behavior of the system in the neighborhood of these states. In other words, we are examining the stability of the system under small disturbances. If the perturbed system eventually returns to the equilibrium state, we say that it is stable; otherwise, we say that it is unstable. These consider- ations are of great practical significance. In order to carry out this study, we set Baty (5) where the y; are taken to be small quantities. Substituting in (2), we obtain the equations Ste Be Hey t+ oye + Yn -.-,ertuy) f-12...,N x, = filener, ..- er) + ) anys to ° 6 at where ay = atti = C1, t: = ey, . 2. , tN = ON (7) and the three dots signify terms involving higher powers of the 4. ‘The behavior of S in the neighborhood of the equilibrium state, {c:}, is thus determined, to an approximation whose accuracy must be care- fully examined, by the linear system with constant coefficients given in (). We have then a powerful motivation for the study of linear systems of this type. Our aim is to determine analytic representations of the solu- tion which will permit us to ascertain its limiting behavior as — «. These questions of stability will be taken up again in Chap. 13. 2. Vector-matrix Notation. To study (1.1), we introduce the vectors yand ¢, possessing the components y; and c,, respectively, and the matrix A = (qj). It is clear from the way that the difference of two vectors is defined that the appropriate way to define the derivative of a vector is the following: ayy dt dn di " « alé dyn atMatrices and Differential Equations 165 Similarly, the integral of y(t) is defined to be ['n(o) as [' vs) ds fiusa-]: @ J wv(s) ds The derivatives and integrals of matrices are defined analogously. It follows that (1.1) can be written Yeay 0) =6 @) The matrix A will, in general, not be symmetric. Consequently, the techniques and results of the first part of this volume can be expected to play a small role, We shall have to develop some new methods for treating general square matrices. A vector whose components are functions of ¢ will be called a vector function, or briefly, a function of t. It will be called continuous if its components are continuous functions of ¢ in the interval of interest. We shall use similar terms in describing matrix functions. EXERCISES 1. Show that wo deen = (Ss) + (28) ® gun - (Bet4G © Jap = (4) BHA (2) @ Jory = -x1 (B) xn ax © 2a = () X$X (%) Xen px 2. Obtain an equation for the derivative of X¥. 8. Norms of Vectors and Matrices. We could, if we so desired, use the scalar function (2,2) a8 a measure of the magnitude of z. However, it is more convenient to use not these Euclidean norms, but the simpler function x Ill = > ted a166 Introduction to Matrix Analysis and, for matrices, lll = 5 la @ fecal It is readily verified that lle + ull < lei + [lvl =A + BI < Al] + LB) zl] < Af} lel] ABI) < [ll [Bll Hevzll = Jes] [zl] les Al] = Jel [Al The reason why we have chosen the foregoing norms for vectors and matrices is that the verification of the results in (3) is particularly simple. As we shall see in the exercises below, there are a large number of choices of norms which are equally useful when dealing with finite dimensional vectors and matrices. It is only when we turn to infinite dimensional vectors and matrices that the choice of a norm becomes critical. EXERCISES 1, Show that we could define as norms satisfying (8) the functions Fl (S lait) = (ey mt ii ~ ( § Tout)" = te (Aasys aon 2. If we set ||z|| = max |z;], what definition should we take for ||A|| in order that ‘ Ali of the inequalities of (3) be valid? 8. Let |2/| be # vector norm satisfying the vector relations in (3) and the condition that ||| 2 0 and [| ~ 0 if and only if = = 0. Show that [|| = mex 42 is a matrix norm which aatiafee the remaining relations in (2). (This i the sandard way of inducing norm for transformations, given a norm for vectors.) 4. If we use the norm for z appearing in Exercise 2, which norm do we obtain for A using the technique of Exercise 3? 5. Show that convergence of a sequence of vectors |2*] to a vector z implies and is implied by convergence of the kth components of the members of the sequence {2} to the kth component of 2. 6. Bhow that convergence of a sequence of vectors {z*] in one norm satisfying the conditions of Exercise 3 implies convergence in any other norm satisfying these conditions. 7. Show that Ilfx(e) dtl] < Sjx( || ae WA] del) < SAI ae 8. Show that ||A"[| < |All" for any norm satisfying the condition in (3). 9. Is there any norm satisfying the conditions in (3) for which ||ABl| = ||A| ||?Matrices and Differential Equations 167 4. Infinite Series of Vectors and Matrices. In the course of establish- ing the existence of solutions of the linear vector equation appearing above, we shall have need of infinite series of vectors and matrices. By the vector > z*, we shall mean the vector whose ith component is the n=o cum of the series ' 2°. Thus, the convergence of the vector series is neo equivalent to the simultaneous convergence of the N series, > a It wo follows that a sufficient condition for convergence of the vector series ), z%is that the scalar series) ||2"|| converge. = neo Similarly, a matrix series of the form )' A, represents N* infinite ko series, and a sufficient condition for convergence is that ) |lAsl fo converge. 5. Existence and Uniqueness of Solutions of Linear Systems, With these preliminaries, we are ready to demonstrate the following basic result. Theorem 1. If A(t) is continuous for t > 0, there is a unique solution to the vector differential equation dx 7 AW x0) > @ This solution exists for t > 0, and may be written in the form r= Xe (2) where X(1) is the unique matrix satisfying the matrix differential equation dX aq ~AOX = XO= (3) Proof. We shall employ the method of successive approximations to establish the existence of a solution of (3). In place of (3), we consider the integral equation X14 f) Ax as (4)168 Introduction to Matrix Analysis Define the sequence of matrices (X,} as follows: Xool © Xue It fl A@Xnds n=0,1,... Then we have Xo —Xa= fPAC(Xa- Xeds n=12%... 6) Let m= max ||A(|| ) oath Here and in what follows, we are employing the norms defined in (3.1) and (3.2). Using (6), we obtain [Xaga — Xall = Uf AG)Xe — Xen) dal S fy WAG Xs — Xnall do Sm fl IX, — Xpalds ®) for0 0, we may take t; arbitrarily large. We thus obtain a solution valid for ¢ > 0. It is easily verified that z = X(Q)c is a solution of (1), satisfying the required initial condition. Let us now establish uniqueness of this solution. Let Y be another solution of (3). Then ¥ satisfies (4), and thus we have the relation X- Y= ff A(x) - YO) ds (uy Hence, the seri Hence ‘ IX - Is [{MA@IIX® - Yellae (12) Since Y is differentiable, hence continuous, define m, = max ||X — YI (13) osignMatrices and Differential Equations 169 From (12), we obtain IX-YI ©, we see that ||X — Y|] <0, Hence X = ¥. Having obtained the matrix X, it is easy to see that X(0)¢ is a solution of (1). Since the uniqueness of solutions of (1) is readily established by means of the same argument as above, it is easy to see X (0)c is the solution. EXERCISE 1, Establish the existence of a solution of (1) under the condition that A(j) is Riemann-integrable function over any finite interval. In this case, need the differen- tial equation be satisfied everywhere? Examine, in particular, the case where AQ) = AO St St, AW = Bt > bo 6. The Matrix Exponential. Consider now the particular case where A(t) is a constant matrix. In the scalar case, the equation S=au uO) =c a) has the solution u = e%c. It would be very convenient to find an analo- gous solution of the matrix equation a= AX XO)=C6 (2) having the form X = eA’, By analogy with the scalar case, or from the method of successive approximations used in Sec. 5, we are led to define the matrix exponential function by means of the infinite series An Peete @ em THAR + Let us now demonstrate170 Introduction to Matrix Analysis Theorem 2. The matriz series defined above exists for all A for any fixed value of t, and for all t for any fized A. It converges uniformly in any finite region of the complex t plane. Proof. We have Arerl] — [Aline at Sool @ Since ||A||"|{/"/n! is a term in the series expansion of el'4i, we see that the series in (3) is dominated by a uniformly convergent series, and hence is itself uniformly convergent in any finite region of the ¢ plane. EXERCISE 1, Using the infinite series representation, show that d/di(et) = Ae‘ = eM'A. 7, Functional Equations—I, ‘The scalar exponential function satisfies the fundamental functional equation ett) = gatgat a) Unless there is an analogue of this for the matrix exponential, we have no right to use the notation of (3). Let us now demonstrate that. Moto = gdrgdt ) Using the series expansions for the three exponentials and the fact that absolutely convergent series may be rearranged in arbitrary fashion, we have “Sen L( 2a) Atel = Ja EO — pan a From (2) we obtain, upon setting s = —t, the important result that em (4) eA = T= ont Hence, e4* is never singular and its inverse is e~, ‘This is a matrix ana- logue of the fact that the scalar exponential never vanishes.Matrices and Differential Equations 17 8. Functional Equations—II. The proof of the functional equation in Sec. 7 was a verification rather than a derivation. In order to under- stand the result, let us turn to the differential equation at G7 Ax a Observe that e“' is a solution with the boundary condition X(0) and that e4* is a solution with the boundary condition X(0) Hence, from the uniqueness theorem, we may conclude that AHO = edtgdt (2) 9. Functional Equations—III. Having derived the functional equa- tion discussed above, the question naturally arises as to the relation between e(4+#” and ete, Since dave a 14 (At BE AEB eta (tae Ee (++ SP --) a " 1 are (at Bet A + ape y BEY... we see that e(A+ Bt — eAteBt = (BA — AB) + see (2) Consequently, o(4+8! = e4te®! for all tonly if AB = BA, which is to say if A and B commute. It is easy to see that this is a sufficient condition. 10. Nonsingularity of Solution. We observed in Sec. 7 that e4‘ is never singular. Let us now point out that this is a special case of the general result that the solution of the equation #2 ax XO) =1 a) is nonsingular in any interval 0 < t < t; in which fr | AQ@| at exists, There are several ways of establishing this result. The first is the most interesting, while two other methods will be given in the exercises below. The first method is based upon the following identity of Jacobi: ixco| = olf sone ) To derive this result, let us consider the derivative of the scalar func- tion |X()|. To simplify the notational problem, consider the two-172 Introduction to Matrix Analysis dimensional case. We have aia Mm Ix@| =|2 % (3) where a = Gyr + ay2t2 aye = anys + ary (4) o = Outs + Oats ou = Guy + Gays and n()=1 (0) =0 0) =0 (0) =1 ® a dy dy| fm on Fl = ad dtl+ dr, dys tote ua Ques + arate og es ty Ya + a n | Qyty + Ort, Onys + Greys nm nM oul oy tonl 7 ve = (tr AO)XO] @) Thus [x(p] = efor soe @ since |X(0)| = 1. EXERCISES 2. Consider the equations 2 = A(X, X(0) = [and d¥ /dt = -YA(, ¥(0) = I. Show that Y = X-1, and conclude that X(¢) is nonsingular in any interval where 14(01 is integrable. %. Consider the second-order differential equation u” + p(Ou’ +q()u = 0. Write uw’ =v and obtain a first-order aystem corresponding to this second-order equation, namely, du a & = pine — aiou $8. Consider the integral J = i * wu” + p(w’ + a(Ou) dt. Integrating by parte we obtain Jebedt I u(w"” + pili! + gi(t)w) dtMatrices and Differential Equations 173 What is the connection between the vector-matrix system for w” + pi(t)o' + qu(}w = 0 and the vector-matrix system for u” + p(w’ + q()u = 0? 4. Consider the foliowing proof of the nonsingular nature of X(t). If |X()| = 0 point ts, 0 < t < 6, there exists a nontrivial relation between the vectors con- stituting the columns of X(), cit! +cst+ +++ tens" =0. Since iz! + solution of the equation dz/dt = A()2, if it is zero at one point, it is zero for all ¢ in (0,4)]. The relation |X(#)| = 0, however, clearly does not hold at ¢ = 0; a contradiction. 11, Solution of Inhomogeneous Equation—Constant Coefficients. Let us now consider the problem of solving the inhomogeneous system & =Artf) 20) =c (1) The utility of the matrix exponential notation shows to advantage here. We have em Gi - 42) = Fletis) = oat @) Hence, otis = c+ fl eafls) de (8) or am etic t [i edtvf(a) ds (4) Observe how the use of matrix exponential function permits us to obtain the solution of (1) in exactly the same fashion as if we were dealing with a scalar equation. 12. Inhomogeneous Equation—Variable Coefficients. Consider the case where A is time-dependent. We wish to solve & = At + fo (0) =¢ qd) Let us use the Lagrange variation of parameters technique and attempt to find a solution of the form z = Xy where X = X(t) is the solution of the equation dX/dt = A()X, X(0) = I. Substituting in (1), we obtain the equation X'y + Xy! = AWXy + Xy = AOXY +410 @) Hence Xy = 10 @) whence y= XM) (4) or yaet ff x-Mafe de (8) Consequently, 2= Xe + f{ XOX) ds 6) a generalization of the result of (11.4).m4 Introduction to Matrix Analysis 18, Inhomogeneous Equation—Adjoint Equation. Let us now pursue a different approach, one of great importance in the general theory of linear functional equations. Take Y(f) to be a variable matrix, as yet unspecified, and integrate between 0 and ¢ the equation VQ # = YOA@s + YOKO w ‘The result ia, upon integrating by parts, YOa - YOe~ [Fate de = f' YeOACaC de + f “Y(@fle) ds (2) Without loss of generality, we can take ¢ = 0, since we can obtain the solution of (12.1) by adding to the solution of this special case the vector X()c. Since our aim is to solve for z(¢), suppose that we make the most convenient assumptions that we can, namely, that x “Y(@)A(t) OS8 O whenever the components of ¢ are non-negative and the components of f(t) are non-negative for t > 0? Referring to (11.4), we see that a sufficient condition is that all ele- ments of e4' be non-negative for ¢ > 0, and it is easily seen that this con- dition is necessary as well. It is rather surprising that there is a very simple criterion for this condition. Theorem 3. A necessary and sufficient condition that all elements of et be non-negative for t > 0 is that ag >O0 tei (2) (3) Proof. Since ote T+ Att: it is clear that the condition in (2) is necessary for the result to be true for small ¢, To establish the sufficiency, let us show that ay > 0, =, implies that the elements of e4* are positive for all 4 It is clear that y > 0 implies that e4* has positive elements for small !. Since eat = (eAtinyn @ for any integer n, the fact that the product of two positive matrices is Positive yields the requisite positivity. Sinee the elements of e4* are continuous functions of the ay, we see that the positivity of the elements of e4‘ for a; > 0, 7 ¥ j, implies the non- negativity of the elements of e4! for ay > 0, 1 ¥ j. Another more direct proof proceeds as follows. Let ¢, be a scalar so that all the elements of A + c:J are non-negative. Then, clearly, all the elements of e+ are non-negative. Also, the elements of e~“!* are non-negative since exponentials are always non-negative. Since At we g(Ataimalt = etAteDtgalt (5) observing that A + c,J and —c,J commute, we have the desired non-negativity.Matrices and Differential Equations 17 EXERCISES 1. Prove the foregoing result by using the system of differential equations x e. dw x(0)—=c §=1,2..., i 2 Show that the result a:;(0) > 0, ¢ sj, is sufficient to ensure that the solution of the corresponding equation with variable coefficients is non-negative if the initial conditions are non-negative, 16. Polya’s Functional Equation. We have seen that Y(t) = e4* satisfies the functional equation Ye+H)=Y(s)¥(t) -x . ¥(ke8)8 (6) where N is determined by the condition that (N + 1)8 =t. Since, from (1) for 8 > 0, ¥(k8) = ¥((k — 1)8)¥(8) = Y(0)* 7)1%8 Introduction to Matrix Analysis we see that » ‘ , f ¥(@) de = lim » ¥(8)5 (8) We also have , (y@) - 7] = - maja Y vans = Y((N + 1)8) —T (9) which leads to the result x lim (29 =4) Q voos)| =¥)-1 (10) We have assumed that ¥(0) = J. Hence, for small t, fc Yo) ds =r + 0) (1) and thus f,' ¥(e) ds is nonsingular for small t. It follows that for fixed x small t and sufficiently small 4, the matrix |) Y(&6)6 is nonsingular and io lim (3 ¥(K8)8)* = ( fe Y@ ds)" (12) Hence (10) yields the fact that [¥(5) — I]/8 has a limit as 8—> 0. Call this limit A. Then A = lim (=) =(%0-n [f ¥(s) asl" (13) for small {, From this we obtain A fk Yds = YQ —-1 (14) and, finally, Y() =AY Y@)=T (15) for small 4, Since Y(!) = e4 for small ¢, the functional equation yields ¥(Q) = e4* for alll t in (0,to). EXERCISES 1. What is wrong with the following continuation of (10): From (10) we have tim [78 — 2] ja [3 vans] = eo -1 a [PO] fi roa = v0 -1Matrices and Differential Equations 179 2 Let ¥ and Z denote the solutions of Peaoy your Baza 20-1 Then the solution of x = AX +XBQ XO) =C is given by X = YCZ. 8. Write the linear equation u” + p(t)u’ + 9(¢)u = Oin the form ED -Lto ro) [2] Use the preceding result to show that the linear equation whose general solution is 1 = ant + ayuauy + anva?, Where u; and usare a aot of linearly independent solutions of the second-order equation, ia ul” + 3pQu” + 2pX(e) + p'O + daw’ + pa) + 2q'Olu = 0 17. The Equation dX/di = AX + XB. Let us now demonstrate Theorem 5. Theorem 8. The solution of 2 ax + XB X00) =¢ @ ie gi ‘eaten by X = etCem (2) ‘The proof follows immediately by direct verification. The result, although simple (and a special case of Exercise 2 of Sec. 16), plays an important role in various parts of mathematical physics. EXERCISES 1. Obtain the solution given above by looking for a solution of the form X = eM, 2 Let ¥() be a square root of X(t). Show that (F ax ()'+°(G)-& 18, The Equation AX + XB = C, Using the foregoing result, we can establish Theorem 6. Theorem 6. If the expression X= fS encom at w exists for all C, it represents the unique solution of AX+XB=C @)180 Introduction to Matrix Analysis Proof. Consider the equation - az+ zB 20) =C () Let us integrate both sides between 0 and ~, under the assumption that lim Z() = 0. The result is be -C=A(f, Zs) + (J, Zas)B @ We see then that = fo Zae= — fo acer ae satisfies (2). In Chap. 12, we shall discuss in some detail the existence of the inte- gral in (1) and the solutions of (2). The uniqueness of solution follows from the linearity of the equation in (2). Considering this matrix equation as NV? linear equations for the elements zy of X, we see that the existence of a solution for all implies that the determinant of the coefficients of the zy is nonzero. Hence, there is a unique solution, MISCELLANEOUS EXERCISES 1, Consider the sequence of matrices {Xn} defined by Xo = Xa(Ql—AX,) Xo B Under what conditions on A and B does this sequence converge, and, if eo, to what does it converge? 2. If AB — BA = 1, and cand cy are scalars, then Ate Dm eridersBenies!t See R. A. Sack,' where the general problem of determining the expansion of f(A + B) = f(A) + +++, for the case where AB = BA, isdiscussed. See also W. 0. Kermack and W. H. McCrea! and R. Kubo," where related expansions in terms of commutators appear. 8. What is the analogue of the result in Exercise 2 if AB — BA = cA? 4 If A is positive definite, B(2) is positive definite for # > 0, and |4| < |B(é)| for all 1>0, then [4] < | f . Bea at| for any non-negative funetion g(f) such that fo oan. 1R. A. Sack, Taylor's Theorem for Shift Operators, Phil. Mag., vol. 3, pp. 497-503, 1958. *W. O. Kermack and W. H. McCrea, Proc. Edinburgh Math. Soc,, vol. 2 (220). *R. Kubo, J. Chem. Phys., vol. 20, p. 770, 1952.Matrices and Differential Equations 181 5. If A is symmetric positive definite and A < J, then the sequence of matrices defined by the recurrence relation Xu = Xu + 3G(A — Xn) Xe =O converges to the positive definite square root of A.t 6. The problem of determining the values of } satisfying the determinantal equation | — NP, — MF, — + + + — MFil = 0, where the Fi are square matrices of order N, is equivalent to determining the characteristic values of FL Fs Fy Aw] I 0 0 of 0 oro See K. G, Guderley, On Nonlinear Kigenvalue Problems for Matrices, J. Ind. and Apol. Math,, vol. 6, pp. 335-353, 1958, 7. Let H be a function of a parameter ¢, H = H(). Then aH non OH oan as, a ot ae 8. Show that taereny = = ft ern etd 9. 1 (4,HO} 10. Show that }H /at, then (A,f(H)| = af(H)/at. Be 4 = B + {A,B} + (A,lA,BI/21 + +++ 11, Write ef = ettet®, Z = x 2ytr, B= y nZ,tv-!. Show that net net f eZ) (ter? dr = A + eMBerta 12. Obtain recurrence relations for the Z, in this fashion, for example Z: = A + B, 2s = (A,B)/2, Zs = ((4,1A,B)] + (1A,B)BII/12, . . . (Baker Campbell-Hasdorff for- mula). For the results of Exercises 7-12 and many further results, see Wilcox! and Eichler.* 18. Show that e(4*#) = lim (e4te')". The result is given in Trotter,* and has interesting applications to the study of the Schrédinger equation and Feynman integrals. See Faris. +. Visser, Notes on Linear Operators, Proc, Acad. Sci. Amsterdam, vol. 40, pp. 270-272, 1937. +R. M. Wilcox, Exponential Operators and Parameter Differentiation in Quantum Physics, J. Math. Phys., vol. 8, pp. 962-982, 1967. *M. Eichler, A New Proof of the Baker-Campbell-Hausdorff Formula, J. Math. Soc. Japan, vol. 20, pp. 23-25, 1968. 1H. F. Trotter, On the Products of Semigroups of Operators, Proc. Am. Math. Soc., pp. 645-551, 1959. “W. G. Faris, The Trotter Product Formula for Perturbations of Semibounded Operators, Bull. Am, Math. See., vol. 73, pp. 211-218, 1967.182 Introduction to Matrix Analysis 14. Let z = y + Az where A depends on a parameter ¢. Write z = y + Ry, defining the resolvent matrix. Show that R’ = (I + R)A'(I + R). 15, Assume that A(!) has distinct characteristic values for 0 «. 19. Consider the case where the interval is {- »,«]. Use Fourier integrals and the Parseval-Plancherel theorem to obtain an expression for min Qn(a) in terms of @ orthogonal polynomials. (The foregoing is connected with differential approzima- tion see Bellman and Kalaba! and Lew.*) 20. Consider the matrix equation X — UXV = W. Show that a formal solution isX = > U*WV+, When does theseries converge and actually represent a solution? Xo See R. A. Smith, Matrix Equation XA + BX = C, SIAM J. Appl. Math,, vol. 16, pp. 198-201, 1963. ‘1. Consider the equation X = C + «(AX + XB), ¢ a scalar parameter. Write X= C+) ees(AsB). Show that oe = Agnes + oneiB, m1, with oo = 6 and ms that on = ANC + (7) AmtCB + + +OB 22, Introduce a position operator P with the property that when it operates on monomial consisting of powers of A and B in any order with C somewhere, it shifts all powers of A in front of C and all powers of B after C. ‘Thus P(ASBY «+» AtBuCAsaBoor +» AtwBbv) = ATCB% Further, define P to be additive, P(m(A,B) + m4(A,B)) = P(m(A,B)) + P(m(4,B)) where m, and m, are monomials of the foregoing types. Show that (A,B) = P(A + B)*C). 98, Hence, show that X, as defined by Exercise 21, may be written X = P({I — (A + BIC). 24, Similarly show that if X = EF + «(AX +XB+CXD), then X = P(t - (4 +B + CP)'£), with P suitably defined. 'R, Bellman and R. Kalaba, Quasilinearization and Nonlinear Boundary-value Problems, American Elsevier Publishing Company, Inc., New York, 1965. +A, Lew, Some Resulte in Differential A pprozimation, University of Southern Cali- fornia Press, Loa Angeles, USCEE-314, 1968.184 Introduction to Matrix Analysis 25. Show that if all the characteristic roots of A are less than one in absolute value, the solution of A*XA — X = —Q is given by X= Qe f (At — PDQ = ates where [denotes integration round the circle (s| = 1. Alternately, x= Gn [" @nrQe de where @ = A — Iei#, See R. A. Smith.! Bibliography and Discussion G1. The equilibrium or stability theory of differential systems was started independently, and almost simultaneously, by Poincaré and Lyapunov. Further references and discussion will be found in Chap. 13 and in the book R. Bellman, Stability Theory of Differential Equations, Dover Publ cations, New York, 1969. Some interesting transformations of matrix differential equations may be found in the following papers: J. H. Barrett, Matrix Systems of Second Order Differential Systems, Portugaticae Math,, vol. 14, pp. 79-89, 1955. J. H. Barrett, A Prufer Transformation for Matrix Differential Equations, Proc. Am, Math. Soc., vol. 8, pp. 510-518, 1957. 2. The matrix exponential lies at the very heart of all advanced work in the field of linear functional equations, Suitably generalized to arbi- trary operators, it is the foundation stone of the theory of semigroups, see E. Hille, Functional Analysis and Semi-groups, Am, Math, Soc. Pub., 1942, and its revised form +R. A. Smith, Matrix Caleulations for Lyapunov Quadratic Forms, J. Dif. Bg. vol. 2, pp. 208-217, 1966. R. A. Smith, Bounds for Quadratic Lyapunov Functions, J. Math. Anal. Appl., vol. 12, pp. 425-435, 1966, 14. J. Levin, On the Matrix Riceati Equation, Proc. Am. Math. Soc., vol. 10, pp. 519-524, 1959.Matrices and Differential Equations 185 E, Hille and R, Phillips, Functional Analysis and Semi-groups, Am. Math, Soc. Collog. Pub., vol. 31, 1958, In a second direction, the problem of expressing e4'e® in the form e%, where 4 and B do not commute, has important ramifications not only in the theory of Lie groups and algebras, but also in modern quantum field theory. The interested reader may consult W. Magnus, Algebraic Aspects of the Theory of Systems of Linear Differential Equations, Research Repl, BR-3, New York University, Institute of Mathematical Sciences, June, 1953; also Comm, Pure Appl. Math., vol. 7, no. 4, 1954. H, F. Baker, On the Integration of Linear Differential Equations, Proc. London Math. Soc., vol. 34, pp. 347-360, 1902; vol. 35, pp. 333-374, 1903; second series, vol. 2, pp. 293-296, 1904. H. F. Baker, Alternants and Continuous Groups, Proc. London Math, Soc., second series, vol. 3, pp. 24-47, 1904. H. B. Keller and J, B. Keller, On Systems of Linear Ordinary Dif- ferential Equations, Research Rept. EM-33, New York University, Washington Square College, Mathematics Research Group, 1957. F. Hausdorff, Die symbolische exponential Formel in der Gruppen- theorie, Saechsischen Akademie der Wissenschaften zu Leipzig, Math.-phys. Klasse, vol. 58, pp. 19-48, 1906. K. Goldberg, The Formal Power Series for log (e*e’), Duke Math. J., vol. 23, pp. 13-21, 1956.7 In another, and related direction, we mest the problem of expressing the solution of a linear system of the form dX/dt = A()X, X() = I, in the form of an “exponential.” This question leads to the study of product integrals”; see L, Schlesinger, Neue Grundlagen far einen Infinitesimalkalkul der Matrizen, Math, Z., vol. 33, pp. 33-61, 1931. L, Schlesinger, Weitere Beitrage zum Infinitesimalkalkul der Matri- zen, Math. Z., vol. 35, pp. 485-501, 1932, G. Rasch, Zur Theorie und Anwendung des Produktintegrals, J. reine angew, Math., vol. 171, pp. 65-119, 1934. B, W. Helton, Integral Equations and Product Integrals, Pacific J. Math., vol. 16, pp. 297-322, 1968. See also Kuo-Teai Chen, Integration of Paths, Geometric Invariants and a Gen- eralized Baker-Hausdorff Formula, Ann. Math., vol. 65, pp. 163-178, 1957.186 Introduction to Matrix Analysis For more recent developments, see W. Magnus, On the Exponential Solution of Differential Equations for a Linear Operator, Comm. Pure Appl. Math., vol. 7, pp. 649-673, 1954. For an understanding of the physical interest in this problem, see R. P. Feynman, An Operator Calculus Having Applications in Quantum Electrodynamics, Phys. Rev., vol. 84, pp. 108-128, 1951. Finally, let us mention that a great deal of effort has been devoted to the generalization of the foregoing results and methods to infinite sys- tems of linear differential equations with constant coefficients. Systems of this type arise in natural fashion in probability theory, particularly in its applications to physics and biology in the study of “birth-and- death” processes. These matters will be discussed again in Chap. 16, See N. Arley and A. Borchsenius, On the Theory of Infinite Systems of Differential Equations and Their Applications to the Theory of Stochastic Processes and the Perturbation Theory of Quantum Mechanics, Acta Math., vol. 76, pp. 261-322, 1945. R. Bellman, The Boundedness of Solutions of Infinite Systems of Linear Differential Equations, Duke Math J., vol. 14, pp. 695-706, 1947. 8. Further results concerning norms of matrices, and further refer- ences, may be found in A. 8. Householder, The Approximate Solution of Matrix Problems, J. Assoc. Comp. Mach., vol. 5, pp. 205~243, 1958. J. von Neumann and H. Goldstine, Numerical Inverting of Matrices of High Order, Bull. Am, Math, Soc., vol. 53, pp. 1021-1099, 1947. A. Ostrowski, Uber Normen von Matrizen, Math. Z., Bd. 63, pp. 2-18, 1955. K, Fan and A. J. Hoffman, Some Metric Inequalities in the Space of Matrices, Proc. Am. Math. Soc., vol. 6, pp. 111-116, 1955. T. E. Essterfield, Matrix Norms and Vector Measures, Duke Math. J., vol. 24, pp. 663-671, 1957. §5. The reader who is familiar with the theory of Lebesgue integration will see that the result of Theorem 1 can be obtained under much weaker conditions on A(t). However, since we have no need for the stronger result, we have not stated it.Matrices and Differential Equations 18? G14. Variation of the characteristic values and characteristic roote of A as the dimension changes can be discussed using the techniques pre- sented in R. Bellman and 8. Lehman, Functional Equations in the Theory of Dynamic Programming-X: Resolvents, Characteristic Values and Functions, Duke Math, J., 1960. 15. Non-negativity results of this type play an important role in probability theory and mathematical economics, where the physical model makes the result intuitively clear. This point will be discussed again in Chap. 16. They also play a role in the study of various classes of nonlinear equations; see R. Bellman, Functional Equations i gramming—V: Positivity and Qi U.S., vol. 41, pp. 743-748, 1955. R. Kalaba, On Nonlinear Differential Equations, the Maximum Operation, and Monotone Convergence, Ph.D. Thesis, New York University, February, 1958. the Theory of Dynamic Pro- linearity, Proc. Natl. Acad. Sci. For an extensive discussion of the positivity of operators, see Chap. 5 of E, F. Beckenbach and R. Bellman, Inequalities, Springer, 1961. The first proof is due to 8. Karlin, and the second proof to O. Taussky. The result was first presented in R. Bellman, I. Glicksberg, and ©. Gross, On Some Variational Problems Occurring in the Theory of Dynamic Programming, Rend. Cire. Palermo, Serie II, pp. 1-35, 1954. §18. The result and proof follow G. Polya, Uber die Funktionalgleichung der Exponentialfunktion in Matrizkalkul, Siteber. Akad. Berlin, pp. 96-99, 1928. The nature of the solutions of ¥(s + ¢) = ¥(s)¥(®) without the normal- izing condition ¥(0) = / is also of interest. See Shaffer.* §17. We are not aware of the origin of this result which may be found in the papers of many authors, For an extensive discussion of this and related equations, see J. A, Lappo-Danilevsky, Mémoires sur la théorie des systtmes des equations différentielles linéaires, Chelsea Publishing Co., New York, 1953, 10, Y. Shaffer, On Singular Solutions to Polya’s Functional Equation, IEEE, vol. AC-13, pp. 135-138, 1968,188 Introduction to Matrix Analysis The fact that the solution of the equation dX/dt = AX + XB has the indicated form becomes quite natural when one examines the way in which it arises in quantum mechanics; cf. D. ter Haar, Elements of Statistical Mechanics, Rinehart & Company, Inc., New York, p. 149, 1954. Y. Nambu, Progr. Theoret. Phys. (Kyoto), vol. 4, p. 331, 1949. H. Primas and H. Gunthard, Eine Methode zur direkten Berech- nung. . . , Helv. Phys, Acta, vol. 31, pp. 413-434, 1958. $18. For a discussion of the operator T' defined by TX = AX — XB, eee M, Rosenbloom, On the Operator Equation BX — XA = Q, Duke Math. J., vol. 23, pp. 263-269, 1956, For a generalization to the operator $ defined by SX = D AxB, see G. Lumer and M. Rosenbloom, Linear Operator Equations, Proc. Am. Math, Soc., vol. 10, pp. 32-41, 1959. Exercises 2 and 3 are taken from R. Bellman, On the Linear Differential Equation Whose Solutions Are the Products of Solutions of Two Given Linear Differential Equations, Bol. unione mat. Italiana, ser. III, anno XII, pp. 12-15, 1957. The intimate connection between matrix theory and the theory of the behavior of the solutions of linear differential equations and thus with the study of linear oscillatory systems leads to the beautiful theory of Gantmacher and Krein, V. Gantmacher and M. Krein, Sur les matrices complétement non négatives et oscillatoires, Comp. math., vol. 4, pp. 445-476, 1937. As pointed out at the end of the remarks on Chap. 16, these results have important applications in probability theory. For a treatment of the equation dz dz AGE + BG +C2=f where A, B, and C are symmetric by variational techniques, under suitable hypotheses concerning A, B, and C, seeMatrices and Differential Equations 189 R. J. Duffin, A Minimax Theory for Overdamped Networks, J. Rat. Mech. and Analysis, vol. 4, pp. 221-233, 1955. See also ‘A. M. Ostrowski, On Lancaster's Decomposition of a Matrix Differ- ential Operator, Arch, Rat, Mech, Anal., vol. 8, pp. 238-241, 1961. H. Langer, Uber Lancaster's Zerlegung von Matrizen-Scharen, Arch. Rat, Mech, Anal,, vol, 29, pp. 75-80, 1968. The study of linear circuits leads in another way to an interesting domain of matrix theory, namely, the study of when a given matrix of rational functions can be considered to be the open-circuit impedance matrix of an n-port network. The concept of a positive real matrix enters here in a fundamental fashion. For work on this subject, and references, see L, Weinberg and P. Slepian, Positive Real Matrices, Hughes Research Laboratories, Culver City, Calif., 1958.11 Explicit Solutions and Canonical Forms 1, Introduction. Although the results obtained in Chap. 10 are of great elegance, it must be confessed that they do not satisfactorily resolve the problem of obtaining explicit scalar solutions. In this chapter, we shall pursue this question and show that it leads us to the question of determining the characteristic roots and vectors of matrices which are not necessarily symmetric. As in the case of sym- metric matrices, this leads us to the study of various canonical forms for matrices. 2. Euler’s Method. In Chap. 10, we demonstrated the fact that the vector-matrix equation daz 20) i c @ possessed a unique solution which could be represented in the form z= ete @) Here, we traverse another road. Following the method of Euler, let us begin by looking for special solutions of the form x = eMc!, where ) is a sealar and c!a vector, ignoring for the moment the initial condition 2(0) = c. Substituting, we see that \ and c! are bound by the relation det = Act (8) Since c! is to be nontrivial, 4 must be a root of the characteristic equation |4-al| =0 (4) and c! must be an associated characteristic vector. From this quite different direction then, we are led to the basic prob- lem treated in the first part of the book—the determination of charac- teristic roots and vectors. Here, however, the analysis is both more complicated and less complete due to the fact that the matrices we encounter are not necessarily symmetric. 190Explicit Solutions and Canonical Forms 191 EXERCISE 1. Find the characteristic roots and vectors of the matrix 8. Construction of a Solution. Let us now see if we can construct a solution of the initial value problem along the preceding lines. As before, denote the characteristic roots by \:, \2, . .. , Av. Since these will, in general, be complex, there is now no attached ordering, To simplify our initial discussion, let us assume initially that the characteristic roots are distinct, Let c', c?, . . . , c%, be a set of associ- ated characteristic vectors, also, in general, complex. To put together the required solution, we use the principle of super- position. Since ect is a solution for k = 1, 2,..., N, the linear combination x z= ) ayertct qd) is a solution of dz/dt = Az for any set of scalars a,. The question is now whether or not these scalars can be chosen so that 2(0) = c. This condition leads to the vector equation x c= ) act @ Ss This system has a unique solution provided that C is nonsingular, where Cae --- 0%) @ and, as before, this represents the matrix whose columns are the veo- tors of. 4. Nonsingularity of C. ‘There are several ways of establishing the fact that C is nonsingular. ‘The first method proceeds as follows. Suppose that |C| = 0. Then there exist a nontrivial set of scalars by, bs, . . . , by such that = back (ty192 Introduction to Matrix Analysis It follows that : = ¥ bite 2: 2 2 @ is a solution of dz/dt = Az satisfying the initial condition 7(0) = 0. ‘The uniqueness theorem asserts that actually 2 = 0 for all ¢. We must then show that the relation x 0= Y byte 3 2 ® cannot hold for all ¢if the {bs} are a nontrivial set of scalars, none of the o* is trivial, and the A; are all distinct. Without loss of generality, assume that by 0. Divide through by " in (8) and differentiate the resultant expression with respect to ¢. The new expression has the form N D, bate — Arde" = 0 (4) A Since the 4 have been assumed distinct, the quantities d+ — ds, k = 2,3,...,N, are all distinct. We may now appeal to an induc- tive argument, or merely repeat the above procedure. It is clear that in this way we eventually reach a relation of the form we (dw — ds) — As) + Qn — Aware =O 6) Since cis nontrivial, we must have by = 0, a contradiction. 5, Second Method. Beginning with the relation in (4.1), multiply both sides by A. ‘The result is 0= bedsc* qd) 2 Repeating this operation (N’ — 1) times, the result is the system of linear equations x o= FY bawe r=0,1,2,...,N-1 (2) ro Considering only the ith components of the ct, the quantities ¢, we obtain the scalar equations 0 x DY bavee = 0,1,2,...,N—1 @) &Explicit Solutions and Canonical Forms 193 Since not all of the bs are zero, and since all of the c/ cannot be zero as iand k take the values 1, 2, . . . , N, we see that (3) possesses a non- trivial solution for some i. It follows then that the determinant of the coefficients [ae] k=1,2,...,N,r=0,1,2,...,N—-1 (4) must be zero, This, however, is not true, as we shall see in a moment. 6. The Vandermonde Determinant. The determinant 1 1 | Ar Me dw MP? Pel = : a) Dod get yt is a very famous one, bearing the name, Vandermonde determinant. We wish to show that [Ai] # 0 if \; # dsfori #j. The simplest way to do this is to evaluate the determinant. Regard |),'| as a polynomial of degree N — lind; Asa polynomial of degree N — 1, |As'| has the roots Ar = An, Ar = Aa, . . » , An = Aw, Since the determinant is zero whenever two columns are equal, It follows that Par] = Qa — Aa)Qa — Aa) + Ar — Aw)aQars, «. - Aw) (2) where g is a function depending only upon dz, As, .. . , Aw. Similarly, viewed as ® polynomial of degree N — 1 in Az, |As*| must possess tie factor (A: — A1)(A2 — Aa) + = (Ap — Aw)- Continuing in this way, we see that, Pel = Th Qs —4Qnds . . - Aw) @) isicigw where ¢ is a polynomial in the \,. Comparing, however, the degrees in the d; of the two sides, we see that ¢ must be a constant. Examining the coefficients of \;A2" - + + Av-1¥-! on both sides, we see that ¢ = 1. From this explicit representation, we see that |As‘| x 0 if \, x A, for inj. EXERCISE 4, Using similar reasoning, evaluate the Cauchy determinant i . lace] teen ae cm194 Introduction to Matrix Analysis 7. Explicit Solution of Linear Differential Equations—Diagonal Matrices. Let us now consider another approach, quite different from that given in the preceding sections. Given the equation 7 =Az 20) =e a) make a change of variable, z = Ty, where T is a constant nonsingular matrix to be chosen ina moment. The equation for y is then Yoruary (0) =T« @) What choice of T will simplify this equation to the point where the solution can be immediately obtained? Suppose that we can find a matrix T such that a TAT = . )) AN, a diagonal matrix. If this can be done, the equation in (2) decomposes into N independent equations of the form We yO) =e F-18000, “ having the elementary solutions y; = e“tc;. Once y has been determined, x is readily determined. 8. Diagonalization of a Matrix, Let us now discuss the possibility of the diagonalization of A, a problem of great interest and difficulty. As we know from the first part of the book, if A is symmetric, real or com- plex, a matrix T possessing the required property can be found, with T-'= T’. As we shall see, there are a number of other important classes of matrices which can be diagonalized, and, what is more important, there are other types of canonical representations which are equally useful. Consider, however, the general case. To begin with, we know that the {u:} must be the same set as the {A,} since the characteristic roots of T-1AT are the same as those of A. It follows then, as in previous chapters, that the columns of T must be characteristic vectors of A.Explicit Solutions and Canonical Forms 195 Conversely, if the 4; are distinct and T is the matrix formed using the associated characteristic vectors as columns, it follows that dw, We have thus established the following important result. Theorem 1. If the characteristic roots of A are distinct, there existe a matrix T such that AL TAT = a (2) aw, Aa we shalll see, the assumption that the roots are distinct is now quite an important one. In the general case, no such simple representation holds, as we shall see in Sec. 10. EXERCISES 1. Show that the Cayley-Hamilton theorem is valid for matrices with distinct characteristic roots. 2. Show that the assumption concerning N distinct characteristic roots may be replaced by one requiring N linearly independent characteristic vectors. 9, Connection between Approaches. As we have seen, one method of solution of the linear equation oe =Az 20) =e a leads to the expression z = ec, while on the other hand, a method based upon characteristic roots and vectors produces scalar exponentials. To obtain the connection between these approaches (still under the196 Tntroduction to Matrix Analysis assumption that A, » 24), which must exist because of uniqueness of solu- tion, in the equation ae =AX X@)=1 @ make the change of variable X = TY, where T is as in Sec. 8. Then Y satisfies the equation Le =TATY Y(0)=T @) or M 0 Me ay . an . Y YO-=r ® 0 Aw. It follows that om 0 om Ys qT ©) 0 ou whence on 0 ow X=et=aT . qT (6) 0 on Let us again note that this representation has been established under the assumption that the characteristic roots of A are distinct. EXERCISES 4, Establish the representation in (6) directly from the exponential series and the representation for A.Explicit Solutions and Canonical Forms 197 4. Une (6) to show that |e4| = et, 8. Use the method of continuity to show that this result is valid for any square matrix A. 10. Multiple Characteristic Roots. Let us now turn to a discussion of the case where A has multiple roots. In order to appreciate some of the difficulties that we may expect to encounter, let us begin by showing that it may not always be possible to obtain a canonical representation such as that given in (8.2). Theorem 2. There exist matrices which cannot be reduced to a diagonal form by means of a nonsingular matrix, as in (8.2). Equivalently, there exist matrices of dimension N which do not possess N linearly independent characteristic vectors, Proof. Consider the particular matrix 11 a={) 4] 0) If there exists a T such that a 0 Lar = Tar | i @) we know that the columns of 7 must be characteristic vectors of A. Let us then determine the characteristic roots and vectors. ‘The characteristic equation is pity 0 1-) Hence 1 is a double root. The characteristic vectors are determined by the equations =(1-a)?=0 (3) ttt = ty By = @ It follows that z, = 0, with x, arbitrary. Consequently, ali character- istic vectors are scalar multiples of en [3] 6 This means that T must be a singular matrix. Observe then the surprising fact that an arbitrary matrix need not possess a full complement of characteristic vectors. This makes us more appreciative of what a strong restriction on a matrix symmetry is. The fact that diagonalization may not be always available greatly complicates the study of general matrices—and, in return, adds equally to their interest. As we shall see below, there are several methods we198 Introduction to Matrix Analysis can use, based upon the use of alternate canonical forms and approxi- mation theorems, to bypass some of these hurdles. EXERCISES 41. Why can't we use a continuity argument to deduce & diagonal representation for general matrices from the result for matrices with distinct characteristic root? 4. Derive the solution of the differential equation Bent n0)=0 on nO=% 1. 7 [o i] cannot be reduced to diagonal form. 11. Jordan Canonical Form. Let us now discuss a canonical form for an arbitrary square matrix which is useful in the treatment of a number of problems. Since its proof is quite detailed and is readily available in a number of texts, and furthermore since its use can always be avoided by one means or another (at least in our encounters), we shall merely state the result without a proof. Theorem 8. Let us denote by Lx(d) a k X k matrix of the form and thus show that the matrix a 10 0 orl 0 L(d) = . (Q) _. x 1 oo rz where L,(\) =». There exists a matrix T such that Tn,0) Ln,(d2) TAT = - @ Lr,Q)Explicit Solutions and Canonical Forms 199 with ki + kat +++ +h, =N. The dare the characteristic roots of A, not necessarily distinct. This representation is called the Jordan canonical form. For example, if A is of the third degree and has ); as a root of multi- plicity three, it may be reduced to one of the three forms » 0 0 s 0 0 Ai=|0 4 0 A:=|0 A 1 ook. oo, mo 10 A:=|0 A 1 0 0 AL EXERCISES @) 1, Using the Jordan canonical form, determine the form of e" for general A. 2. Use this result to obtain the necessary and sufficient condition that e/— 0 as to a, 8. Prove that the As, i = 1, 2, 3, are distinct in the sense that there exists no T' for which T-1A:7’ = Aj,fori sj. Give both an algebraic proof and one depending upon the solutions of the associated linear differential equations. 4. Show that Z(\) ~ AJ when raised to the kth power yields the null matrix. 12, Multiple Characteristic Roots. Using the Jordan canonical form, we possess a systematic method for obtaining the explicit analytic struc- ture of the solutions of linear differential equations with constant ooef- ficients. Thus, if the equation is reduced to au Mty nO)=a é a) GEN ¥3(0) = er we have ys = ees, and y; determined as the solution of W cry ter 1100) =e @ Using the integrating factor e~, this yields Sue") = 02 @) whence ya = cre’ + cate (4)200 Introduction to Matrix Analysis This is one way of seeing why terms of the form te, and generally terms of the form tte*, enter into the solution in the case of multiple roots, Let us now discuss an alternative approach. If A has distinct characteristic roots, \1, 2, . . . , Av, We may write N particular solutions of dy/dt = Ay in the form y = c(A)e“, 4 = dy, Az, +» , Aw, Where c(A) is @ polynomial function of A. This we can see in the following way. As we know, to obtain a solution of the equation, we set y = ec, where ¢ is a scalar, and determine A and c from the equation Ac=r or (A—A)e=0 O) Once \ has been determined from the characteristic equation JA -al] =0 we must determine a solution of the linear system in (5). Let by = [A — My @) Cre the a of the term ay — 4; in the determinantal expansion of |A — Al]. Then Gc=be t= 1,2, iN @ is a solution of (5), with the property that the c; are polynomials in 2. It may happen that this is a trivial solution in the sense that all the gare zero. In that case, we may try the alternative sets of solutions a=bs §=1,2..0,N ® fork = 2,3,...,N. Let us now show that one of these sets of solutions must be nontrivial if the characteristic roots of A are all distinct. In particular, let us show that for any particular characteristic root, 4 = Xi, not all the expressions by can be zero if A, is not 8 multiple root. Consider the characteristic equation a-ha tt aw My JO) =|4-Ml=| | =0 (9) au ana sts aww)Explicit Solutions and Canonical Forms 201 ‘Then, using the rule for differentiating a determinant, we have -~1 0 + 0 Qn an—h ++ aw fQ) = ani ane "<* dex =X an—- Gy -+° aw 0 -1-:: 0 al ani Sn °)) fe= * a-ha aw an ay — aay tet 0 O vee =i = bu — bn — ++ = byw (10) If, for \ = d,, we have bu(hi) = 0,1 = 1,2,..., N, then fu) = 0, which means that 2, is a multiple root, a contradiction. It follows that we have a solution of the desired form which we use in the following fashion. If A, and Az are two distinct characteristic roots, then a= eye = erat an is also a solution of the differential equation. The case where Xy is a multiple root, can be considered to be a limiting case of the situation where A; approaches 4. Taking the limit as \2—> x, we find, as a candidate for a solution, the expression de( [= Oy weed) | (12) We leave as an exercise for the reader the task of putting this on a rigorous foundation. Let us merely point out that there are two ways of establishing this result, direct verification, or as a consequence of general theorems concetning the continuous dependence of solutions upon the matrix A.202 Introduction to Matrix Analysis EXERCISE 1. How do we obtain solutionsif Ais. double root, using a direct algebraicapproach? 18, Semidiagonal or Triangular Form—Schur's Theorem. Let us now prove one of the most useful reduction theorems in matrix theory. Theorem 4. Given any matriz A, there exists a unitary matrix T such that Aba bw Ae T*AT = - (1) 0 dw. where, as the notation indicates, the elements below the main diagonal are zero. Proof. Let us proceed by induction, beginning with the 2 X 2 case. Let A. be a characteristic root of A and é an associated characteristic veetor normalized by the condition (c'c}) = 1. Let T be a matrix whose first column is c! and whose second column is chosen so that T is unitary. Then evaluating the expression T-'AT as the product T-(AT), we see that saga [es bs roar =[ be] @ The quantity 62: must equal A; since T-'AT has the same characteristic roots as A. Let us now show that we can use the reduction for Nth-order matrices to demonstrate the result for (N + 1)-dimensional matrices. As before, let c! be a normalized characteristic vector associated with A,, and let N other vectors a, a?, . . . , a%, be chosen so that the matrix 7), whose columns are c!,a!,a?,.. . ,a%,isunitary. Then, asfor N = 2, we have Ay bast + Bawa? 0 TAT, = : By (3) 0 where By isan N X N matrix. Since the characteristic equation of the right-hand side is Qi — A)By — Ml = 0 4)Explicit Solutions and Canonical Forms 203 it follows that the characteristic roots of By are As, Xa, . . . , Awsi, the remaining N characteristic roots of A. The inductive hypothesis asserts that there exists a unitary matrix Ty such that Ae ca Cw As Cw Tw"'ByTy = st (6) 0 Dw Let Ty. be the unitary (N + 1) X (N + 1) matrix formed as follows: 10:++: 0 0 Try = Tr 6) 0 Consider the expression (TiP vgs) A(T iT nga) = Trg (POAT) Pegs Drie Dawes Ae - yt @ 0 Dw ‘The matrix T; Tsu is thus the required unitary matrix which reduces A to semidiagonal form. EXERCISES 1. Show that if A has k simple roots, there exists a matrix T such that MO “+ 0 baw’ 0 om - 0 baw TAT = /Q ++ ve he Bates baw Dee 0 0 aw &. Using the semidiagonal form derived above, obtain the general solution of dz/dt = Az, 2(0) = ¢. 8. Determine a necessary and sufficient condition that no terms of the form te occur.204 Introduction to Matrix Analysis 4 Determine the general solution of 2” + Az = 0, where A is positive definite. 5. Use the triangular form to obtain necessary and sufficient. conditions that ets O ast ©, . 6. Consider the difference equation x(n + 1) = Az(n), n= 0, 1,..., where 2(0) = c. Show that the solution is given by 2(n) = Ave, 1. Find particular solutions by setting 2(n) = dc, and determine the set of possible and c. 8. Show that the general solution has the form N a(n) = 2 pe(n)r* roa where the pi(n) are vectors whose components are polynomials in n of degree at most, 9. What is the necessary and sufficient condition that every solution of 2(n + 1) = Az(n) approach zero as n—> ©? 10. Consider the difference equation 2((n + 1)) = 2(nd) + A Az(nd), n = 0, 1, 2(0) = c, where A is a positive scalar. As A—» 0, show that 2(nd) + z(0), the solution of the differential equation dz/dt = Az, z(0) = ¢, provided that nA — t. Give two proofs, one using the explicit form of the solution, and the other without this aid. 14. Normal Matrices. Real matrices which commute with their transposes, or, more generally, complex matrices which commute with their conjugate transposes, are called normal, Explicitly, we must have AA* = AtA @ The utility of this concept lies in the following theorem. Theorem 5. If A is normal, it can be reduced to diagonal form by a unitary matrix. Proof. As we know, we can find @ unitary transformation, T, which reduces A to semidiagonal form, Da bis baw Me baw A=T . T- = TST 2) 0 dw Then, since T is unitary, x _._ 0 T= TST) @)Explicit Solutions and Canonical Forms 205 Henee, AA* = TSS*T-! = A*A = TS*ST“! @) ‘Thus, we must have Ss* = s*s (6) Equating corresponding elements in the products, we see that by=0 1Si 0 where At = B, then A is normal (Putnam, Proc. Am. Math. Soc., vol. 8, pp. 768-768, 1957). 9. If A and B commute, then A* and B commute if A is normal. 15, An Approximation Theorem. Let us now state a very useful approximation theorem which can often be used to treat questions involv- ing general square matrices. Theorem 6, We can find a matrix T such that Ar bi tet bw de TOAT = a Q) 0 Aw with ) |bs| < ¢, where ¢ és any preassigned positive constant. & Proof. Let T be a matrix which reduces A to semidiagonal form. ‘Then the change of variable y = Tyz converts dy/dt = Ay into206 Introduction to Matrix Analysis Wo Net brats + ++ + bine tong t+ + baven de; ae = ven (2) It is now ensy to see how to choose 7’ so as to obtain the stated result. In (2) make the change of variable ae = rites! (3) Then the new variables 2;! satisfy the system of equations 1 > = Met ft ribaseet to = ers awew! 1 ee ie ) dex yen! With a suitable choice of r:, the sum of the absolute values of the off-diagonal terms can be made as small as possible. The last change of variable is equivalent to a transformation z = Hz', where E is non- singular. Hence we can take T to be T.E. At first sight, the above result may seem to contradict the result that @ general matrix possessing multiple characteristic roots may not be reducible to diagonal form. The point is that T depends upone. If we attempt to let ¢— 0, we find that either T approaches a singular matrix, or else possesses no limit. 16. Another Approximation Theorem. A further result which can similarly be used to reduce the proof of results involving general matrices to @ proof involving diagonal matrices is given in Theorem 7. Theorem 7. Given any matrit A, we can find a matriz B with distinct characteristic roots such that ||A — Bll < ¢, where ¢ is any preassigned positive quantity. Proof. Consider the matrix A + E, where E = (es) with the ; inde- pendent complex variables. If A + / has a multiple characteristic root, then f(A) = |A + E — AJ] and f(A) have a root in common. If f(A)Explicit Solutions and Canonical Forms 207 and f‘(A) have a root in common, the resultant of the two polynomials, R(E), @ polynomial in the ¢; must vanish. We wish to show that we can find a set of values of the efor which ) leu| is arbitrarily small, and for which R(EZ) #0. The negation of this is the statement that R(E) vanishes identically in the neighborhood of the origin in ey space. If this is true, the polynomial R(E) vanishes identically, which means that f() always has a multiple root. Consider, however, the following choice of the e:, ej mayb G esimay §=1,2...,N 0 The matrix A + E clearly does not have multiple roots. Hence, R(E) does not vanish identically, and we can find a matrix B = A + E with the desired properties. EXERCISE 1. Construct a proof of this result which depends only upon the fact that A can be reduced to triangular form. (The point of this is that the foregoing proof, although short and rigorous, makes use of the concept and properties of the resultant of two polynomials which are not as easy to establish rigorously as might be thought.) 17. The Cayley-Hamilton Theorem. Using the approximation theo- rem established in Sec. 16, we can finally establish in full generality the famous Cayley-Hamilton theorem. Theorem 8. Every matriz satisfies its characteristic equation. Proof. Let A + E, with ||E|| < «, be a matrix with distinct charac- teristic roots. Then, as we know, A + E satisfies its characteristic equa- tion. The characteristic polynomial of A + Bis f(A,£) = |A + E - XII, a polynomial in \ whose coefficients are polynomials in the elements of E, and thus continuous in the elements of E. Hence, lim f(A + E,E) = f(A) a) E40 Since f(A + E,#) = 0, we see that f(A) = 0. EXERCISES 1. Establish the Cayley-Hamilton theorem using the Jordan canonical form. 4. Establish the Cayley-Hamilton theorem under the assumption that A possesses a full set of characteristic vectors which are linearly independent. 8. Use the Cayley-Hamilton theorem to derive a representation for A~! as a poly- nomial in A, provided that A is nonsingular. 18. Alternate Proof of Cayley-Hamilton Theorem. Once we know what we wish to prove, it is much easier to devise a variety of short and208 Introduction to Matrix Analysis elegant proofs. Let us present one of purely algebraic nature, making no appeal to continuity. Consider the matrix inverse to A — MI, (A — M)~, for A not a charac- teristic value. We see that (A — 31) = BQY/fQ) q@) where the elements of B(A) are polynomials ind of degree N — 1 and 40) is the characteristic polynomial of A — AJ, the determinant |A — XJ]. Hence, (A — DBE) = fA)T 2) Write BOQ) = By, + MBya too ++ + Bo JO) = (<1 OME ew » where the B; are matrices independent of A. Equating coefficients, we obtain a sequence of relations — By. = (-1)"T ABy-1 — By-2 = cf @® ABy_1 — By-a = col and so on. We see that each B; is a polynomial in A with scalar coef- ficients, and hence that B.A = AB, for each i. It follows then that the identity in (2) is valid not only for all scalar quantities A, but also for all matrices which commute with A. In particular, it is valid for 4 = A. Substituting in (2), we see that J(A) = 0, the desired result. 19, Linear Equations with Periodic Coefficients. Let us now examine the problem of solving a linear differential equation with periodic coefficients, dz aw P@z 20) =e () where P(é) is a continuous function of # satisfying the condition Pt+1) = PO @) for all ¢. Surprisingly, the problem is one of extreme difficulty. Even the relatively simple scalar equation du Get @ +bc08 Du = 0 @ the Mathieu equation, poses major difficulties, and has essentially a theory of its own. In obtaining a canonical representation of the solution of (1), we areExplicit Solutions and Canonical Forms 209 led to discuss a problem of independent interest, namely the representa- tion of a nonsingular matrix as an exponential. As usual, it is convenient to discuss the matrix equation first. We shall prove Theorem 9. Theorem 9. The solution of dX GA POX XO =7 (4) where P(t) is periodic of period 1, may be written in the form XW = Me (5) where Q(t) is periodic of period 1. Proof. It is clear from the periodicity of P(t), that X(t + 1) is a solu- tion of the differential equation in (4) whenever X(J) is. Since X(1) is not singular, we see that X({ + 1)X(1)~! is a solution of (4) with the initial value 7. It follows from the uniqueness theorem that X@+ YX) = XO (6) Suppose that it were possible to write X(1) = ef @ Consider, then, the matrix Q() = X (We. We have QU + 1 = Xt Yee” = XU+ Ve = X(t 4+ I)X() et = X (He = QH) (8) This establishes (5). It remains to prove that (7) is valid. 20, A Nonsingular Matrix Is an Exponential. Let us now establish Theorem 10. Theorem 10. A nonsingular malriz is an exponential. Let A be nonsingular with distinct characteristic roots, so that A may be written dw.210 Introduction to Matrix Analysis Then A = e° where log Xs log Az log dw. which proves the result for this ease. If A has multiple characteristic roots, we can use the Jordan canonical form with equal effect. Let Tr,Q1) Lx,Q:) L1,Q1) It is sufficient to show that each of the matrices Zs,(X,) has a logarithm. For if B, is a logarithm of Ls,(\,), then By is a logarithm of A. We have noted above in Exercise 4 of See. 11 that (LkQ) — MIF = (5)Explicit Solutions and Canonical Forms 21 It follows then that the formal logarithm = log LL) = log [AT + LQ) - NJ = Flog } + ye ("nay - = Tog h + oS Ye" Qs) = A exists and is actually o logarithm in the sense that e® = A. EXERCISES 1. A nonsingular matrix has a kth root for k = 2,3, . @. The solution of dX/dt = Q()X, X(0) = J, can be put in the form eFePt + - + ervey where P= f.' Q() de, and Pa =f) Qu ds, with “1 Os ePeiQaasermt + fo! orm ‘The infinite product converges if ¢ is sufficiently emall (F, Fer, Bull. classe sci., Acad. roy. Belg., vol. 44, no. 5, pp. 818-829, 1958). 21. An Alternate Proof. Since we have not proved the Jordan representation, let us present an independent proof of an inductive nature. Assume that the result holds for V X N matrices and that we have already converted the nonsingular (N + 1)-dimensional matrix under consideration to triangular form, dua =[4" 2 ] w daa etPant dp where Ay is an N-dimensional matrix in triangular form and ay denotes an N-dimensional column vector. Let By be a logarithm of Ay, which is nonsingular if Ay,, is, and write Ben = [2* 3] @ where I = log Avy: and z is an unknown N-dimensional vector. It remains to show that 2 may be determined so that e?' = Ayy.. It is easy to see induetively that Brut = - (By) + tt seek ae @) fork=1,2,....212 Introduction to Matrix Analysis Hence an ele » Bi + BA FP Da/EL 0 Awa where the first two terms are taken to be 0 and 1. If Lis not a characteristic root of By, we have CO = Y By + Bet o> FID /A iM = Y Gxt = BD(Bw — U)-/ht = (eh = (By — IY 6) iS Hence _ [ell Tene leo = =F = Ul mae ® where ri, rs, . . . , tw are the characteristic roots of By. Since |C(D| is a continuous function of I for all J, as we see from the series, it follows that (6) derived under the assumption that 1 » rz holds for all I. If I x rs or any of the other values of the logarithms of the A, it is clear that [C()| #0. Ifl =r, then the factor (e* — e)/(rz — 1) reduces to e 0, Restricting ourselves to principal values of the logarithms, we see that C({) is never singular. Hence, we see that + may be determined so that C(I)z = aw. This completes the demonstration. 22. Some Interesting Transformations. The reduction of differential equations with variable matrix, s =A()z 20) =e oy to canonical form is a problem of some difficulty, Since ft lies more within the province of the theory of differential equations than of matrix theory, we shall not pursue it further here. Some interesting transformations, however, arise at the very beginning of the investigation. Set z = 7'y, where T is a function of ¢. Then y satisfies the equation Wo yar — ar/any @)Explicit Solutions and Canonical Forms 213 Write HA,T) = T-(AT — dT /dt) 3) Then we see from the derivation that f(A,T) satisfies the functional equation HA,ST) = {(7-(AT = dT/dt), 8) (4) EXERCISE 1, Use the scalar equation au HH + ax) TM $+ st asQu = 0 and the transformations u = or, (= (8), to derive similar classes of functional equations. Biorthogonality. In the first part of the book, dealing with sym- metric matrices, we observed that every N-dimensional vector could be written as a linear combination of N orthonormal characteristic vectors associated with the N characteristic roots of an N X N matrix A. Thus, if x z= 2 ait q@ the coefficients are determined very simply by means of the formula a; = (2,2') (2) If A is not symmetric, we face two difficulties in obtaining an analogue of this result. In the first place, A may not have a full quota of charac- teristic vectors, and in the second place, these need not be mutually orthogonal. To overcome the first difficulty, we need merely make an assumption that we consider only matrices that do possess N linearly independent characteristic vectors. To overcome the second difficulty, we shall use the adjoint matrix and the concept of biorthogonality, rather than orthogonality. Let y', y?, y, ..., y% be a set of linear independent: characteristic vectors associated with the characteristic roots Ai, Ax, . . . , Aw of A. Then ae) 3) is nonsingular and possesses the property of transforming A to diagonal form, T= (iy, .214 Introduction to Matrix Analysis AL 0 Aa TAT = (4) 0 Aw, From this, it follows that ‘| 0 i THAT) = 6) 0 Hence, A* also possesses the property of possessing N linearly inde- pendent characteristic vectors, furnished by the columns of (T*)-! and associated with the characteristic roots X;, Xz... , Xw. Call these characteristic vectors 2', 2%,.. . , 2%, respectively. To determine the coefficients in the representation w z= Y ay (6) mH we proceed as follows, From Ay! = dy! () Atel = Kai we obtain the further relations (Ay) = datz) (8) and thus (iA) = (vid) = Qa) Henee, Or Ayia) = 0 @) Henee, if A; 7 4,, we have (yz) = 0 (10) It is clear that (yz) »¢ 0, since #, being nontrivial, cannot be orthogonal to alll of the y'.Explicit Solutions and Canonical Forms 215 Thus, in the special case where the A, are all distinct, we can write, for the coefficients in (6), ay ‘Two sets of vectors {y'], {2'], satisfying the condition Wiz =0 t¥5 (12) are said to be biorthogonal, It is clear that we can normalize the vectors 80 as to obtain the additional condition (vz) = 1 (13) This technique for expanding z as a linear combination of the y‘ bears out a comment made earlier that often the properties of a matrix are most easily understood in terms of the properties of its transpose. EXERCISES 1. How does one treat the case of multiple characteristic roots? 2. What simplifications ensue if A is normal? 24. The Laplace Transform. Once we know that the solution of a linear differential equation with constant coefficients has the form y z=) op ® mm where p,(t) is a vector whose components are polynomials of degree at most NV — 1, we can use the Laplace transform to determine the solution without paying any attention to a number of details of rigor which usu- ally necessitate an involved preliminary discussion. The value of the Laplace transform lies in the fact that it can be used to transform a transcendental function into an algebraic function cifically, it transforms the simplest exponential function into the rational function, ‘The integral ae) =f” ergo at ®) is called the Laplace transform of f(t), provided that it exists. Since we shall be interested only in functions f() having the form given in (1), it is clear that 9(s) will always exist if Re (s) is sufficiently large.216 Introduction to Matrix Analysis The basic formula is 1 (3) [[rtea ; for Re (s) > Re (a). It follows that if we know that f(t) has the form FO = ce (4) then the equality fo" 90 dt = ex/te - a) (5) means that c, = cs and A; a Similarly, from the additivity of the integral, it follows that if f(t) is of the form given in (1), and if g(s) has the form 9(8) = ¢:/(8 — a1) + ¢2/(8 — a2) ® then we must have HO) = exe + eer a) 25. An Example. Consider the problem of solving the linear differen- tial equation ul! — Bu’ + 2u=0 my uo) =1 —-w(0) =0 Since the characteristic equation is M-BA+2=0 (2) with roots \ = 1 and = 2, we see that the solution of (1) will have the form u = ce" + cre! (3) where ¢, and cs will be determined by the initial conditions in (1). Thus we obtain the linear system ate=l Yr ter = @ yielding a=-l c=2 (6) so that the solution is u = Qe — et (6) In place of this procedure, let us use the Laplace transform. From (1), for Re (s) > 2, we obtain the relation fo wena —s fy wet dt +2 f° wen at = 0 i)Explicit Solutions and Canonical Forms 217 In order to evaluate the first two terms, we integrate by parts, obtaining [yo were dt = ue |" $0 f° ue at = -lts fue dt fo wen dt = wen]? +s f” wen at =O45 [5 wewat = meter fue at (8) upon referring to the relation for i we-* dt. Thus, (7) yields the equation (st = 36-42) f° vende = 5-3 (9) or ft ue" dt = 3 ea Fe The rational function on the right side of (10) has a partial fraction decomposition (10) (i) with 12 ay = tim 2(s- 3) _ 8-3) Ly (12) 1 =Be$2 ~e= Ife Henee, - i f ue" dt = — (13) whence u(t) has the form stated in (6). 26. Discussion. If, in place of the second-order system appearing in (5.1), we had studied a tenth-order system, the straightforward approach, based upon undetermined coefficients, would have required the solution of ten simultaneous linear equations in ten unknowns—a formidable problem. The use of the Laplace transform technique avoids this. Actually, this is only a small part of the reason why the Laplace trans- form plays a fundamental role in analysis, EXERCISES 1, Show that * inenstest dtm nl @-ariIntroduction to Matrix Analysis by parts (®) Differentiation with respect to s or a 2. Using this formule, show how to solve linear differential equations whose char- acteristic equations have multiple roots. In particular, solve ut — du tuned uO) =1 w'(0) =O 8. Use the Laplace transform solution of an associated differential equation to obtain a representation for the solution of the system of linear equations by = Mz, i = 1,2, .. , N, where the 2; are the unknowns, 4. How does one trent the corresponding problem when the z; and N are unknowns? 27. Matrix Case. Consider the equation dz Ga4t 10) = @ where A is a constant matrix. Taking transforms, we have ° dz a = - 1 , Borun f zen" dl (e2) whence, integrating by parts, (A = a0) f° ze" dt = ~e @) ‘Thus, the Laplace transform of the solution is given by i ze" dt = (81 — A)“c (4) In order to solve this, we can use the same type of partial fraction decomposition referred to above, namely y 2, Aus)" © wah (sI — A) ‘We shall assume, for the sake of simplicity, that the characteristic roots of A are distinct. Then Ay = lim (8 — &)(s7 — A) (6) =e EXERCISES 1. Find an explicit representation for Ay, using the Sylvester interpolation formula of Exercise 34, Miscellaneous Exercises, Chap. 6. 2. What happens if A has multiple characteristic roots?Explicit Solutions and Canonical Forms 219 ‘8. Use the Laplace transform of a linear differential equation of the form u‘™? + aw) $+. fay =0, uO) = ay wO) = ay...) uP) = ayy to obtain a solution of the linear system x D wa = or r=0,1,...,N—-1 oa} where As # A; for i #7. MISCELLANEOUS EXERCISES 1. If A = I +B, under what conditions does Atm 1 BEB oe? 2. By considering the equation 2”” + Az = 0, where A is positive definite, show that the characteristic roots of a positive definite matrix are positive, 3. Let A = HU bea representation of the complex matrix A where H is a positive definite Hermitian matrix and U is unitary. Show that A is normal if and only if H and U commute. 4. Prove that if A, B and AB are normal, then BA is normal (N. A, Wiegmann). 5. A necessary and sufficient condition that the product of two normal matrices be normal is that each commute with the H matrices of the other. By the H matrix of A, we mean the non-negative definite square root of AA* (N. A. Wiegmann). 6.'A* = I for some n if and only if the characteristic roots of A are roots of unity. 1. If Be = 0 for some &, then |A +B) = ||. 8. If|A + AJ! = Ohssrociprocal roots, A iseither orthogonal, or A? = I (Burgatti). 8 Let P, Q, R, X be matrices of the second order. Then every characteristic root of a solution X of PX? + QX + R = Disa root of |PM + Qd + Rl = 0 (Sylvester). 10. Prove that the result holds for NV-dimensional matrices and the equation AoX™ + AXP sb AQ AX + Aw 0 11. The scalar equation uw” + p(t)u’ + q(t) = Ois converted into the Riccati equa- tion v' +0? + ply + a) = 0 by means of the change of variable u = exp (fv dt) Show that there is a similar connection between the matrix Riccatian orve wh equation X’ = A(®) + BOX +XCWX and a second-order linear differential equation. 42. Every matrix A with |Aj = 1 can he written in the form A = BCB-'C-t (Shoda). 18. 1f |X] = ]¥| #0, then wo matrices C and D can be found sucht that X= cp“¥eD (0, Taussky). 14, Let 0 m0 -i 00 ie -ih Hy = 0 aya iy 0220 Introduction to Matrix Analysis Show that y im, flow [Aly +A] = im nd Jog 1 + 001 where 29(A) represents the continued fraction Bn(d) = Ant/D A Angt8/A ov > Hint: Start with the recurrence relation Hin (AjMiAa oe 6 pAwer) = AHWau(AGAgAg, oe 6 Awan) FAH WAa(APAa Ae eo Awet) for N > 3, where Hx(A) = |Hw + AI. 16. If A and B are both normal, then if cA + cB has the characteristic values cis + Crs for all ¢) and cz, we must have AB = BA (N. A, Wiegmann-H. Wielandi), 16. Write hem GAT BHNA.B Show that (A,B) = [4,B1/2 + 9(4,B) + h(4,B), [A,B] = AB — BA, where g(A,B) is a homogeneous polynomial of degree 3 in A and B satisfying the relation g(A,B) = 9(B,A), and A(A,B) is a sum of homogeneous polynomials beginning with one of degree 4, 17. Show that SBC) + $(A, B+ C + $(B,C)) = f(A,B) + f(A + B + f(A,8), C) 18. Using this result, show that [4,[B,€]] + (8,{C,A]] + [C,[4,B]] = 0 where [4,8] is the Jacobi bracket symbol. 19. A necessary and sufficient condition that lim B* = 0, for B real or complex, is that there exist a positive definite Hermitian matrix H such that H — B*HB is positive definite (P. Stein). 20. A is diagonalizable if and only if HAH is normal for some positive definite Hermitian matrix H (Mitchell) 21, Let r be a root of the quadratic equation r* + air +a: = 0, Write y = zo + air, yr = zor + 2[—a1r — as] = —asr1 + (Zo — ma). Introduce the matrix xf. 7 a 2-20 and write X ~ 2» + zr. If X ~ 2» + air and ¥ ~ yo + yir, to what does XY correspond? 22, Determine the characteristic roots of X. 28. Generalize the foregoing results to the case where r is a root of the Nth-order polynomial equation PY fay + +++ tay =0 24, There exists an orthogonal matrix B(!) such that y = B(0z transforms dy/dt = A(Oy into de/dt = Ay(Oz where A,(t) is semidiagonal (Diliberto). 26. B(0) can be chosen to be bounded and nonsingular and A,(() diagonal (Ditiberto). 6. Given one characteristic vector of an N X N matrix A, can this information be used to reduce the problem of determining all characteristic vectors of A to that of determining all characteristic vectors of an (N — 1) X (N ~ 1) matrix? 27. The matrix A is called circular if A* = A-!, Show that e** is circular if R is real.Explicit Solutions and Canonical Forms 221 28, The matrix A has the form 5 where § is real skew-symmetric if and only if Ais a real orthogonal matrix with [Al = 1 (Taber). 29. Every nonsingular matrix can be expressed as the product of (a) Asymmetric matrix and an orthogonal matrix, if real () A Hermitian matrix and 9 unitary matrix, if complex If the first factor is chosen to be positive definite, the factors are uniquely deter- mined. ‘These and many other results are derived by DeBruijn and Szekeres in the paper cited below,! using the logarithm of a matrix. 80. Derive the result of Exercise 28 from the canonical representation given above for orthogonal matrices, BL. Let u, us, « . « , uy be a set of linearly independent solutions of the Nth-order linear differential equation WO + pQuD $+ pu = 0 ‘The determinant ow uy 4 uy uy Weayus, os. ux) = : OD OD ee uylrD is called the Wronskian of the function wu, uz, ..., un. Show that 1 WO = Wows... nu) = We Siciom For an extension of the concept of the Wronskian, see A. Ostrowski.* ‘The corresponding determinant associated with the solution of a linear difference equation of the form ule +N) + predu(e +N 1) + + + pwl@u(e) = 0 2=0,1,2,..., Juste) us(z) se uta) uz +1) ux(z + 1) ss une +1) Clu, guy) = wlt+N = 1) wie +N ~1) s+ un tN = 1h is called a Casorati determinant.’ Many further results concerning Wronskians can be found in G. Polya and G. Szego.* 1N, G, DeBruijn and G, Szekeres, Nieuw. Arch. Wisk., (3), vol. 111, pp. 20-32, 1958. +A. Ostrowski, Uber ein Analogon der Wronskischen Determinante bei Funktionen mehrerer Verdnderlicher, Math. Z., vol. 4, pp. 223-230, 1919. 1 For some of its properties, see P. Montel, Lecons sur les recurrences et leurs appli- cations, Gauthier-Villars, Paris, 1957. Bee also D. M. Krabill, On Extension of Wronskian Matrives, Bull. Am. Math. Soc., vol. 49, pp. 593-601, 1943. 4G, Polya and G. Szego, Aufgaben und Lehrsatee aus der Analysis, Zw p. 113, Dover Publications, New York, 1945. r Band,222 Introduction to Matrix Analysis 82. A unitary matrix U can be expressed in the form U = V-"W-!VW with unitary V, W, if and only if det U = 1. 88. The unitary matrices U, V can be expressed in the form U = W,Wils, V = W.W.W, with unitary Wi, Ws, Ws if and only if det U = det V. 84. If AA* — A*A has all non-negative elements, then actually A is normal, 86. If H, and H, are Hermitian and one, at least, is positive definite, then H,Hy can be transformed into diagonal form (0. Taussky). 86, If A4 + uB can be dingonalized for all scalars X and », then AB = BA (I. 8. Motzkin and 0. Taussky). 87, Show that a matrix A, which is similar to a real diagonal matrix D in the sense that A = T'DT™', is the product of two Hermitian matrices (0. Taussky). 88. The characteristic roots of are given by =a —2/be cos k0, k= 1, 2,..., N, where 9=x/(N +1), Show that Aw(a) = |A — A1| satisfies the difference equation Aw (d) = (@ — A)Aw-1(A) — bedy-a(d) 89. The characteristic roots of z—b 2 b 2a 2 %@ b b 2 2 mM db B= b 2% s 2 b bm 2 2a b 2 2b, are given by Ye 2 — 2b — ba? — (@ — 2 cos ko)*} k= 1,2,...,N (Rutherford, Todd) Obtain the result by relating B to A*. (A detailed discussion of the behavior of the characteristic roots matrix A (a)'A(a) may be found in A. Ostrowski.') The matrix A(q) is given by the expression ind vectors of the 1 A. Ostrowski, On the Spectruin of a One-paramettic Family of Matrices, J. reine angew. Math., Bd. 193, pp. 143-160, 1984.Explicit Solutions and Canonical Forms 223 (Another detailed discussion of the properties of a particularly interesting matrix may be found in T. Kato.') 40. Denote by #(A) the function max [Ac ~ a/{, and by [lll the quantity (Sis) (A) < (atatt - 2 Ite ap)” Show that and thus that «(A) < 25[Al| (L. Mirsky). 41. Let [4,8] = AB — BA, as above. If [4,[4,A*l] =0, then A is normal (Putnam). See T. Kato and O. Tanssky.* 42. If M1, Any... , Aware the characteristic values of A, ¢ and c; are two nonzero complex numbers, and the characteristic values of cA + crA* ate chi + cahjeoy where j(j) is 8 permutation of the integers 1, . . . , N, then A is normal. 48. Let A and B commute. Then the characteristic valnes of (A,B) are (Nui), where A and yu: are the characteristic values of A and B arranged in a fixed order, independent of f (@. Frobeniua). (Pairs of matrices A-and B for which cA + caB has the characteristic roots oN + Cui are said to possess property L.!) 44. Prove that A is normal if and only if x (Ata) = SIME (i. Shur) 46. Use this result to establish Wiegmann’s result that BA is normal if A, B and AB are normal (I. Mirsky). 46. If A is a Hermitian matrix, show that with s(A) defined as in Exercise 40, 8(A) = 2 sup |(d,Av)| where the upper bound is taken with respect to all pairs of orthogonal vectors wand v (L. Mirsky). 47. If A is Hermitian, then s(A) > 2 max laro| (Parker, Miraky). rH 48. If A is normal, then 2A + 2A copsye (4) < sup #( t ) (L. Mirsky) 1, Kato, On the Hilbert Matrix, Proc. Am. Math, Soc., vol. 8, pp. 73-81, 1957. ?. Kato and O. Taussky, Commutators of A and A*, J. Washingion Acad. Sci., vol. 46, pp. 38-40, 1956. + For a discussion and further references, see T. 8, Motzkin and O. Taussky, Pairs of Matrices with Property L, Trans. Am. Math. Soc., vol. 73, pp. 108-114, 1952. 4 For the preceding results, and further ones, see L, Mirsky, Inoqualities for Normal and Hermitian Matrices, Duke Math, J., vol. 24, pp. 592-600, 1957.224 Introduction to Matrix Analysis 49. If A and B are normal matrices with characteristic values X; and ys, respec- tively, then there exists a suitable rearrangement of the characteristic values so that x Ds = wilt $14 = Bll, where |All? = y lacjl%, a8 above (Hoffman-Wielandt), 7 u 60, As we know, every matrix A can be written in the form B + iC, where Band C are Hermitian. Let the characteristic roots of B be contained between b; and by and those of C be between c, and cs. ‘Then if we consider the rectangle in the complex plane with the vertices bi + ic, bs + ics, bs + ici, bs + ics, all the characteristic roots of A are contained within this rectangle (Bendizton-Hirack). 61, By the domain D of a matrix A, let us mean the set of complex values assumed by the quadratic form (z,A2) for values of z satisfying the constraint (2,£) = 1. Show that the characteristic values of A belong to the domain of A.* 62, Let K be the smallest convex domain which includes all characteristic values of A, and let D be a domain of A. If A is normal, K and D are the sam 58. Show, by considering 2 X 2 matrices, that the result need not be valid if A is not normal. 64. The boundary of D is a convex curve, whether or not A is normal. 56. In addition, D itself is a convex set. For a bibliography of recent results concerning this and related problems, see O. Taussky.? 66, Associated with the equation 2’ = Az, where zis an N-dimensional vector and Aan N XN matrix, is an Nth-order linear differential equation satisfied by each component of 2. Suppose that every solution of the corresponding equation associ- ated with y’ = By is a solution of the equation associated with z’ = Az. What can be said about the relation between the matrices 4 and B? 57. Consider the equation dz/dt = Bz, 2(0) = c, where B is a constant matrix to be chosen so that the function (z,2) remains constant over time. Show that B will have the required property if and only if it is skew-symmetric. From this, conclude that e? is orthogonal. 68. Using the fact that the vector differential equation A d*z/dt? + B dz/dt + Cz = O can be considered to be equivalent to the system dz/dl = y, A dy/dt + By + Cz = 0, obtain a determinantal relation equivalent to |Ad? + BA + C| = O involving only linear terms in . 68, Let 21, 21, ..., 2m be scalars, and F(zyts, .. . tm) denote the matrix Auti + Asty + +++ + Ante, where Ay are pairwise commutative. Then if Mut . tm) = [F(zyzn . .. tn)[ (the determinant of F), we have f(XyXs, +.» Xm) = O if the X; are matrices such that AiXy + A:X+-+ +++ + AwXm = 0. (This generalisation of the Hamilton-Cayley theorem is due to H. B. Phillips.) 60. Show that the result remains valid if the linear form > Aja; is replaced by a 7 polynomial in the 2; with matrix coefficients, See Chao Ko and H. C. Lee,‘ where references to earlier work by Ostrowski and Phillips will be found. 1 For this and the following results see O. Toeplitz, Das algebraische Analogon zu einem Batze von Fejer, Math. Z., vol. 2, pp. 187-197, 1919. °F, Hausdorff, Der Wertvorrat einer Bilinearform, Math, Z., vol. 3, pp. 814-316, 1919, 20, Taussky, Bibliography on Bounds for Characteristic Roots of Finite Matrices, National Bureau of Standards, September, 1951. Chao Ko and H. C. Lee, A Further Generalization of the Hamilton-Cayley Theorem, J. London Math. Soe., vol. 18, pp. 163-158, 1940.Explicit Solutions and Canonical Forms 225 61. If By i = 1,2, ... ,n, are 4 X 4 matrices satisfying the relations Bt = —1, BiB; = —EyBs, i xj, then n < 5 (Eddington). 62. If the B, are restricted to be either real or pure imaginary, then 2 are real and 3 are imaginary (Eddington). For a generalization, see M. H. A. Newman.! 68. If B commutes with every matrix that commutes with A, then B is » scalar polynomial in A, See J, M, Wedderburn. 64. Let A be a complex matrix with characteristic roots 41, Ay... , Av. Then, if we set [XI]? — y fault, we have x ing Iasi = Yani where the lower bound is taken with respect to all nonsingular $. ‘The lower bound is attained if and only if A is diagonizable (L. Mirsky). 85. Let A, B, and X denote N x N matrices. Show that a sufficient condition for the existence of at least one solution X of the matrix equation X* — 2AX +B = 0 is that the characteristic values of al Re (a ~B 4) be distinct (Treuenfels), 68. Let {X} bea set of NX N matrices with the property that every real linear combination has only real characteristic values. Then Amax(X), the largest charac- teristic value, is a convex matrix function, and Anis(X), the smallest characteristic value, is a concave matrix funetion (P. D. Laz). 67. Let X and ¥ be two matrices all of whose linear combinations have real charac- teristic values, and suppose that those of X are negative, Then any characteristic root of X + i¥ has negative real part (P, D. Laz). €8. Let [X} be a set of NX N real matrices with the property that every real linear combination has only real characteristic values. Then if Xi and X; are two matrices in the set with the property that X; — Xz has non-negative characteristic roots, we have A:(X;) 2 (Xs), i = 1,2, .. . , N, where 4¢(X) is the ith character- istic root arranged in order of magnitude (P. D. Laz). For proofs of these results based upon the theory of hyperbolic partial differential equations, see P. D, Lax.t For proofs along more elementary lines, see A, F. Weinberger.t @8. Let {ac}, {8} (1 »? See R. Bellman, k=0 A Note on Linear Differential Equations with Quasiperiodic Coefficients, J. Math. Anal. Appl., to appear. Bibliography and Discussion §1 to 810. The results and techniques follow R. Bellman, Stability Theory of Differential Equations, Dover Pub- lications, New York, 1969. 8. Lefschetz, Lectures on Differential Equations, Ann. Math, Studies, no. 14, 1946. 8. For a discussion of the problem of determining the characteristic polynomial, see A.8, Householder and F. L, Bauer, On Certain Methods for Expand- ing the Characteristic Polynomial, Numerische Mathematik, vol. 1, pp. 29-40, 1959. $6. For some further results, and additional references, see XK. A. Hirsch, A Note on Vandermonde's Determinant, J. London Math, Soc., vol. 24, pp. 144-145, 1949. 11. For a proof of the Jordan canonical form, see 8. Lefschetz, Differential Equations, Geometric Theory, Interscience Publishers, Inc., New York, Appendix I, 1957. J. H. M. Wedderburn, Lectures on Matrices, Am. Math. Soc. Collog. Publs., vol. 17, 1934. $13. The result is given by I. Schur in I. Schur, Uber die characteristischen Wurzeln einer linearen Substitu- +R. L. Causey, On Closest Normal Matrices, Computer Sciences Division, Stanford University, Technical Report CB10, June, 1964. 4A, J. Hoffman and O. Taussky, On Characterizations of Normal Matrices, J. Res. Bur, Standards, vol. 52, pp. 17-19, 1954.228 Introduction to Matrix Analysis tion mit einer Anwendung auf die Theorie der Integral Gleichungen, Math. Ann., vol. 66, pp. 488-510, 1909. $14. The concept of a normal matrix appears to have been introduced by Toeplitz, O. Toeplitz, Math. Z., vol. 2, p. 190, 1919. §15. This approximate diagonalization was introduced by Perron and used to establish stability theorems. See O, Perron, Math. Z., vol. 29, pp. 129-160, 1929. 816. For a discussion of the requisite theorems concerning resultants, see B, L. Van der Werden, Moderne Algebra, Springer-Verlag, Berlin, 1931. §17. This result was given by Hamilton for quaternions and by Cayley for matrices. On page 18 of Macduffee's book, there is an account of a generalization of the Hamilton-Cayley theorem due to Phillips, For fur- ther generalizations, see the paper by Ko and Lee referred to in Exercise 60 of the Miscellaneous Exercises. $19. This derivation follows the monograph by Lefschetz, cited above. The determination of C' is a problem of great difficulty, An excellent survey of results in this field is contained in V. M. Starzinski, Survey of Works on Conditions of Stability of the Trivial Solution of a System of Linear Differential Equations with Periodic Coefficients, Trans, Am. Math. Soc., series 2, vol. 1, pp. 189-239. $20. Observe what this result means. Every nonsingular point trans- formation y = Az can be considered to arise from a family of continuous transformations furnished by the differential equation dZ/dt = CZ, Z(0) = I, where A = e®. 22. Some problems connected with these transformations may be found in the set of Research Problems of the Bull. Am, Math. Soc. for 1958. For the second-order linear differential equation (pu)’ + qu = 0 there exist a number of interesting representations for the solution. One such, called the Priifer transformation, was generalized by J. H. Barrett to cover the matrix equation (P()X')’ + Q()X = 0 (Proc. Am. Math. Soc.,Explicit Solutions and Canonical Forms 229 vol. 8, pp. 510-518, 1957). Further extension, and refinement were given by W. T. Reid, A Prufer Transformation for Differential Systems, Pacific J. Math., vol. 8, pp. 575-584, 1958. For an application to stability theory, see R. Conti, Sulla to-similitudine tra matrici e Vequivalenza asinto- tica dei sistemi differenziali lineari, Riv. Mat. Univ. Parma, vol. 8, pp. 43-47, 1957. 823. For a detailed account of the use of biorthogonalization in con- nection with the basic problems of inversion of matrices and the determi- nation of characteristic roots and vectors, see M. R. Hestenes, Inversion of Matrices by Biorthogonalization and Related Results, J. Soc. Ind. Appl, Math., vol. 6, pp. 51-90, 1958. One of the most important problems in the theory of linear differential equations is that of determining the structure of the solutions of linear systems of equations having the form x Y rlD)y-0 20) =a F=1%...,N j=l where D is the operator d/dt. ‘An equivalent problem in matrix analysis is that of determining the characteristic values and vectors of matrices A(A) whose elements are polynomials in X. Since any thorough discussion of this problem, which we have merely touched upon in treating A — XJ, would take us too far afield in this volume, we would like to refer the reader to R, A. Frazer, W. J. Duncan, and A. R. Collar, Blementary Matrices, The Macmillan Company, New York, 1946, where many applications are given to the flutter problem, etc. A consideration of the analytic properties of these matrices brings us in contact with the general theory of matrices whose elements are functions of a complex variable. These questions are not only of importance in themselves, but have important ramifications in modern physics and pre- diction theory. See P. Masani, The Laurent Factorization of Operator-valued Functions, Proc. London Math. Soc., vol. 6, pp. 59-69, 1956. P. Masani and N. Wiener, The Prediction Theory of Multivariate230 Introduction to Matrix Analysis Stochastic Processes, Acta Math., I, vol. 98, pp. 111-150, 1957; II, vol. 99, pp. 96-137, 1958. H. Helson and D. Lowdenslager, Prediction Theory and Fourier Series in Several Variables, Acta Math,, vol. 99, 1958. Results on the structure of matrices with quaternion elements may be found in N. A. Wiegmann, The Structure of Unitary and Orthogonal Qua- ternion Matrices, Iilinois J. Math,, vol. 2, pp. 402-407, 1958, where further references may be found. Some recent papers of interest concerning the location of charaeteristic roots are J.L, Brenner, New Root-location Theorems for Partitioned Matrices, SIAM J. Appl. Math., vol. 16, pp. 889-896, 1968. A. 8. Householder, Norms and the Location of Roots of Matrices, Bull. Am. Math. Soc., vol. 74, pp. 816-830, 1968, J. L, Brenner, Gersgorin Theorems by Householder's Proof, Bull. Am. Math, Soc., vol. 74, pp. 625-627, 1968. D. G, Feingold and R. 8. Varga, Block Diagonally Dominant Matri- ces and Generalizations of the Gersgorin Circle Theorem, Pacific J. Math., vol. 12, pp. 1241-1250, 1962. R. 8, Varga, Minimal Gerschgorin Sets, Pacific J. Math,, vol. 15, pp. 719-729, 1965. B. W. Levinger and R. 8. Varga, Minimal Gerschgorin Sets, II, Pacific J. Math., vol. 17, pp. 199-210, 1966. §24, For discussions of analytic and computational aspects of the Laplace transform, see R. Bellman and K. L. Cooke, Differential Difference Equations, Academic Press Inc., New York, 1963. R. Bellman, R. E. Kalaba and J. Lockett, Numerical Inversion of the Laplace Transform, American Elsevier, New York, 1966.12 Symmetric Functions, Kronecker Products and Circulants 1, Introduction. In this chapter, we shall show that some problems of apparently minor import, connected with various symmetric functions of the characteristic roots, lead to a quite important concept in matrix theory, the Kronecker product of two matrices. As a limiting case, we obtain the Kronecker sum, a matrix function which plays a basic role in the theory of the matrix equation AX+XB=C a We have already met this equation in an earlier chapter, and we shall encounter it again in connection with stability theory. Furthermore, the Kronecker product will crop up in a subsequent chapter devoted’ to stochastic matrices. Following this, we shall construct another class of compound matrices which arise from the consideration of certain skew-symmetric functions. Although these are closely tied in with the geometry of N-dimensional Euclidean space, we shall not enter into these matters here. Finally, again motivated by conditions of symmetry, we shall discuss circulants. Throughout the chapter, we shall employ the same basic device to determine the characteristic roots and vectors of the matrices‘under dis- cussion, namely, that of viewing a matrix as an equivalent of a trans- formation of one set of quantities into another. A number of problems in the exercises will further emphasize this point of view. 2. Powers of Characteristic Roots. A very simple set of symmetric functions of the characteristic roots are the power sums x p= yu @ a 231232 Introduction to Matrix Analysis x Suppose that we wish to determine ) #, Since isl we can determine the sum of the squares of the characteristic roots in terms of two of the coefficients of the characteristic polynomial. More- N over, since ) A#,k = 1,2, .. . , isa symmetric function of the charac- & teristic roots, we know, in advance, that it can be written as a polynomial in the elementary symmetric functions. For many purposes, however, this is not the most convenient repre- sentation, We wish to demonstrate Theorem 1. Fork = 1,2,..., we have x YM = tr (44) @) & Proof. If A has distinct characteristic roots, the diagonal represen tation AL 0 Ap A=T rT 4) 0 aw, which yields the representation for A¥, at AF 0 A=T Tm (5) 0 dy makes the rosult evident. It is easily seen that an appeal to continuity along by now familiar lines yields (3) for general matrices.Symmetric Functions, Kronecker Products and Circulants 233 If we do not like this route, we can employ the triangular representation A dw. where the elements above the main diagonal are not necessarily zero and the A: are not necessarily distinct. It is easily seen that at dit At=T . [ro 3) Ant, ‘This representation yields an alternative proof of (3). 3. Polynomials and Characteristic Equations. Although we know that every characteristic equation produces a polynomial, it is not clear that every polynomial can be written as the characteristic polynomial of a matrix. Theorem 2. The matrix -a, -a a1 —a, 1 0 0 0 o 1 0 0 A= a) o 0 1 0 has the characteristic equation JA =A) =O=M+ade' +--+ tay (2) We leave the proof as an exercise, EXERCISE 4. Using the result given above, determine the sum of the cubes of the roots of the equation M+ aN! + + +a, =O234 Introduction to Matrix Analysis 4 Symmetric Functions. Although we have solved the problem of determining matrices which have as characteristic roots the quantities A, Ast)... , Awt, We do not as yet possess matrices whose character- istic roots are prescribed functions of the characteristic roots, such as AM iG =1,2,...,N In order to solve this problem, we begin with the apparently more difficult problem of determining 2 matrix whose characteristic roots are Aaj, Where A, are the characteristic roots of A and u, those of B. Speeial- izing «= us, we obtain the solution of the original problem. Let us discuss the 2 X 2 case where the algebra is more transparent. Start with the two sets of equations Mei = ati tant, way Mite = ant + Gt. mays a and perform the following multiplications: Auatigs = Gubutiys + auibitiys + Qubutays + arabrtaye Mmrtiys = Qubutiyn + aubsotiys + aibetayr + aradrotays (2) Amity = anbutiys + anbistiys + aabrtays + arbrstays Awataye = Qnbatiys + anbestiys + andatays + arbrrays We note that the four quantities r1y:, t1y2, Z2y1, 222, occur on the right and on the left in (2). Hence if we introduce the four-dimensional vector my nye 2) aan ® aay, and the 4 X 4 matrix abi aibi2 Qiabi aibie anda aber diebny aiabae (4) dnb nbs ambi anbis Ando Grabr2 Gaabnr @aabaa. ¢c we may write the equations of (2) in the form Ame = Cz (5) It follows that z is a characteristic vector of C with the associated characteristic value A,u:. In precisely the same way, we see that C has Aus, Ant, and Azus as characteristic roots with associated characteristic vectors formed in the same fashion as in (3). ‘We thus have solved the problem of determining a matrix whose char- acteristic roots are Auu;, #, 9 = 1, 2.Symmetric Functions, Kronecker Products and Circulants 235 5, Kronecker Products. Since even the 2 X 2 case appears to intro- duce an unpleasant amount of calculation, let us see whether or not we can introduce a better notation. Referring to (4.4), we note that there is a certain regularity to the structure of C. Closer observation shows that C may be written as a compound matrix ayB auB c Fer os ® er, even more simply, C = (a;B) (2) Once this representation has been obtained, it is easy to see that it is independent of the dimensions of A dnd B. Hence Definition. Let A be an M-dimensional matriz and B an N-dimensional matrix. The MN-dimensional matrix defined by (2) is called the Kronecker product of A and B and written AX B= (asB) @) The above argumentation readily yields Theorem 3. Theorem 8. The characteristic roots of A X B are diu; where d; are the characteristic roots of A and y; the characteristic roots of B. The characteristic vectors have the form ay! aity! _ a) ruty’, Here by af, k = 1, 2, teristic vector x! of A, w , M, we mean the components of the charac: y is a characteristic vector of B. EXERCISES 1, Show that tr (A X B) = (tr A)(tr B). 2. Determine |A X Bl. 8. Show that ‘B 0 B IXB=236 Introduction to Matrix Analysis 6. Algebra of Kronecker Products. In order to justify the name “Kronecker product” and the notation we have used, we must show that A X B possesses a number of the properties of a product. We shall leave as exercises for the reader proofs of the following results: AXBXC=(AXB)XC=AX(BXC) (la) (A+B)xX(C+D)=AXC+AXD+BXC+BXD (1d) (A X B)(C X D) = (AC) x (BD) (le) 1. Kronecker Powers—I. We shall also consider the Kronecker power and write Aa AXA Ate a) AX AW If A and B do not commute, (AB)* # A*B* in general; and never if k= 2. It is, however, true that (AB) = AMR @) for all A and B. This important property removes many of the difficulties due to non- commutativity, and will be used for this purpose in a later chapter devoted to stochastic matrices, EXERCISE 1. Prove that All = AM xX Alt, 8. Kronecker Powers—II. If we are interested only in Kronecker powers of a particular matrix, rather than general products, we can define matrices with these properties which are of much smaller dimension. Starting with the equations duty = anti + diets a) Mita = nity + Gaote we form the products Ave? = au2e? + Zayas + ais*e2? Avaie = Audet? + (Guides + ar2de1)ri%2 + aisdesr2* (2) Aitae? = antes? + Zanraearite + areas? It is clear then that the matrix ay? 2anai a? Ay = [a@nan (aides + aid) aides @) ay)? Zander ys” possesses the characteristic roots 4:2, Ards, and As’. The corresponding characteristic vectors are readily obtained.Symmetric Functions, Kronecker Products and Circplants 237 To obtain the expression for Aj, we proceed in a more systematic fashion. 9. Kronecker Products—III. Consider the equations, for a fixed integer k, (Quits + aiets)* = aytes* + kay ayers! ley + ++ + (@nti + aizts)*'(ant, + e2%2) = an‘ anz* + ((& — Dan'*ande + aut an)z'z9 (@uti + aiete)*(ant, + aget2)) = anaes + + @ where i = 0, 1, 2, ih The matrix Aw is then defined by the tableau of coefficients of the terms xi~'x44, i = 0, 1, 2, shy at kana aut an ((& — Van %ay2ae1 + ais* aes) Aw = : 2) This representation holds only for 2 X 2 matrices. ‘The representation of Aw for matrices of dimension N can be obtained similarly. EXERCISES 1. Show that (4B)u) = AuBuy 2. Show that Ais = AwAuy 10. Kronecker Logarithm. Let us now derive an infinite matrix which we can think of as a Kronecker logarithm. We begin by forming a generalized kth power for nonintegral k. To do this, we consider the infinite sequence of expression (az; + aiszs)*~(anz1 + anszs)', i = 0, 1, 2,.. . , for k nonintegral. This yields, in place of the finite tableau of (8.2), an infinite array, since the binomial series (a1.21 + aist2)*~ is now not finite. The change of variable a} _pf[t [=]-°[2) © (AB) = AmBuy (2) shows that holds for arbitrary values of k.238 Introduction to Matrix Analysis Let us now expand each element in Aj about k = 0. Write Auy = Ata + BL(A) + + - @) This defines an infinite matrix L(A), which we can think of as a Kronecker logarithm. Substituting in (2), we see that, L(AB) = AwL(B) + L(A)Bia (4) an analogue of the functional equation for the scalar logarithm. Observe that Aj is not an identity matrix. EXERCISE 1, Determine the ijth element in L(A), for i = 0, 1, j = 0, 1. 11, Kronecker Sum—I, Let us now turn to the problem of deter- mining matrices which possess the characteristic values 4; + ;. Follow- ing a method used before, we shall derive the additive result from the multiplicative case. Consider, for a parameter ¢, the relation (Iu + €A) X (In + B) =InXIn+luXB+AX Ix) +@AXB (1) Since the characteristic roots of (Iu + ¢A) X (Iw + eB) are (+d) + ey) = 1+ Qe + a) + Oy we see that the characteristic roots of Im X B + A X In must beds + 4j. EXERCISE 1. Determine the characteristic vectors of Iw X B +4 X In. 12. Kronecker Sum—II. The matrix Iy X B+ A X Iw can also be obtained by use of differential equations, Consider the equations Fann tan YY = bunt bane az, dys ® aaa + agery un begs + bay Let us now compute the derivatives of the four products zis, zi 220, Za» We have S ceun) = (nts + asda + abun + dav) = Gu + budzin + bistiys + array (2)Symmetric Functions, Kronecker Products and Circulants 239 and so on. It is easy to see that the matrix we obtain is precisely AXIn+ Iu XB. 18. The Equation AX + XB = C, the Lyapunov Equation. In Chap. 11, we showed that the equation AX+XB=C (1) possessed the unique solution X= — [5 ecorat 2) provided that the integral on the right exists for all C. Let us now complete this result. Consider the case where A, B, C, and X are 2 X 2 matrices. The equations to determine the unknown components =, are atu + asta + tubu + tba = en Qysig + Grete2 + Zubia + tiadr = C2 (3) Gat + Geta + taba + abn = cn Ante + Gaston + Labrie + Loeb = C22 The matrix of coefficients is an + bn bn ais 0 bir i + bas oO ay an 0 antbu bn ® 0 an biz dar + bag, which we recognize as the matrix A X I +1 X BY. The characteristic roots are thus \: + 4, since B and B’ have the same characteristic roots. It is easy to see that the same results generalize to matrices of arbitrary dimension so that we have established Theorem 4. Theorem 4. A necessary and sufficient condition that (1) have a solution for all C is that di + uj # O where d: are the characteristic roots of A and us the characteristic roots of B. EXERCISES 1, Prove that @ necessary and sufficient condition that AX + XA’ = C have a unique solution for all C is that ds + Ay x 0. 2. Prove the foregoing result in the following steps: (a) Let 1 be a matrix reducing A to triangular form, bu bin bw bu +> baw TAT =B=240 Introduction to Matrix Analysis () Then AX + XA’ = C becomes B'(T"XT) + (1’XT)B = T'CT. (©) Let Y = 7’XT and consider the linear equations for the elements of Y. N Show that the determinant of the coefficients is [[ (bu + 6y,) and thus Sint derive the stated results (Hahn). 14, An Alternate Route. We observed in the foregoing sections how Kronecker products arose from the consideration of various symmetric functions of the roots of two distinct matrices, A and B, and how a par- ticular Kronecker power could be formed from a single matrix A, Let us now consider a different type of matrix “ power” which is formed if we consider certain skew-symmetric functions, namely, determinants, There is strong geometric motivation for what at first sight may seem to be quite formal manipulation. The reader who is interested in the geo- metric background will find references to books by Bocher and Klein listed in the Bibliography at the end of the chapter. To emphasize the basic idea with the arithmetic and algebraic level kept at a minimum, we shall consider a 3 X 3 matrix and a set of 2 X 2 determinants formed from the characteristic vectors. In principle, it is clear how the procedure can be generalized to treat R X R determinants associated with the characteristic vectors of N X N matrices. In practice, to carry out the program would require too painstaking a digression into the field of determinants. Consequently, we shall leave the details, by no means trivial, to the interested reader. Consider the matrix a a ay A=/b, be bs q) C1 Ce Ca where we have momentarily departed from our usual notation to simplify keeping track of various terms, whose characteristic roots are A1, As, As, with associated characteristic vectors z!, 2%, 2°, For any two characteristic veetors, 2 and 2/, i » j, we wish to obtain a set of linear equations for the 2 X 2 determinants ys, ys, ya given by the relations [aut ae] [hart Nae ge ee ee (2) Adis = | dt! Nee! wa Nat Nat Since Dans = anai! + are’ + ars Dita = dita + dara’ + buts’ (3) dew = erty’ + cota + cots!Symmetric Functions, Kronecker Products and Circulants 241 with similar equation for the components of 2/, we see that ayes + ages! + ayes! aye! + aged + ast 4) biti! + dave! + bata! bits + bated + bars! If a is the vector whose components are the a,, and b the vector whose components are the b,, we see that (4) is equivalent to the equation (a,z') (a,z/) (b,2) (be) What we want at this point is a generalization of the expansion for the Gramian given in Sec. 5 of Chap. 4. Fortunately, it exists. It is easy to verify that dy = diy = (5) (a2) (a2)| far bff oe (b,x!) (bya!) ay bel [aot ae jaz bil] ast ay a; bs||aat ait Flas ballze ae) t)a, bs || zt Al ®) From this it follows immediately from consideration of symmetry that n gi2(ajb) gar(a,b) gui(a,b) | { a AM] ye] = | aixlae) gra(ac) gar(a,c) | | v2 @ Ys Gralb,c) gaa(bye) — gailbse) I Lys, where aulay) = | x| 6) Our first observation is that 2, is a characteristic root and y an associ- ated characteristic vector of the matrix occurring in (7). Again, since the matrix is independent of i and j, we see that the three characteristic roots must be AiAa, AAs, and Ards. We now have a means of obtaining a matrix whose characteristic roots are MA, tj. EXERCISES 4. Reverting to the usual notation for the elements of A, write out the representa- tion for the éjth element of the matrix appearing in (7). 2 Let A be an NX N matrix, Write out the formula for the elements of the N(N — 1)/2-dimensional matrix, obtained in the foregoing fashion, whose roots are Dadi be 8. For the case where A is symmetric, show that we can obtain the inequality from the preceding results, and many similar inequalities. 4. Let A bean NX N matrix and let r be an integer between I and N, Denote by S, the ensemble of all sets of r distinct integers chosen from the integers 1,2, . . . ,242 Introduction to Matrix Analysis Let # and £ be two elements of 8, and Aw denote the matrix formed from A by delcoing all rows whose indices do not belong to # and ail columns whose elementa do not belong tot. Let the elements of S, be enumerated in some fixed order, say in numerical value, au. The M X M matrix, where M = N1/rl(N —1)!, C(A) = (lAsel) is called the rth compound or rth adjugate of A, Establish the following results. (@) CAB) = C,(A)C,(B) @) CA) = CA)’ © C(A7) = C4) @) [CA = 1A ke W- Or = DI! (©) The characteristic roots of C,(A) are the expressions i, yn... » aay where mi tus + +++ + wwis the rth elementary symmetric function of M1, Mt - + + 4 Amy Bey wi = AWAg so AR, we wae + ARIA» + For a discussion of when a given matrix is a compound of another, see D. E, Ruther- ford, Compound Matrices, Koninkl. Ned. Akad, Wetenschap. Amsterdam, Proc. Sect. Sci. ser. A, vol. 54, pp. 16-22, 1951. Finally, for the connection with invariant theory, see R. Weitzenbuck, Znvariant- entheorie, Groningen, 1923, 16. Circulants. Matrices of the form 3 By coy se ew cN-1 co st owes cw-2 Cn-1 Co + ** Crus C= : (a) c1 ce st co oecur in a variety of investigations. Let us determine the characteristic roots and vectors. Let r; be a root of the scalar equation r” = 1, and set th = cot ew + cya? (2) Then we see that y; satisfies the following system of equations: th = cote tos) ewan righ = ena + coms + + + + owoar sh? @) rity, = er bcos cari! } Bee, for further results and references, H. J. Ryser, Inequalities of Compound and Induced Matrices with Applications to Combinatorial Analysis, IUtinoie J. Math., vol. 2, pp. 240-253, 1958,Symmetric Functions, Kronecker Products and Circulants 243 It follows that y; is a characteristic root of C with associated charac- teristic vector rN} Since the equation r” = 1 has N distinet roots, we see that we obtain N distinet characteristic vectors. Consequently, we have the complete set of characteristic roots and vectors in this way. EXERCISES 1, Use the scalar equation r$ = ayr + 1 ina similar fashion to obtain the character- istic roota and vectors of the matrix ek e e ce 1 tem tem cote co +O atom tea tea ted tea OF oe fe eo + Ce a cr cy 2. Generalize, using the defining equation r¥ = be! + -- - + dy. MISCELLANEOUS EXERCISES 1. Let f(X) be a function of the N? variables 2; possessing a power series develop- ment in these variables about zero. Show that we may write s00 = Fw oxnicy Mo 2. Is it true that Fx, kt . eke vo 3. If A and B are positive definite, then A X B is positive definite. 4. If A and B are eymmetric matrices with A > B, then Atl > Bol, for n = 1,2,.... 5. If r satisfies the equation in Exercise 23 of Chap. 11 and s the equation s¥ + built + + +. + by = 0, how docs one form an equation whose roots are r:a;? Let 1, a, -.., a1 be the elements of a finite group. Write z = 2» + aia. + +++ + zy-raN~-; where the z; are sealars. Consider the products aiz = ax(0)z0 + ai(1)zr + ++ + + as(N ~ )zw-s where the elements aj(j) are the ay in some order, or aiz = zo(i) + aili)ar + +» - + ty—rG@)aw—1, where the 2:(j) are the 7244 Introduction to Matrix Analysis in some order, Introduce the matrix X = (2,(3)) and write X~2, If Xwe ¥ ~y, does XY ~ zy? 1. Let 1, a, . . . , avi be the elements of a finite group G, and 1, fi, .. . , Bua the elementa of a finite group H. Consider the direct product of G and H defined as the group of order MN whose elements are aif;. Using the procedure outlined in the preceding exercise, form a matrix corresponding to (Sho) ‘What is the connection between this mnatrix and the matrices No Ma xX~ > mas YX > s6s? So io For an interesting relation between the concept of group matrices and normality, see 0, Tauseky.t 8 The equation AX — XA = AX posseases nontrivial solutions for X if and only if X= — A where Ai, As, . +. , Aw are the characteristic values of A (Lappo- Danileveky). 9% Let F be a matrix of order NV%, portioned into an array of N submatrices fe,, each of order N, such that each fiz ional function, f:s(A), of a fixed matrix A of order N. If the characteristic values of A are Xs, As, . . « , dw, then those of F are given by the characteristic values of the N matrices, (f:;(M)), k = 1,2, . +. , N, each of order N, See J, Williamson? and 8, N. Afriat.* For an application to the solution of partial differential equations by numerical techniques, see J. Todd‘ and A. N. Lowan.* 10. Let f(@) be a real function of @ for —» < @ < x, and form the Fourier coeffi- cients of f(6), antl" Se" de n=O, +1, 42... The Gnite matrices Ty= (a) b= 0,1,2...,N are called Toeplitz matrices of order N. Show that T'y is Hermitian, 11, Show that if we denote by Ax, 4s, . . . , Avi, the N + 1 characteristic values of 7'y, then Pa ie tani 1 fr im NHI = +0. Taussky, A Note on Group Matrices, Proc. Am. Math. Soc., vol. 6, pp. 984-986, 1955, 2 J, Williamson, The Latent Roots of a Matrix of Special Type, Bull. Am. Math. Soc., vol. 87, pp. 585-590, 1931. +8. N. Afriat, Composite Matrices, Quart, J. Math., Ozford 24, ser. 5, pp. 81-98, 1954, «J. Todd, The Condition of Certain Matrices, III, J. Research Natl. Bur. Standards, vol. 60, pp. 1-7, 1958. * AN. Lowan, The Operator Approach to Problems of Stability and Convergence of Solutions of Difference Equations, Scripta Math., no. 8, 1957.Symmetric Functions, Kronecker Products and Circulants 245 12. Show that ion QUOD QAM sss tw LL fr fin, SE OA 8 [Eras (These last two results are particular results of a general result established by Szego in 1917. A discussion of the most recent results, plus a large number of references to the ways in which these matrices enter in various parts of analysis, will be found in Kae, Murdock, and Szego.’) Generalizations of the Toeplits matrix have been considered in U. Grenander.* ‘See also H. Widom.* 18. Having defined what we mean by the Kronecker sum of two matrices, A ® B, define a Kronecker derivative of a variable matrix X(() as follows: BX _ tig X@+h) @ (= XO) li 0 , 14 Consider the differential equation BY. yt AX¥ YO HT Does it have a solution, and if so, what is it? 18. Establish the Schur result that C = (a;;b:3) is positive definite if A and B are by considering C as a suitable submatrix of the Kronecker product A x B (Marcus). 16. Let a be the ith column of A. Introduce the “stacking operator,” S(4), which transforms a matrix into a vector a) a” (A) = ae Prove that S(PAQ) = (Q’ @P)S(A). Bee D. Nissen, A Note on the Variance of a Matrix, Econometrica, vol. 36, pp. 603-604, 1968. y 17, Consider the Selberg quadratic form (see Appendix B), Q(z) = ) ()' z.)*: ier Si where a., k = 1,2... , WN, it'a set of positive integers, 21 = 1, and the remaining zy are real. Consider the extended form Q,(z,2) = 2 (a + > 2)" ond write os a 42) = min Q,(z,2), where the minimization is over 24, tay, --., zy. Obtain a recurrence relation connecting /,(e) with f,1(2). 1M. Kae, W. L. Murdock, and G. Szego, On the Eigenvalues of Certain Hermitian Forms, J. Rat. Mech. Analysis, vol. 2, pp. 767-800, 1953. *U. Grenander, Trans. Am. Math, Soc., 1958. +11. Widom, On the Eigenvalues of Certain Hermitian Operators, Trans. Am. Math, Soc., vol. 88, 491-522, 1958,246 Introduction to Matrix Analysis 18, Use the fact that J,(c) is quadratic in z to simplify this recurrence relation; see Chap. 9. 19, Discuss the use of the recurrence relation for analytic and computational pur- poses. Seo R. Bellman, Dynamic Programming and the Quadratic Form of Selberg, J. Math, Anal. Appl., vol. 15, pp. 30-32, 1966. 20. What conditions must 6 and c satisfy so that Xb = c where X is positive definite? (Wimmer) 21, Establish the Schur result concerning (a,j64;) using Kronecker producta and a suitable submatrix (Marcus). Bibliography and Discussion §1. The direct product of matrices arises from the algebraic concept of the direct product of groups. For discussion and treatment from this point of view, see ©. C. MacDufiee, The Theory of Matrices, Chelsea Publishing Co., chapter VII, New York, 1946. F, D, Mumaghan, The Theory of Group Representation, Johns Hop- kins Press, Baltimore, 1938. We are interested here in showing how this concept arises from two different sets of problem areas, one discussed here and one in Chap. 15. See also E. Cartan, Ocwures Completes, part 1, vol. 2, pp. 1045-1080, where the concept of a zonal harmonic of a positive definite matrix is introduced. §4. This procedure is taken from R. Bellman, Limit Theorems for Non-commutative Operations—I, Duke Math, J., vol. 21, pp. 491-500, 1954. §5. Some papers concerning the application and utilization of the Kronecker product are F. Stenger, Kronecker Product Extensions of Linear Operators, SIAM J. Numer, Anal., vol. 5, pp. 422-435, 1968. J. H. Pollard, On the Use of the Direct Matrix Product in Analyzing Certain Stochastic Population Models, Biometrika, vol. 53, pp. 397- 415, 1966. $10. The Kronecker logarithm is introduced in the paper cited above. §11. The result of Theorem 4 is given in the book by MacDuffeeSymmetric Functions, Kronecker Products and Circulants 247 referred to above. It was independently derived by W. Hahn, using the procedure outlined in the exercises; see W. Hahn, Eine Bemerkung aur zweiten Methode von Lyapunov, Math. Nachr., 14 Band., Heft 4/8, pp. 349-354, 1956. The result was obtained by R. Bellman, as outlined in the text, unaware of the discussion in the book by MacDuffee, cited above, R. Bellman, Kronecker Products and the Second Method of Lyapunov, Math. Nachr., 1959. 818, A great deal of work has been done on the corresponding operator equations, partly for their own sake and partly because of their central position in quantum mechanics. See M. Rosenbloom, On the Operator Equation BX — XA = C, Duke Math. J, vol. 23, 1956. E, Heinz, Math. Ann,, vol. 123, pp. 415-438, 1961. D. E. Rutherford, Koninkl. Ned. Akad. Wetenschap., Pros., ser. A, vol. 35, pp. 54-59, 1932. §14. The motivation for this section lies within the theory of invari- ants of linear transformations and the geometric interconnections. See M. Booher, Introduction to Higher Algebra, The Macmillan Company, New York, reprinted, 1947. F. Klein, Elementary Mathematics from an Advanced Standpoint, Geometry, Dover Publications, New York. H. Weyl, The Classical Groups, Princeton University Press, Prince- ton, N.J., 1946. The 2 X 2 determinants are the Plucker coordinates. The basic idea is that every linear transformation possesses a latge set of associated induced transformations that permit us to derive certain properties of the original transformation in a quite simple fashion. As indicated in the text, we have not penetrated into this area in any depth because of a desire to avoid a certain amount of determinantal manipulation. See also H. Schwerdtfeger, Skew-symmetric Matrices and Projective Geom- etry, Am. Math. Monthly, vol. LI, pp. 137-148, 1944. It will be clear from the exercises that once this determinantal ground- work has been laid, we have a new way of obtaining a number of the inequalities of Chap. 8.248 Introduction to Matrix Analysis For an elegant probabilistic interpretation of Plucker coordinates, and the analogous multidimensional sets, see Exercises 15 and 16 of Chap. 15, where some results of Karlin and MacGregor are given. §18. Circulants play an important role in many mathematical-physical theories. See the paper T. H, Berlin and M. Kae, The Spherical Model of a Ferromagnet, Phys. Rev., vol. 86, pp. 821-835, 1952. for an evaluation of some interesting circulants and further references to work by Onsager, et al, connected with the direct product of matrices and groups. Treatments of the theory of compound matrices, and further references may be found in H. J. Ryser, Inequalities of Compound and Induced Matrices with Applications to Combinatorial Analysis, ZUinois J. Math., vol. 2, pp. 240-253, 1958, N. G. deBruijn, Inequalities Concerning Minors and Eigenvalues, Nieuw. Arch. Wisk, (3), vol. 4, pp. 18-35, 1956, D. E. Littlewood, The Theory of Group Characters and Matrix Repre- sentations of Groups, Oxford, New York, 1950. I. Schur, Uber eine Klasse von Matrizen die sich einer gegebenen Matrix zuordnen lassen, Dissertation, Berlin, 1901. J. H. M, Wedderburn, Lectures on Matrices, Am. Math. Soc. Coltog. Publ., vol. 17, 1984. C.C. MacDuffee, The Theory of Matrices, Chelsea Publishing Com- pany, New York, 1946. O. Toeplitz, Das algebraische Analogon zu einem Satze von Fejer, Math. Z., vol. 2, pp. 187-197, 1918. M. Marous, B. N. Moyls, and R. Westwick, Extremal Properties of Hermitian Matrices II, Canad. J. Math., 1959.13 Stability Theory 1, Introduction, A problem of great importance is that of determining the behavior of a physical system in the neighborhood of an equilibrium state. If the system returns to this state after being subjected to small disturbances, it is called stable; if not, it is called unstable. Although physical systems can often be tested for this property, in many cases this experimental procedure is both too expensive and too time-consuming. Consequently, when designing a system we would like to have mathematical criteria for stability available. It was pointed out in Chap. 11 that a linear equation of the form dz Ga4t 2) =e a can often be used to study the behavior of a system in the vicinity of an equilibrium position, which in this case is x = 0. Consequently, we shall begin by determining a necessary and sufficient condition that the solution of (1) approach zero ast—> ©. The actual economic, engineering, or physical problem is, however, more complicated, since the equation describing the process is not (1), but nonlinear of the form Ye ay+ay) 0) =6 @) The question then is whether or not criteria derived for linear systems are of any help in deciding the stability of nonlinear systems. It turns out that under quite reasonable conditions, the two are equivalent. This is the substance of the classical work of Poincaré and Lyapunov. How- ever, we shall not delve into these more recondite matters here, restrict- ing ourselves solely to the consideration of the more tractable linear equations. 2. A Necessary and Sufficient Condition for Stability. Let us begin by demonstrating the fundamental result in these investigations. 249250 Tatroduction to Matrix Analysis Theorem 1. A necessary and sufficient condition that the solution of = =Ar 20) =e a regardless of the value of c, approach zero as t—> ©, is that all the charac- teristic roots of A have negative real parts. Proof. If A has distinct characteristic roots, then the representation ent ew “aT T (2) establishes the result. We cannot make an immediate appeal to con- tinuity to obtain the result for general matrices, but we can proceed in the following way. In place of reducing A to diagonal form, let us transform it into tri- angular form by means of a similarity transformation, T-'AT = B. The system of equations in (1) takes the form ¢ =Be 20) =c (3) where B is a triangular matrix upon the substitution z = Tz. Written out in terms of the components, we have Hm duet bart ot biew 2100) =F 2. Dutt + + bavew —29(0) = cf . « dey To bwwen — an(0) = cy Since the by are the characteristic roots of A, we have, by assumption, Re (ba) < Ofori? =1,2,...,N. Solving for zw, ay = cyebvet (5) we see that zy Oasi— ,Stability Theory 251 In order to show that all 2: 0 as t—> @, we proceed inductively based upon the following result. If of) +0 as t—> =, then u(t) as determined by a = bu + 0(t) u(0) = a (6) approaches zero as t—> ©, provided that Re (b:) <0. Since u(t) = arch + tf! e-bn0(e) ds @ it is easy to see that the stated result is valid. Starting with the result for zw based upon (5), we obtain successively the corresponding results for zw—t) 6. - 5 21 EXERCISE 1, Prove Theorem 1 using the Jordan canonical form. 3, Stability Matrices. To avoid wearisome repetition, let us introduce @ new term. Definition. A matrix A will be called a stability matrix if all of its characteristic roots have negative real parts. EXERCISES 1. Derive a necessary and sufficient condition that a real matrix be a stability matrix in terms of the matrix (tr (A“)) (Bass). 2. What is the corresponding condition if A is complex? 4. A Method of Lyapunov. Let us now see how we can use quadratic forms to discuss questions of asymptotic behavior of the solutions of linear differential equations. This method was devised by Lyapunov and is of great importance in the modern study of the stability of solu- tions of nonlinear functional equations of all types. Consider the equation dz _ Gr At 20) =e qa) where cand A are taken to be real, and the quadratic form u = (x,¥2) @) where ¥ is a symmetric constant matrix as yet undetermined, We have due , qm @s¥2) + @ Ye!) = (Az,Yx) + (,YAz) = @(A'Y + YA)z) 3)252 Introduction to Matrix Analysis Suppose that we can determine Y so that AY+YA=-I 4) with the further condition that Y be positive definite. Then the relation in (3) becomes (5) which yields (6) where Aw is the largest characteristic root of Y. From (6) we have u ©, 2, Extend the foregoing argument to treat the case of complex A. 5. Mean-square Deviation. Suppose that A is a stability matrix and that we wish to calculate T= fy Bs) at q@ where x is a solution of (4.1).Stability Theory 253 It is interesting to note that J can be calculated as a rational function of the elements of A without the necessity of solving the linear differ- ential equation for x, In particular, it is not necessary to calculate the characteristic roots of A. Let us determine a constant matrix Y such that (2,Br) = 7 (2,¥x) (2) We see that BoA'Y+YA @) With this determination of Y, the value of J is given by J = —(¢,¥e) (4) Since A is a stability matrix, (3) has a unique solution which can be found using determinants. 6. Effective Tests for Stability. The problem of determining when a given matrix is a stability matrix is formidable one, and at the present time there is no simple solution, What complicates the problem is that we are not so much interested in resolving the problem for a particular matrix A as we are in deriving conditions which enable us to state when various members of a class of matrices, A(u), are stability matrices, Questions of this type arise constantly in the design of control mecha- nisms, in the field of mathematical economies, and in the study of com- putational algorithms. Once the charaeteristic polynomial of A has been calculated, there are a variety of criteria which can he applied to determine whether or not all the roots have negative real parts, Perhaps the most useful of these are the criteria of Hurwitz. Consider the equation IAT — Al = 8 Faa!-$ 0+ fay tay =0 a) and the associated infinite array a 1000 0 a, a a 1 0 0 as ay @ a a, O tas Gy mG) ay @ where a, is taken to be zero for k > N.254 Introduction to Matrix Analysis A necessary and sufficient condition that all the roots of (1) have negative real parts is that the sequence of determinants a1 a 1 0 Ay = lala -| : ha=|a, a2 a (3) a a Gs Ge as formed from the preceding array, be positive. There are no simple direct proofs of this result, although there are a number of elegant proofs. We shall indicate in the following section one line that can be followed, and in Appendix C discuss briefly the chain of ideas, originating in Hermite, giving rise to Hurwitz’s proof, Both of these depend upon quadratic forms, References to other types of proof will be found at the end of the chapter. The reason why this result is not particularly useful in dealing with stability matrices is that it requires the evaluation of |Z — Al, some- thing we wish strenuously to avoid if the dimension of A is high. EXERCISES 1. Using the foregoing criteria, show that a necessary and sufficient condition that M4 aid + az be a stability polynomial is that a; > 0, a, >0. By thie we mean that the roots of the polynomial have negative real parte, 2. For M+ ai! + ash + as show that corresponding conditions are 41,002 >0 aids > ay 8%. For ¢ + a: +a, + ash +a, show that the conditions are a1, a4 >0, G10y > Gs, O4(a102 — as) > asta. 7. A Necessary and Sufficient Condition for Stability Matrices. Let us now show that the results of Sec. 4 yield Theorem 2. Let Y be determined by the relation AY+YA=-I q) Then a necessary and sufficient condition that the real matrix A be a sta- bility matrix is that Y be positive definite. Proof. Referting to Sec. 5, we see that JV" a) dt = (2(0),¥2(0)) - CT), ¥2(7)) @) or (2(T),¥2(7)) + fy” (a2) dt = (2(0),¥2(0)) @) Here 2 isa solution of the equation d2/di = Az. If Y is positive definite, f," (2,2) dt is uniformly bounded, which means that z(t) + Oast— ©, whence A isa stability matrix. We have already established the fact that Y is positive definite if A is a stability matrix,Stability Theory 255 8 Differential Equations and Characteristic Values. Consider the differential equation Ac" +2Be +Cr=0 20)=c 20) = ct a) which, if considered to arise from the study of electric circuits possessing capacitances, inductances, and resistances, is such that A, B, and C are non-negative definite. On these grounds, it is intuitively clear that the following result holds. Theorem 3. If A, B, andC are non-negative definite, and either A or C positive definite, then PA + 2B + Cl =0 Q) has no roots with positive real parts, If A and C are non-negative definite and B is positive definite, then the only root with zero real part is \ = 0. Proof. Let us give a proof which makes use of the physical back- ground of the statement. In this case, we shall use energy considerations. Starting with (1), let us write (x',Az") + 2(2',Ba") + (2/,Cz) = 0 @) Thus, for any s > 0, [eazy + 2@',Be) + @Cn] dt = 0 (4) or Ar’) |, +4 ff" @,Be!) at + (2,02) |} = 0 (6) This is equivalent to the equation (2'(s),Az'(s)) + 4 f (w',Bx') dt + (x(s),Cx(s)) = es (6) where cs = (c?,Act) + (¢!,Ce!). If Ais a root of (2), then (1) has a solution of the form ee, cis real. If) is complex, \ = r: + ir, then the real part of ec, which has the form e'(a! cos r3f + a? sin 21), is also a solution. Substituting in (6), we see that e*(b1,Ab!) + 4 i e*(b2(¢), Bb%(t)) dt + e*7#(b,Cb8) = cs (7) where b! and b? are constant vectors and b? is a variable vector given by (a'r: + ar) cos ret + (a*ri — a'r) sin rot. If A or C is positive definite, with B > 0, we see that r: > 0 leads to a contradiction as s—> ©, If A, C > 0, then B positive definite requires that r, < 0, Further- more, sinoc a! cos rst + a? sin rf is periodic, we see that the integral ff @,Bd*() at diverges to plus infinity as s— ©, unless rz = 0, if mn =0.256 Introduction to Matrix Analysis We have presented this argument in some detail since it can be extended to treat similar questions for equations in which the coefficients A, B, and C are variable. EXERCISES 1. Following Anke," let us use the foregoing techniques to evaluate f,” ut a, given that u is a solution of x" + a.u" + au’ + aon = 0, all of whose solutions tend to zero, Let u(0) = co, u’(0) = cr, u""(0) = cz. Establish the results i uudt = wall - fr ullwl dt = ee, + c,2/2 [feted awl = [Tatar ee, = [una ; [feu -con 2. Derive from the equations i WO” + au" + au! +a) db=0 §=0, 1,2 and the results of Exercise 1, a set of linear equations for the quantities f utdt, i (wy at, i (uy de, 8. Using these lincar equations, express f * dt as a quadratic form in co, c1, ox 4, Using this quadratic form, obtain a sct of nocessary and sufficient conditions that r? + ay? + air + ay be a stability polynomial. 5. Show that these conditions are equivalent to the Hurwitz conditions given in Exercise 2 of Sec. 6. 9 Effective Tests for Stability Matrices. As indicated in the fore- going sections, there exist reasonably effective techniques for determin- ing when the roots of a given polynomial have negative real parts. Since, however, the task of determining the characteristic polynomial of a matrix of large dimension is a formidable onc, we cannot feel that we have a satisfactory solution to the problem of determining when a given matrix is a stability matrix, In some special cases, nonetheless, very elegant criteria exist. Thus: Theorem 4. If A has the form ath a: 1 by as A=| - -lb um (y) —1 bya aw 1 by, 1B. angew. Math, u, Phys., vol. VI, pp. 327-332, 1955.Stability Theory 257 where the dots indicate that all other terms are zero, the a; are real, and the b; are zero or pure imaginary, then the number of positive terms in the sequence of products a1, aidz, .. . , did2 . . . v-10n is the number of character istic roots of A with positive real part. For the proof, which is carried out by means of the theory of Sturmian sequences, we refer to the paper by Schwarz given in the Bibliography at the end of the chapter. In this paper it is also shown that any matrix B with complex elements can be transformed into the form appearing in (1) by means of a transformation of the type A = T—'BT. ‘As we shall see in Chap. 16, the theory of positive matrices gives us a foothold on the stability question when A is a matrix all of whose off-diagonal elements are non-negative. MISCELLANEOUS EXERCISES 1. Obtain the solution of the scalar equation u” + u = f(t) by writing u’ = », v = —u + full) and determining the elements of e# where o1 An [3 a 2, Let the characteristic roots of A be distinct and B(!) + Oa {+ ». Then the characteristic roots of A + B(0), which we shall call 2(), are distinct for f > lo 3 If, inaddition, f “4B (|| dt < ©, there existe matrix 7() having the property that the change of variable y = Teconverts y’ = (A + B(O)yintos’ = (LO) + C)e where a 0 r() L(t) = 0 . Aw(0). and [Iola < @. x a 2 J," last +450] dt < ©, all solutions of v/ = A(Oy are bounced as fa t-4 0, 5. There exists an orthogonal matrix B(!) such that the transformation y = B(z converts y' = A()y into 2’ = C(0)2 where C(0) is semidiagonal (Diliberto). 6. There exists a bounded nonsingular matrix B(!) with the property that C(0) is diagonal. 7. The usual rule for differentiation of a product of two functions u and » has the form d(wo)/dt = udv/dt + (du/di)», Consider the linear matrix function d(X) defined by the relation a(X) = AX - XA258 Introduction to Matrix Analysis where A isa fixed matrix. Show that a(XY) = Xa(¥) + d(X)¥ 8 Obtain a representation for d(X:,X2 - - + Xx) and thus for d(X*). 9. Let da(X) = AX — XA, da(X) = BX - XB. Show that aa(da(X)) = da(da(X)) 10, When does the equation d(X) = AX, \ a scalar, have a solution, and what is it? For an extensive and intensive discussion of these and related questions, see J. A. Lappo-Danilevsky.! 11. When do the equations a(X1) = anXs + aXe (Xs) = anXi + aXe have solutions, and what are they? 12, Since d(X) is an analogue of a derivative, are there analogues of a Taylor series expansion? 18. Given a real matrix A, can we always find a diagonal matrix B with elements £1 such that all solutions of Bdz/dt = Az approach zero as !—> © (Brock)? 14, Let A be nonsingular. Then a permutation matrix P and a diagonal matrix D can be chosen so that DPA is a stability matrix with distinct characteristic roots (Folkman). 15, What is the connection between the foregoing results and the idea of solving Az = b by use of the limit of the solution of y’ = Ay — bast =? (Unfortunately, the result of Exercise 14 is an existence proof. No constructive way of choosing D and P is known.) 16. If C has all characteristic roots less than one in absolute value, there exists a Positive definite G such that @ — CGC" is positive definite (Stein). 17. Establish Stein’s result using difference equations. For extensions of the results in Exercises 14-17 and additional references, see O. Taussky, On Stable Matrices, Programmation en mathématiques numériques, Besangon, pp. 75-88, 1966. 18, Let A be an N X N complex matrix with characteristic roots Xs, Xs + Xa x 0. ‘Then the N X N matrix @ = G*, the solution of AG + GA* = I is nonsingular and has as many positive characteristic roots as there are A; with positive real parts. (. Tausaky, A Generalization of a Theorem of Lyapunov, J. Soc. Ind. Appl. Math., vol. 9, pp. 640-643, 1961.) 19, Given the equation A" + aM"! +--+ + aq = 0, with the roots Ay, Ay « «++ Aw find the equation of degree n* whose roots are Xi ++ Ay iF = 1,2)... 20. Find the equation of degree n(n + 1)/2 whose roota are Ai ++ Ai = 1,2, 0+ +5 nj =1,2,...,4,and the equation of degree n(n — 1)/2 whose roots are A: + Aj SOR. aig 41,4. 7, fe 1 1, Use these equations to determine necessary and sufficient conditions that all of the roots of the original equation have negative real parts (Cliford-Routh). See Fuller, where many additional results will be found, and Barnett and Storey? and the survey paper by O. Taussky cited above. 1J. A. Lappo-Danilevsky, Mémoires aur la théorie des syatdmea des équations différ- entielles linéaires, vol. 1, Chelsea Publishing Co., New York, 1953. +A. T, Fuller, Conditions for a Matrix to Have Only Characteristic Roots with Negative Real Parts, J. Math. Anal. Appl., vol. 23, pp. 71-98, 1968, 48, Barnett and C. Storey, Analysis and Synthesis of Stability Matrices, J. Dig. Eq, vol. 3, pp. 414-422, 1967.Stability Theory 259 Bibliography and Discussion §1. We are using the word “stability” here in the narrow sense that all solutions of dz/dt = Ax tend to the null solution ast— ©. Fora wider definition and further results, see R. Bellman, Stability Theory of Differential Equations, Dover Pub- lications, New York, 1969. The problem of determining the asymptotic behavior of the solutions of dz/dt = A(i)z in terms of the nature of the characteristic roots of A(t) as functions of ¢ is much more complex. See the book cited above, and E. A. Coddington and N. Levinson, Theory of Ordinary Differential Equations, McGraw-Hill Book Company, Inc., New York, 1955. $2. An extensive generalization of this result is the theorem of Poinearé-Lyapunov which asserts that all solutions of dz/dt = Ax + g(t) for which |[z(0)|| is sufficiently sriall tend to zero ast — ©, provided that A is a stability matrix and g(2) is nonlinear, that is, llg(z)l|/llz|| +> 0 as l|z|| +0. See the books cited above for several proofs, one of which is sketched in Exercise 1 of Seo. 4. See also R, Bellman, Methods of Nonlinear Analysis, Academic Press, Inc., New York, 1970. §4. This method plays a fundamental role in the modern theory of the stability of functional equations, It was first presented by Lyapunov in his memoir published in 1897 and reprinted in 1947, A. Lyapunov, Probleme general de la stabilite du mouvement, Ann, Math, Studies, no 17, 1947. For a report on its current applications, see H. Antosiewiez, A Survey of Lyapunov's Second Method, Applied Mathematics Division, National Bureau of Standards, 1956, H, Antosiewiez, Some Implications of Lyapunov’s Conditions for Stability, J. Rat. Mech. Analysis, vol. 3, pp. 447~457, 1954. §5. We follow the results contained in R. Bellman, Notes on Matrix Theory—X: A Problem in Control, Quart. Appl. Math., vol. 14, pp. 417-419, 1957.260 Introduction to Matrix Analysis K. Anke, Eine neue Berechnungsmethode der quadratischen Regel- flache, Z. angew. Math. u. Phys., vol. 6, pp. 327-331, 1955. H. Buckner, A Formula for an Integral Ocourring in the Theory of Linear Servomechanisms and Control Systems, Quart. Appl. Math, vol. 10, 1952. P, Hazenbroek and B. I. Van der Waerden, Theoretical Consider- ations in the Optimum Adjustment of Regulators, Trans. Am. Soc. Mech. Engrs., vol. 72, 1950. §6. The results of Hurwitz were obtained in response to the problem of determining when a given polynomial was a stability polynomial; see A, Hurwitz, Uber die Bedingungen unter welchen eine Gleichung nur Wurzeln mit negativen reellen Teilen besitat, Math. Ann., vol. 46, 1895 (Werke, vol. 2, pp. 533-545). A large number of different derivations have been given since. A critical review of stability criteria, new results, and an extensive bibliography may be found in er and F. H. Effertz, Uber die algebraische Kriterien fir die it von Regelungssystemen, Math. Annalen, vol. 137, pp. 328-350, 1959. A. T. Fuller, Conditions for a Matrix to Have Only Characteristic Roots with Negative Real Parts, J. Math. Analysis Appl., vol. 23, pp. 71-98, 1968. §7. A great deal of work using this type of argument has been done by R. W. Bass in unpublished papers. For a discussion of stability problems as they arise in mathematical economics, see A. C. Enthoven and K. J. Arrow, A Theorem on Expectations and the Stability of Equilibrium, Econometrica, vol. 24, pp. 288-293, 1956. K. J. Arrow and M. Nerlove, A Note on Expectations and Stability, Econometrica, vol. 26, pp. 297-305, 1958, References to earlier work by Metzler and others will be found there. 9 See the paper H. R. Schwarz, Ein Verfahren zur Stabilititsfrage bei, Matrizen- Eigenwert-Problemen, Z. anyew. Math. u. Phys., vol. 7, pp. 473-500, 1956.Stability Theory 261 For the problem of the conversion of a given matrix into a Jacobi matrix, see the above paper by Schwarz and W. Givens, Numerical Computation of the Characteristic Values of a Real Symmetric Matrix, Oak Ridge National Laboratory, Report ORNL-1574, 1954. For closely related results, see E, Frank, On the Zeros of Polynomials with Complex Coefficients, Bull. Am. Math. Soc., vol. 52, pp. 144-157, 1946. E. Frank, The Location of the Zeros of Polynomials with Complex Coefficients, Bull. Am. Math. Soc., vol. 52, pp. 890-898, 1946. Two papers dealing with matrices all of whose characteristic values lie inside the unit cirele are 0. Taussky, Analytical Methods in Hypercomplex Systems, Comp. Math., 1937. O. Taussky and J. Todd, Infinite Powers of Matrices, J. London Math. Soc., 1943, An interesting and important topic which we have not at all touched upon is that of determining the asymptotic behavior of the solutions of the vector-matrix equation x(n + 1) = (A + B(n))z(n) asn— , where the elements of B(n) tend to zero as n—+ &, ‘The study of this problem was begun by Poincaré; see H, Poincaré, Sur les équations linéaires aux différentielles ordinaires et aux différences finies, Am. J. Math., vol. 7, pp. 203-258, 1885. and continued by Perron, O. Perron, Uber die Poincarésche lineare Differenzgleichung, J. reine angew. Math., vol. 137, pp. 6-64, 1910. and his pupil Ta Li, Ta Li, Die Stabilitiitsfrage bei Differenzengleichungen, Acta Math., vol. 68, pp. 99-141, 1934, The most recent result in this field is contained in G. A. Freiman, On the Theorems of Poincaré and Perron, Uspekhi Mat. Nauk (N.S.), vol. 12, no. 3(75), pp. 241-246, 1957. (Russian.) Other discussion may be found in R. Bellman, A Survey of the Theory of the Boundedness, Stability, and Asymplotic Behavior of Solutions of Linear and Non-linear Differ-262 Introduction to Matrix Analysis ential and Difference Equations, Office of Naval Research, Depart- . ment of the Navy, January, 1949. R, Bellman, Methode of Nontinear Analysis, Academic Press, Inc., New York, 1970. These questions, as well as the corresponding questions for the differ- ential equation dz/dt = (A + B(t))z, belong more to the theory of linear differential and difference equations than to the theory of matrices per se. For a discussion of the stability of linear differential systems with random coefficients, see ©. Sef, On Stability of a Randomized Linear System, Sei. Sinica, vol. 7, pp. 1027-1034, 1958. §. Sninivasan and R. Vasodevan, Linear Differential Equations with Random Coefficients, American Elsevier Publishing Company, Inc., New York, 1970 (to appear). J. C. Samuels, On the Mean Square Stability of Random Linear Systems, IRE Trans. on Circuit Theory, May, 1959, Special Supple- ment (Transactions of the 1959 International Symposium on Circuit. and Information Theory), pp. 248-259. where many additional references to earlier work by Rosenbloom and others will be found.14 Markoff Matrices and Probability Theory 1. Introduction. In this and the following chapters, which constitute the last third of the book, we wish to study a class of matrices which are generated by some fundamental questions of probability theory and mathematical economics. The techniques we shall employ here are quite different from those utilized in the study of quadratic forms, or in con- nection with differential equations. ‘The basic concept is now that of non-negativity, non-negative matrices and non-negative vectors. A matrix M = (mj) is said to be non-negative if my > O for alliandj. Similarly, a vector z is said to be non-negative if all its components are non-negative, x; > 0. ‘The subclass of non-negative matrices for which the stronger condition my > 0 holds is called positive. These matrices possess particularly interesting and elegant properties. The terms “positive” and “non-negative” are frequently used to describe what we have previously called “positive definite” and “non- negative definite.” Since the two types of matrices will not appear together in what follows, we feel that there is no danger of confusion, The adjective “positive” is such a useful and descriptive term that it is understandable that it should be a bit overworked. We have restricted ourselves in this volume to a discussion of the more elementary properties of non-negative matrices. Detailed discussions, which would require separate volumes, either in connection with proba- bility theory or mathematical economics, will be found in references at the end of the chapter. This chapter, and the one following, will be devoted to stochastic processes, while the next chapter will cover various aspects of matrix theory and mathematical economics, 2, A Simple Stochastic Process. In order to set the stage for the entrance of Markoff matrices, let us review the concept of a delerministic process. Consider a system S which is changing over time in such a way that its state at any instant ¢ can be described in terms of a finite 263264 Introduction to Matrix Analysis dimensional vector z, Assume further, that the state at any time s + , 8 > 0, can be expressed as a predetermined function of the state at time ¢, and the elapsed time s, namely, 2(8 + t) = g(2(t),8) qa) Under reasonable conditions of continuity on the function g(t), we can derive, by expansion of both sides in powers of s, a differential equation for x(t), of the form # = Mew) @ Finite-dimensional deterministic systems of the foregoing type are thus equivalent to systems governed by ordinary differential equations. If we introduce functions of more complicated type which depend upon the past history of the process, then more complicated functional equations than (2) result. ‘The assumption that the present state of the system completely deter- mines the future states is a very strong one. It is clear that it must always fail to some degree in any realistic process. If the deviation is slight, we keep the deterministic model because of its conceptual sim- plicity. It is, however, essential that other types of mathematical models be constructed, since many phenomena cannot, at the present time, be explained in the foregoing terms. Let us then consider the following stochastic process. To simplify the formulation, and to avoid a number of thorny questions which arise otherwise, we shall suppose that we are investigating a physical system S which can exist only in one of a finite number of states, and which can change its state only at discrete points in time. Let the states be designated by the integers 1, 2, , N, and the times by ¢=0, 1, 2,.... To introduce the random element, we assume that there is a fixed probability that a system in stage j at time ¢ will transform into state iat time ¢ + 1. This is, of course, again a very strong assumption of regularity. A “random process” in mathematical parlance is not at all what we ordinarily think of as a “random process” in ordinary verbal terms, Pursuant to the above, let us then introduce the transition matriz M = (my), where my = the probability that a system in state j at time ¢ will be in state ¢ at time ¢+ 1 Observe that we take M to be independent of time. This is the most important and interesting case. In view of the way in which M has been introduced, it is clear that we wish to impose the following conditions:Markoff Matrices and Probability Theory 265 m; 20 (8a) x Ymgat f212..., (38) i That my is non-negative is an obvious requirement that it be con- sidered a probability. The second condition expresses the fact that a particle in state j at time ¢ must be somewhere, which if to say in one of the allowable states, at time ¢ + 1. 3. Markoff Matrices and Probability Vectors. Let us now introduce some notation. A matrix M whose elements satisfy the restrictions of (2.4) will be called a Markoff matriz. A vector z whose components 2; satisfy the conditions u20 (la) x » mel (1b) will be called a probability vector. Generally, a veetor all of whose com- ponents are positive will be called a positive vector. EXERCISES 1, Prove that AP + (1 — )Qis a Markoff matrix for 0 << 1 whenever P and Q are Markoff matrices. 2. Prove that PQ is a Markoff matrix under similar assumptio: 8. Prove that Mzis a probability vector whenever 2 is a probability vector and Mf ig. a Markoff matrix. 4. Prove that Markoff matrices can be characterized in the following way: A matrix M is a Markoff matrix if and only if Mz is a probability vector whenever 2 is a probability vector. 8. Prove that if Mf’ is the transpose of a Markoff matrix M and M’ = (ai;), then x Do =, for = 1,2, oo. i 6. Prove that a positive Markoff matrix transforms non-trivial vectors with non- negative components into vectors with positive components, 4. Analytic Formulation of Discrete Markoff Processes, A stochastic process of the type described in Sec. 2 is usually called a discrete Markoff process, Let, us now see how this process can be described in analytic terms. Since the state of the system S at any time ¢ is a random variable, assuming any one of the values 7 = 1, 2,... , N, we introduce the V functions of { defined as follows: 2,(t) = the probability that the system is in state zat timet (1)266 Tntroduction to Matrix Analysis At t = 0 we have the relation 26(0) = 8 (2) where k is the initial state of S. The following relations must then be valid: x ati) = Ym) ¢=1,2,...,0 @) del One aspect of the problem of predicting regular behavior of $ is that of studying the behavior of the solutions of the system of equations in (3), which we may write more compactly in the form at+l)=Mz® ¢=0,1,2,... (4) The full problem is one of great complexity and interest, with analytic, algebraic, topological, and physical overtones. A number of monographs have been written on this topic, and there seems to be an unlimited area for research. It turns out that particularly simple and elegant results can be obtained in the case in which the elements of M are all positive. Consequently, we shall restrict ourselves principally to a discussion of this case by several methods, and only lightly touch upon the general case. EXERCISE 1. Prove that z(t) = M'z(0), z(t + s) = M'z(s). 5. Asymptotic Behavior. ‘The next few sections will be devoted to two proofs of the following remarkable result: Theorem 1. If M is a positive Markoff matrix and if x(t) satisfies (4) of See. 4, then lim z(t) = y, where y is a probability vector, (1a) ie y is independent of x(0), (18) y isa characteristic vector of M with associated characteristic root! (Le) That z(t) should settle down to a fixed probability vector y is interest- ing and perhaps not unexpected in view of the mixing property implied in the assumption m; > 0. That this limit should be independent of the initial state is certainly surprising, 6. First Proof. We shall present two proofs in this chapter. A third proof can be derived as a consequence of results obtained for general positive matrices in Chap. 16, The first proof, which we present first because of its simplicity, illustrates a point we have stressed before,Markoff Matrices and Probability Theory 267 namely, the usefulness of considering transformations together with their adjoints, Since ¢ assumes only the discrete sequence of values 0, 1, 2,..., let us replace it by n and speak of x(n) in place of 2(f). To establish the fact that x(n) has a limit as n — ©, we consider the inner product of z(n) with an arbitrary vector b. Since, by virtue of (4.4), z(n) = M*z(0), ‘we see, upon setting 2(0) = ¢, that (a(n),b) = (Mre,b) = (¢,(M»)’b) = (c,(M’)"b) ql) where M’ is the transpose of M. If we show that (Jf’)") converges as n— ©, for any given b, it will follow that z(n) converges as n — ©, since we can choose for b first the vector all of whose components are zero, except for the first, then the vector all of whose components are zero except for the second, and so on. Let us then introduce the vector 2(n) = (M’)*b (2) which satisfies the difference equation 2(n + 1) = M’e(n) 2(0) = 6 (3) Let, for each n, u(n) be the component of z(n) of largest value and v(n) the component of 2(n) of smallest value. We shall show that the properties of M’ imply that u(n) — v(x) > Oasn— «. Since x a(n + 1) =) mys(n) ® x and since )' my = 1, my 2 0, we see that ms u(n + bt) S un) v(n + 1) > v(n) Since {u(n)} is a monotone decreasing sequence, bounded from below by zero, and {v(n)} is a monotone increasing sequence, bounded from above by one, we see that both sequences converge. Let un) un) (6) (6) ‘To show that u =v, which will yield the desired convergence, we pro- ceed as follows. Using the component form of (3), we see that u(n + 1) < (1 — d)u(n) + do(n) ” v(n + 1) > (1 = d)v(n) + du(n) where d is the assumed positive lower bound for mi, 4,7 = 1,2,...,N.268 Introduction to Matrix Analysis From (7) we derive u(n + 1) — o(n + 1) < [(1 — d)u(n) + dv(n) — 1 — d)v(n) — du(n)] S (1 — 2d)(u(n) — v@)) 8) We see that (8) yields u(n) — v(n) < (1 — 2d)"(u(0) — v(0)) (9) and hence, since d < }4 if N > 2, that u(n) — v(n) > 0asn— @, Thus, not only does 2(n) converge as n—> ©, but it converges to a vector all of whose components are equal. From the convergence of 2(n) we deduce immediately the convergence of x(n), while the equality of the components will yield, as we shall see, the independence of initial value. Let lim z(n) =z and lim a(n) = y. As we know, all of the compo- nents of z are equal. Let us call each of these components a). Then (yb) = lim (2(n),b) = (¢,2) = ailer + cz + ++ +n] =a, (10) where a; is a quantity dependent only upon b, It follows that y is inde- pendent of c. That y is a characteristic vector of M with characteristic root 1 follows ensily. We have y = lim Mc = M lim Mre = My (11) In other words, y is a “fixed point” of the transformation represented by M. EXERCISE 1. Use the fact that 7 is a non-negative Markoff matrix to show that conditions of Positivity cannot be completely relaxed. 7. Second Proof of Independence of Initial State. Another method for establishing independence of the initial state is the following. We already know that lim M"c exists for any initial probability vector c. Let y and z be two limits corresponding to different initial probability vectors ¢ and d, respectively. Choose the scalar ¢, so that y — tz has at least one zero component, with all other components positive. As we know, M(y — tz) must then be a positive vector if y — tz is nontrivial. However, M(y — tz) = My — Mz =y — he a a contradiction unless y — ts = 0. ‘This means that f, = 1, since y and z are both probability vectors.Markoff Matrices and Probability Theory 269 8, Some Properties of Positive Markoff Matrices. Combining the results and techniques of the foregoing sections, we can derive some inter- esting results concerning positive Markoff matrices. To begin with, what we have shown by means of the preceding argument, is that a positive Markoff matrix cannot possess two linearly independent positive characteristic vectors associated with the characteristic root one. Furthermore, the same argument shows that a positive Markoff matrix cannot possess any characteristic vector associated with the character- istic root one which is linearly independent of the probability vector y found above. For, let z be such a vector.’ Then for ¢; sufficiently large, z+ tay is a positive vector. Dividing each component by the scalar ty = (2 + ty, ©), where c is a vector all of whose components are ones, we obtain a probability vector. Hence (z+ tiy)/ts = y, which means that z and y are actually linearly dependent. It is easy to see that no characteristic root of M can exceed one in absolute value. For, let x be a characteristic vector of M’, \ bea charac- teristic root, and m be the absolute value of a component of z of greatest magnitude. Then the relation Az = M’z shows that x im ©, we see that Mwtiztal+iy)=M(wteytietey)) (4) converges, because of the positivity of w + ey and z + ey, to a certain scalar multiple of y. The vector y*(w + iz), however, converges as n—> © only if » = 1, if w is constrained to have absolute value one. This completes the proof. We see then that we have established Theorem 2, ‘It is readily seen that it suffices to take ¢ real.270 Tntroduction to Matrix Analysis Theorem 2. If M is a positive Markoff matrix, the characteristic root of largest absolute value is 1. Any characteristic vector associated with this characteristic root i a scalar multiple of a probability vector. EXERCISE 1. If dia @ characteristic root with an associated characteristic veotor which is positive, then \ = 1. 9 Second Proof of Limiting Behavior. Let us now turn to a second direct proof of the fact that x(n) has a limit asn— ©. Writing out the equations connecting the components of z(n) and z(n + 1), we have x a(n +1) =) moan) a dm This yields x a(n + 1) ~ an) = Y-maledn) — an — 1) @) fs i x Since d x(n) = 1 for all n, it is impossible to have x(n) > 2(n — 1) i for all j, unless we have equality. In this case x(n) = 2(n — 1), a rela- tion which yields 2(m) = x(n — 1) for m > n, and thus the desired convergence. In any case, for each n let S(n) denote the set of j for which x(n) > x(n — 1) and T(n) the set of j for which zj(n) < a(n — 1). The com- ment in the preceding paragraph shows that there is no loss of generality in assuming that S(n) and T(n) possess at least one element for each n. Referring to (2), we see that d melaj(n) — a(n — I) < zn + 1) — a(n) ‘GT (n) < meglz,(n) — zn — 1) (3) ito Hence Yn +1) - adm < } { ¥ ms} tem) - am - D1 48 (n+1) GeS(n) eS(n$ 1) which yields, since mi > d > 0 for all ¢ and j, fain + Yat Sa) DY bam) — a(n 1) a(n) 55m)Markoff Matrices and Probability Theory 271 Similarly, summing over #eT(n + 1), we have a=4d Y am -am-nis JS dnt — 20m] © HeTin) eT (nt1) Combining (5) and (6), we see that x x D +) — am) $d) Y iam) -aXn- 9) @) A A and henee that the series )° |jx(n) — 2(n — 1)|| converges. con This completes the proof of convergence of x(n). EXERCISES 1. If M is a Markoff matrix and Mf? is a positive Markoff matrix, show that the sequences |M?c} and (M?*'c} converge. Does [M*c} converge? 2. What is the limit of Mf* if M is a positive Markoff matrix? 10. General Markoff Matrices. As mentioned above, the study of the general theory of Markoff matrices is quite complex, and best carried on from a background of probability theory. In order to see how complex roots of absolute value 1 can arise, con- sider a system with only two states, 1 and 2, in which state 1 is always transformed into state 2 and state 2 into state 1. The corresponding transition matrix is then M= [ al QQ) 1¢ which has the characteristic roots A; = 1, 42 = —1. By considering cyclic situations of higher order, we can obtain charac- teristic roots which are roots of unity of arbitrary order. Let us, however, discuss briefly an important result which asserts that even in the case where there is no unique limiting behavior, there is an average limiting behavior. Let us suppose for the moment that M has only simple characteristic roots, so that we may write dw,272 Introduction to Matrix Analysis Then > Ae/n 0 ro » ait/n (Sones ma ae ee If [Ad <1, we have ) At/n—+0asn— «. If /Aj| = 1, the limit ro lim Y t/n = a (4) me eet exists. If \; = e', where @/2m is an integer, then a; = 1; if @/2m is not an integer, then aj = 0. In any case, under the assumption of distinct characteristic roots, we can assert the existence of a quasi-limiting state -. y= fim eb ves tis) oy which is a characteristic vector of M, that is, My = y. The limit veo- tor y is a probability vector if x is a probability vector. ‘The general case can be discussed along similar, but more complicated, lines; see the exercises at the end of the chapter. EXERCISE 4, It \ is o characteristic root of a Markoff matrix M, with the property that 1] = 1, must » be @ root of unity? 11. A Continuous Stochastic Process. Let us now consider the stochastic process described in Sec. 2 under the assumption that the sys- tem is observed continuously. We begin with the discrete process in which observations are made at the times ¢ = 0, A, 24,.... Since we want the continuous process to be meaningful, we introduce a con-Markoff Matrices and Probability Theory 273 tinuity hypothesis in the form of a statement thet the probability of the system remaining in the same state over any time interval of length A is 1 - 0(4). ‘To make this statement precise, we define the following quantities: a; = the probability that S is in state ¢ at t + 4, given that it is in state j at time t, for i j q@) 1 — ay = the probabi that S is in state 7 at t + A, given that it is in state 7 at time ¢ ‘The quantities a;; are assumed to satisfy the following conditions: ay 20 (2a) ass = ) a (2b) Then the equations governing the quantities x,(f) are nt +4) = (1 aida) +4 Yaz) §=1,2,...,N @) i for! =0,4,24,.... Writing, in a purely formal way, a(t + A) = x(t) + dz,(t) + 0(4?) (4) and letting 4 — 0 in (2), we obtain the system of differential equations Ge ayes + y ast, (0) = ©) i i=1,2,...,N,wherec;,? = 1,2, ... ,N, are the initial probabilities, We shall bypass here all the thorny conceptual problems connected with continuous stochastic processes by defining our process in terms of this system of differential equations. In order for this to be an oper- ationally useful technique, we must show that the functions generated (5) act like probabilities. This we shall do below. There are a number of interesting questions arising in this way which we shall not treat here. Some of these are: 1. How does one define a continuous stochastic process directly and derive the differential equations of (5) directly? 2, In what sense can the continuous stochastic process defined by (5) be considered the limit of the discrete stochastic process defined by (3)? Since these are problems within the sphere of probability theory, we shall content ourselves with mentioning them, and restrain ourselves here to the matrix aspects of (5). As usual, however, we shall not scruple to use these ideas to guide our analysis.274 Introduction to Matrix Analysis 12, Proof of Probabilistic Behavior. Let us now demonstrate that the system of differential equations Gm ant Yas nee ¢=1,2...,¥ @ i where (2) (2) (2c) produces a set of functions satiafying the relations 2) 20 t20 (3a) Ya -1 (3b) i” Let us demonstrate (3b) first. From (1) we have a()+) ~ J (ant Ya) =0 ® by virtue of (2c). Hence, for all ¢, 2 2) =) 240) = 1 (5) To show that 2; > 0, write (1) in the form 4 west) = erst » Gijty im Since 2; > 0 at ¢ = 0, this shows that the quantities ze are monotone increasing. In place of following this procedure, we could use the easily established fact that the solutions of (1) are the limits of the solutions of (11.3) as A— 0. Hence, properties of the solutions of (11.3) valid for all A must carry over to the solutions of (1). We recommend that the reader sup- ply a rigorous proof. 13. Generalized Probabilities—Unitary Transformations. Any set of non-negative quantities {z,} satisfying the conditions y HzO (la) mal (1b) *Markoff Matrices and Probability Theory 275 ies, where 2; is the proba- can be considered to represent a set of probabi bility that a system S is in state i. A set of transformations 2 = get... tv) $=142,...,N @ which preserves the relations in (1) can then be considered to represent the effects of certain physical transformations upon S. ‘The simplest such transformations are the linear, homogeneous transformations a=) ayy @) ‘As we know, a necessary and sufficient condition that the relations in (1) be preserved is that A = (aj) be a Markoff matri: Let us now consider a quadratic transformation. Let x be a complex vector with components 2; and let r= Tr (4) where 7’ is a unitary transformation. Then (z',z”) = (z,2). Hence, if we consider |z,|", |z2|*, . . . , |zw|? to represent a set of proba- bilities, we see that |zx'|?, |zsl?, . . . , [z'v|? can also be considered to be a set of probabilities. We leave it to the reader to establish the relevant limit theorems for the sequence {z(n)} defined recurrently by x(n + 1) = Tx(n) (6) where 7’ is unitary and (z(0),2(0)) = 1 14. Generalized Probabilities—Matrix Transformations. Let us now generalize the concept of probabilities in the following fashion. If {Xi} is @ finite set of symmetric matrices satisfying the conditions X20 §=1,2...,N (1a) x Dea =t (1b) we shall call them a set of probabilities. The condition X; > 0 here signifies that X; is non-negative definite. Consider the transformation x Y= YAWKAy §=12...,¥ 2)276 Introduction to Matrix Analysis It is easy to establish the following analogue of the result concerning Markoff matrices. Theorem 3, A necessary and sufficient condition that the transformation in (2) preserve the relations in (1) is that x AyAg =I f= 1,2...,N @) mt Although analogues of the limit theorems established for the usual Markoff transformations can be established for the transformations of (2), the results are more difficult to establish, and we shall therefore not discuss these matters any further here. EXERCISES 1, Let Mf be a Markoff matrix. Consider a proof of the fact that lim > Mn ne ye exista along the following lines. Using the Schur transformation, we can write ‘eth bus bw eis where the @ are realand |A| <1,é—k+1,...,N. Tt suffices to consider the iterates of the triangular matrix. “To do this, consider the system of equations zi(n + 1) = effzy(n) + breza(n) + + buvzw(n) a(n +1) = eitiz,(n) + + bane (n) an(n + 1) = dwaw(n) n=0, 1,2 .... Show that |zi(n)| Scr, 0 1 if $10) = 1, do() = + do (W. Ledermann and G. E. Reuter 2. Thus, or otherwise, show that the characteristic roots are A? are distinct and negative or zero, and separate those of A‘ (W, Ledermann and @. , Reuter). 8. In the notation of Exercise 27 of the Miscellaneous Exercises of Chap. 4, is A) symmetrizable? 4. By 9 permutation matriz P of order N, we mean a matrix possessing exactly one nonsero element of value one in each row and column, Show that there are NI Permutation matrices of order . 5. Prove that the product of two permutation matrices is again a permutation matrix, and likewise the inverse. What are the possible values for the determinant of permutation matrix? 6. By a doubly stochastic matriz of order N we mean a matrix A whose elements satisfy the following conditions: (@) ay 20 ® Vapet §=1,%...,¥ } © Yaya G42..." Is the product of two doubly stochastic matrices doubly stochastic? 7. Prove that any doubly stochastic matrix A may be written in the form A=Jwr rata...,Nt m20 Punt where (P,} is the set of permutation niatrices of order N. The result is due to Birkhoff.! ‘Another proof, together with a number of applications to scheduling theory, may be found in T. C. Koopmans and M. J. Beckman.* 1G, Birkhoff, Tres observaciones sobre el algebra lineal, Rev. univ. nac, Tucumdn, ser. A, vol. 5, pp. 147-151, 1946. s and M. J. Beckman, Assignment Problems and the Location of 8, Econometrica, vol. 25, pp. 53-76, 1957.278 Introduction to Matrix Analysis For some further results concerning doubly stochastio matrices, see 8. Schreiber! and L. Mirsky. Mirsky’s paper contains a particularly simple proof of the Birkhoff results cited above and of an interesting connection between doubly stochastic matrices and the theory of inequalities. 8. Let A be a 2X2 Markoff matrix. Show that A is a square of » Markoff matrix if and only ifay > ays. 1. Hence, show that if A is a square of a Markoff matrix, it can be written as the 2 power of a Markoff matrix for n = 1, 2, 10. From this, conclude that if A has a square root which is a Markoff matrix, it has a root of every order which is a Markoff matrix. 11, From this, conclude that we can construct a family of Markoff matrices A(0) with the property that A(s + 0) = A(s)A(0), for s, ¢ > 0, A(0) = I, and A(1) = A. 12, Using this result, or otherwise, show that A = e®, where B haa the form e=[S f] enzo 18. If B is » matrix of the type described in Sec. 12, we know that e® is a Markoff matrix, Let By and By be two matrices of this type and let A; =e, Ar =e% Is AxAs 4 matrix of this type? 14. Let A be a Markoff matrix of dimension N with the property that it has a Toot of any order which is again a Markoff matrix. Let A, denote a particular 2"-th root which is a Markoff matrix, and write A(t) = An,Aa,. . . , when ¢ has the form t=2" +2 +4. , where the sum is finite, Can we define A(t) for0 ©. This depends upon the nature of the characteristic roots of A = (a,,), and, in particular, upon those of greatest absolute value. As we know from our discussion of Markoff matrices, this problem is one of great complexity. We suspect, however, that the problems may be quite simple in the special case where all the ay; are positive, and this is indeed the case. 3. Notation. As in Chap. 14, we shall call a matrix positive if all of its elements are positive. If A is positive, we shall write A >0. The notation A > B is equivalent to the statement that A — B > 0. Simi- larly, we may conceive of non-negative matrices, denoted by A > 0. This notation definitely conflicts with previous notation used in our discussion of positive definite matrices. At this point, we have the option of proceeding with caution, making sure that we are never dis- cussing both types of matrices at the same time, or of introducing a new notation, such as A >> B, or something equally barbarous. Of the alternatives, we prefer the one that uses the simpler notation, but which requires that the render keep in mind what is being discussed. This is, after all, the more desirable situation. i=1,2,..., N, determine the initial288 Introduction to Matrix Analysis ‘A vector z will be called positive if all of its components are positive, and non-negative if the components are merely non-negative. We shall write z > 0, and > O/inthe second. The relation z > y is equivalent tor-—y20. EXERCISES 1, Show that z > y, A > 0, imply that Az > Ay. 2. Prove that Ax > 0 for all z > 0 implies A > 0. 4. The Theorem of Perron, Let us now state the fundamental result of Perron. Theorem 1, if A is a positive matrix, there is a unique characteristic root of A, (A), which has greatest absolute value, This root is positive and simple, and its associated characteristic vector may be taken to be positive. There are many proofs of this result, of quite diverse origin, structure, and analytical level. ‘The.proof we shall give is in some ways the most important since it provides a variational formulation for A(A) which makes many of its properties immediate, and, in addition, can be extended to handle more general operators. 6. Proof of Theorem 1. Our proof of Theorem 1 will be given in the course of demonstrating Theorem 2. Theorem 2. Let A be a positive matrix and let X(A) be defined as above. Let S(d) be the set of non-negative d for which there exist non-negative vec- tors x such that Ax > dx. Let T(A) be the set of positive d for which there exist positive vectors z such that Az < dz. Then AA) = max A —_eS(A) =mind dP) 0) Proof of Theorem 2. Let us begin by normalizing all the vectors we shall consider, so that x tl = Y= @) mH This automatically excludes the null vector. Let us once more set 4 = 5 ay, If dx < Az, we have ifr Mlzll s or o< All [ell a< [All (3) This shows that S(A) is a bounded set, which is clearly not empty if A is positive.Positive Matrices and Mathematical Economics 289 Let Xo = Sup A for AeS(A); let {Ac} be a sequence of As in S(A) com verging to Ao; and let {2} be an associated set of vectors, which is to say me S Ae = 1,2... (4) Since ||z|] = 1,.we may choose a subsequence of the x, say (x} which converges to 2, a non-negative, nontrivial vector. Since dot < Ax, it follows that AocS(A), which means that the supremum is actually a maximum. Let us now demonstrate that the inequality is actually an equality, that is, rox = Ax, The proof is by contradiction. Let us suppose, without loss of generality, that ” y ayjt; — dot = d > 0 i (5) Danses — dots 2 0 /N fA where 2;, 4 = 1,2, .. . , N are the components of 2, Consider now the vector dy/2d0 0 w+] | @) 0 It follows from (5) that Ay > Ao. This, however, contradicts the maxi- mum property of hs. Hence d, = 0, and there must be equality in all the terms in (5). Consequently Xo is a characteristic root of A and 2 a charaeteristic vector which is necessarily positive. Let us now show that Ay = A(A). Assume that there exists a charac- teristic root ) of A for which |A| > do, with 2 an associated characteristic veetor. Then from Az = Xz we have In el < Alel @ where |z| denotes the vector whose components are the absolute values of the components of 2. It follows from this last inequality and the defi- nition of Xo, that |A| < Ao. Since |A| = Ao, the argument above shows that the inequality |A| jz] < Alz| must be an equality. Hence |Az| = Ale|; and thus 2 = cw, with w > 0 and c, a complex number. Consequently, Az = z is equivalent to Aw = dw; whence » is real and positive and hence equal to do.290 Introduction to Matrix Analysis To show that w is equivalent to z, let us show that apart from scalar multiples there is only one characteristic vector associated with Xo. Let # be another characteristic vector of A, not necessarily positive, associ- ated with X(A). Then x + ez, for all scalar ¢, is a characteristic vector of A. Varying ¢ about ¢ = 0, we reach a first value of ¢ for which one or several components of x + ez are zero, with the remaining components positive, provided that x and z are actually linearly independent. How- ever, the relation A(v + ez) = MA)(2 + e2) (8) shows that x + ez > 0 implies 2 + ez > 0. Thus a contradiction. EXERCISES 1. Show that A > B > 0 implies that (4) > X(B), 4. Show by means of 2 X 2 matrices that 4(4B) < (A)A(B) is not universally valid. 6, First Proof that \(A) Is Simple. Let us now turn to the proof that (A) is a simple characteristic root. We shall give two demonstrations, one based upon the Jordan canonical form and the other upon more elementary considerations. Assume that (A) is not a simple root. The Jordan canonical form shows that there exists a vector y with the property that (A - MA)IYy = 0 (A — MA) ty #0 Q) for’some k > 2. This means that (A — \(A)J)*-ly is a characteristic vector of A and hence a scalar multiple of c. By suitable choice of y we may take this multiple to be one. Hence 2m (4 — MAME y @) Now let = (A — MA) y (3) Then Az = MA)e + 24 > A(A)e a) This yields Ale| > d(A)|el, which contradicts the maximum definition of MA). 7. Second Proof of the Simplicity of A(A). Let us now present an alternate proof of the simplicity of (A) based upon the minimum property. We begin by proving the following lemma. Lemma. Let Ay be a positive N XN matriz (aj), and Ay any (N = 1) x (N = 1) matrix obtained by striking out an ith row and jthPositive Matrices and Mathematical Economics 291 column. Then \(Ay) > (Ana) a) Proof. Let us proceed by contradiction. Write Aw = MAw) Anna = A(Aw-a) Assume for the moment that Aw < dy, and take, without loss of generality, Ay-1 = (a,), i,j = 1,2,...,.N— 1. We have then the equations nat DY aay = Awe §=42...,N- Ly >0 (2) y and D ais = Ave, i=1,2,...,N,a>0 (3) i= Using the first N — 1 equations in (3), we obtain N= Gyj2; = Ant; — Gin mm = Aw(2i — aintw/Aw) < Anas (4) which contradicts the minimum property of Av—1. Let us now apply this result to show that \(A) is a simple root of {Q) =|A —X1| =0. Using the rule for differentiating a determinant, we obtain readily SQ) = —|Ar — AI] — [Ae — Mo — [Aw — AT (6) where by Ay we denote the matrix obtained from A by striking out the kth row and column. Since \(A) > (Ax) for each k, and each expres- sion |Ax — A/| is a polynomial in with the same leading term, each poly- nomial |A, — AZ| has the same sign at \ = (A). Hence, f’(A(A)) ¥ 0. 8. Proof of the Minimum Property of \(4). We have given above a proof of the maximum property of (A), and used, in what has preceded, the minimum property. Let us now present a proof of the minimum property which uses the maximum property rather than just repeating the steps already given. The technique we shall employ, that of using the adjoint operator, is one of the most important and useful in analysis, and one we have already met in the discussion of Markoff matrices. Let A’ be, as usual, the transpose of A. Since the characteristic roots of A and A’ are the same, we have \(A) = 4(A’). As we know, (Az,y) = (z,A'y). If Ay < Ay for some y > 0, we have for any z > 0, New) = Ay) = (A’2y) a)292 Introduction to Matrix Analysis Let # be a characteristic vector of A’ associated with X(A). Then Ae,y) 2 (A’ey) = MADE) @) Since (z,y) > 0, we obtain 4 > (A). This completes the proof of the minimum property. 9. An Equivalent Definition of \(A). In place of the definition of 4(A) given in Theorem 1 above, we may also define A(A) as follows. Theorem 3 (A) = max min { 5 aises/25} 7 ‘ gal = min max { 5 aie,/2i} a ee Here z varies over all non-negative vectors different from zero. The proof we leave as an exercise. 10, A Limit Theorem, From Theorem 1, we also derive the following important result. Theorem 4. Let c be any non-negative vector. Then v = lim Ane/X(A)" exists and is a characteristic vector of A associated with (A), unique up to a scalar multiple determined by the choice of c, but otherwise independent of the initial state c. This is an extension of the corresponding result proved in Chap, 14 for the special case of Markoff matrices. EXERCISE 1, Obtain the foregoing result from the special case established for Markoff matrices. 11, Steady-state Growth. Let us now see what the Perron theorem predicts for the asymptotic behavior of the growth process discussed in Sec, 2, We shall suppose that A = (a;) is a positive matrix. Then, as n—> ©, the asymptotic behavior of the vector 2(n), defined by the equation in (8), is given by the relation a(n) ~ dy ay where ) is the Perron root and 7 is a charaeteristic vector associated with \. We know that 7 is positive, and that it is a positive multiple of one particular normalized characteristic vector, say 5. ‘This normali- zation can be taken to be the condition that the sum of the components totals one. ‘The constant of proportionality connecting and 6 is thenPositive Matrices and Mathematical Economics 293 determined by the values of the initial components of c, cj, the initial population. What this means is that regardless of the initial population, as long as we have the complete mixing afforded by a positive matrix, asymptoti- cally as time increases we approach a steady-state situation in which the total population grows exponentially, but where the proportions of the various species remain constant. ‘As we shall see, the same phenomena are predicted by the linear models of economic systems we shall discuss subsequently. 12. Continuous Growth Processes. Returning to the mathematical model formulated in Sec. 2, let us suppose that the time intervals at which the system is observed get closer and closer together. The limiting case will be a continuous growth process. Let aA = the number of type i produced by a single item of type J in a time interval A, j # i w 1+ aud = the number of type ¢ produced by a single item of type i in a time interval 4 £=1,2,...,. ‘Then, we have, as above, the equations ai + 4) = (1+ and)n@ +4 > ay) i=1,2,...,N (2) Ke Observe that the a; are now rates of production. Proceeding formally to the limit as 4 — 0, we obtain the differential equations das Fm + Ya. f=1,2,...,¥ @) im We can now define the continuous growth process by means of these equations, with the proviso that this will be meaningful if and only if we can show that this process is in some sense the limit of the discrete process, The asymptotic behavior of the 2;(t) as t—> © is now determined by the characteristic roots of A of largest real part. 13. Analogue of Perron Theorem. Let us now demonstrate Theorem 5. Theorem 5. Jf aj >0 ivf @ then the rool of A with largest real part, which we shall call p(A), is real and simple. There is an associated posilive characteristic vector which is unique up to a mulliplicative constant.294 Introduction to Matrix Analysis Furthermore, (A) = max min { 5 ais,/2s} ef ist = min max { $ aziz} @ * jm Proof. The easiest way to establish this result is to derive it as a limiting ease of Perron’s theorem. Let p(A) denote a characteristic root of largest real part. It is clear that a root of e'4 of largest absolute value is given by deb) = etal (@) for § > 0. Since e'4 is a positive matrix under our assumptions, as fol- lows from Sec, 15 of Chap. 10, it follows that p(A) is real and positive, and simple. Using the variational characterization for A(e*4), we have x ee = max min (1+ 8) as,/z:) + 0(64) 4) _- i Letting 8 0, we obtain the desired representation. EXERCISES 1. If B > 0, and A is as above, show that p(A + B) > p(A). 4. Derive Theorem 5 directly from Perron’s theorem for s/ + A, where s has been chosen so that sJ + A is a positive matrix. 14. Nuclear Fission. The same type of matrix arises in connection with some simple models of nuclear fission. Here the N types of objects may be different types of elementary particles, or the same particle, say a neutron, in N different energy states. References to extensive work in this field will be found at the end of the chapter. 16, Mathematical Economics. As a simple model of an economic system, let us suppose that we have N different industries, the operation of which is dependent upon the state of the other industries. Assuming for the moment that the state of the system at each of the discrete times n=0,1,..., can be specified by a vector x(n), where the ith com- ponent, x(n), in some fashion describes the state of the ith industry, the statement concerning interrelation of the industries translates into a sys- tem of recurrence relations, or difference equations, of the following type: a(n +1) = ge(n),raln), . . . tw(n)) 5) with 20) =o,¢=1,2,...,¥.Positive Matrices and Mathematical Economics 295 This is, however, much too vague a formulation. In order to see how relations of this type actually arise, let us consider a simple model of three interdependent industries, which we shall call for identification purposes the “auto” industry, the “steel” industry, and the “tool” industry. Using the same type of “lumped-constant” approach that has been so successful in the study of electric circuits, we shall suppose that each of these industries can be specified at any time by its stockpile of raw material and by its capacity to produce new quantities using the raw materials available. Let ui troduce the following state variables: x(n) = number of autos produced up to time n x(n) = capacity of auto factories at time n za(n) = stockpile of steel at time n (a z4(n) = capacity of steel mills at time n x,(n) = stockpile of tools at time n a(n) apacity of tool factories at time n In order to obtain relations connecting the z,(n + 1) with the a(n), we must make some assumptions concerning the economic interdepend- ence of the three industries: To increase auto, steel, or tool capacity, we require only steel and tools. (3a) Production of autos requires only auto capacity and steel. (3b) Production of steel requires only steel capacity, (3c) Production of tuols requires only tool capacity and steel. (3d) The dynamies of the production process are the following: At the beginning of a time period, n to n + 1, quantities of stéel and tools, taken from their respective stockpiles, are allocated to the production of additional steel, tools, and autos, and to increasing existing capacities for production. Let 2(n) = the amount of steel allocated at time n for the purpose of increasing a(n) 1 =1,2,...,6 (4) w(n) = the amount of tools allocated at time » for the purpose of inereasing 2.(n) =12...,6 Referring to the assumptions in (3), we see that a(n) = 0 (5a) wi(n) = wr(n) = ws(n) = 0 (5b)296 Introduction to Matrix Analysis In order to obtain relations connecting 2.(n + 1) with the 2,(n) and 2(n) and w,(n), we must make some further assumptions concerning the relation between input and output. The simplest assumption is that we have a linear production process where the output of a product is directly proportional to input in shortest supply. Thus production is propor- tional to capacity whenever there is no constraint on raw materials, and proportional to the minimum raw material required whenever there is no limit to capacity. Processes of this type are called “bottleneck processes.” To obtain the equations describing the process, we use the principle of conservation. The quantity of an item at time n + 1 is the quantity at time n, less what has been used over [n, n + 1], plus what has been pro- duced over [n,n + 1]. The constraints on the choice of the 2; and w; are that we cannot allocate at any stage more than is available in the stockpiles, and fur- ther, that there is no point to allocating more raw materials than the productive capacity can accommodate. ‘The conservation equations are then, taking account of the bottleneck aspects, a(n + 1) = x(n) + min (y:22(n),cr21(n)) ay(n + 1) = 22(n) + min (a229(n),8.002(n)) a(n + 1) = ta(n) — 2n(n) — ax(n) — 2a(n) — es(n) — 26(n) + vard(n) (6) a(n + 1) = xan) + min (auea(n),Batra(n)) By(n + 1) = xo(n) — wa(n) — wan) — we(m) + min [ye20(n),cr425(n)] Le(n + 1) = zo(m) + min (aeze (12) ,Bewe(n)) where ay, 8;, yi are constants. The stockpile constraints on the choice of the z; and w; are a, wi 20 (7a) aterteateot ee Sts (7b) We + We + We S te (7e) Applying the capacity constraints, we can reduce the equations in (5) to linear form. We see that We (8a) as (88) Bowe (8c) este wee Beto Using these relations, we can eliminate the w; from (7) and obtain the linear equationsPositive Matrices and Mathematical Economics 297 a(n + 1) = 21(n) + onei(n),21(0) = 4 ay(n + 1) = 22(n) + agea(n),22(0) = c2 a(n + 1) = 2a(n) — 2n(n) — 22(n) — 24(n) — 2a(n) — een) + vat<(n),2a0) = cs (9) za(n + 1) = z4(n) + cxaza(n),24(0) = 4 as(n + 1) = x9(n) — esta(n) — eata(n) — eze(n) + aszs(n) «= ai/B, 24(0) =e ae(n + 1) = ae(n) + aeze(n),26(0) = cs The constraints on the choice of the 2; are now 220 (10a) atatatatacz (106) yita + reat vets S os (10c) aS fata (10a) a S fore (10e) ‘Suppose that we satisfy these conditions by means of a choice 2. = Cel, te uy oe = Cate, (ul) where the scalars és, ¢4, and e¢ are chosen to satisfy (10c). aft. 2 = fete (12) and suppose that (108) is satisfied ‘Then the foregoing equations have the form an $1) = (1 adedn) + Y ayn(n) §=1,2,...,6 (13) where a; > 0 116 (14) The continuous version of these equations is Gf = anit Yom #=1,2,.. ,6 (15) int What we have wished to emphasize is that a detailed discussion of the economic model shows that a special type of matrix, sometimes called an “‘input-output” matrix, plays a paramount role, Unfortunately, in the most interesting and important cases we cannot replace (14) by the stronger condition of positivity. ‘The result is that the study of the asymptotic behavior of the x(n) as given by a linear recurrence relation of the form appearing in (13) is quite complex. References to a great deal of research will be found at the end of the chapter.298 Introduction to Matrix Analysis 16, Minkowski-Leontieff Matrices, Let us now discuss a particular class of non-negative matrices specified by the conditions y OS 8 (a) Da<1 (1b) They arise in connection with the solution of linear equations of the form N m= Yaa tm i 12..-.4 (2) which occur in the treatment of models of interindustry production processes quite similar to that formulated above. Let us demonstrate Theorem 6. Theorem 6 If 0 < ay and x Yas 0, then (I — A)-" is positive, Proof. Toshow that (J — A)-1is a positive matrix under the assump- tion that a; > 0, it is sufficient to show that its transpose is positive, which is to say that (J — A’)-' is positive. Consider then the adjoint system of equations x, n=) og tu i=1,2,...,N (4) A It is easy to see that the solution obtained by direct iteration con- verges, in view of the condition in (3), and that this solution is positive if w>Ofori=1,2,...,N. The assumption that a;; > 0 shows that z; is positive whenever w is a nontrivial non-negative vector. The fact that (J — A)-' exists and is positive establishes the first part of the theorem. 17, Positivity of J — A]. Since the linear system in (16.2) has a unique solution for all y; under the stated conditions, it follows that |I — A] #0. To show that |7 — A| > 0, let us use the method of continuity as in Sec. 4 of Chap. 4. If \ > 0, the matrix AA satisfies the same conditions as A, Thus |J — AA| is nonzero for 0 << 1. Since the determinant is positive at \ = 0 and continuous in A, it is positive at \ = 1.Positive Matrices and Mathematical Economics 299 x 18, Strengthening of Theorem 6, It is clear that if ) aj = 1 for all & j, then 1 is a characteristic root of A, whence |f— A] =0. On the other hand, it is reasonable to suppose that the condition Jay < 1 can 7 be relaxed. It suffices to assume that 1>ay>0 (la) Das < 1 for at least one j (1b) 7 Day < 1 for all j (Ie) To prove this, we show that A? is a matrix for which (1b) holds for all j. 19. Linear Programming. In the first part of this book, we considered the problem of maximizing quadratic forms subject to quadratic con- straints, and the problem of maximizing quadratic forms subject to linear constraints, We have, however, carefully avoided any questions involv- ing the maximization of linear forms subject to linear constraints. A typical problem of this type would require the maximization of N Lee) = Yeats () A subject to constraints of the form x Dati Sb §=1,2,...,M (2a) n>0 (20) The reader will speedily find that none of the techniques discussed in the first part of the book, devoted to quadratic forms, are of any avail here. Questions of this type are indeed part of a new domain, the theory of linear inequalities, which plays an important role in many investiga- tions. The theorem of Perron, established above, is closely related, as far as result and method of proof, to various parts of this theory. In addition to establishing results concerning the existence and nature of solutions of the problem posed above, it is important to develop vari- ous algorithms for numerical solution of this problem. This part of the general theory of linear inequalities is called linear programming.300 Introduction to Matrix Analysis Let us now present a very simple example of the way problems of this type arise in mathematical economies. Suppose that we possess quanti- ties 21, 22, . .. , zu of M different resources, e.g., men, machines, and money, and that we can utilize various amounts of these resources in connection with N different activities, e.g,, drilling for ail, building auto- mobiles, and 80 on. Let a; = the quantity of the ith resource alloeated to the jth activity (8) so that x Yas F=4%...,M (4a) =k 420 (4b) Suppose, and this is usually a crude approximation, that the utility of allocating 2y is determined by simple proportionality, which is to say that the utility is ay. Making the further assumption that these utilities are additive, we arrive at the problem of maximizing the linear form L@) = Y ayes 6) subject to the constraints of (2). As another example of the way in which these linear variational prob- lems arise in mathematical economics, consider the model presented in Sec. 15. In place of assigning the quantities 2; and w; as we did there, we can ask for the values of those quantities which maximize the total output of steel over V stages of the process. 20. The Theory of Games. Suppose that two players, A and B, are engaged in a game of the following simple type. ‘The first player can make any of M different moves and the second player can make any of N different moves. If A makes the ith move and B the jth move, A receives the quantity a, and B receives —ay. The matrix A = (ai) qa) is called the payoff matrix. We now require that both players make their choices of moves simul- taneously, without knowledge of the other’s move. An umpire then examines both moves and determines the amounts received by the players according to the rule stated above. Suppose that this situation is repeated over and over. In general, it will not be advantageous for cither playor to make the same choice at each play of the game, To guard himself against his opponent takingPositive Matrices and Mathematical Economics 301 advantage of any advance information, each player will randomize over choices. Let 2; = probability that A makes the ith choice, i yj = probability that B makes the jth choice, j Then the expected quantity received by A after any particular play is Seu) = 2 ara @) The problem is to determine how the 7; and y; should be chosen. A can reason in the following way: “Suppose that B knew my choice. ‘Then he would choose the yj 80 as to minimize f(z,y). Consequently, I will choose the 2; 80 as to maximize.” Proceeding in this fashion, the expected return to A will be v4 = max min f(z,y) (4) where the z and y regions are defined by my 20 (5a) Ya=Pun (6b) Similarly, B can guarantee that he cannot lose more than vp = min max f(z,y) (6) The fundamental result of the theory of games is that 14 = vs. This is the min-maz theorem of von Neumann. What is quite remarkable is that it can be shown that these problems and the problems arising in the theory of linear inequalities, as described in Sec. 19, are mathematically equivalent. This equivalence turns out to be an analytic translation of the duality inherent in N-dimensional Euclidean geometry. 21. A Markovian Decision Process. Let us now discuss some problems that arise in the theory of dynamic programming. Consider a physical system S which at any of the times¢=0,1,..., must be in one of the states Si, S:,..., Sy. Let x(n) = the probability that S is in state S, at time n,¢ = 1,2,...,N qd)302 Introduction to Matrix Analysis For each g, which represents a vector variable, let MQ) = (m(Q)) (2) represent a Markoff matrix. Instead of supposing, as in an ordinary Markoff process, that the proba- bility distribution at time n is determined by M(q)*z(0), for some fixed g, we shall suppose that q can change from stage to stage. Specifically, we shall suppose that this process is being supervised by someone who wants to maximize the probability that the system is in state I at any particular time. In place of the usual equations, we obtain the relations x a(n + 1) = max Y maqam) <7 @) aint 1) = Y ma(gran) a where q* is a value of g which maximizes the expression on the first line. We leave to the reader the quite interesting problem of determining simple conditions under which a “ateady~state” or equilibrium solution exists, In Sec. 22, we shall discuss @ more general system. 22. An Economic Model. Suppose that we have N different types of resources, Let x(n) = the quantity of the ith resource at time n a) Suppose that, as in the foregoing pages, the following linear relations exist: x nin +1) =) ax(a(n) @ im where q is a vector parameter, as before. If A(g) is a positive matrix, the Perron theorem determines the asymptotic behavior of the system. ‘Assume, however, as in See. 21, that the process is supervised by some- one who wishes to maximize each quantity of resource at each time. The new relations are then x a(n + 1) = max > ay(gan) i=1,2,...,N (3) ‘ ist The following generalization of the Perron theorem can then be obtained.Positive Matrices and Mathematical Economics 303 Theorem 7. If q runs over a set of values (qu, . . « qx) which allow the maximum to be obtained in (3) (4a) 0 0. This solution is unique up to a multipli- cative constant and d= max A(A(g)) (6) e Furthermore, as n—> ©, a(n) ~ ayid® 2) where ay depends on the initial values. The proof may be found in references cited at the end of the chapter. MISCELLANEOUS EXERCISES 1, Let A = (ai;) be a non-negative matrix. A necessary and sufficient condition that all characteristic roots of A be less than 1 in absolute value is that all the prineipal minors of J ~ A be positive (Metzler) 2. If A has all negative diagonal elements, and no negative off-diagonal elements, if D isa diagonal matrix, and if the real parts of the characteristic roots of both A and DA are negative, then the diagonal elements of D are positive (K. D. Arrow and A.C. Enthoven). 8. Consider a matrix Z(t) = (2,)(t)), where the 2;; possess the following properties: (@ %4>0 ® i 2qdl > 1 for some i © f, ‘Then there is @ positive vector z and a positive number so for which (ff zeva) == Furthermore, #eis the root of | i- fr few a| = O with greatest real part, and i yer dl < c for some a > 0 ‘a simple root (Bohnenblus!); see R. Bellman.’ 4. Let A = (ay), and a = lal — > Jail. Is >0, 151 0, For any arbitrary partition 0 = te < ti go [1 — (a/a + 1)|” as well as a term less than this quantity (Ky Fan). 8. Let A bea matrix of theform ai < 0,1 ¥ jy ayy + Sou =0,5=1,2...,N. a If Nis a characteristic root, then either 4 = O or Re (A) <0 (A. Brauer-O. Taussky). ” 9 Mug 20, 4) 20,4 #4, ) aay = 04, then Fa autm —@, +++ om an anti s+ aw Dyu) =| | 20 say aye +> + ax tun (Minkowski) 10. If aij 2 0, ij, and by > > 4s, then all cofactors of B bh —an aw -an bs ~aw Dy =Positive Matrices and Mathematical Economics 305 11, A real matrix X for which 2a > [zu is called on Hadamard matriz. Let me A and B both be Hadamard matrices. Then |A +B] > |A| + 1B (E. V. Hayna- worth).' See also A. Ostrowski Hadamard matrices play an important role in computational analysis. Matrices of this nature, and of more general type, the dominant matrices, enter in the study of m-part networks. See P. Slepian and L. Weinberg, Synthesis Applications of Paramount and Dominant Matrices, Hughes Research Laboratories, 1958, R. 8. Burington, On the Equivalence of Quadrics in m-affine n-space and Its Relation to the Equivalence of 2m-pole Networks, Trans. Am. Math. Soc., vol. 38, pp. 163-176, 1935, R. 8. Burington, R-matrices and Equivalent Networks, J’, J. Math. and Phys., vol. 16, pp, 85-103, 1937. 12, If lag S mlau F< hf 12%... Nylad < Mlaul, J >4i— 12... , N —1, then A is nonsingular if m/(1 +m)" < M(1+M)%, and m 0 i=1,%...,N i () A is nonsingular and all elements of A-t are non-negative (©) All principal minors of A are positive. M matrices were first introduced by Ostrowski, Sec also E. V. Haynsworth, Note on Bounds for Certain Determinants, Duke Math. J., vol. 24, pp. 313-320, 1957, ‘and by the same author, Bounds for Determinants with Dominant Main Diagonal, Duke Math. J., vol. 20, pp. 199-209, 1953, where other references may be found. 14. Let A and B be two positive matrices of ordet N, with AB + BA, and cagiven positive N-dimensional vector. Consider the vector zy = ZyZy-1 + +> ZsZie, where each Z; is either an A or a B. Suppose that the Z; are to be chosen to maximize the inner product (zy,b) where b is @ fixed vector. Define the function Inf) = wal (zw,b) Then file) = max ((Ac,b),(Be,b)) Jule) = max (fy-(Ae),fv-(Bo)) N= 2,3,... 1, V. Haynsworth, Bounds for Determinants with Dominant Main Diagonal, Duke Math, J., vol. 20, pp. 199-209, 1953. 2 A. Ostrowski, Note on Bounds for Some Determinants, Duke Math. J., vol. 22, Pp. 95-102, 1955. A. Ostrowski, Uber die Determinaten mit uberwiegender Haupt-Diagonale, Com- ment. Math. Helveticit, vol. 10, pp. 69-96, 1937-1938. +A, Ostrowski, On Nearly Triangular Matrices, J. Research Natl. Bur. Standards, vol. 52, pp. 319-345, 1954. “A. Ostrowski, Comment. Math. Helveticii, vol. 30, pp. 175-210, 1956; ibid., vol. 10, pp. 69-96, 1937.306 Introduction to Matrix Analysis 16, Does there exist a scalar d such that fy(e) ~ »%g(c)? 16. Consider the corresponding problem for the case where we wish to maximize the Perron root of ZyZn-1 ++ + ZZ. 11, Let A be a positive matrix. ‘There exist two positive diagonal matrices, D; and Ds, such that DADs is doubly stochastic (Sinkkorn; see Menon'). 18. Let A be a non-negative matrix. When does there exist a positive diagonal matrix such that DAD is positive and symmetric? (The question arose in some work by Pimbley in neutron transport theory. Sec Parter,? Parter and Youngs,* and Hearon.*) 19. Let A be a non-negative matrix, and let P(A) be the finite set of non-negative matrices obtained from A by permuting ite elements arbitrarily. Let BC P(A). (a) What are the maxima and minima of tr (B*)? (®) What are the maximum and minimum values of the Perron root of B? (B. Schware, Rearrangements of Square Matrices with Nonnegative Elements, University of Wisconsin, MRC Report #282, 1961.) 20. Let A be positive matrix. Prove that the Perron root satisfies the inequalities max{max aw (V — 1) min ay + min au) < MA) < (N — 1) max ay + max ay i ind (Sender) 21, Consider the differential equation 2’ = Q(z, 0 <¢ 0, inj. If ff vera © 8 8 0, where ¥(0) inf (qciais), then the equation has positive solution, unique ea up to a constant factor. (@. Birkhoff and L. Kolin, Linear Second-order Differential Equations of Positive Type, J. Analyse Math., yol. 18, pp. 43-52, 1967.) 22, Consider the matrix product C = A - B where ce; = —aijbi;, 4 5, cu Show that if A and B are M matrices (see Exercise 13), then C is an M matrix. many further deeper results concerning this matrix product introduced by Ky Fan, see Ky Fan, Inequalities for M-matrices, Zndag. Math., vol. 26, pp. 602-610, 1944. M matrices play an important role in the study of iterative methods for solving sys- tems of linear equations, and in the numerical solution of elliptic partial differential equations using difference methods; see Ostrowski® and Varga.* ‘The Fan product is analogous to the Schur product (ci; = aisbs,), previously intro- duced for positive definite matrices. In general, there is an amazing parallel between results for positive definite and M matrices. ‘This phenomenon, noted some time 1M. V, Menon, Reduction of a Matrix with Positive Elements to a Doubly Sto- chastic Matrix, Proc. Am. Math. Soc., vol. 18, pp. 244-247, 1967. +8, V. Parter, On the Eigenvalues and Eigenvectors of a Class of Matrices, S7AM Journal, to appear. 48, V. Parter and J. W. T. Youngs, The Symmetrization by Diagonal Matrices, J. Math. Anal. Appl., vol. 4, pp. 102-110, 1962. +5. Z. Hearon, The Kinetics of Linear Systems with Special Reference to Periodic Reactions, Bull. Math. Biophye., vol. 15, pp. 121-141, 1953. +A, M. Ostrowski, Iterative Solution of Linear Systems of Functional Equations, J. Math. Anal. Appl., vol. 2, pp. 351-369, 1961. #R.8. Varga, Matriz Herative Analysis, Pron 1962. Tall, Ine., Englewood Cliffs, N.J.,Positive Matrices and Mathematical Economics 307 ago by O. Taussky (and also by Gantmacher-Krein, and by D. M, Kotelyanskii, Math, Rev., vol. 12, p. 793), is discussed and analyzed in the following papers: Ky Fan, An Inequality for Subadditive Functions on a Distributive Lattice, with Applications to Determinantal Inequalities, Lin. Algebra Appl., vol. 1, pp. 33-38, 1966. Ky Fan, Subadditive Functions on a Distributive Lattice, and an Extension of Szasz’s Inequality, J. Math. Anal. Appl., vol. 18, pp. 262-268, 1967. Ky Fan, Some Matrix Inequalities, Abh. Math. Sem. Hamburg Univ., vol. 29, Pp. 185-196, 1966, 98, If A and B are M matrices such that B — A > 0, then {A + Blt > |Aji™ + 1BI™, @A + (1 — OB) < (1 — OB" for any ¢ in (0,1), Many further results are given in Ky Fan. 24. Call a non-negative matrix A irreducible if A” > 0 for some NV. Show that every irreducible A has a positive characteristic root f(A) which is at least as large as the absolute value of any other characteristic root. This characteristic root is simple and the characteristic vector can be taken to be positive (Frobenius). Sec Blackwell for a proof using the min-max theorcm of game theory. 25, A matrix A is said to be of monotone kind if Az > O implies that > 0. Show that A-! exists if A is of monotone kind, and A~1 > 0. See Collatz,* and for an extension to rectangular matrices see Mangasarian.* 96, Let A = (ay) be a Markov matrix, and let A~! = (b;). Under what con- ditions do we have bi; > 1, bi; $0? See Y. Uekawa, P-Malrices and Three Forms of the Stolper-Samuelson Criterion, Institute of Social and Eeonomic Research, Osaka University, 1968. 27. Let X > X*>0. Thon the real characteristic values of X must lie between land (1 — »/2)/2 (R. E. DeMarr). 28, Let B bea real square matrix such that by; < O fort sj and the leading minors of Bare positive. ‘Then B-' is a non-negative matrix. Hint: Use the representation ft eM dt = BM}, See J. Z. Hearon, The Washout Curve in Tracer Kinetics, Math. Biasci., vol. 3, pp. 31-39, 1968, Bibliography and Discussion §1. A very large number of interesting and significant analytic prob- lems arise naturally from the consideration of various mathematical models of economic interaction. Vor an interesting survey of mathe- matical questions in this field, see I. N. Herstein, Some Mathematical Methods and Techniques in Economics, Quart. Appl. Math., vol. 11, pp. 249-261, 1953. 1Ky Fan, Inequalities for the Sum of Two M-matrices, Inequalities, Academic Preas Ine., New York, pp- 105-117, 1967. *D. Blackwell, Minimax and Irreducitle Matrices, J. Math. Anal. Appl., vol. 3, pp. 33 1961 3L, Collatz, Functional Analysis and Numerical Mathematics, Academic Press., Ine,, New York, 1966. “0. L, Mangasarian, Characterization of Real Matrices of Monotone Kind, 8/4 Af Review, vol. 10, pp. 439-441, 1968.