See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/289442345

Matrix Theory and Linear Algebra

Book · September 2015

CITATIONS READS

0 24,056

1 author:

Athanasios I Margaris
University of Thessaly
51 PUBLICATIONS   107 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Neuroevolution strategy View project

All content following this page was uploaded by Athanasios I Margaris on 06 January 2016.

The user has requested enhancement of the downloaded file.

MATRIX THEORY AND LINEAR
ALGEBRA

Athanasios I. Margaris
BSc in Physics
MSc, PHD in Computer Science

TZIOLAS PUBLICATIONS
 Περιεχόµενα
Table of Contents

Preface i

1 SET THEORY AND ALGEBRAIC STRUCTURES 1

1.1 The concept of set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Intersection, union, difference and complement of sets . . . 6
1.1.2 Set operations and properties of them . . . . . . . . . . . . 8
1.1.3 Cartesian product, relations and functions . . . . . . . . . 9
1.1.4 Generalized intersection and union of sets . . . . . . . . . 13
1.1.5 Set algebra and σ-algebra . . . . . . . . . . . . . . . . . . . 14
1.1.6 Operations on sets elements . . . . . . . . . . . . . . . . . 14
1.2 Algebraic structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.2.1 Semigroup . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.2.2 Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.2.3 Ring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.2.4 Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
1.3 Numbers and set of numbers . . . . . . . . . . . . . . . . . . . . . . . 28
1.3.1 A historical perspective . . . . . . . . . . . . . . . . . . . . 29
1.3.2 The evolution of the concepts of zero and infinity . . . . . . 33
1.4 Natural numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
1.4.1 The method of mathematical induction . . . . . . . . . . . 42
1.5 Integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
1.5.1 Number theory . . . . . . . . . . . . . . . . . . . . . . . . . 51
1.6 Rational numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
1.6.1 Operations and ordering . . . . . . . . . . . . . . . . . . . 56
1.6.2 The properties of rational numbers . . . . . . . . . . . . . . 57
1.6.3 The decimal form of rational numbers . . . . . . . . . . . . 58
1.6.4 Countability and density . . . . . . . . . . . . . . . . . . . 61
1.7 Irrational and real numbers . . . . . . . . . . . . . . . . . . . . . . . . 63
1.7.1 Defining the irrational numbers . . . . . . . . . . . . . . . 64
1.7.2 Defining the real numbers . . . . . . . . . . . . . . . . . . 68
1.7.3 Types of irrational numbers . . . . . . . . . . . . . . . . . . 70
1.7.4 Properties of real numbers . . . . . . . . . . . . . . . . . . 71
1.7.5 Intervals of real numbers . . . . . . . . . . . . . . . . . . . 74
1.7.6 Real analysis - Lebesgue integral . . . . . . . . . . . . . . . 76
1.7.7 Sequences of real numbers . . . . . . . . . . . . . . . . . . 87
1.8 Complex numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

i
ii TABLE OF CONTENTS

1.8.1 Representation of complex numebrs . . . . . . . . . . . . . 99

1.8.2 The vector form of complex numbers . . . . . . . . . . . . . 100
1.8.3 Inner and outer product of complex numbers . . . . . . . . 104
1.8.4 Roots of complex numbers . . . . . . . . . . . . . . . . . . 105
1.8.5 Logarithms of complex numbers . . . . . . . . . . . . . . . 108

2 LINEAR VECTOR SPACES 109

2.1 Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
2.2 Defining the vector space . . . . . . . . . . . . . . . . . . . . . . . . . 112
2.3 Examples of vector spaces . . . . . . . . . . . . . . . . . . . . . . . . . 114
2.4 Ordered vector spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
2.4.1 Ordered sets and nets . . . . . . . . . . . . . . . . . . . . . 122
2.5 The spaces R2 and R3 - Analytic geometry . . . . . . . . . . . . . . . . 123
2.5.1 Conal sections . . . . . . . . . . . . . . . . . . . . . . . . . 125
2.5.2 Quadratic surfaces . . . . . . . . . . . . . . . . . . . . . . 129
2.6 Operators and geometric transformations . . . . . . . . . . . . . . . . 136
2.6.1 The algebra of linear operators . . . . . . . . . . . . . . . . 137
2.6.2 Inverse operators . . . . . . . . . . . . . . . . . . . . . . . 138
2.6.3 Matrices and linear operators . . . . . . . . . . . . . . . . . 140
2.6.4 Rotations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
2.6.5 Other linear geometric transformations . . . . . . . . . . . 152
2.7 The concept of subspace . . . . . . . . . . . . . . . . . . . . . . . . . 157
2.7.1 Complementary subspaces . . . . . . . . . . . . . . . . . . 158
2.7.2 Invariant subspaces . . . . . . . . . . . . . . . . . . . . . . 160
2.8 Linear dependence and independence . . . . . . . . . . . . . . . . . . 161
2.9 Linear span . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
2.10 The row space, the column space and the null space of a matrix . . . 169
2.11 Base and dimension of a vector space . . . . . . . . . . . . . . . . . . 171
2.11.1 The dimension of the sum of vector subspaces . . . . . . . 175
2.11.2 The properties of invariant subspaces . . . . . . . . . . . . 180
2.12 Vector projection and projection matrices . . . . . . . . . . . . . . . . 183
2.12.1 The properties of the projection matrix . . . . . . . . . . . . 185
2.13 Vector norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
2.13.1 Cauchy-Bunyakovskii-Schwarz (CBS) inequality . . . . . . 189
2.13.2 Triangular inequality . . . . . . . . . . . . . . . . . . . . . 190
2.13.3 The p-norm . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
2.13.4 Hölder’s inequality . . . . . . . . . . . . . . . . . . . . . . . 194
2.13.5 Minkowski’s inequality . . . . . . . . . . . . . . . . . . . . 195
2.13.6 Relations between vector norms . . . . . . . . . . . . . . . 197
2.14 Defining the inner product . . . . . . . . . . . . . . . . . . . . . . . . 199
2.14.1 The angle between vectors . . . . . . . . . . . . . . . . . . 200
2.14.2 The angle between complementary subspaces . . . . . . . . 200
2.14.3 Orthogonal and orthonormal vector sets . . . . . . . . . . . 201
2.14.4 Distance functions . . . . . . . . . . . . . . . . . . . . . . . 201
2.14.5 Returning to the CBS inequality . . . . . . . . . . . . . . . 202
2.15 Vector norms and inner products . . . . . . . . . . . . . . . . . . . . . 203
2.16 Gram-Schmidt Orthonormalization . . . . . . . . . . . . . . . . . . . . 207
TABLE OF CONTENTS iii

2.17 Orthogonal complementary subspaces . . . . . . . . . . . . . . . . . . 210

2.18 Metric and normed vector spaces . . . . . . . . . . . . . . . . . . . . . 212
2.18.1 Vector sequences and sequence convergence . . . . . . . . 214
2.18.2 The concept of continuity in metric and normed spaces . . 215
2.18.3 The continuity of the inner product . . . . . . . . . . . . . 216
2.18.4 The equivalence between vector norms . . . . . . . . . . . . 217
2.18.5 Sets and the diameter of them . . . . . . . . . . . . . . . . 219
2.18.6 Metric properties of vector spaces . . . . . . . . . . . . . . 221
2.18.7 Completeness of vector spaces . . . . . . . . . . . . . . . . 225
2.18.8 The theorem of Riesz-Fischer . . . . . . . . . . . . . . . . . 231
2.18.9 Compactness and related theorems . . . . . . . . . . . . . 233
2.19 Banach space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
2.19.1 A useful inequality . . . . . . . . . . . . . . . . . . . . . . . 237
2.19.2 A fundamental theorem . . . . . . . . . . . . . . . . . . . . 239
2.19.3 Vector sequences in Banach spaces . . . . . . . . . . . . . 239
2.19.4 Cauchy sums in Banach spaces . . . . . . . . . . . . . . . 240
2.20 Hilbert space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
2.20.1 The concept of orthogonality in Hilbert space . . . . . . . . 244
2.20.2 Cauchy sums in Hilbert spaces . . . . . . . . . . . . . . . . 250
2.20.3 Bessel inequality . . . . . . . . . . . . . . . . . . . . . . . . 250
2.20.4 Orthonormal bases . . . . . . . . . . . . . . . . . . . . . . 252
2.20.5 Parseval’s identity . . . . . . . . . . . . . . . . . . . . . . . 255
2.21 Applications of the infinite dimensional Hilbert space . . . . . . . . . . 256
2.21.1 Orthonormal bases in the vector space L2 ([−π, π]) . . . . . 256
2.21.2 Orthogonal polynomials . . . . . . . . . . . . . . . . . . . . 262
2.21.3 Vector approximations and normal equations . . . . . . . . 272
2.22 Linear varieties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
2.23 Linear transformations . . . . . . . . . . . . . . . . . . . . . . . . . . 278
2.23.1 Composite transformations . . . . . . . . . . . . . . . . . . 281
2.23.2 Euclidean transformations . . . . . . . . . . . . . . . . . . 281
2.23.3 The principle of superposition . . . . . . . . . . . . . . . . 284
2.23.4 Linear transformations and linear span . . . . . . . . . . . 285
2.23.5 The kernel and the image of a linear transformation . . . . 286
2.23.6 Isomorphisms, invertibility and automorphisms . . . . . . 293
2.23.7 Change of base in a vector space . . . . . . . . . . . . . . . 298
2.24 Similarity transformations . . . . . . . . . . . . . . . . . . . . . . . . 304
2.24.1 Similarity transformations and projection matrices . . . . . 307
2.24.2 Building the orthogonal projection matrices . . . . . . . . . 309
2.24.3 Unitary transformations . . . . . . . . . . . . . . . . . . . . 312
2.25 Linear transformations and normed spaces . . . . . . . . . . . . . . . 313
2.25.1 The vector space of bounded transformations . . . . . . . . 316
2.25.2 Compactness of transformations in normed spaces . . . . . 317
2.25.3 Linear functionals and dual spaces . . . . . . . . . . . . . 317
2.25.4 Riesz representation theorem . . . . . . . . . . . . . . . . . 319
2.25.5 Dual transformations . . . . . . . . . . . . . . . . . . . . . 319
2.25.6 The concept of optimization in Hilbert spaces . . . . . . . . 323
iv TABLE OF CONTENTS

3 MATRIX THEORY 327

3.1 The concept of matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
3.2 Row transformations and echelon form . . . . . . . . . . . . . . . . . 330
3.3 Elementary matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
3.3.1 Relating a matrix with its echelon form . . . . . . . . . . . 340
3.4 Diagonal, triangular and transpose matrices . . . . . . . . . . . . . . 346
3.5 Symmetric, conjugate, and hermitian matrices . . . . . . . . . . . . . 349
3.6 Matrix norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
3.6.1 Frobenius norm . . . . . . . . . . . . . . . . . . . . . . . . 356
3.6.2 Induced matrix norms . . . . . . . . . . . . . . . . . . . . . 357
3.6.3 Matrix norm of an inverse matrix . . . . . . . . . . . . . . . 365
3.6.4 Spectral norm of a diagonal matrix . . . . . . . . . . . . . . 366
3.6.5 Relationships between matrix norms . . . . . . . . . . . . . 367
3.6.6 Equivalence between matrix norms . . . . . . . . . . . . . 369
3.7 Matrix operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
3.7.1 Matrix addition . . . . . . . . . . . . . . . . . . . . . . . . 371
3.7.2 Scalar - matrix product . . . . . . . . . . . . . . . . . . . . 372
3.7.3 Matrix - matrix product . . . . . . . . . . . . . . . . . . . . 374
3.7.4 Vector - matrix product . . . . . . . . . . . . . . . . . . . . 383
3.7.5 Hadamard product . . . . . . . . . . . . . . . . . . . . . . 386
3.7.6 Kronecker product . . . . . . . . . . . . . . . . . . . . . . . 388
3.7.7 Kronecker sum . . . . . . . . . . . . . . . . . . . . . . . . . 397
3.7.8 Vectorizing matrices - the vec operator . . . . . . . . . . . . 398
3.7.9 Converting vectors to matrices - the diag operator . . . . . 402
3.8 Permutation matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
3.8.1 Building permutation matrices . . . . . . . . . . . . . . . . 407
3.8.2 Permutation matrices and Kronecker product . . . . . . . . 409
3.9 Orthogonal and unitary matrices . . . . . . . . . . . . . . . . . . . . . 412
3.10 Powers of square matrices . . . . . . . . . . . . . . . . . . . . . . . . . 418
3.11 Fundamental subspaces and the index of a matrix . . . . . . . . . . . 420
3.11.1 The rank of matrix product . . . . . . . . . . . . . . . . . . 423
3.11.2 The rank of matrix sum . . . . . . . . . . . . . . . . . . . . 425
3.11.3 Base and dimension of fundamental subspaces . . . . . . . 426
3.12 The trace of a square matrix . . . . . . . . . . . . . . . . . . . . . . . 432
3.12.1 The trace of Kronecker product . . . . . . . . . . . . . . . . 432
3.12.2 Matrix trace and matrix multiplication . . . . . . . . . . . . 433
3.12.3 Matrix trace and vectorization operator . . . . . . . . . . . 434
3.12.4 The trace of matrices AAT and AT A . . . . . . . . . . . . . 436
3.13 Matrix inner product . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
3.14 The concept of the inverse matrix . . . . . . . . . . . . . . . . . . . . . 438
3.14.1 Invertibility and elementary matrices . . . . . . . . . . . . 443
3.14.2 The formula of Sherman-Morrison . . . . . . . . . . . . . . 445

4 THE DETERMINANT OF A SQUARE MATRIX 449

4.1 Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
4.2 Introduction to the concept of determinant . . . . . . . . . . . . . . . 451
4.3 Principal minor and algebraic complement . . . . . . . . . . . . . . . 456
TABLE OF CONTENTS v

4.3.1 Row and column expansion of a determinant . . . . . . . . 458

4.3.2 Higher order minors and algebraic complements . . . . . . 463
4.3.3 Cofactor expansion . . . . . . . . . . . . . . . . . . . . . . 464
4.4 Laplace expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
4.5 Properties of determinants . . . . . . . . . . . . . . . . . . . . . . . . 469
4.6 Determinants and elementary matrices . . . . . . . . . . . . . . . . . 476
4.7 Applications of determinants . . . . . . . . . . . . . . . . . . . . . . . 479
4.8 Estimating the inverse matrix . . . . . . . . . . . . . . . . . . . . . . . 480
4.9 Jacobi identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484
4.10 Bordered determinants . . . . . . . . . . . . . . . . . . . . . . . . . . 486
4.11 Wronski determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
4.11.1 Derivatives of Wronski determinant . . . . . . . . . . . . . 494
4.11.2 Wronski determinant of special functions . . . . . . . . . . 495
4.11.3 The method of Wronski determinant . . . . . . . . . . . . . 504
4.11.4 Wronski determinant of a self-adjoint differential equation . 506
4.12 Other special determinants . . . . . . . . . . . . . . . . . . . . . . . . 507
4.12.1 The determinant of Vandermonde . . . . . . . . . . . . . . 507
4.12.2 Symmetric and antisymmetric determinants . . . . . . . . 511
4.12.3 Centrosymmetric determinants . . . . . . . . . . . . . . . . 519
4.12.4 The determinant of Toeplitz . . . . . . . . . . . . . . . . . . 520
4.12.5 Cyclic determinants . . . . . . . . . . . . . . . . . . . . . . 521
4.12.6 Hilbert matrix and its determinant . . . . . . . . . . . . . . 523
4.12.7 The determinant of Cauchy . . . . . . . . . . . . . . . . . . 527
4.12.8 The determinant of Casorati . . . . . . . . . . . . . . . . . 528
4.12.9 The determinant of Hankel . . . . . . . . . . . . . . . . . . 531
4.12.10 The determinant of Hessenberg . . . . . . . . . . . . . . . . 533
4.13 Representing polynomials as determinants . . . . . . . . . . . . . . . 534
4.14 Matrix partitioning and submatrices . . . . . . . . . . . . . . . . . . . 535
4.14.1 Determinant properties of complex matrices . . . . . . . . . 539
4.14.2 Schur complement . . . . . . . . . . . . . . . . . . . . . . . 539
4.14.3 Primary submatrices and their determinants . . . . . . . . 540
4.15 Matrix and determinant derivatives . . . . . . . . . . . . . . . . . . . 541
4.15.1 Derivative of a composite vector function . . . . . . . . . . 546
4.15.2 Properties and derivative relations . . . . . . . . . . . . . . 547
4.15.3 Properties of matrix derivatives . . . . . . . . . . . . . . . . 551
4.15.4 Matrix derivative with respect to matrix . . . . . . . . . . . 553
4.15.5 The derivative of a determinant with respect to matrix . . . 555
4.15.6 Derivatives of matrix traces . . . . . . . . . . . . . . . . . . 557
4.15.7 Derivatives of matrix norms . . . . . . . . . . . . . . . . . . 560
4.15.8 Second order derivatives . . . . . . . . . . . . . . . . . . . 561
4.15.9 Other matric and determinant functions . . . . . . . . . . . 565

5 LINEAR SYSTEMS AND MATRIX FACTORIZATION 571

5.1 Definitions and basic concepts . . . . . . . . . . . . . . . . . . . . . . 572
5.2 Matrix representation of linear systems . . . . . . . . . . . . . . . . . 574
5.3 Linear systems and linear dependence . . . . . . . . . . . . . . . . . . 576
5.4 Solving square linear systems . . . . . . . . . . . . . . . . . . . . . . . 578
vi TABLE OF CONTENTS

5.4.1 The method of substitution . . . . . . . . . . . . . . . . . . 579

5.4.2 The method of inverse matrix . . . . . . . . . . . . . . . . . 580
5.4.3 Cramer’s method . . . . . . . . . . . . . . . . . . . . . . . 581
5.4.4 Gauss elimination method . . . . . . . . . . . . . . . . . . 583
5.4.5 Gauss elimination method with pivoting . . . . . . . . . . . 591
5.4.6 Gauss elimination and the augmented matrix . . . . . . . . 601
5.4.7 Diagonally dominant matrices . . . . . . . . . . . . . . . . 604
5.4.8 Pivoting strategies . . . . . . . . . . . . . . . . . . . . . . . 606
5.4.9 Gauss-Jordan elimination method . . . . . . . . . . . . . . 613
5.4.10 Identifying the inverse matrix . . . . . . . . . . . . . . . . . 615
5.5 LU factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
5.5.1 Direct estimation of L and U matrices . . . . . . . . . . . . 620
5.5.2 LU factorization and pivoting . . . . . . . . . . . . . . . . . 621
5.5.3 Identifying determinants via LU factorization . . . . . . . . 623
5.5.4 LU factorization and linear systems . . . . . . . . . . . . . 624
5.5.5 LDU factorization . . . . . . . . . . . . . . . . . . . . . . . 625
5.6 Cholesky decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 626
5.6.1 Identifying determinants via Cholesky decomposition . . . 630
5.6.2 Cholesky decomposition and linear systems . . . . . . . . . 631
5.7 Solving non-square linear systems . . . . . . . . . . . . . . . . . . . . 632
5.7.1 Under-determined linear systems . . . . . . . . . . . . . . 635
5.7.2 Over-determined linear systems . . . . . . . . . . . . . . . 642
5.8 QR factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 646
5.8.1 QR factorization and linear systems . . . . . . . . . . . . . 650
5.8.2 Householder transformation . . . . . . . . . . . . . . . . . 650
5.8.3 Householder transformation and QR factorization . . . . . 656
5.8.4 QR factorization and least squares . . . . . . . . . . . . . . 665
5.9 URV factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667
5.9.1 Householder transformation and URV factorization . . . . . 674
5.10 SVD factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 680
5.10.1 Full and reduced SVD factorization . . . . . . . . . . . . . 684
5.10.2 Matrix rank and normalized eigenvectors . . . . . . . . . . 685
5.10.3 SVD factorization and geometric transformations . . . . . . 686
5.11 Condition number and linear systems . . . . . . . . . . . . . . . . . . 688
5.11.1 Variation of all system parameters . . . . . . . . . . . . . . 691
5.11.2 Geometric interpretation of the condition number . . . . . 692
5.11.3 Properties of the condition number . . . . . . . . . . . . . . 693
5.11.4 Condition number and matric norms . . . . . . . . . . . . 694
5.12 Moore-Penrose pseudo-inverse . . . . . . . . . . . . . . . . . . . . . . 696
5.12.1 Relationships between inverse and pseudo-inverse . . . . . 701
5.12.2 The orthogonal projection operator . . . . . . . . . . . . . . 701
5.12.3 Returning to the least squares method . . . . . . . . . . . . 702

6 EIGENVALUES AND MATRIX DIAGONALIZATION 705

6.1 Eigenvalues and eigenvectors . . . . . . . . . . . . . . . . . . . . . . . 706
6.1.1 Spectral radius and spectral norm . . . . . . . . . . . . . . 708
6.1.2 Schur triangularization theorem . . . . . . . . . . . . . . . 710
TABLE OF CONTENTS vii

6.1.3 Spectral norm and matrix powers . . . . . . . . . . . . . . 712

6.1.4 Gelfand’s formula . . . . . . . . . . . . . . . . . . . . . . . 712
6.1.5 Spectral norm and Frobenius norm . . . . . . . . . . . . . 713
6.1.6 Spectral radius of normal matrices . . . . . . . . . . . . . . 714
6.1.7 Eigenvalue and eigenvector estimation examples . . . . . . 714
6.1.8 Geometric interpretation of eigenvalues and eigenvectors . 722
6.2 Similarity invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723
6.3 Singular values and eigenvalues . . . . . . . . . . . . . . . . . . . . . 724
6.3.1 Symmetric (hermitian) matrices . . . . . . . . . . . . . . . 725
6.3.2 Condition number and spectral radius . . . . . . . . . . . . 725
6.4 The coefficients of the characteristic polynomial . . . . . . . . . . . . . 729
6.4.1 Symmetric eigenvalue functions . . . . . . . . . . . . . . . 735
6.5 The properties of eigenvalues . . . . . . . . . . . . . . . . . . . . . . . 736
6.5.1 Characteristic polynomial of inverse matrix . . . . . . . . . 740
6.5.2 Eigenvalues, trace and determinant of Kronecker product . 741
6.5.3 Singular values and rank of Kronecker product . . . . . . . 742
6.5.4 Gerschgorin cycles . . . . . . . . . . . . . . . . . . . . . . . 744
6.5.5 Schur inequality . . . . . . . . . . . . . . . . . . . . . . . . 747
6.6 Eigenvalues and eigenvectors of special matrices . . . . . . . . . . . . 749
6.6.1 Eigenvalues and eigenvectors of normal matrices . . . . . . 749
6.6.2 Eigenvalues and eigenvectors of hermitian matrices . . . . 750
6.6.3 Eigenvalues and eigenvectors of unitary matrices . . . . . . 751
6.7 Eigenvalue spectrum of matrix product . . . . . . . . . . . . . . . . . 753
6.7.1 Permutation of normal matrices . . . . . . . . . . . . . . . 756
6.8 The theorem of Cayley-Hamilton . . . . . . . . . . . . . . . . . . . . . 758
6.8.1 Estimating the inverse matrix . . . . . . . . . . . . . . . . 760
6.8.2 Cayley - Hamilton theorem and Kronecker product . . . . . 761
6.8.3 The minimal polynomial . . . . . . . . . . . . . . . . . . . . 762
6.9 Diagonalization of square matrices . . . . . . . . . . . . . . . . . . . . 763
6.9.1 Diagonalization and cardinality of eqigenvalues . . . . . . . 769
6.9.2 Relating the eigenvalues with the diagonal elements . . . . 771
6.9.3 The theorem of Courant-Fischer . . . . . . . . . . . . . . . 772
6.9.4 Weyl’s monotonocity theorem . . . . . . . . . . . . . . . . . 776
6.9.5 Spectral analysis . . . . . . . . . . . . . . . . . . . . . . . . 777
6.10 Jordan form of a nilpotent matrix . . . . . . . . . . . . . . . . . . . . 780
6.10.1 Building a base for the nullspace nullsp (L) . . . . . . . . . 788
6.10.2 Extending the base of nullsp (L) with Jordan chains . . . . 796
6.11 Jordan form of a square matrix . . . . . . . . . . . . . . . . . . . . . . 802
6.11.1 Building the Jordan form . . . . . . . . . . . . . . . . . . . 807
6.12 Positive definite matrices . . . . . . . . . . . . . . . . . . . . . . . . . 812
6.12.1 Properties of positive definite matrices . . . . . . . . . . . . 814
6.12.2 Properties of positive semidefinite matrices . . . . . . . . . 817
6.12.3 Roots of positive definite matrices . . . . . . . . . . . . . . 820
6.12.4 Analysis and factorization . . . . . . . . . . . . . . . . . . . 822
6.12.5 Positive definite matrices and inner product . . . . . . . . . 823
6.12.6 Schur theorem . . . . . . . . . . . . . . . . . . . . . . . . . 825
6.12.7 Hadamard’s inequality . . . . . . . . . . . . . . . . . . . . 826
viii TABLE OF CONTENTS

6.12.8 Minkowski’s inequality . . . . . . . . . . . . . . . . . . . . 828

6.12.9 Fielder’s inequality . . . . . . . . . . . . . . . . . . . . . . . 829
6.12.10 Schur’s complement . . . . . . . . . . . . . . . . . . . . . . 830
6.12.11 Oppenheim’s inequality . . . . . . . . . . . . . . . . . . . . 831
6.12.12 Congruence of matrices . . . . . . . . . . . . . . . . . . . . 834
6.12.13 Löwner ordering . . . . . . . . . . . . . . . . . . . . . . . . 836
6.13 Quadratic forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 838
6.13.1 Graphical representation of a quadratic form . . . . . . . . 841

7 MATRIX FUNCTIONS AND THEIR APPLICATIONS 843

7.1 Defining a matrix function . . . . . . . . . . . . . . . . . . . . . . . . 844
7.2 The exponential function eA . . . . . . . . . . . . . . . . . . . . . . . . 844
7.2.1 The exponential function of a nilpotent matrix . . . . . . . 848
7.2.2 The exponential function of a matrix sum . . . . . . . . . . 848
7.2.3 Exponential function and hermitian matrices . . . . . . . . 849
7.3 The property of spectral mapping . . . . . . . . . . . . . . . . . . . . . 849
7.4 The matrix power An . . . . . . . . . . . . . . . . . . . . . . . . . . . . 852
7.4.1 Z transform and matrix powers . . . . . . . . . . . . . . . 854
7.4.2 Neumann’s identity . . . . . . . . . . . . . . . . . . . . . . 857
7.5 The exponential function eAt . . . . . . . . . . . . . . . . . . . . . . . 859
7.5.1 The case of complex eigenvalues . . . . . . . . . . . . . . . 867
7.6 The logarithm of a matrix . . . . . . . . . . . . . . . . . . . . . . . . . 871
7.7 Functions of non-diagonalizable matrices . . . . . . . . . . . . . . . . 873
7.7.1 Spectral projection of non-diagonalizable matrices . . . . . 880
7.7.2 The properties of minimal polynomial . . . . . . . . . . . . 881
7.7.3 Krylov sequences . . . . . . . . . . . . . . . . . . . . . . . 883
7.8 Polynomial representation a matrix functions . . . . . . . . . . . . . . 885
7.8.1 Returning to the exponential matrix function . . . . . . . . 888
7.9 Power series expansion of a matrix function . . . . . . . . . . . . . . . 888
7.10 Representing matrix functions via Cauchy formula . . . . . . . . . . . 891
7.10.1 An alternative approach for estimating a matrix power . . . 896
7.10.2 The equivalence with the power series expansion . . . . . . 897
7.11 Analytic matrix functions . . . . . . . . . . . . . . . . . . . . . . . . . 898
7.11.1 Spectral mapping and eigenvalue cardinality . . . . . . . . 901
7.12 Systems of differential equations . . . . . . . . . . . . . . . . . . . . . 904
7.12.1 Other forms for the solution of a differential equation . . . 908
7.12.2 Eigenfunctions and eigenfrequencies . . . . . . . . . . . . . 910
7.12.3 The method of Putzer . . . . . . . . . . . . . . . . . . . . . 911
7.12.4 Non-diagonalizable homogenous differential system . . . . 916
7.12.5 Wronskian and the Abel formula . . . . . . . . . . . . . . . 918
7.12.6 Solving the complete differential system . . . . . . . . . . . 922
7.12.7 The general solution of the complete differential system . . 925
7.12.8 Sylvester’s and Lyapunov’s differential equations . . . . . . 926
7.13 Systems of difference equations . . . . . . . . . . . . . . . . . . . . . . 927
7.13.1 The behavior of the solution for n → ∞ . . . . . . . . . . . 930
7.13.2 Other forms for the solution of the difference system . . . . 931
7.13.3 The method of Putzer . . . . . . . . . . . . . . . . . . . . . 937
 TABLE OF CONTENTS ix

7.14 Higher order equations . . . . . . . . . . . . . . . . . . . . . . . . . . 941

Α PAULI AND DIRAC MATRICES 945

Α.1 Pauli and Dirac matrices . . . . . . . . . . . . . . . . . . . . . . . . . 946

Literature 957

Index 962

View publication stats