Algebra Math Notes Study Guide

Linear Algebra
1
1-1

Vector Spaces
Vector Spaces
A vector space (or linear space) V over a field F is a set on which the operations addition (+) and scalar multiplication, are defined so that for all and all , 0. Closure and are unique elements in V. 1. Commutativity of Addition 2. Associativity of Addition 3. There exists such that for every . Existence of Additive Identity (Zero Vector) 4. There exists an element such that Existence of Additive . Inverse 5. Multiplicative Identity 6. Associativity of Scalar Multiplication 7. Left Distributive Property 8. Right Distributive Property Elements of F, V are scalars, vectors, respectively. F can be Examples: n-tuples with entries from F sequences with entries from F mxn matrices with entries from F functions from set S to F polynomials with coefficients from F continuous functions on and , then . , etc.

or

Cancellation Law for Vector Addition: If Corollary: 0 and -x are unique. For all ,

1-2

Subspaces
A subset W of V over F is a subspace of V if W is a vector space over F with the operations of addition and scalar multiplication defined on V. is a subspace of V if and only if 1. whenever 2. whenever . A subspace must contain 0.

Any intersection of subspaces of V is a subspace of V. If S1, S2 are nonempty subsets of V, their sum is . V is the direct sum of W 1 and W 2 ( ) if W 1 and W 2 are subspaces of V such that and . Then each element in V can be written uniquely as where . are complementary. is the smallest subspace of V containing W 1 and W 2, i.e. any subspace containing W 1 and W 2 contains . For a subspace W of V, is the coset of W containing v. iff . The collection of cosets is called the quotient (factor) space of V modulo W. It is a vector space with the operations o o

1-3

Linear Combinations and Dependence

A vector vectors is a linear combination of vectors of if there exist a finite number of and scalars such that . v is a linear combination of . The span of S, span(S), is the set consisting of all linear combinations of the vectors in S. By definition, . S generates (spans) V if span(S)=V. The span of S is the smallest subspace containing S, i.e. any subspace of V containing S contains span(S). A subset vectors is linearly (in)dependent if there (do not) exist a finite number of distinct and scalars , not all 0, such that . is linearly dependent iff

Let S be a linearly independent subset of V. For .

1-4

Bases and Dimension

A (ordered) basis for V is a (ordered) linearly independent subset of V that generates V. Ex. is the standard ordered basis for . A subset of V is a basis for V iff each combination of vectors of . can be uniquely expressed as a linear

Any finite spanning set S for V can be reduced to a basis for V (i.e. some subset of S is a basis). Replacement Theorem: (Steinitz) Suppose V is generated by a set G with n vectors, and let L be a linearly independent subset of V with m vectors. Then and there exists a

subset H of G containing vectors such that Pf. Induct on m. Use induction hypothesis for

generates V. ; remove a and replace by

Corollaries: If V has a finite basis, every basis for V contains the same number of vectors. The unique number of vectors in each basis is the dimension of V (dim(V)). Suppose dim(V)=n. Any finite generating set/ linearly independent subset contains n/n elements, can be reduced/ extended to a basis, and if the set contains n elements, it is a basis. Subsets of V, dim(V)=n

Basis (n elements)

Linearly Independent Sets (n elements)

Generating Sets (n elements)

Let W be a subspace of a finite-dimensional vector space V. Then dim(W)dim(V). If dim(W)=dim(V), then W=V.

The dimension of V/W is called the codimension of V in W.

1-5

Infinite-Dimensional Vector Spaces

Let be a family of sets. A member M of is maximal with respect to set inclusion if M is contained in no member of other than M. ( is partially ordered by .) A collection of sets is a chain (nest, tower) if for each A, B in , either or .( is totally ordered by .) Maximal Principle: [equivalent to Axiom of Choice] If for each chain , there exists a member of containing each member of , then contains a maximal member. A maximal linearly independent subset of is a subset B of S satisfying (a) B is linearly independent. (b) The only linearly independent subset of S containing B is B. Any basis is a maximal linearly independent subset, and a maximal linearly independent

subset of a generating set is a basis for V. Let S be a linearly independent subset of V. There exists a maximal linearly independent subset (basis) of V that contains S. Hence, every vector space has a basis. Pf. = linearly independent subsets of V. For a chain , take the union of sets in , and apply the Maximal Principle. Every basis for a vector space has the same cardinality. Suppose basis such that , S1 is linearly independent and S2 generates V. Then there exists a . so

Let be a basis for V, and S a linearly independent subset of V. There exists is a basis for V.

1-6

Modules
A left/right R-module scalar multiplication ( 1. Distributive 2. Distributive 3. Associative 4. Identity Modules are generalizations of vector spaces. All results for vector spaces hold except ones depending on division (existence of inverse in R). Again, a basis is a linearly independent set that generates the module. Note that if elements are linearly independent, it is not necessary that one element is a linear combination of the others, and bases do not always exist. A free module with n generators has a basis with n elements. V is finitely generated if it contains a finite subset spanning V. The rank is the size of the smallest generating set. Every basis for V (if it exists) contains the same number of elements. / over the ring R is an abelian group (M,+) with addition and or ) defined so that for all and Left Right ,

1-7

Algebras
A linear algebra over a field F is a vector space over F with multiplication of vectors defined so that for all , 1. Associative 2. Distributive 3. If there is an element so that , then 1 is the identity element. is commutative if . Polynomials made from vectors (with multiplication defined as above), linear transformations, and matrices (see Chapters 2-3) all form linear algebras.

2 2-1

Matrices
Matrices
A matrix has m rows and n columns arranged filled with entries from a field F (or ring R). denotes the entry in the ith column and jth row of A. Addition and scalar multiplication is defined component-wise:

The

matrix of all zeros is denoted

or just O.

2-2

Matrix Multiplication and Inverses

Matrix product: Let A be a and B be a matrix. The product AB is the matrix with entries

Interpretation of the product AB: 1. Row picture: Each row of A multiplies the whole matrix B. 2. Column picture: A is multiplied by each column of B. Each column of AB is a linear combination of the columns of A, with the coefficients of the linear combination being the entries in the column of B. 3. Row-column picture: (AB)ij is the dot product of row I of A and column j of B. 4. Column-row picture: Corresponding columns of A multiply corresponding rows of B and add to AB. Block multiplication: Matrices can be divided into a rectangular grid of smaller matrices, or blocks. If the cuts between columns of A match the cuts between rows of B, then you can multiply the matrices by replacing the entries in the product formula with blocks (entry i,j is replaced with block i,j, blocks being labeled the same way as entries). The identity matrix In is a nxn square matrix with ones down the diagonal, i.e.

A is invertible if there exists a matrix A -1 such that . The inverse is unique, and for square matrices, any inverse on one side is also an inverse on the other side. Properties of Matrix Multiplication (A is mxn): 1. Left distributive 2. Right distributive 3. Left/ right identity 4. Associative 5. 6. (A, B invertible) : Not commutative Note that any 2 polynomials of the same matrix commute. A nxn matrix A is either a zero divisor (there exist nonzero matrices B, C such that ) or it is invertible.

The Kronecker (tensor) product of pxq matrix A and rxs matrix B is . If v and w are column vectors with q, s elements, . Kronecker products give nice eigenvalue relations- for example the eigenvalues are the products of those of A and B. [AMM 107-6, 6/2000]

2-3

Other Operations, Classification

The transpose of a mxn matrix A, At, is defined by . The adjoint or Hermitian of a matrix A is its conjugate transpose: Name Symmetric Self-adjoint/ Hermitian Skew-symmetric Skew-self-adjoint/ Skew-Hermitian Upper triangular Lower triangular Diagonal Definition Properties is real for any complex z.

Properties of Transpose/ Adjoint 1. (For more matrices, reverse the order.) 2. 3. , 4. is symmetric. The trace of a matrix A is the sum of its diagonal entries:

The trace is a linear operator. The direct sum of and matrices A and B is the ,

(augmented) matrix C given by

3
3-1

Linear Transformations
Linear Transformations
For vector spaces V and W over F, a function (homomorphism) if for all and , (a) (b) It suffices to verify is automatic. . is a linear transformation

Ex. Rotation, reflection, projection, rescaling, derivative, definite integral Identity Iv and zero transformation T 0 An endomorphism (or linear operator) is a linear transformation from V into itself. T is invertible if it has an inverse T-1 satisfying . If T is invertible, V and W have the same dimension (possibly infinite). Vector spaces V and W are isomorphic if there exists a invertible linear transformation (an isomorphism, or automorphism if V=W) . If V and W are finite-dimensional, they are isomorphic iff dim(V)=dim(W). V is isomorphic to . The space of all linear transformations from V to W is a vector space over F. The inverse of a linear transformation and the composite of two linear transformations are both linear transformations. The null space or kernel is the set of all vectors x in V such that T(x)=0. The range or image is the subset of W consisting of all images of vectors in V. Both are subspaces. nullity(T) and rank(T) denote the dimensions of N(T) and R(T), respectively. If is a basis for V, then .

Dimension Theorem: If V is finite-dimensional, nullity(T)+rank(T)=dim(V). Pf. Extend a basis for N(T) to a basis for V by adding . Show is a basis for R(T) by using linearity and linear independence. T is one-to-one iff N(T)={0}. If V and W have equal finite dimension, the following are equivalent: (a) T is one-to-one. (b) T is onto. (c) rank(T)=dim(V) (a) and (b) imply T is invertible.

A linear transformation is uniquely determined by its action on a basis, i.e., if is a basis for V and , there exists a unique linear transformation such that . A subspace W of V is T-invariant if T on W. for every . TW denotes the restriction of

3-2

Matrix Representation of Linear Transformation

Matrix Representation: Let be an ordered basis for V and for W. For , define so that be an ordered basis

The coordinate vector of x relative to is

Note is an isomorphism from V to Fn. The ith coordinate is Suppose is a linear transformation satisfying

The matrix representation of T in and is above. (i.e. load the coordinate representation of

with entries as defined into the jth column of A.)

Properties of Linear Transformations (Composition) 1. Left distributive 2. Right distributive 3. Left/ right identity 4. Associative (holds for any functions) 5. 6. (T, U invertible) Linear transformations [over finite-dimensional vector spaces] can be viewed as leftmultiplication by matrices, so linear transformations under composition and their corresponding matrices under multiplication follow the same laws. This is a motivating factor for the definition of matrix multiplication. Facts about matrices, such as associativity of matrix multiplication, can be proved can be proved using linear transformations, or vice versa. Note: From now on, definitions applying to matrices can also apply to the linear transformations they are associated with, and vice versa. The left-multiplication transformation matrix). is defined by (A is a mxn

Relationships between linear transformations and their matrices: 1. To find the image of a vector under T, multiply the matrix corresponding to T

on the left: i.e. where . 2. Let V, W be finite-dimensional vector spaces with bases , . T he function defined by is an isomorphism. So, for linear transformations , a. b. for all scalars a. c. has dimension mn. 3. For vector spaces V, W, Z with bases , , and linear transformations , . 4. T is invertible iff is invertible. Then .

3-3

Change of Coordinates
Let and be two ordered bases for finite-dimensional vector space V. The change of coordinate matrix (from -coordinates to -coordinates) is . Write vector j of in terms of the vectors of , take the coefficients and load them in the jth column of Q. (This is so (0,1,0) gets transformed into the jth column.) 1. changes -coordinates into -coordinates. 2. Two nxn matrices are similar if there exists an invertible matrix Q such that . Similarity is an equivalence relation. Similar matrices are manifestations of the same linear transformation in different bases.

3-4

Dual Spaces
A linear functional is a linear transformation from V to a field of scalars F. The dual space is the vector space of all linear functionals on V: . V** is the double dual. If V has ordered basis , then dual basis) is an ordered basis for V*, and for any (coordinate functionsthe ,

To find the coordinate representations of the vectors of the dual basis in terms of the standard coordinate functions: 1. Load the coordinate representations of the vectors in into the columns of W. 2. The desired representations are the rows of . 3. The two bases are biorthogonal. For an orthonormal basis (see section 5-5), the coordinate representations of the basis and dual bases are the same. Let V, W have ordered bases , . For a linear transformation , define its t transpose (or dual) by . T is a linear transformation satisfying . Define by fixed point), and (input is a function, output is the value of the function at a . (The input is a function; the output is a function

by

evaluated at a fixed point.) If V is finite-dimensional, is an isomorphism. Additionally, every ordered basis for V* is the dual basis for some basis for V. The annihilator of a subset S of V is a subspace of :

4
4-1

Systems of Linear Equations

Systems of Linear Equations
The system of equations

can be written in matrix form as Ax=b, where

and

. The

augmented matrix is (the entries of b placed to the right of A). The system is consistent if it has solution(s). It is singular if it has zero or infinitely many solutions. If b=0, the system is homogeneous. 1. Row picture: Each equation gives a line/ plane/ hyperplane. They meet at the solution set. 2. Column picture: The columns of A combine (with the coefficients ) to produce b.

4-2

Elimination
There are three types of elementary row/ column operations: (1) Interchanging 2 rows/ columns (2) Multiplying any row/ column by a nonzero scalar (3) Adding any multiple of a row/ column to another row/ column An elementary matrix is the matrix obtained by performing an elementary operation on I n. Any two matrices related by elementary operations are (row/column-)equivalent. Performing an elementary row/ column operation is the same as multiplying by the corresponding elementary matrix on the left/ right. The inverse of an elementary matrix is an elementary matrix of the same type. When an elementary row operation is performed on an augmented matrix or the equation , the solution set to the corresponding system of equations does not change. Gaussian elimination- Reduce a system of equations (line up the variables, the equations are the rows), a matrix, or an augmented matrix by using elementary row operations. Forward pass 1. Start with the first row. 2. Excluding all rows before the current row (row j), in the leftmost nonzero column (column k), make the entry in the current row nonzero by switching rows as necessary. (Type 1 operation) The pivot di is the first nonzero in the current row, the row that does the elimination. [Optional: divide the current row by the pivot to make the entry 1. (2)] 3. Make all numbers below the pivot zero. To make the entry a ik in the ith row 0, subtract row j times the multiplier from row i. This corresponds to multiplication by a type 3 elementary matrix . 4. Move on to the next row, and repeat until only zero rows remain (or rows are exhausted). Backward pass (Back-substitution) 5. Work upward, beginning with the last nonzero row, and add multiples of each row to

the rows above to create zeros in the pivot column. When working with equations, this is essentially substituting the value of the variable into earlier equations. 6. Repeat for each preceding row except the first. A free variable is any variable corresponding to a column without a pivot. Free variables can be arbitrary, leading to infinitely many solutions. Express the solution in terms of free variables. If elimination produces a contradiction (in A|b, a row with only the last entry a nonzero, corresponding to 0=a), there is no solution. Gaussian elimination produces the reduced row echelon form of the matrix: (Forward/ backward pass accomplished 1, (2), 3/ 4.) 1. Any row containing a nonzero entry precedes any zero row. 2. The first nonzero entry in each row is 1. 3. It occurs in a column to the right of the first nonzero entry in the preceding row. 4. The first nonzero entry in each row is the only nonzero entry in its column. The reduced row echelon of a matrix is unique.

4-3

Factorization
Elimination = Factorization Performing Gaussian elimination on a matrix A is equivalent to multiplying A by a sequence of elementary row matrices. If no row exchanges are made, , so A can be factored in the form

where L is a lower triangular matrix with 1s on the diagonal and U is an upper triangular matrix (note the factors are in opposite order). Note and differ only in the sign of entry (i,j), and the multipliers go directly into the entries of L. U can be factored into a diagonal matrix D containing the pivots and U an upper triangular matrix with 1s on the diagonal: The first factorization corresponds to the forward pass, the second corresponds to completing the back substitution. If A is symmetric, . Using , 1. Solve 2. Solve can be split into two triangular systems: for c. for x.

A permutation matrix P has the rows of I in any order; it switches rows. If row exchanges are required, doing row exchanges 1. in advance gives . 2. after elimination gives .

4-4

The Complete Solution to Ax=b, the Four Subspaces

The rank of a matrix A is the rank of the linear transformation L A, and the number of pivots after elimination.

Properties: 1. Multiplying by invertible matrices does not change the rank of a matrix, so elementary row and column matrices are rank-preserving. 2. rank(At)=rank(A) 3. Ax=b is consistent iff rank(A)=rank(A|b). 4. Rank inequalities Linear transformations T, U Matrices A, B rank(TU) min(rank(T), rank(U)) rank(AB) min(rank(A), rank(B)) Four Fundamental Subspaces of A 1. The row space C(AT) is the subspace generated by rows of A, i.e. it consists of all linear combinations of rows of A. a. Eliminate to find the nonzero rows. These rows are a basis for the row space. 2. The column space C(A) is the subspace generated by columns of A. a. Eliminate to find the pivot columns. These columns of A (the original matrix) are a basis for the column space. The free columns are combinations of earlier columns, with the entries of F the coefficients. (See below) b. This gives a technique for extending a linearly independent set to a basis: Put the vectors in the set, then the vectors in a basis down the columns of A. 3. The nullspace N(A) consists of all solutions to . a. Finding the Nullspace (after elimination) i. Repeat for each free variable x: Set x=1 and all other free variables to 0, and solve the resultant system. This gives a special solution for each free variable. ii. The special solutions found in (1) generate the nullspace. b. Alternatively, the nullspace matrix (containing the special solutions in its columns) is when the row reduced echelon form is . If columns are switched in R, corresponding rows are switched in N. 4. The left nullspace N(AT) consists of all solutions to or . Fundamental Theorem of Linear Algebra (Part 1): Dimensions of the Four Subspaces: A is mxn, rank(A)=r (If the field is complex, replace by .)

Row space Dimension r

Row rank = column rank

Column space Dimension r

Nullspace Dimension n-r

Left nullspace Dimension m-r

The relationships between the dimensions can be shown using pivots or the dimension theorem. The Complete Solution to Ax=b 1. Find the nullspace N, i.e. solve Ax=0. 2. Find any particular solution xp to Ax=b (there may be no solution). Set free variables to 0. 3. The solution set is ; i.e. all solutions are in the form , where is in the nullspace and is a particular solution.

4-5

Inverse Matrices
A is invertible iff it is square (nxn) and any one of the following is true: 1. has rank n, i.e. has n pivots. 2. has exactly 1 solution. 3. Its columns/ rows are a basis for . Gauss-Jordan Elimination: If A is an invertible nxn matrix, it is possible to transform (A|I n) into (In|A-1) by elementary row operations. Follow the same steps as in Gaussian elimination, but on (A|In). If A is not invertible, then such transformation leads to a row whose first n entries are zeros.

5
5-1

Inner Product Spaces

Inner Products
An inner product on a vector space V over F ( or ) is a function that assigns each ordered pair a scalar , such that for all and , 1. 2. (The inner product is linear in its first component.)1 3. (Hermitian) 4. for . (Positive) V is called an inner product space, also an Euclidean/ unitary space if F is / . The inner product is conjugate linear in the second component: 1. 2. If for all then . The standard inner product (dot product) of and is

The standard inner product for the space of continuous complex functions H on

is

A norm of a vector space is a real-valued function 1. 2. , equality iff . 3. Triangle Inequality: The distance between two vectors x, y is .

satisfying

In an inner product space, the norm (length) of a vector is Cauchy-Schwarz Inequality:

5-2

Orthogonality
Two vectors are orthogonal (perpendicular) when their inner product is 0. A subset S is orthogonal if any two distinct vectors in S are orthogonal, orthonormal if additionally all vectors have length 1. Subspaces V and W are orthogonal if each is orthogonal to each . The orthogonal complement (V perp) of V is the subspace containing all vectors orthogonal to V. (Warning: holds when V is finite-dimensional, not necessarily when V is infinite-dimensional.) When an orthonormal basis is chosen, every inner product on finite-dimensional V is similar to the standard inner product. The conditions effectively determine what the inner product has to be. Pythagorean Theorem: If x and y are orthogonal, .

In some books (like Algebra, by Artin) the inner product is linear in the second component and conjugate linear in the first. The standard inner product is sum of instead.

Fundamental Theorem of Linear Algebra (Part 2): The nullspace is the orthogonal complement of the row space. The left nullspace is the orthogonal complement of the column space.

5-3

Projections
Take 1: Matrix and geometric viewpoint The [orthogonal] projection of onto is

The last two expressions are for (row) vectors in , using the dot product. (Note: this shows that for 2 and 3 dimensions.) Let be a finite orthogonal basis. A vector y is the sum of its projections onto the vectors of S:

Pf. Write y as a linear combination and take the inner product of y with a vector in the basis; use orthogonality to cancel all but one term. As a corollary, any orthogonal subset is linearly independent. To find the projection of onto a finite-dimensional subspace W, first find an orthonormal basis for W (see section 5-5), . The projection is

and the error is . is perpendicular to , and minimal. (Proof uses Pythagorean theorem) Bessels Inequality: ( a basis for a subspace) , equality iff If . Alternatively: Let W be a subspace of

is the vector in W so that

is

is an orthonormal basis, then for any linear transformation T,

generated by the linearly independent set , the projection of onto W is

. Solving

where P is the projection matrix. In the special case that the set is orthonormal,

A matrix P is a projection matrix iff

Take 2: Linear transformation viewpoint If then the projection on W 1 along W2 is defined by T is an orthogonal projection if orthogonal projection iff and . . A linear operator T is an

5-4

Minimal Solutions and Least Squares Approximations

When is consistent, the minimal solution is the one with least absolute value. 1. There exists exactly one minimal solution s, and . 2. s is the only solution to in : . The least squares solution makes as small as possible. (Generally, is inconsistent.) Project b onto the column space of A. To find the real function in the form the points i.e. such that the error let A be the matrix with , . Then for fixed functions that is closest to is least,

is equivalent to the system

. Now find the projection of onto the columns of , by multiplying by and solving . Here, p is the values estimated by the best-fit curve and e gives the errors in the estimates. Ex. Linear functions .The equation A has orthogonal columns when letting . Then squares line is
Row space Dimension r

: becomes .

. To produce orthogonal columns, shift the times by is diagonal and . The least

.
Column space Dimension r

Least squares solution Minimal solution to

Nullspace Dimension n-r

Left nullspace Dimension m-r

5-5

Orthogonal Bases
Gram-Schmidt Orthogonalization Process: Let be a linearly independent subset of V. Define and by

Then S is an orthogonal set having the same span as S. To make S orthonormal, divide every vector by its length. (It may be easier to subtract the projections of on for all at step , like in elimination.) Ex. Legendre polynomials (integration from -1 to 1). Factorization A=QR From , Gram-Schmidt constructs orthonormal vectors are an orthonormal basis for

. Then

Note R is upper triangular. Suppose is an orthonormal set in n-dimensional inner product space V. Then (a) S can be extended to an orthonormal basis for V. (b) If W=span(S), is an orthonormal basis for . (c) Hence, and .

5-6

Adjoints and Orthogonal Matrices

Let V be a finite-dimensional inner product space over F, and let transformation. The unique vector such that for all be a linear is given by

Let be a linear transformation, and and be bases for inner product spaces V, W. Define the adjoint of T to be the linear transformation such that . (See section 2.3) Then is the unique (linear) function such that for all and . A linear operator T on V is an isometry if for all . If V is finitedimensional, T is orthogonal for V real and unitary for V complex. The corresponding matrix representations, as well as properties of T, are described below.

Real Complex Linear Transformation

Commutative property Normal Normal

Inverse property Orthogonal Unitary

Symmetry property Symmetric Self-adjoint/ Hermitian

( A real matrix has orthonormal columns iff . If is square it is called an orthogonal matrix, and its inverse is its transpose. A complex matrix has orthonormal columns iff . If is square it is a unitary matrix, and its inverse is its adjoint. If has orthonormal columns it leaves lengths unchanged ( for every x) and preserves dot products ( . is invertible iff A has linearly independent columns. More generally, has the same rank as A.

5-7

Geometry of Orthogonal Operators

A rigid motion is a function satisfying for all . If V is finite-dimensional is also called an isometry. Each rigid motion is the composition of a translation and an orthogonal operator. A (orthogonal) linear operator is a 1. rotation (around ) if there exists a 2-dimensional subspace orthonormal basis for W, and such that . and for 2. reflection (about for all and . ) if W is a one-dimensional subspace of V such that for all .

and an

Structural Theorem for Orthogonal Operators: 1. Let T be an orthogonal operator on finite-dimensional real inner product space V. There exists a collection of pairwise orthogonal T-invariant subspaces of V of dimension 1 or 2 such that . Each is a rotation or reflection; the number of reflections is even/ odd when / . It is possible to choose the subspaces so there is 0 or 1 reflection. 2. If A is orthogonal there exists orthogonal Q such that

where p, q are the dimensions of N(T-I), N(T+I)

and

Eulers Theorem: Every orthonormal 3x3 matrix represents a rotation. Alternate method to factor QR: Q is a product of reflection matrices

and plane rotation matrices (Givens rotation)

in the form (1s on diagonal. Shown are rows/ columns i, j).

Multiply by

to produce 0 in the (i,j) position, as in elimination.

where the factors are reversed in the second product.

6
6-1

Determinants
Characterization
The determinant (denoted or ) is a function from the set of square matrices to the field F, satisfying the following conditions: 1. The determinant of the nxn identity matrix is 1, i.e. . 2. If two rows of A are equal, then , i.e. the determinant is alternating. 3. The determinant is a linear function of each row separately, i.e. it is n-linear. That is, if are rows with n elements,

These properties completely characterize the determinant. 4. Adding a multiple of one row to another row leaves unchanged. 5. The determinant changes sign when two rows are exchanged. 6. A matrix with a row of zeros has . 7. If A is triangular then is the product of diagonal entries. 8. A is singular iff . 9. 10. has the same determinant as A. Therefore the preceding properties are true if row is replaced by column.

6-2

Calculation
1. The Big Formula: Use n-linearity and expand everything.

where the sum is over all

permutations of {1,n} and

2. Cofactor Expansion: Recursive, useful with many zeros, perhaps with induction. (Row)

(Column)

where is A with the ith row and jth column removed. 3. Pivots: If the pivots are and , (P a permutation matrix, L is lower triangular, U is upper triangular) where det(P)=1/ -1 if P corresponds to an even/ odd permutation. a. Let denote the matrix consisting of the first k rows and columns of A. If

there are no row exchanges in elimination,

4. By Blocks: a. b. Tips and Tricks Vandermonde determinant (look at when the determinant is 0, gives factors of polynomial)

Circulant Matrix (find eigenvectors, determinant is product of eigenvalues)

For a real matrix A, If A has eigenvalues In particular, if M has rank 1, , then

6-3

Properties and Applications

Cramers Rule: If A is a nxn matrix and then has the unique solution given by

Where

is A with the ith column replaced by b.

Inverses: Let C be the cofactor matrix of A. Then

The cross product of

and

is

a vector perpendicular to u and v (direction determined by the right-hand rule) with length

. Geometry: The area of a parallelogram with vertices sides satisfy the same properties as determinants.) The area of a parallelepiped with sides is , is . (Oriented areas , and

The Jacobian used to change coordinate systems in integrals is

7
7-1

Eigenvalues and Eigenvectors, Diagonalization

Eigenvalues and Eigenvectors
Let T be a linear operator (or matrix) on V. A nonzero vector of T if there exists a scalar , called the eigenvalue, such that of is the set of all eigenvectors corresponding to : The characteristic polynomial of a matrix A is the eigenvalues of A. For each eigenvalue solve eigenvalues that span the eigenspace. is an (right) eigenvector . The eigenspace .

. The zeros of the polynomial are to find linearly independent

Multiplicity of an eigenvalue : 1. Algebraic ( )- the multiplicity of the root in the characteristic polynomial of A. 2. Geometric ( )- the dimension of the eigenspace of . . . For real matrices, complex eigenvalues come in conjugate pairs. The product of the eigenvalues (counted by algebraic multiplicity) equals The sum of the eigenvalues equals the trace of A. An eigenvalue of 0 implies that A is singular. Spectral Mapping Theorem: Let A be a nxn matrix with eigenvalues (not necessarily distinct, counted according to algebraic multiplicity), and P be a polynomial. Then the eigenvalues of are . Gerschgorins Disk Theorem: Every eigenvalue of A is strictly in a circle in the complex plane centered at some diagonal entry with radius (because . Perron-Frobenius Theorem: Any square matrix with positive entries has a unique eigenvector with positive entries (up to multiplication by a positive factor), and the corresponding eigenvalue has multiplicity one and has strictly greater absolute value than any other eigenvalue. Generalization: Holds for any irreducible matrix with nonnegative entries, i.e. there is no reordering of rows and columns that makes it block upper triangular. A left eigenvalue of A satisfies instead. Biorthogonality says that any right eigenvector of A associated with is orthogonal to all left eigenvectors of A associated with eigenvalues other than . .

7-2

Invariant and T-Cyclic Subspaces

The subspace is the T-cyclic subspace generated by x. W is the smallest T-invariant subspace containing x. 1. If W is a T-invariant subspace, the characteristic polynomial of T W divides that of T. 2. If k=dim(W) then is a basis for W, called the T-cyclic basis

generated by x. If with , the characteristic polynomial of TW is . 3. If , each is a T-invariant subspace, and the characteristic polynomial of is , then the characteristic polynomial of T is . Cayley-Hamilton Theorem: A satisfies its own characteristic equation: if is the characteristic polynomial of A, then . Pf. See http://holdenlee.wordpress.com/2010/06/01/cayley-hamilton-theorem/

7-3

Triangulation
A matrix is triangulable if it is similar to an upper triangular matrix. (Schur) A matrix is triangulable iff the characteristic polynomial splits over F. A real/ complex matrix A is orthogonally/ unitarily equivalent to a real/ complex upper triangular matrix. (i.e. , Q is orthogonal/ unitary) Pf. T=LA has an eigenvalue iff T* has. Induct on dimension n. Choose an eigenvector z of T*, and apply the induction hypothesis to the T-invariant subspace .

7-4

Diagonalization
T is diagonalizable if there exists an ordered basis for V such that diagonalizable if there exists an invertible matrix S such that Let is diagonal. A is is a diagonal matrix.

be the eigenvalues of A. Let be a linearly independent subset of for . Then is linearly independent. (Loosely, eigenvectors corresponding to different eigenvalues are linearly independent.) T is diagonalizable iff both of the following are true: 1. The characteristic polynomial of T splits (into linear factors). 2. For each eigenvalue, the algebraic and geometric multiplicities are equal. Hence there are n linearly independent eigenvectors T is diagonalizable iff V is the direct sum of eigenspaces of T. To diagonalize A, put the linearly independent eigenvectors into the columns of A. Put the corresponding eigenvalues into the diagonal entries of . Then For a linear transformation, this corresponds to

Simultaneous Triangulation and Diagonalization Commuting matrices share eigenvectors, i.e. given that A and B can be diagonalized, there exists a matrix S that is an eigenvector matrix for both of them iff . Regardless, AB and BA have the same set of eigenvalues, with the same multiplicities. More generally, let be a commuting family of triangulable/ diagonalizable linear operators on V. There exists an ordered basis for V such that every operator in is simultaneously represented by a triangular/ diagonal matrix in that basis.

7-5

Normal Matrices

(For review see 5-6) A nxn [real] symmetric matrix: 1. Has only real eigenvalues. 2. Has eigenvalues that can be chosen to be orthonormal. ( ) (See below.) 3. Has n linearly independent eigenvectors so can be diagonalized. 4. The number of positive/ negative eigenvalues equals the number of positive/ negative pivots. For real/ complex finite-dimensional inner product spaces, T is symmetric/ normal iff there exists an orthonormal basis for V consisting of eigenvectors of T. Spectral Theorem (Linear Transformations) Suppose T is a normal linear operator ( on a finite-dimensional real/ complex inner product space V with distinct eigenvalues (its spectrum). Let be the eigenspace of T corresponding to and the orthogonal projection of on . 1. T is diagonalizable and . 2. is orthogonal to the direct sum of with . 3. There is an orthonormal basis of eigenvectors. 4. Resolution of the identity operator: 5. Spectral decomposition: Pf. The triangular matrix in the proof of Schurs Theorem is actually diagonal. 1. If then . 2. W is T-invariant iff is -invariant. 3. Take a eigenvector v; let . From (1) v is an eigenvector of ; from (2) is T-invariant. 4. Write . Use induction hypothesis on . (Matrices) Let A be a normal matrix ( of eigenvectors: where is diagonal and U in unitary. Condition Factorization U unitary, real diagonal Real eigenvalues (because ) Unitary U unitary, diagonal Eigenvalues have absolute value 1 Symmetric (real) Q orthogonal, real diagonal Real eigenvalues Orthogonal (real) Q unitary, diagonal Eigenvalues have absolute value 1

. Then A is diagonalizable with an orthonormal basis

Type of Matrix Hermitian (Self-adjoint)

7-6

Positive Definite Matrices and Operators

A real matrix A is positive (semi)definite if ( ) for every nonzero vector x. A linear operator T on a finite-dimensional inner product space is positive (semi)definite if T is self-adjoint and ( ) for all . The following are equivalent: 1. A is positive definite. 2. All eigenvalues are positive. 3. All upper left determinants are positive. 4. All pivots are positive. Every positive definite matrix factors into with positive pivots in D. The Cholesky factorization is

7-7

Singular Value Decomposition

Every matrix A has a singular value decomposition in the form

where U and V are unitary matrices and

is diagonal. The singular values

for are positive and are in decreasing order, with zeros at the end (not considered singular values). If A corresponds to the linear transformation , then this says there are orthonormal bases and such that Letting be the standard ordered bases for V, W,

Orthogonal elements in the basis are sent to orthogonal elements; the singular values give the factors the lengths are multiplied by. To find the SVD: 1. Diagonalize , choosing orthonormal eigenvectors. The eigenvalues are the squares of the singular values and the eigenvector matrix is V.

2. Similarly, If V and the singular values have already been found, the columns of U are just the images of under left multiplication by A: , unless this gives 0. 3. If A is a mxn matrix: a. The first r columns of V generate the row space of A. b. The last n-r columns generate the nullspace of A. c. The first r columns of U generate the column space of A. d. The last m-r columns of U generate the left nullspace of A.

The pseudoinverse of a matrix A is the matrix such that for , is the vector x in the row space such that , and for , . For a linear transformation, replace with and with . In other words, 1. is the projection matrix onto the column space of A. 2. is the projection matrix onto the row space of A. Finding the pseudoinverse:

The shortest least squares solution to See Section 5-4 for a picture.

is

The polar decomposition of a complex (real) matrix A is where Q is unitary (orthogonal) and H is semi-positive definite Hermitian (symmetric). Use the SVD: If A is invertible, Q is positive definite and the decomposition is unique.

Summary
Type of matrix Real symmetric Orthogonal Skew-symmetric Self-adjoint Positive definite Eigenvalues Real Absolute value 1 (Pure) Imaginary Real Positive Eigenvectors (can be chosen)

Orthogonal

Canonical Forms
A canonical form is a standard way of presenting and grouping linear transformations or matrices. Matrices sharing the same canonical form are similar; each canonical form determines an equivalence class. Similar matrices share Eigenvalues Trace and determinant Rank Number of independent eigenvectors Jordan/ Rational canonical form

8-1

Decomposition Theorems
A minimal polynomial of T is the (unique) monic polynomial such that . If then ; in particular, polynomial of T. of least positive degree divides the characteristic

Let W be an invariant subspace for T and let . The T-conductor (T-stuffer) of x into W is the set which consists of all polynomials g over F such that . (It may also refer to the monic polynomial of least degree satisfying the condition.) If , T is called the T-annihilator of x, i.e. it is the (unique) monic polynomial of least degree for which . The T-conductor/ annihilator divides any other polynomial with the same property. The T-annihilator is the minimal polynomial of TW, where W is the T-cyclic subspace generated by x. The characteristic polynomial and minimal polynomial of T W are equal or negatives. Let L be a linear operator on V, and W a subspace of V. W is T-admissible if 1. W is invariant under T. 2. If , there exists such that . Let T be a linear operator on finite-dimensional V. Primary Decomposition Theorem (leads to Jordan form): Suppose the minimal polynomial of T is

where are distinct irreducible monic polynomials and are positive integers. Let be the null space of (a generalized eigenspace). Then 1. . 2. Each is invariant under T. 3. The minimal polynomial of is . Pf. Let . These polynomial have gcd 1, so we can find so that . is the projection onto space V. . So the direct sum of the eigenspaces is the vector

Cyclic Decomposition Theorem (leads to rational canonical form):2 Let T be a linear operator on finite-dimensional V and (often taken to be ) a proper Tadmissible subspace of V. There exist nonzero with (unique) T-annihilators , called invariant factors such that 1. 2. for . Pf. 1. There exist nonzero vectors in V such that a. b. If and then has maximum degree among all T-conductors into . 2. Let . If then for some and for some . (Stronger form of condition that each is T-admissible.) 3. Existence: Let . implies and . 4. Uniqueness: Induct. Show is unique. If is unique, operate on both sides of 2 decompositions of V to show that and vice versa.

8-2

Jordan Canonical Form

is a Jordan canonical form of T if

where each

is a Jordan block in the form

with an eigenvalue. Nonzero is a generalized eigenvector corresponding to if some p. The generalized eigenspace consists of all generalized eigenvectors corresponding to : for

If

is the smallest positive integer so that

} is a cycle of generalized eigenvectors corresponding to . Every such cycle is linearly independent. Existence (the in the Primary Decomposition Theorem) has an ordered basis consisting of a union of disjoint cycles of generalized eigenvectors corresponding to . Thus every linear transformation (or matrix) on a finite-dimensional vector space, whose characteristic
2

This is a terribly ugly way to prove the rational canonical form. A nicer approach is with the structure theorem for modules. See Abstract Algebra notes, section 5-2.

polynomial splits, has a Jordan canonical form. V is the direct sum of the generalized eigenspaces of T. Uniqueness and Structure The Jordan canonical form is unique (when cycles are listed in order of decreasing length) up to ordering of eigenvalues. Suppose is a basis for . Let be the restriction of to . Suppose is a disjoint union of cycles of generalized eigenvectors with lengths . The dot diagram for contains one dot for each vector in , and 1. has columns, one for each cycle. 2. The jth column consists of dots that correspond to the vectors of , starting with the initial vector. The dot diagram of is unique: The number of dots in the first r rows equals , or if is the number of dots in the jth row, . In particular, the number of cycles is the geometric multiplicity of . The Jordan canonical form is determined by the eigenvalues and for every eigenvalue . So now we know Supposing splits, let be the distinct eigenvalues of T, and let be the order of the largest Jordan block corresponding to . The minimal polynomial of T is

T is diagonalizable iff all exponents are 1.

8-3

Rational Canonical Form

Let T be a linear operator on finite-dimensional V with characteristic polynomial

where the factors integers. Define

are distinct irreducible monic polynomials and

are positive

Note this is a generalization of the generalized eigenspace. The companion matrix of the monic polynomial is

because the characteristic polynomial of c(p) is

Every linear operator T on finite-dimensional V has a rational canonical form (Frobenius normal form) even if the characteristic polynomial does not split.

where each

is the companion matrix of an invariant factor

Uniqueness and Structure: The rational canonical form is unique under the condition for each The rational canonical form is determined by the prime factorization of f(t) and for every positive integer r. Generalized Cayley-Hamilton Theorem: Suppose the characteristic polynomial of T is

where are distinct irreducible monic polynomials and minimal polynomial of T is

are positive integers. Then the

where

8-4

Calculation of Invariant Factors

For a matrix over the polynomials F[x], elementary row/ column operations include: (1) Interchanging 2 rows/ columns (2) Multiplying any row/ column by a nonzero scalar (3) Adding any polynomial multiple of a row/ column to another row/ column However, note arbitrary division by polynomials is illegal in F[x]. For such a (mxn) polynomial F[x], the following are equivalent: 1. P is invertible. 2. The determinant of P is a nonzero scalar. 3. P is row-equivalent to the mxm identity matrix. 4. P is a product of elementary matrices. A matrix is in Smith normal form if 1. Every entry not on the diagonal is 0. 2. On the main diagonal of N, there appear polynomials .

such that

Every matrix is equivalent to a unique matrix N in normal form. For a this algorithm to find it: 1. Make the first column .

matrix A, follow

a. Choose the nonzero entry in the first column that has the least degree. b. For each other nonzero entry , use polynomial division to write , where is the remainder upon division. Subtract times the row with from the row with . c. Repeat a and b until there is (at most) one nonzero entry. Switch the first row with that row if necessary. 2. Put the first row in the form by following the steps above but

exchanging the words rows and columns. 3. Repeat 1 and 2 until the first entry is the only nonzero entry in its row and column. (This process terminates because the least degree decreases at each step.) 4. If does not divide every entry of A, find the first column with an entry not divisible by g and add it to column 1, and repeat 1-4; the degree of g will decrease. Else, go to the next step. 5. Repeat 1-4 with the matrix obtained by removing the first row and column. Uniqueness: Let be the gcd of the determinants of all submatrices of M ( ). Equivalent matrices have all these values equal. The polynomials in the normal form are . Let A be a matrix, and be its invariant factors. The matrix is equivalent to the diagonal matrix with diagonal entries . Use the above algorithm.

Summary
Diagonalization -Diagonal form has only entries on diagonal -Condition: All eigenvalues have same algebraic and geometric multiplicity- n linearly independent eigenvectors -Determined by eigenvalues -V is the direct sum of eigenspaces E. -All irreducible factors in minimal polynomial have exponent 1. -T=1P1+...+kPk, where Pi are projections onto eigenspaces. -I=P1+...+Pk Jordan Canonical Form -Jordan blocks on diagonal -Characteristic polynomial splits -Determined by eigenvalues and nullity [(T-I)r] -V is the direct sum of generalized eigenspaces K. -Exponent of linear term in minimal polynomial is order of largest Jordan block. -Primary decomposition theorem Rational Canonical Form -Companion matrices on diagonal, each polynomial (invariant factor) is multiple of the next. -No condition -Determined by prime factorization and nullity(p(T)r) -Exponent of irreducible factor in minimal polynomial is nullity(f(T)a)/deg(f) -Cyclic decomposition theorem

8-5

Semi-Simple and Nilpotent Operators

A linear operator N is nilpotent if there is a positive integer r such that The characteristic and minimal polynomials are in the form . .

A linear operator is semi-simple if every T-invariant subspace has a complementary Tinvariant subspace. A linear operator (on finite-dimensional V over F) is semi-simple iff the minimal polynomial has no repeated irreducible factors. If F is algebraically closed, T is semi-simple iff T is diagonalizable. Let F be a subfield of the complex numbers. Every linear operator T can be uniquely decomposed into a semi-simple operator S and a nilpotent operator N such that 1. 2. N and S are both polynomials in T. Every linear operator whose minimal (or characteristic) polynomial splits can be uniquely decomposed into a diagonalizable operator D and a nilpotent operator N such that 1. 2. N and D are both polynomials in T. If are the projections in the Primary Decomposition Theorem (Section 8.1) then .

9 9-1

Applications of Diagonalization, Sequences

Powers and Exponentiation
Diagonalization helps compute matrix powers: To find , write x as a combination of the eigenvectors (Note S is a change of base formula that finds the coordinates )

Then

If diagonalization is not possible, use the Jordan form:

Use the following to take powers of a

Jordan block

For a matrix in Jordan canonical form, use this formula for each block. The spectral radius is the largest absolute value of the eigenvalues. If it is less than 1, the matrix powers converge to 0, and it determines the rate of convergence. The matrix exponential is defined as

Thus the eigenvalues of

are

. For a Jordan block,

For nilpotent A, we have

for some functions of t for

. Letting

for every eigenvalue .

Use the system of n equations to solve for the coefficients. If AB=BA, . When A is skew-symmetric, is orthogonal.

9-2

Markov Matrices
Let be a column vector where the ith entry represents the probability that at the kth step the system is at state i. Let A be the transition matrix, that is, contains the probability that a system in state j at any given time will be at state i the next step. Then where contains the initial probabilities or proportions.

The Markov matrix A satisfies: 1. Every entry is nonnegative. 2. Every column adds to 1. A contains an eigenvalue of 1, and all other distinct eigenvalues have smaller absolute value. If all entries of A are positive, then the eigenvalue 1 has only multiplicity 1. The eigenvector corresponding to 1 is the steady state- approached by the probability vectors and describing the probability that a long time late the system will be at each state.

9-3

Recursive Sequences
System of linear recursions: To find the solution to the recurrence with n variables

let

and use

Pells Equation: If D is a positive integer that is not a perfect square, then all positive solutions to are in the form with

where is minimal.

and

is the fundamental solution, that is, the solution where

Homographic recurrence: A homographic function is in the form

defined by by

. .

is the corresponding matrix. Define the sequence Then where .

Linear recursions: A sequence of complex numbers satisfies a linear recursion of order k if

Solve the characteristic equation . If the roots are with multiplicities , then where is a polynomial of degree at most . Determine the polynomials from solving a system involving the first k terms of the sequence. (Note the general solution is a k-dimensional subset of .)

10
10-1

Linear Forms
Multilinear Forms
A function L from , where V is a module over R, to R is

1. Multilinear (n-linear) if it is linear in each component separately: 2. Alternating if whenever with is denoted by . , and the collection of all

The collection of all multilinear functions on alternating multilinear functions is . If L and M are multilinear functions on is the function on defined by where

, respectively, the tensor product of L and M

. The tensor product is linear in each component and is associative. and the linear transformation

For a permutation define by

If V is a free module of rank n, is a free R-module of rank where is a basis for . When , and L is a r-linear form in ,

, with basis

where A is the rxn matrix with rows . is a free R-module of rank , with basis the same as before, but combinations of ( . Where the Determinant fits in: 1. 2. If T is a linear operator on

are

, the and

standard coordinate functions. ,

The determinant of T is the same as the determinant of any matrix representation of T. 3. The special alternating form ( ) is the determinant of the rxr matrix A defined by is the standard dual basis. , also written as , where

10-2

Exterior Products
Let G be the group of all permutations which permute and within themselves. For alternating r and s-linear forms L and M, define by . For a coset , define . The exterior product of L and M is

Then 1. characteristic 0. 2. 3. Laplace Expansions: Define and

; in particular

if R is a field of

and . Then , giving

where

For a free R-module V of rank n, the Grassman ring over and has dimension

is defined by

. (The direct sum is treated like a Cartesian product.)

10-3

Bilinear Forms
A function is a bilinear form on V if H is linear in each variable when the other is held fixed: 1. 2. The bilinear form is symmetric (a scalar product) if for all and skew-symmetric if . The set of all bilinear forms on V, denoted by , is a vector space. An real inner product space is a symmetric bilinear form. A function is a quadratic form if there exists a symmetric bilinear form H such that . If F is not of characteristic 2,

Let be an ordered basis for V. The matrix the matrix representation of H with respect to . 1. is an isomorphism. 2. Thus has dimension . 3. If is a basis for then 4. is (skew-)symmetric iff H is. 5. A is the unique matrix satisfying

with

is

is a basis for . such that

Square matrix B is congruent to A if there exists an invertible matrix

Congruence is a equivalence relation. For 2 bases , and are congruent; conversely, congruent matrices are 2 representations of the same bilinear form. Define and .The rank of H is . For n-dimensional V, the following are equivalent:

1. rank(H)=n 2. For , there exists y such that . 3. For , there exists y such that . Any H satisfying 2 and 3 is nondegenerate. The radical (or null space) of H, Rad(H), is the kernel of or , in other words, it is the set of vectors orthogonal to all other vectors. Nondegenerate Nullspace is .

10-4

Theorems on Bilinear Forms and Diagonalization

A bilinear form H on finite-dimensional V is diagonalizable if there is a basis such that is diagonal. If F does not have characteristic 2, then a bilinear form is symmetric iff it is diagonalizable. If V is a real inner product space, the basis can be chosen to be orthonormal. where Q is the change-of-coordinate matrix changing standard -coordinates into coordinates and . Diagonalize the same way as before, choosing Q to be orthonormal so . A vector v is isotropic if (orthogonal to itself). A subspace W is isotropic if the restriction of H to W is 0. A subspace is maximally isotropic if it has greatest dimension among all isotropic subspaces. Orthogonality, projections, and adjoints for scalar products are defined the same way as orthogonality for inner products: v and w are orthogonal if , and . 1. If then the restriction of H to W, HW, is nondegenerate. 2. If H is nondegenerate on subspace , . 3. If H is nondegenerate, there exists an orthogonal basis for V. Sylvesters Law of Inertia: Let H be a symmetric form on finite-dimensional real V. Then the number of positive diagonal entries (the index p of H) and negative diagonal entries in any diagonal representation of H is the same. The signature is the number of positive entries and the number of negative entries. The rank, index, and signature are all invariants of the bilinear form. 1. Two real symmetric nxn matrices are congruent iff they have the same invariants. 2. A symmetric nxn matrix is congruent to

3. For nondegenerate H: a. The maximal subspace W such that is positive/ negative definite is p/ n-p. b. The maximal isotropic subspace W has dimension If is the adjoint of linear transformation f, and . is the dual (transpose), then

Let H be a skew-symmetric form on n-dimensional V over a subfield of . Then r=rank(H) is even and there exists such that is the direct sum of the zero matrix and copies of .

10-5

Sesqui-linear Forms
A sesqui-linear form f on or is Linear in the first component Conjugate-linear in the second component The form is Hermitian if . A sesqui-linear form f is Hermitian if is real for all x. [Note: Some books reverse x and y for sesqui-linear forms and inner products.] The matrix representation A of f in basis reversal.) Then . is given by . (Note the

If V is a finite-dimensional inner product space, there exists a unique linear operator T f on V such that . This map is an isomorphism from the vector space of sesqui-linear forms onto . is Hermitian iff is self-adjoint. f on or is positive/ nonnegative if it is Hermitian and for / positive form is simply an inner product. f is positive if its matrix representation is positive definite. Principal Axis Theorem: (from the Spectral Theorem) For every Hermitian form f on finite-dimensional V, there exists an orthonormal basis in which f has a real diagonal matrix representation. .A

Summary

Linear Transfromation L -VW - Matrix representation Aij=fi(T(vj)) -Evaluation: [T(v)]=[T][v] - Change of basis: [T] =Q-1[T]Q, Q changes to -coordinates - Representations in different bases are similar/ equivalent

Bilinear Form H -VxVF - Matrix representation Aij=H(vi,vj) -Evaluation: [x]TA[y] - Change of basis (H)=QT(H)Q - Representations in different bases are congruent. -Diagonalizable iff symmetric.

Sesqui-linear/ Hermitian Form f -VxVC -Matrix representation Aij=f(vj,vi) -Evaluation: [y]*A[x] - Change of basis (H)=P*(H)P

10-6

Application of Bilinear and Quadratic Forms: Conics, Quadrics and Extrema

An equation in 2/ 3 variables of degree 2 determines a conic/ quadric. 1. Group all the terms of degree 2 on one side, and represent them in the form where n=2/ 3 and A is a symmetric of is then . If the coefficient of and write the terms as is matrix. If the coefficient then . . The axes

Diagonalize

the conic/ quadric are oriented along are given by the eigenvectors. 2. Write the linear terms with respect to the new coordinates, and complete the square in each variable. Name of Quadric Ellipsoid 1-sheeted hyperboloid 2-sheeted hyperboloid Elliptic paraboloid Equation

Hyperbolic paraboloid Elliptic cone

The Hessian matrix

of

is defined by

Second Derivative Test: Let be a real-valued function for which all third-order partial derivatives exist and are continuous. Let be a critical point (i.e. for all i). (a) If all eigenvalues of are positive, f has a local minimum at p. (b) If all eigenvalues are negative, f has a local maximum at p. (c) If has at least one positive and one negative eigenvalue, p is a saddle point. (d) If (an eigenvalue is 0) and does not have both positive and negative eigenvalues, the test fails.

11
11-1

Numerical Linear Algebra

Elimination and Factorization in Practice
Partial pivoting- For the kth pivot, choose the largest number in row k or below in that column. Exchange that row with row k. Small pivots create large roundoff error because they must be multiplied by large numbers. A band matrix A with half-bandwidth w has Operation counts (A is Process Forward elimination (AU), A=LU factorization Forward elimination on band matrix with halfbandwidth w Forward elimination, right side (b) Back-substitution Factorization into QR (Gram-Schmidt) when .

and invertible) (Multiply-subtract counted as one operation) Reason Count ( . When there are k rows left, for all k1 rows below, multiply-subtract k times. . There are no more than w-1 nonzeros below any pivot. when w small . When there are k rows left, multiplysubtract for all entries below the current one. . For row k, divide by pivot and substitute into previous k-1 rows. . When there are k columns left, divide the th vector by its norm, find the projection of all remaining columns onto it ( ) then subtract ( ). for A=LU, for right side- no work is required on the kth column on the right side until row k, back substitution

(Gauss-Jordan elimination)

Note: For parallel computing, working with matrices (more concise) may be more efficient.

11-2

Norms and Condition Numbers

The norm of a matrix is the maximum magnification of a vector x by A:

For a symmetric matrix, value. Finding the norm:

is the absolute value of the eigenvalue with largest absolute

The condition number of A is

When A is symmetric,

. Anyway,

. to error. Problem error is be the solution error and

The condition number shows the sensitivity of a system inaccuracy in or due to measurement/ roundoff. Let be the problem errors. 1. When the problem error is in b,

2. When the problem error is in A,

11-3

Iterative Methods
For systems: General approach: 1. Split A into S-T. 2. Compute the sequence Requirements: 1. (2) should be easy to solve for , so the preconditioner S should be diagonal or triangular. 2. The error should converge to 0 quickly: Thus the largest eigenvalue of should have absolute value less than 1.

Useful for large sparse matrices, with a wide band. Method Jacobis method Gauss-Siedel method S Diagonal part of A Lower triangular part of A Remarks About twice as fast: Often is the square of the for Jacobi. Combination of Jacobi and Gauss-Siedel. Choose to minimize spectral radius. Set small nonzero in L, U to 0.

Successive overrelaxation

Incomplete LU method

S has diagonal of original A, but below, entries are those of . Approximate L times approximate U

Conjugate Gradients for positive definite A: Set Formula Description 1. Step length 2. 3. 4. 5.

. to

Approximate solution New residual Improvement Next search direction

Computing eigenvalues 1. (Inverse) power methods: Keep multiplying a vector u by A. Typically, u approaches the direction of the eigenvector corresponding to the largest eigenvalue. Convergence is quicker when is small, where are eigenvalues with largest, second largest absolute values. For the smallest eigenvalue, apply the method with (but solve rather than compute the inverse). 2. QR Method: Factor , reverse R and Q (eigenvalues dont change), multiply them to get , and repeat. Diagonal entries approach the eigenvalues. When the last diagonal entry is accurate, remove the last row and column and continue. Modifications: a. Factor into . . Choose c near an unknown eigenvalue. b. (Hessenberg) Obtain off-diagonal entries first by changing A to a similar matrix. Zeros in lower-left corner stay.

12

Applications

12-1 Fourier Series (Analysis)

Use the orthonormal system series: Use projections (Section 5.3) to find the coefficients. (Multiply by the function youre trying to find the coefficient for, and integrate from 0 to ; orthogonality makes all but one term 0.) The orthonormal system is closed, meaning that f is actually equal to the Fourier series. Fourier coefficients offer a way to show the isomorphism between Hilbert spaces (complete, separable, infinite-dimensional Euclidean spaces). The exponential Fourier series uses the orthonormal system applies to functions in . instead. This to express a function in as a Fourier

12-2 Fast Fourier Transform

Let . The Fast Fourier Transform takes as input the coefficients of and outputs the value of the function at . The matrix for F satisfies when the rows and columns are indexed from 0. Then

The inverse of F is . The inverse Fourier transform gives the coefficients from the functional values. To calculate a Fourier transform quickly when , break

Dn/2 is the diagonal matrix with (n/2)th roots of unity. The last matrix has n/2 columns with 1s in even locations (in increasing order starting from 0) and the next n/2 rows in odd locations. Then break up the middle matrix using the same idea, but now there s two copies. Repeating to , the operation count is . The net effect of the permutation matrices is that the numbers are ordered based on the number formed from their digits reversed.

http://cnx.org/content/m12107/latest/

Set . The first and last m components of transforms and , i.e. for

are combinations of the half-size ,

12-3 Differential Equations

The set of solutions to a homogeneous linear differential equation with constant coefficients

is a n-dimensional subspace of . The functions ( a root of the auxiliary polynomial , where m is the multiplicity of the root) are linearly independent and satisfy the equation. Hence they form a basis for a solution space. The general solution to the system of n linear differential equations solutions of the form is any sum of

where the x are the end vectors of distinct cycles that make up a Jordan canonical basis for A, is the eigenvalue corresponding to x, p is the order of the Jordan block, and is a polynomial of degree less than p.

12-4 Combinatorics and Graph Theory

Graphs and applications to electric circuits The incidence matrix A of a directed graph has a row for every edge and a column for every node. If edge i points away from/ toward node j, then , respectively. Suppose the graph is connected, and has n nodes and m edges. Each node is labeled with a number (voltage), and multiplying by A gives the vector of edge labels showing the difference between

the nodes they connect (potential differences/ flow). 1. The row space has dimension n-1. Take any n-1 rows corresponding to a spanning tree of the graph to get a basis for the row space. Rows are dependent when edges form a loop. 2. The column space has dimension n-1. The vectors in the column space are exactly the labeling of edges such that the numbers add to zero around every loop (when moving in the reverse direction as the edges, multiply by -1). This corresponds to all attainable sets of potential differences (Voltage law). 3. The nullspace has dimension 1 and contains multiples of (1,,1) T. Potential differences are 0. 4. The left nullspace has dimension m-n+1. There are m-n+1 independent loops in the graph. The vectors in the left nullspace are those where the flow in equals the flow out at each node (Current law). To find a basis, find m-n+1 independent loops; for each loop choose a direction, and label the edge 1 if it goes around the loop in that direction and -1 otherwise. Let C be the diagonal matrix assigning a conductance (inverse of resistance) to each edge. Ohms law says . The voltages at the nodes satisfy where f tells the source from outside (ex. battery). Another useful incidence matrix is where A has a row and column for each vertex, and if vertices i and j are connected by an edge, and 0 otherwise. (For directed graphs, use -1/ 1.) Sets The incident matrix A for a family of subsets . Exploring

containing elements

has

and using properties of ranks, determinants, linear on problems

dependency, etc. may give conclusions about the sets. Working in the field dealing with parity may help.

12-5 Engineering
Discrete case: Springs Vector/ Equation Description Movements of the n masses Kinematic equation: Elongations of the m springs Constitutive law: Tensions (internal forces) in the m springs Static/ balance equation: External forces on n masses There are four possibilities for A: Case Description Matrix A FixedThere are n+1 springs; each mass has fixed 2 springs coming out of it and the top and bottom are fixed in place. Matrix A gives the elongations of the springs. C is a diagonal matrix that applies Hookes Law for each spring, giving the forces. Internal forces balance external forces on masses.

Equations

Fixedfree

There are n springs; one end is fixed and the other is not. (Here we assume the top end is fixed.) No springs at either end. n-1 springs.

Freefree

Circular The nth spring is connected to the first one. n springs.

Each spring is stretched or compressed by the difference in displacements. Facts about K: 1. K is tridiagonal except for the circular case: only nonzero entries are on diagonal or one entry above or below. 2. K is symmetric. 3. K is positive definite for the fixed-fixed and fixed-free case. 4. has all positive entries for the fixed-fixed and fixed-free case. in the fixed-fixed and fixed-free case give the movements from the forces. For the singular case: 1. The nullspace of K is , if the whole system moves by the same amount the forces

stay the same. 2. To solve , the forces must add up to 0 (equilibrium). Continuous case: Elastic bar becomes the differential equation

The discrete case can be used to approximate the continuous case. When going from the continuous to discrete case, multiply by .

12-6 Physics: Special Theory of Relativity

For each event p occurring at at time t read on clock C relative to S, assign the space-time . Suppose S and S have parallel axes and S moves at

coordinates relative to C and S

constant velocity v relative to S in the +x direction, and they coincide when their clocks C and C read 0. The unit of length is the light second. Define coordinates represent the same event with respect to S and S Axioms: , where the two sets of

1. The speed of light is 1 when measured in either coordinate system. 2. Tv is an isomorphism. 3. implies .

4.

implies

5. The origin of S moves in the negative x-axis of S at velocity v as measured from S. These axioms complete characterize the Lorentz transformation Tv, whose representation in the standard bases is

1. If a light flash at time 0 at the origin is observed at . 2. Time contraction: 3. Length contraction:

is observed at time t, then

12-7 Computer Graphics

3-D computer graphics use homogeneous coordinates: point at infinity if c=0). The transformation Translation by represents the point (the

is like multiplying (on the left side) by

Scaling by a, b, c in x, y, and z directions

Rotation around z-axis (similar for others) by

Projection onto plane through (0,0,0) perpendicular to unit vector n Projection onto plane passing through Q, perpendicular to unit vector n

where T is the translation taking Q to the origin, and P is as above

Reflection through plane through (0,0,0) perpendicular to unit vector n The matrix representation for an affine transformation is

12-8 Linear Programming

Linear programming searches for a nonnegative vector x satisfying that minimizes (or maximizes) the cost . The dual problem is to maximize subject to . The extremum must occur at a corner. A corner is a vector with positive entries that satisfies the m equations with at most m positive components. Duality Theorem: If either problem has a best solution then so does the other. Then the minimum cost equals the maximum income . Simplex Method: 1. First find a corner. If one cant easily be found, create m new variables, start with their sum as the cost, and follow the remaining steps until they are all zero, then revert to the original problem. 2. Move to another corner that lowers the cost. Repeat for each zero component: Change it from 0 to 1, find how the nonzero components would adjust to satisfy , then compute the change in the total cost . Let the entering variable be the one that causes the most negative change (per single unit). Reduce the entering variable until the first positive component hits 0. 3. When every other adjacent corner has higher cost, the current corner is the optimal x.

12-9 Economics
A consumption matrix A has the amount of product j needed to produce product i in entry (i,j). Then where v/ u are the input/ output column vectors containing the amount of product i in entry i. If the column vector y contains the demands for each product, then for the economy to meet the demands, there must exist a vector p with nonnegative entries satisfying

if the inverse exists. If the largest eigenvalue is greater than 1 is equal to 1 is less than 1 then has negative entries fails to exist has only nonnegative entries

If the spectral radius of A is less than 1, then the following expansion is valid:

References
Introduction to Linear Algebra (Third Edition) by Gilbert Strang Linear Algebra (Fourth Edition) by Friedberg, Insel, and Spence Linear Algebra (Second Edition) by Kenneth Hoffman and Ray Kunze Putnam and Beyond by Titu Andreescu and Razvan Gelca MIT OpenCourseWare, 18.06 and 18.700

Notes
I tried to make the notes as complete yet concise and understandable as possible by combining information from 3 books on linear algebra, as well as put in a few problem-solving tips. Strangs book offers a very intuitive view of many linear algebra concepts; for example the diagram on Orthogonality of the Four Subspaces is copied from the book. The other two books offer a more rigorous and theoretical development; in particular, Hoffman and Kunzes book is quite complete. I prefer to focus on vector spaces and linear transformations as the building blocks of linear algebra, but one can start with matrices as well. These offer two different viewpoints which I try to convey: Rank, canonical forms, etc. can be described in terms of both. Big ideas are emphasized and I try to summarize the major proofs as I understand them, as well as provide nice summary diagrams. A first (nontheoretical) course on linear algebra may only include about half of the material in the notes. Often in a section I put the theoretical and intuitive results side by side; just use the version you prefer. I organized it roughly so later chapters depend on earlier ones, but there are exceptions. The last section is applications and a miscellany of stuff that doesnt fit well in the other sections. Basic knowledge of fields and rings is required. Since this was made in Word, some of the math formatting is not perfect. Oh well. Feel free to share this; I hope you find it useful! Please report all errors and suggestions by posting on my blog or emailing me at [email protected]. (Im only a student learning this stuff myself so you can expect errors.) Thanks!