Chapter 2 Linear Algebra Linear algebra is a branch of mathematics that is widely used throughout science and engineering. Yet because linear algebra is a form of continuous rather than discrete mathematics, many computer scientists have little experience with it. A good understanding of linear algebra is essential for understanding and working with many machine learning algorithms, especially deep learning algorithms. We therefore precede our introduction to deep learning with a focused presentation of the key linear algebra prerequisites. If you are already familiar with linear algebra, feel free to skip this chapter. If you have previous experience with these concepts but need a detailed reference sheet to review key formulas, we recommend The Matrix Cookbook (Petersen and Pedersen, 2006). If you have had no exposure at all to linear algebra, this chapter will teach you enough to read this book, but we highly recommend that you also consult another resource focused exclusively on teaching linear algebra, such as Shilov (1977). ‘This chapter completely omits many important linear algebra topics that arc not essential for understanding decp learning. 2.1 Scalars, Vectors, Matrices and Tensors ‘The study of linear algebra involves several types of mathematical objects: Scalars: A scalar is just a single number, in contrast to most of the other objects studied in linear algebra, which are usually arrays of multiple numbers. We write scalars in italics. We usually give scalars lowercase variable names. When we introduce them, we specify what kind of number they are. For 29CHAPTER 2, LINEAR ALGEBRA example, we might say “Let s € R be the slope of the line,” while defining a real-valued scalar, or “Let n € N be the number of units,” while defining a natural number scalar. * Vectors: A vector is an array of numbers, The numbers are arranged in order, We can identify each individual number by its index in that ordering. ‘Typically we give vectors lowercase names in bold typeface, such as &. The elements of the vector are identified by writing its name in italic typeface, with a subscript. The first element of a is 1, the second element is «2, and so on. We also need to say what kind of numbers are stored in the vector. If cach element is in R, and the vector has n elements, then the veetor lies in the set formed by taking the Cartesian product of R n times, denoted as R”. When we need to explicitly identify the elements of a vector, we write them as a column enclosed in square brackets: a x2 x . (2.1) | =| We can think of vectors as identifying points in space, with each el giving the coordinate along a different axis ment Sometimes we need to index a set of elements of a vector. In this case, we define a set containing the indices and write the set as a subscript. For example, to access 111, 3 and 6, we define the set S = {1,3,6} and write xg. We use the — sign to index the complement of a set. For example #1 is the vector containing all elements of a except for #1, and #g is the vector containing all elements of # except for a1, x3 and 2% « Matrices: A matrix is a 2-D array of numbers, so each element is identified by two indices instead of just one. We usually give matrices uppercase variable names with bold typeface, such as A. If a real-valued matrix A has a height of m and a width of n, then we say that A € 2”, We usually identify the elements of a matrix using its name in italic but not bold font, and the indices are listed with separating commas. For example, A1,1 is the upper left entry of A and Ayan is the bottom right entry. We can identify all the numbers with vertical coordinate i by writing a “:” for the horizontal coordinate. For example, Ai, denotes the horizontal cross section of A with vertical coordinate i. This is known as the +th row of A. Likewise, A. is 30CHAPTER 2, LINEAR ALGEBRA the i-th column of A. When we need to explicitly identify the elements of a matrix, we write them as an array enclosed in square brackets: f Air Aiz ” 2.2) | Azi Ava (22) Sometimes we ma a single letter. In this case, we use subscripts after the expression but do not convert anything to lowercase. For example, f(A); gives element (i,j) of the matrix computed by applying the function f to A y need to index matrix-valued expressions that are not just * Tensors: In some ca: In the general case, an array of numbers arranged on a regular grid with a variable number of axes is known as a tensor. We denote a tensor named “A” with this typeface: A. We identify the clement of A at coordinates (i, j,k) by writing Aj, we will need an array with more than two axes. One important operation on matrices is the transpose. The transpose of a matrix is the mirror image of the matrix across a diagonal line, called the main diagonal, running down and to the right, starting from its upper left comer. See figure 2.1 for a graphical depiction of this operation. We denote the transpose of a matrix A as A’, and it is defined such that (A )ig = Aja (2.3) Vectors can be thought of as matrices that contain only one column. The efore a matrix with only one row. Sometimes we transpose of a vector is the: Figure 2.1: The transpose of the matrix can be thought of as a mirror image across the main diagonal. 31CHAPTER 2, LINEAR ALGEBRA define a vector by writing out its elements in the text inline as a row matrix, then using the transpose operator to turn it into a standard column vector, for example w@ = [x1 2, 13] ‘A scalar can be thought of as a matrix with only a single entry. From this, we see that a scalar is its own transpose: a =a We can add matrices to each other, as long as they have the same shape, just by adding their corresponding clements: C = A+ B where Cj, = Ajj + Big We can also add a scalar to a matrix or multiply a matrix by a scalar, just by performing that operation on cach clement of a matrix: D =a- B +e where Dig = a+ Bg +e. In the context of deep learning, we also use some less conventional notation. We allow the addition of a matrix and a vector, yielding another matrix: C =A +, where Ci,j = Ajj +bj. In other words, the vector b is added to each row of the matrix. This shorthand eliminates the need to define a matrix with b copied into each row before doing the addition. This implici is called broadcasting. copying of & to many locations 2.2 Multiplying Matrices and Vectors One of the most important operations involving matrices is multiplication of two matrices. The matrix product of matrices A and B is a third matrix C. In order for this product to be defined, A must have the same number of columns as B has rows. If A is of shape m x n and B is of shape n x p, then C is of shape mx p. We can write the matrix product just by placing two or more matrices together, for example, C= AB. (2.4) The product operation is defined by Cig = DO Ain Bes (2.5) D Note that the standard product of two matrices is not just a matrix containing the product of the individual elements. Such an operation exists and is called the element-wise product, or Hadamard product, and is denoted as A © B. The dot product between two vectors e and y of the same dimensionality is the matrix product a'y. We can think of the matrix product C = AB as computing Ci as the dot. product between row i of A and column j of B 32CHAPTER 2, LINEAR ALGEBRA Matrix product operations have many useful properties that make mathematical analysis of matrices more convenient. For example, matrix multiplication is distributive: A(B+C)=AB+ AC. (2.6) It is also associative A(BC) = (AB)C. (2.7) Matrix multiplication is not commutative (the condition AB = BA does not always hold), unlike scalar multiplication. However, the dot product between two vectors is commutative: alysy'a. (2.8) The transpose of a matrix product has a simple form (AB)" = BAT (2.9) This enables us to demonstrate equation 2.8 by exploiting the fact that the value of such a product is a scalar and therefore equal to its own transpose: aly (e"y)" aya (2.10) Since the focus of this textbook is not linear algebra, we do not attempt to develop a comprehensive list of useful properties of the matrix product here, but the reader should be aware that many more exist. ‘We now know enough linear algebra notation to write down a system of linear equations: Ag=b (2.11) » ACR™* is a known matrix, b € R is a known vector, and a € R" is vector of unknown variables we would like to solve for. Each element 2; of x is one of these unknown variables. Each row of A and cach element of 6 provide another constraint. We can rewrite equation 2.11 as wher a (2.12) (2.18) (2.14) (2.15) or even more explicitly as Ayiti + (2.16)CHAPTER 2, LINEAR ALGEBRA Agari + Agata +--+ Aanan = be (2.17) (2.18) Amati + Am2t2t+++ + Amntn = bm (2.19) Matrix-vector product notation provides a more compact representation for equations of this form. 2.3 Identity and Inverse Matrices Linear algebra offers a powerful tool called matrix inversion that enables us to analytically solve equation 2.11 for many values of A. To describe matrix inversion, we first need to define the concept of an identity matrix. An identity matrix is a matrix that does not change any vector when we multiply that vector by that matrix. We denote the identity matrix that preserves n-dimensional vectors as I. Formally, I, € R"™*", and Va €R", Inv = 2. (2.20) The structure of the identity matrix is simple: all the entries along the main diagonal are 1, while all the other entries are zero, See figure 2.2 for an example. The matrix inverse of A is denoted as A~', and it is defined as the matrix such that ATA=I, (2.21) We can now solve equation 2.11 using the following steps: Az =b (2.22) AlAx = Aq1b (2.23) Ina = An b (2.24) 100 010 001 ure 2.2: Example identity matrix: This is I. 34CHAPTER 2, LINEAR ALGEBRA c= Ab. Of course, this process depends on it being possible to find A~!. We dis the conditions for the existence of A~ in the following section. When A? exists, several different algorithms can find it in closed form. In theory, the same inverse matrix can then be used to solve the equation many times for different values of b. A~? is primarily useful as a theoretical tool, however, and should not actually be used in practice for most software applications. Because A~! can be represented with only limited precision on a digital computer, algorithms that make use of the value of & can usually obtain more accurate estimates of x. 2.4 Linear Dependence and Span For A7! to exist, equation 2.11 must have exactly one solution for every value of b. It is also possible for the system of equations to have no solutions or infinitely many solutions for some values of & It is not possible, however, to have more than one but less than infinitely many solutions for a particular & if both # and y are solutions, then z=08+(1—a)y (2.26) is also a solution for any real a To analyze how many solutions the equation has, think of the columns of A as specifying different directions we can travel in from the origin (the point specified by the vector of all zeros), then determine how many ways there are of reaching b. In this view, each clement of a specifi directions, with x; specifying how far to move in the direction of column é: Aa = SoaiA 4 (2.27) 's how far we should travel in each of these In general, this kind of operation is called a linear combination. Formally, a linear combination of some set of vectors {v"!),...,v("} is given by multiplying each vector v) by a corresponding scalar coefficient and adding the results: Lei. (2.28) The span of a set of vectors is the set of all points obtainable by linear combination of the original vectors. 35CHAPTER 2, LINEAR ALGEBRA Determining whether Az = b has a solution thus amounts to testing whether 6 is in the span of the columns of A. This particular span is known as the column space, or the range, of A. In order for the system Ax = 6 to have a solution for all values of b R™, we therefore require that the column space of A be all of R™. If any point in R’” is excluded from the column space, that point is a potential value of b that has no solution. ‘The requirement that the column space of A be all of R” implies immediately that A must have at least m columns, that is, n > m. Otherwise, the dimensionality of the column space would be less than m. For example, consider a 3 x2 matrix. The target bis 3-D, but a is only 2-D, so modifying the value of x at best enables us to trace out a 2-D plane within R*. The equation has a solution if and only if 6 lies on that plane Having n > m is only a necessary condition for every point to have a solution. It is not a sufficient condition, because it is possible for some of the columns to be redundant. Consider a 2 x2 matrix where both of the columns are identical This has the same column space as a 2 x 1 matrix containing only one copy of the replicated column, In other words, the column space is still just a line and fails to encompass all of R2, even though there are two columns. Formally, this kind of redundancy is known as linear dependence. A sct of vectors is linearly independent if no vector in the set is a linear combination of the other vectors. If we add a vector to a set that is a linear combination of the other vectors in the set, the new vector does not add any points to the set’s span. This means that for the column space of the matrix to encompass all of R”, the matrix must contain at least one set of m linearly independent columns. ‘This condition is both necessary and sufficient for equation 2.11 to have a solution for every value of b. Note that the requirement is for a set to have exactly 7m linearly independent columns, not at least m. No set of m-dimensional vectors more than m mutually linearly independent columns, but a matrix with more than m columns may have more than one such set. can have For the matrix to have an inverse, we additionally need to ensure that equa- tion 2.11 has at most one solution for each value of b. To do so, we need to make certain that the matrix has at most m columns. Otherwise there is way of parametrizing each solution. more than one Together, this means that the matrix must be square, that is, we require that m =n and that all the columns be linearly independent. A square matrix with linearly dependent columns is known as singular. If Ais not square or is square but singular, solving the equation is still possible, but we cannot use the method of matrix inversion to find the solution. 36CHAPTER 2, LINEAR ALGEBRA So far we have discussed matrix inverses as being multiplied on the left. It is also possible to define an inverse that is multiplied on the right: AA?=I. (2.29) For square matrices, the left inverse and right inverse are equal. 2.5 Norms Sometimes we need to measure the size of a vector. In machine learning, we usually measure the size of vectors using a function called a norm. Formally, the I? norm is given by llellp = (Su) , Norms, including the L? norm, are functions mapping vectors to non-negative values. On an intuitive level, the norm of a vector & measures the distance from the origin to the point x. More rigorously, a norm is any function f that satisfies the following properties: for pe R,p>1 ¢ f(z) =0>"=0 © f(@+y) < f(w) + f(y) (the triangle inequality) lel fw: The L? norm, with p = 2, is known as the Euclidean norm, which is simply the Euclidean distance from the origin to the point identified by x The L? norm is used so frequently in machine learning that it is often denoted simply as ||ar||, with the subscript 2 omitted. It is also common to measure the size of a vector using the squared L? norm, which can be calculated simply as «x Va e R, flaw) The squared L? norm is more convenient to work with mathematically and computationally than the L? norm itself. For example, each derivative of the squared L? norm with respect to each element of x depends only on the corre- sponding element of x, while all the derivatives of the L? norm depend on the entire vector. In many contexts, the squared L? norm ma it increases very slowly near the origin. In several machine learning applications, it is important to discriminate between elements that are exactly zero and elements that are small but nonzero. In these cases, we turn to a function that grows at the be undesirable because 37CHAPTER 2, LINEAR ALGEBRA same rate in all locations, but that retains mathematical simplicity: the L! norm The L? norm may be simplified to lelln = >> lal. (2.31) The L! norm is commonly used in machine learning when the difference between zero and nonzero elements is very important. Every time an element of & moves away from 0 by ¢, the L? norm increases by €. v ize of the vector by counting its number of nonzero elements. Some authors refer to this function as the “ZL? norm,” but this is incorrect terminology. The number of nonzero entries in a vector is not a norm, b’ scaling the vector by a does not change the number of nonzero entries. The L! norm is often used as a substitute for the number of nonzero entries, ¢ sometimes measure the use One other norm that commonly aris: also known as the max norm. This norm simplifies to the absolute value of the clement with the largest magnitude in the vector, in machine learning is the L® norm, [x00 = max ||. (2.32) Sometimes we may also wish to measure the size of a matrix. In the context of deep learning, the most common way to do this is with the otherwise obscure Frobenius norm: (2.33) which is analogous to the L? norm of a vector. The dot product of two vectors can be rewritten in terms of norms. Specifically, a! y = ||2||2||yll2 cosd, (2.34) where 0 is the angle between a and y. 2.6 Special Kinds of Matrices and Vectors Some special kinds of matrices and vectors are particularly useful. Diagonal matrices consist mostly of zeros and have nonzero entries only along the main diagonal. Formally, a matrix D is diagonal if and only if Dj = 0 for all i # j. We have already seen one example of a diagonal matrix: the identity 38CHAPTER 2, LINEAR ALGEBRA matrix, where all the diagonal entries are 1. We write diag(v) to denote a square diagonal matrix whose diagonal entries are given by the entries of the vector v. Diagonal matrices are of interest in part because multiplying by a diagonal matrix is computationally efficient. To compute diag(v)a, we only need to scale each element tj by 1%. In other words, diag(v)x = v © a. Inverting a square diagonal matrix is also efficient. The inverse exists only if every diagonal entry is nonzero, and in that case, diag(v)~' = diag([1/v1,...,1/u]"). In many cases, we may derive some general machine learning algorithm in terms of arbitrary matric but obtain a less expensive (and less descriptive) algorithm by restricting some matrices to be diagonal Not all diagonal matrices need be square. It is possible to construct a rectangular diagonal matrix. Nonsquare diagonal matrices do not have inverses, but we can still multiply by them cheaply. For a nonsquare diagonal matrix D, the product Da: will involve scaling each element of # and either concatenating some zeros to lt, if D is taller than it is wide, or discarding some of the last elements of the vector, if D is wider than it is tall the re Asymmetric matrix is any matrix that is equal to its own transpose: A=A™ (2.35) Symmetric matrices often arise when the entries are generated by some function of two arguments that does not depend on the order of the arguments. For example, if A is a matrix of distance measurements, with Aj; giving the distance from point ito point j, then Aj; = Aj; because distance functions are symmetric. A unit vector is a vector with unit norm: Ilelb = 1 (2.36) A vector # and a vector y are orthogonal to each other if ay = 0. If both ors have nonzero norm, this means that they are at a 90 degree angle to each other. In R", at most n vectors may be mutually orthogonal with nonzero norm. If the vectors not only are orthogonal but also have unit norm, we call them orthonormal. vect An orthogonal matrix is a square matrix whose rows are mutually orthonor- mal and whose columns are mutually orthonormal: A’A=AA =I. (2.37) This implics that (2.38)CHAPTER 2, LINEAR ALGEBRA so orthogonal matrices are of interest because their inverse is very cheap to compute. Pay careful attention to the definition of orthogonal matrices. Counterintuitively, their rows are not merely orthogonal but fully orthonormal. There is no s term for a matrix whose rows or columns are orthogonal but not orthonormal ecial 2.7 Eigendecomposition Many mathematical objects can be understood better by bre constituent parts, or finding some properties of them that are universal, not caused by the way we choose to represent them. king them into For example, integers can be decomposed into prime factors. The way we represent the number 12 will change depending on whether we write it in base ten or in binary, but it will always be true that 12 = 2x 2 3. From this representation we can conclude useful properties, for example, that 12 is not divisible by 5, and that any integer multiple of 12 will be divisible by 3. Much as we can discover something about the true nature of an integer by decomposing it into prime factors, we can also decompose matrices in ways that show us information about their functional properties that is not obvious from the representation of the matrix as an array of elements One of the most widely used kinds of matrix decomposition is called eigen- decomposition, in which we decompose a matrix into a set of eigenvectors and eigenvalues. An eigenvector of a square matrix A is a nonzero vector v such that multi- plication by A alters only the scale of v: Av = dv (2.39) The scalar \ is known s the eigenvalue corresponding to this eigenvector. (One can also find a left eigenvector such that v'A = Av', but we are usual concerned with right eigenvectors.) If v is an eigenvector of A, then so is any rescaled vector sv for s € R,s # 0. sv still has the same eigenvalue. For this reason, we usually look only for unit eigenvectors. Moreover, Suppose that a matrix A has n linearly independent eigenvectors {v,..., yl} with corresponding eigenvalues {A1,..., An}. We may concatenate all the eigenvectors to form a matrix V with one eigenvector per column: V = [v")),..., v)], Likewise, we can concatenate the eigenvalues to form a vector A = Di,..-, 40CHAPTER 2, LINEAR ALGEBRA An]. The eigendecomposition of A is then given by A= Viiag(A)V (2.40) We have seen that constructing matrices with specific eigenvalues and eigen- vectors enables us to stretch space in desired directions. Yet we often want to decompose matrices into their eigenvalues and eigenvectors. Doing so can help us analyze certain properties of the matrix, much as decomposing an integer into its prime factors can help us understand the behavior of that integer. Not every matrix can be decomposed into eigenvalues and cigenvee! cases, the decomposition exists but involves complex rather than real numbers. Fortunately, in this book, we usually need to decompose only a specific class of ors. In some pifect of eigenvectors and eigenvalues Before multiplication After multiplication 3 vf B 2] eo he 0) f Gu fia) Ve y 1 z 2| “3 2-1 0 Tt 2 38 2 o 1 2 3 Figure 2.3: An example of the effect of eigenvectors and eigenvalues. Here, we have a matrix A with two orthonormal eigenvectors, uv!) with eigenvalue A; and v) with eigenvalue Ap. (Left)We plot the set of all unit vectors u € R? as a unit circle. (Right)We plot the set of all points Au. By observing the way that A distorts the unit circle, we can sce that it scales space in direction v() by Xv. ALAR ALGEBRA matrices that have a simple decompe matrix can be decomposed into an expression using only real-valued eigenvectors and eigenvalues: ition. Specifically, every real symmetric A=QAQ’, (2.41) where Q is an orthogonal matrix composed of eigenvectors of A, and A is a diagonal matrix. The eigenvalue Aj, is associated with the eigenvector in column i of Q, denoted as Q.;. Because Q is an orthogonal matrix, we can think of A a scaling space by Aj in direction v"). See figure 2.3 for an example. While any real symmetric matrix A is guaranteed to have an cigendecomposi- tion, the eigendecomposition may not be unique. If any two or more eigenvectors share the same eigenvalue, then any set of orthogonal vectors lying in their span are also eigenvectors with that eigenvalue, and we could equivalently choose a Q using those eigenvectors instead. By convention, we usually sort the entries of A in descending order. Under this convention, the cigendecomposition is unique only if all the eigenvalues are unique. The eigendecomposition of a matrix tells us many useful facts about the matrix. The matrix is singular if and only if any of the eigenvalues are zero. The eigendecomposition of a real symmetric matrix can also be used to optimize quadratic expressions of the form f (a) = a! Aa subject to |{z|]2 = 1. Whenever is equal to an eigenvector of A, f takes on the value of the corresponding eigenvalue. The maximum value of f within the constraint region is the maximum eigenvalue and its minimum value within the constraint region is the minimum eigenvalue. A matrix whose eigenvalues are all positive is called positive definite. A matrix whose eigenvalues are all positive or zero valued is called positive semidefi- nite. Likewise, if all eigenvalues are negative, the matrix is negative definite, and if all eigenvalues are negative or zero valued, it is negative semidefinite. Positive semidefinite matrices are interesting because they guarantee that Va, a’ Aa: > 0. Positive definite matrices additionally guarantee that 2’ Ax =0 > x =0. 2.8 Singular Value Decomposition In section 2.7, we saw how to decompose a matrix into eigenvectors and eigenvalues. The singular value decomposition (SVD) provides another way to factorize a matrix, into singular vectors and singular values. The SVD enables us to discover some of the same kind of information as the eigendecomposition reveals; however, the SVD is more generally applicable. Every real matrix has a singular value decomposition, but the same is not true of the eigenvalue decomposition 42