Appendix

A.1 Linear Algebra, Vectors, and Matrices

Vectors provide an efficient way to compute with multidimensional data.
An ordered pair of numbers can be regarded as a point or a vector in
two-dimensional space, as shown in Figure A.1.
Similarly, a point in three-dimensional space is represented by a vector
with three components. A vector with n dimensions is written using the
notation xZ [x1, x2, ., xn]. For example, an input vector with values for n
variables is an n-dimensional vector. A vector can be written in row format
or column format. For example:

x Z ½ 3; 2; 1 

is in row format, while

2 3
3
6 7
y Z425
1
is in column format. Because the individual values and their order are the
same for both x and y, y is called the transpose of x, written as y Z xT, or x
is the transpose of y, written as x Z yT.

A.1.1 Addition of Vectors

Vectors can be added or subtracted following the usual way addition or
subtraction is carried out on an element-by-element basis. Consider two

555

q 2006 by Taylor & Francis Group, LLC

556 & Neural Networks for Applied Sciences and Engineering

X2 X2

(2, 4) (2, 4)

X = [2, 4]

X1 X1

Figure A.1 A point in two-dimensional space (left) and the same point viewed as a
vector (right).

vectors, x and y:
x Z ½ 2; 3; 1 
y Z½1 6 5
xCy Z½3 9 6
xK y Z ½ 1 K3 K4 :

A.1.2 Multiplication of a Vector by a Scalar

A vector can be multiplied by a scalar (a constant value). For example,
multiplying x Z [3 2 1] by 2 produces a vector twice as long, 2x Z [6 4 2].

A.1.3 The Norm of a Vector

The length (also called norm or magnitude) of a vector denotes the distance
to the point referred to by its coordinates. The norm or magnitude of a
vector x with n components is
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
kxk Z x12 C x22 C /C xn2 :

Therefore, for the vector x Z [3, 2, 1], the norm is

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffi
kxk Z 32 C 22 C 12 Z 14:

q 2006 by Taylor & Francis Group, LLC

 Appendix & 557

A.1.4 Vector Multiplication: Dot Products

The dot product provides an efficient method of multiplying two vectors.
The dot product of two vectors xZ[x1, x2, ., xn] and wZ[w1, w2, ., wn] is

x$w Z x1 w1 C x2 w2 C . C xn wn ;

which is the sum of products of vector components. The dot product can
also be written as

x$w Z kxkkwkðcos qÞ;

where q represents the angle between the two vectors. This is presented in
Figure A.2.
Example. Find the dot product of the vectors [K1 3 6 K2] and
[1 2 2 K3]. Find the angle between the two vectors.
Solution. The dot product is

½1 3 6 2$½1 2 2 3

Z 1 !1 C 3 !2 C 6 !2 C 2 !3 Z 23:

The angle between the two vectors is (from the above equation)
23
cos q Z pffiffiffiffiffipffiffiffiffiffi Z 39:948 :
50 18
pffiffiffiffiffi pffiffiffiffiffi
The 50 and 18 in the above expression are norms or the lengths of
the two vectors x and w. When two vectors are perpendicular or
orthogonal to each other, their dot product is zero. This is because when
q Z908, cos q Z0.

X2
x

X1

Figure A.2 Two vectors, x and w, and the angle between them.

q 2006 by Taylor & Francis Group, LLC

558 & Neural Networks for Applied Sciences and Engineering

A.2 Matrices
Operating with many vectors can be greatly simplified by organizing them
in matrices. A matrix is a rectangular array of numbers. An m ! n matrix is a
matrix with m rows and n columns that represent vectors. Each element of a
matrix can be indexed by its row and column position. The order of
elements is therefore written as

2 3
a11 a12 a13 . a1n
6a a22 a23 a2n 7
6 21 . 7
6 7
A Z6
6 a31 a32 a33 . a3n 7:
7
6 7
4 « 5
am1 am2 am3 . amn

For the 3 ! 2 matrix B, given below, the element b32 ZK5 and b21 Z 2:
2 3
K3 3
6 7
B Z4 2 9 5:
1 K5

A.2.1 Matrix Addition

Matrices can be added by summing their corresponding elements. For
example, if CZ ACB, then cij Z aij C bij. For example, if
2 3 2 3
3 1 K2 1
6 7 6 3 27
A Z42 45 and B Z 4 5;
1 5 1 4

then
2 3 2 3
3K 2 1C1 1 2
6 7 6 7
C ZACB Z42C3 4 C 2 5 Z 4 5 6 5:
1C1 5C4 2 9

A.2.2 Matrix Multiplication

To multiply two matrices, A and B (written as AB), the number of columns in
A must match the number of rows in B. If A is an m ! n matrix and B is an

q 2006 by Taylor & Francis Group, LLC

 Appendix & 559

n ! s matrix, then the matrix product AB is an m ! s matrix. If C Z AB,

then an element of this product is defined as

Cij Z ðith row vector of AÞ$ð jth column vector of BÞ

X
n
Z aik bkj :
kZ1

Example. Compute the product

2 3
" # 4 2 1 1
K2 3 1 6 7
CZ 4K3 K1 4 35
1 2 4
1 4 5 1

Solution. The elements c11 and c23 are calculated as

" #
K2 !4 C 3 !K 3 C 1 !1. . .
CZ ;
. .1 !1 C 2 !4 C 4 !5 .

and the complete solution is

" #
K16 K3 15 8
CZ :
2 16 29 11

Using this procedure, any number of matrices can be multiplied together

by sequentially carrying out the computation.

A.2.3 Multiplication of a Matrix by a Vector

A matrix can be multiplied by a vector in a similar manner to the
multiplication of two matrices. For example, for a matrix w of size 2 ! 2 and
a vector x of size 2 ! 1, given as
" # " #
1 2 5
wZ ; xZ ;
3 4 6

multiplication gives a 2 ! 1 vector, u, as

" #" # " # " #
1 2 5 5 C 12 17
uZ Z Z :
3 4 6 15 C 24 39

In this manner, many matrices and vectors can be multiplied together.

q 2006 by Taylor & Francis Group, LLC

560 & Neural Networks for Applied Sciences and Engineering

A.2.4 Matrix Transpose

Similar to vector transpose, matrices are transposed by arranging original
rows as columns, or vice versa. If matrix B is the transpose of matrix A, then
the element bij is the same as element aji.
Example. Write the matrix B if B Z AT and
" #
K2 3 1
AZ :
1 2 4

Solution.
2 3
K2 1
6 7
B Z AT Z 4 3 2 5
1 4

References
1. Callan, R. The Essence of Neural Networks, Pearson Education, London, UK,
1999.

q 2006 by Taylor & Francis Group, LLC