Advanced Computational Fluid Dynamics

AA215A Lecture 1
Vector and Function Spaces

Antony Jameson

Winter Quarter, 2016, Stanford, CA

Last revised on January 7, 2016
Contents

1 Vector and Function Spaces 2

1.1 Basic concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Convex and concave functions: Jensen’s theorem . . . . . . . . . . . . . . . . . . . . 2
1.3 The generalized AM-GM inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Hölder’s inequality and the Cauchy-Schwarz inequality . . . . . . . . . . . . . . . . . 3
1.5 Vector Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5.1 Minkowski’s inequality - triangle inequality for the p norm . . . . . . . . . . . 5
1.5.2 Equivalence of Norms for Finite Dimensional Vectors . . . . . . . . . . . . . . 5
1.6 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.6.1 Infinity Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6.2 1 Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6.3 2 Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.7 Norms of Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.7.1 Function Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.7.2 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.7.3 Non-equivalence of Norms for Functions . . . . . . . . . . . . . . . . . . . . . 9
1.8 Inner Products of Functions: Orthogonality . . . . . . . . . . . . . . . . . . . . . . . 10
1.8.1 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.8.2 Triangle Inequality for the ∞ and 1 Norms . . . . . . . . . . . . . . . . . . . 10
1.8.3 Cauchy Schwarz Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.8.4 Triangle Inequality for the Euclidean Norm . . . . . . . . . . . . . . . . . . . 11
1.8.5 Pythagorean Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.8.6 Linear Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.8.7 Weierstrass Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.9 Portraits of Courant and Hilbert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1
Lecture 1

Vector and Function Spaces

1.1 Basic concepts

The well known concepts of Euclidean space can be generalized to n dimensional vectors and also
to functions. These concepts are extremely useful in the analysis of numerical methods and are
briefly reviewed in this appendix. The idea of function space was one of the great achievements of
nineteenth century mathematics, particularly due to Banach and Hilbert. Hilbert was the first to
introduce spaces with abstract inner product. As a preliminary, it is useful to derive some basic
inequalities due to Jensen, Hölder, Cauchy and Schwarz.

1.2 Convex and concave functions: Jensen’s theorem

A function of f (x) of a real variable x is convex if

f (tx1 + (1 − t)x2 ) ≤ tf (x1 ) + (1 − t)f (x2 ), (1.1)

and concave if
f (tx1 + (1 − t)x2 ) ≥ tf (x1 ) + (1 − t)f (x2 ), (1.2)
whenever 0 < t < 1. It is strictly convex or concave if equality in these expressions implies x1 = x2 .
If f is twice differentiable, it is convex if f 00 ≥ 0 and concave if f 00 ≤ 0, and strictly convex or
concave if f 00 > 0 or f 00 < 0.
Jensen’s theorem:
If f (x) is a concave function, Ã n !
Xn X
ti f (xi ) ≤ f ti xi , (1.3)
i=1 i=1
Pn
whenever t1 , ..., tn ∈ (0, 1) and i=1 ti = 1.
This may be proved by induction. It is true for n = 2 by the definition of a concave function.
Suppose n ≥ 3 and the assertion holds for smaller values of n. Then, for i = 2, ..., n, set t0i =

2
LECTURE 1. VECTOR AND FUNCTION SPACES 3

Pn 0
ti /(1 − t1 ), so that i=2 ti = 1. Then,
n
X n
X
ti f (x1 ) = t1 f (x1 ) + (1 − t1 ) t0i f (xi )
i=1 i=2
à n !
X
≤ t1 f (x1 ) + (1 − t1 )f t0i xi
i=2
à n
!
X
0
≤f t1 x1 + (1 − t1 ) ti xi
i=2
à n
!
X
=f ti xi
i=1

1.3 The generalized AM-GM inequality

The geometric mean (GM) of n real numbers does not exceed the arithmetic mean (AM). This
result follows from Jensen’s
P theorem applied to the function log x, which is strictly concave Thus
if p1 , ..., pn > 0 and ni=1 pi = 1
n
à n !
X X
pi log ai ≤ log pi ai (1.4)
i=1 i=1

and hence
n
Y n
X
api i ≤ pi a i (1.5)
i=1 i=1
1
This is known as the generalized AM-GM inequality. Setting pi = n, recovers the basic AM-GM
inequality
à n !1 n
Y n
1X
ai ≤ ai (1.6)
n
i=1 i=1

1.4 Hölder’s inequality and the Cauchy-Schwarz inequality

1 1
Suppose p, q > 1 and p + q = 1. Then
¯ ¯ Ã !1 Ã n !1
¯Xn ¯ n
X p X q
¯ ¯ p q
¯ ak bk ¯ ≤ |ak | |bk | (1.7)
¯ ¯
k=1 k=1 k=1

Set x1 = ap , x2 = bq , p1 = p1 , p2 = 1q . By the generalized GM-AM theorem

ap bq
ab = xp11 xp22 ≤ p1 x1 + p2 x2 = + (1.8)
p q
Take vectors such that
n
X n
X
|ak |p = |bk |q = 1 (1.9)
k=1 k=1
LECTURE 1. VECTOR AND FUNCTION SPACES 4

Then ¯ n ¯
¯X ¯ X n n µ
X ¶
¯ ¯ |ak |p |bk |q 1 1
¯ a b
k k¯ ≤ |a b
k k | ≤ + = + =1 (1.10)
¯ ¯ p q p q
k=1 k=1 k=1

When p = 2 this reduces to the Cauchy-Schwartz inequality:

¯ n ¯ Ã n !1 Ã n !1
¯X ¯ X 2 X 2
¯ ¯
¯ ak bk ¯ ≤ a2k b2k (1.11)
¯ ¯
k=1 k=1 k=1

This may be proved directly by noting that for any value of ρ

n
X
(ak + ρbk )(ak + ρbk ) ≥ 0 (1.12)
k=1

and consequently ¯ n ¯
n
X ¯X ¯ n
X
2 ¯ ¯ 2
ak + 2ρ ¯ ak bk ¯ + ρ b2k ≥ 0 (1.13)
¯ ¯
k=1 k=1 k=1

Then setting P
| nk=1 ak bk |
ρ = − Pn 2 (1.14)
k=1 bk
Pn 2
and multiplying by k=1 bk à !à ! ¯ n ¯
n
X n
X ¯X ¯
¯ ¯
a2k b2k ≥¯ ak bk ¯ (1.15)
¯ ¯
k=1 k=1 k=1

1.5 Vector Norms

The size of an n-dimensional vector can be conveniently represented by a variety of measures. Such
measures are called norms, which are required to satisfy the following axioms:
To each vector x assign a number ||x|| where

1. ||x|| ≥ 0

2. ||αx|| = |α|||x|| for scalar α

3. ||x + y|| ≤ ||x|| + ||y|| (triangle inequality)

4. ||x|| = 0 if and only if x = 0

Some widely used norms are

P 1
• ||x||p = ( |xi |p ) p for p = 1, 2, 3, ...∞
P
• ||x||1 = |xi |
¡P ¢1
• ||x||2 = |xi |2 2

• ||x||∞ = max |xi |

i
LECTURE 1. VECTOR AND FUNCTION SPACES 5

Conditions (1) and (2) are evident. For (3)

P P
• ||x + y||1 = |xi + yi | ≤ (|xi | + |yi |) ≤ ||x||1 + ||y||1

• ||x + y||∞ = max |xi + yi | ≤ max |xi | + max |yi | ≤ ||x||∞ + ||y||∞
i i i

To verify (3) for ||x||2 use Cauchy-Schwartz inequality

|xT y| ≤ ||x|| ||y|| (1.16)

Then

¡ ¢1
||x + y||2 = (x + y)T (x + y) 2
¡ T T T
¢ 12 ³ T T 1
T 1
T
´1
2
≤ x x + 2|y x| + y y ≤ x x + 2(x x) (y y) + y y
2 2

= ||x||2 + ||y||2

1.5.1 Minkowski’s inequality - triangle inequality for the p norm

||x + y||p ≤ ||x||p + ||y||p (1.17)

Proof:
X X X
|xk + yk |p ≤ |xk + yk |p−1 |xk | + |xk + yk |p−1 |yk | (1.18)
Then by Hölder’s inequality
X ³X ´ 1 µ³X ´1 ³X ´1 ¶
p (p−1)q q p p p p
|xk + yk | ≤ |xk + yk | |xk | + |yk |
³X ´ 1 µ³X ´1 ³X ´1 ¶
p q p p
= |xk + yk | |xk |p + |yk |p (1.19)

since p − 1 = pq . Divide both sides by

³X ´1
q
|xk + yk |p (1.20)

to obtain the triangle inequality.

1.5.2 Equivalence of Norms for Finite Dimensional Vectors

For finite dimensional vectors all norms are equivalent in the sense that for 2 norms M (x) and
N (x) there exist constants c1 , c2 such that for all x

c1 M (x) ≤ N (x) ≤ c2 M (x) (1.21)

1 1
N (x) ≤ M (x) ≤ N (x) (1.22)
c2 c1
LECTURE 1. VECTOR AND FUNCTION SPACES 6

This need only be proved for M (x) = N∞ (x) since if M and N are both equivalent to N∞ then
they are also equivalent to each other. Suppose that

c1 N∞ ≤ M ≤ c2 N∞ (1.23)

and
d1 N∞ ≤ N ≤ d2 N∞ (1.24)
Then
c1 N ≤ d2 M (1.25)
and
d1 M ≤ c2 N (1.26)
where
c1 c2
N ≤M ≤ N (1.27)
d2 d1
Consider the unit cell S of vectors for which Ns (x) = 1. Let x0 and x1 be elements such that

N (x0 ) = min N (x), N (x1 ) = max N (x) (1.28)

x∈S x∈S

(These exist since N (x) is a continuous function of the elements of x.)

y
Now for any y, is in S, so
N∞ (y)
µ ¶
y
N (x0 ) ≤ N ≤ N (x1 ) (1.29)
N∞ (y)
or
N (x0 )N∞ (y) ≤ N (y) ≤ N (x1 )N∞ (y) (1.30)
This is the desired result with
c1 = N (x0 ), c2 = N (x1 ) (1.31)

1.6 Matrix Norms

||Ax||
The induced norm of a matrix is defined as ||A|| = sup .
x6=0 ||x||
||Ax|| is a continuous function of ||x|| so with ||x|| = 1

||A|| = sup ||Ax|| (1.32)

where the max is attained.

Let x be such that with ||x|| = 1

||A + B|| = ||(A + B)x|| ≤ ||Ax|| + ||Bx|| ≤ ||A|| + ||B|| (1.33)

so triangle inequality is satisfied. Also note that

||ABx|| ≤ ||A|| ||Bx|| ≤ ||A|| ||B|| ||x|| (1.34)

so that
||AB|| ≤ ||A|| ||B|| (1.35)
LECTURE 1. VECTOR AND FUNCTION SPACES 7

1.6.1 Infinity Norm

Corresponding to ||x||∞ we have
X
||A||∞ = max |aij | (max absolute row sum) (1.36)
i
j

since ¯ ¯ ¯ ¯ ¯ ¯
¯X ¯ ¯X ¯ ¯X ¯
||Ax||∞ ¯
= max ¯ ¯ ¯
aij xj ¯ ≤ max ¯ ¯ ¯
aij ¯ max |xj | ≤ max ¯ aij ¯¯ ||x||∞ (1.37)
i i j i
j j j

where the max is attained with xj = sgn(aij ).

1.6.2 1 Norm
Corresponding to ||x||1 we have
X
||A||1 = max |aij | (max absolute column sum) (1.38)
j
i

since
à !
X ¯¯ X ¯
¯ XX X X
||Ax||1 = ¯ aij xj ¯¯ ≤ |aij ||xj | = |aij | |xj |
¯
i j i j j i
à ! à !
X X X
≤ max |aij | |xj | = max |aij | ||x||1 (1.39)
j j
i j i

with the max attained when x = ej , where j is the index of the max sum.

1.6.3 2 Norm
Corresponding to ||x||2 we have
||Ax||22 xH AH Ax
= (1.40)
||x||22 xH x
where AH is the Hermitian transpose of A. P
Let ui be an eigenvector of AH A with real eigenvalue σ 2 . Set x = αi ui , then
P 2 2
xH AH Ax α σ
H
= P i 2 i ≤ σ12 (1.41)
x x αi

where σ12 is the largest eigenvalue of AH A. The decomposition is possible because ui are indepen-
dent.

1.7 Norms of Functions

1.7.1 Function Space
For considering errors we need to measure differences between functions. It is convenient to re-
gard functions as vectors in an infinite dimensional space. It turns out that many of the familiar
LECTURE 1. VECTOR AND FUNCTION SPACES 8

concepts of three dimensional Euclidean geometry carry over to function spaces. In particular we
can introduce norms to measure the size of a function, and an abstract definition of a generalized
inner product of two functions, corresponding to the scalar product of two vectors. This leads to
the concept of orthogonal functions. The introduction of axiomatic definition leads to proofs which
hold for a variety of different norms and inner products. In order to enable standard arguments of
real analysis to be carried over, function spaces are required to satisfy an axiom of completeness
that the limit of a sequence of elements in a given space is contained in the space. Spaces with an
inner product are known as Hilbert spaces in honor of their inventor.

1.7.2 Norms
Norms will be a measure of the size of a function, regarded as a vector. For example
µZ 1 ¶ 12
2
||f || = f dx (1.42)
0

is a generalization of
à 3
! 12
X
||f || = fi2 dx (1.43)
1
for Euclidean space.
To be more precise we require the norm of a function to satisfy the following axioms:

1. ||f || ≥ 0

2. ||αf || = |α|||f || for scalar α

3. ||f + g|| ≤ ||f || + ||g||

4. ||f || = 0 if and only if f = 0

If a function is given by a table

fi = f (xi ) for i = 0, 1, 2, ..., n (1.44)

it may be convenient to use as a measure

à n
!1
X 2

||f || = fi2 dx (1.45)

0

Then we can have

||f || = 0 for f 6= 0 (1.46)
This measure satisfies the first 3 axioms only. Such a measure is called a semi-norm.
Examples of norms are:

||f ||∞ = max |f (x)| Maximum norm (1.47)

x∈[a,b]

µZ b ¶ 21
2
||f ||2 = f (x)dx Euclidean norm (1.48)
a
LECTURE 1. VECTOR AND FUNCTION SPACES 9

Z b
||f ||1 = |f (x)|dx (1.49)
a
These are special cases of
µZ b ¶ p1
p
||f ||p = |f (x)| dx for p ≥ 1 (1.50)
a

Note that, for example, we can regard ||f ||2 as the limit of
à n
!1
1X 2 2

||f || = fi dx (1.51)
n
0

where fi is the table of values representing f (x).

We can also introduce weighted norms such as the weighted Euclidean norm
µZ b ¶ 12
||f || = f 2 (x)w(x)dx (1.52)
a

where w(x) is a non-negative weight function.

All these norms can be shown to satisfy the axioms.

1.7.3 Non-equivalence of Norms for Functions

For functions norms are not equivalent. If p > q then

||f ||p → 0 implies ||f ||q → 0 (1.53)

but not the other way round.

For example consider ½ 1 1
1 if − 2n ≤x≤ 2n
fn = (1.54)
0 otherwise
Then Z 1
1
fn2 dx = (1.55)
−1 n
and
1
||fn ||2 = √ → 0 as n → ∞ (1.56)
n
while
||fn ||∞ = 1 (1.57)
On the other hand if
||f ||∞ = ² (1.58)
then Z b
f 2 dx ≤ ²2 (b − a) (1.59)
a
so √
||f ||2 ≤ ² b − a → 0 as ² → 0 (1.60)
LECTURE 1. VECTOR AND FUNCTION SPACES 10

1.8 Inner Products of Functions: Orthogonality

1.8.1 Orthogonality
It is also convenient to introduce the idea of orthogonality in function space. Define the inner
product as
Z b
(f, g) = f (x) g(x) w(x) dx in the continuous case
a
n
X
= fi gi wi in the discrete case (1.61)
i=0

Then if (f, g) vanishes the functions are said to be orthogonal.

We have
(f, g) = (g, f ) commutative
(c1 f + c2 g, Φ) = c1 (f, Φ) + c2 (g, Φ) linearity (1.62)
(f, f ) ≥ 0 positivity
These can be taken as axioms for an abstract inner product.

1.8.2 Triangle Inequality for the ∞ and 1 Norms

We have
max |f + g| ≤ max(|f | + |g|) ≤ max |f | + max |g| (1.63)
and Z Z
|f + g|dx ≤ (|f | + |g|)dx (1.64)

Thus the ∞ and 1 norms satisfy the triangle inequality. They also already satisfy axioms (1), (2)
and (4).

1.8.3 Cauchy Schwarz Inequality

(f, g)2 ≤ (f, f )(g, g) (1.65)

Proof:

0 ≤ (f + pg, f + pg) ≤ (f, f ) + 2p|(f, g)| + p2 (g, g) (1.66)

But since this is a quadratic function of p it has imaginary roots or coincident real roots. The
discriminant yields the desired inequality.
Or set
|(f, g)|
p=− (1.67)
(g, g)
and multiply by (g, g) to get

0 ≤ (f, f ) (g, g) − 2|(f, g)|2 + |(f, g)|2 (1.68)

LECTURE 1. VECTOR AND FUNCTION SPACES 11

1.8.4 Triangle Inequality for the Euclidean Norm

1
||f + g||2 = (f + g, f + g) 2
1
≤ ((f, f ) + 2|(f, g)| + (g, g)) 2
³ 1 1
´1 (1.69)
2
≤ (f, f ) + 2(f, f ) 2 (g, g) 2 + (g, g)
= ||f || + ||g||
using the Schwartz inequality.

1.8.5 Pythagorean Theorem

In the Euclidean norm, if (f, g) = 0 then

||f + g||2 = ||f ||2 + ||g||2 (1.70)

Proof:

||f + g||2 = (f + g, f + g) = (f, f ) + (f, g) + (g, f ) + (g, g) (1.71)

1.8.6 Linear Independence

A set of functions φ0 , φ1 , ..., φn is said to be linearly independent if
¯¯X ¯¯
¯¯ ¯¯
¯¯ ci φi ¯¯ = 0 (1.72)

implies
ci = 0 for i = 0, 1, ..., n (1.73)
Orthogonal functions are independent since then from the Pythagorean theorem
¯¯ n ¯¯2
¯¯X ¯¯ n
X
¯¯ ¯¯
¯¯ ci φi ¯¯ = c2i ||φi ||2 (1.74)
¯¯ ¯¯
i=0 i=0

1.8.7 Weierstrass Theorem

Let f (x) be given in interval [a, b]. Let the lower bound of the error in the maximum norm for all
polynomials of order n be
En (f ) = min ||f − pn (x)||∞ (1.75)
pn (x)

Then if f is continuous
lim En (f ) = 0 (1.76)
n→∞
That is, a continuous function can be arbitrarily well approximated in a closed interval by a
polynomial of sufficiently high order.
The proof is by construction of the required pn (x) (Isaacson and Keller, p183).

1.9 Portraits of Courant and Hilbert

LECTURE 1. VECTOR AND FUNCTION SPACES 12

Figure 1.1: Richard Courant (1888-1972)

Figure 1.2: David Hilbert (1862-1943)