Lecture

Notes 2021
Analysis 2
Lecturer: Upendra Kulkarni
Scribe: Soham Chatterjee
 Introduction

This is the lecture notes scribed by me. If you find any mistakes in the notes please email me at sohamc@
cmi.ac.in.
The whole course is taken by Prof. Upendra Kulkarni, online. If you want the lecture videos then you can
find them in this link. Sir mainly followed Prof. Pramath Sastry’s Notes (https://www.cmi.ac.in/~pramath/
teaching.html#ANA2). You can find all the assignments problems in the following drive link. Through out the
course the book we followed is Principles of Mathematical Analysis by Walter Rudin.

2
 Contents

Chapter 1 Normed Linear Space Page 5

1.1 Defination 5
1.2 Open and Closed Ball 7
1.3 Limit of a Sequence 8
1.4 Continuity 8

Chapter 2 Metric Space Page 9

2.1 Definition 9
2.2 Open and Closed Ball and Set 9
2.3 Topological Space 12

Chapter 3 Continuity in Metric Space Page 15

3.1 Limit Point and Closure 15
3.2 Continuity 16

Chapter 4 Equivalence of Norms Page 20

Chapter 5 Compactness Page 23

5.1 Sequentially Compact 23
5.2 Open Cover and Compactness 25

Chapter 6 Differentiation Page 30

6.1 Partial Derivatives 30
6.2 Differentiation 31

Chapter 7 Examples on Multivariable Differentiation Page 34

Chapter 8 Chain Rule of Differentiation and Operator Norm Page 36

8.1 Operator Name 36

3
 8.2 Chain Rule 37
8.3 Special Case of Chain Rule: When m = k = 1 38

Chapter 9 Mean Value Theorem Page 40

Chapter 10 Higher Derivatives Page 42

1
10.1 Class C Functions 42
10.2 Higher Derivatives and Class C k functions 44

Chapter 11 Multivariable Taylor Theorem Page 47

Chapter 12 Maximum and Minimum of Multivariable Functions Page 50

Chapter 13 Examples of Functions and Analyze Critical Points Page 54

Chapter 14 Constrained Optimizations and Lagrange Multipliers Page 58

14.1 Tangent Space 58
14.2 Lagrange Multiplier 59
14.3 Some Examples for Applications 59
14.4 Generalized Lagrange Multiplier 60

Chapter 15 Inverse Function Theorem Page 62

Chapter 16 Implicit Function Theorem Page 66

Chapter 17 Complex Differentiation Page 69

4
Chapter 1

Normed Linear Space

Definition 1.0.1: Limit of Sequence in R

Let {sn } be a sequence in R. We say

lim sn = s
n→∞

where s ∈ R if ∀ real numbers ϵ > 0 ∃ natural number N such that for n > N

s − ϵ < sn < s + ϵ i.e. |s − sn | < ϵ

Want to generalize this to a sequence in Rn i.e. sn ∈ Rn ∀ n ∈ N. Now the s − sn makes no sense. So it is

useful to have a notion of whether vectors are big or small. We have magnitude of a vector. So lets revisit this
Definition 1.0.2: Limit of Sequence in Rn

Let {sn } be a sequence in Rn . We say

lim sn = s
n→∞

where s ∈ Rn if ∀ real numbers ϵ > 0 ∃ natural number N such that for n > N

∥s − sn ∥ < ϵ

The same definition works if we interpret ∥v∥ = length of the vector v.

From school, for v=v1 , v2 , · · · , vn we had
q
∥v∥ = v12 + v22 + · · · + vn2
But it will be useful to have a more general notion of length (of which the above will be an example)

1.1 Defination
Definition 1.1.1: Normed Linear Space and Norm ∥ · ∥

Let V be a vector space over R (or C). A norm on V is function ∥ · ∥ V → R≥0 satisfying
1 ∥x∥ = 0 ⇐⇒ x = 0 ∀ x ∈ V

2 ∥λx∥ = |λ|∥x∥ ∀ λ ∈ R(or C), x ∈ V

3 ∥x + y∥ ≤ ∥x∥ + ∥y∥ ∀ x, y ∈ V (Triangle Inequality/Subadditivity)

And V is called a normed linear space.

• Same definition works with V a vector space
√ over C (again ∥·∥ → R≥0 ) where 2 becomes ∥λx∥ = |λ|∥x∥
∀ λ ∈ C, x ∈ V , where for λ = a + ib, |λ| = a2 + b2

5
 Example 1.1.1 (p−Norm)
V = Rm , p ∈ R≥0 . Define for x = (x1 , x2 , · · · , xm ) ∈ Rm
  p1
∥x∥p = |x1 |p + |x2 |p + · · · + |xm |p

(In school p = 2)

Special Case p = 1: ∥x∥1 = |x1 | + |x2 | + · · · + |xm | is clearly a norm by usual triangle inequality.
Special Case p → ∞ (Rm with ∥ · ∥∞ ): ∥x∥∞ = max{|x1 |, |x2 |, · · · , |xm |}
For m = 1 these p−norms are nothing but |x|. Now exercise
Question 1

Prove that triangle inequality is true if p ≥ 1 for p−norms. (What goes wrong for p < 1 ?)

Solution: For Property 3 for norm-2

When field is R :
We have to show
 2
X sX sX
(xi + yi )2 ≤  x2i + yi2 
i i i
v" #" #
u
X X u X X X
=⇒ (x2i + 2xi yi + yi2 ) ≤ x2 + 2 t
i x2 y2 + i y2 i i
i i i i i
" #2 " #" #
X X X
=⇒ xi yi ≤ x2i yi2
i i i
2
So in other words prove ⟨x, y⟩ ≤ ⟨x, x⟩⟨y, y⟩ where
X
⟨x, y⟩ = xi yi
i

Note:-
• ∥x∥2 = ⟨x, x⟩
• ⟨x, y⟩ = ⟨y, x⟩
• ⟨·, ·⟩ is R−linear in each slot i.e.

⟨rx + x′ , y⟩ = r⟨x, y⟩ + ⟨x′ , y⟩ and similarly for second slot

Here in ⟨x, y⟩ x is in first slot and y is in second slot.

Now the statement is just the Cauchy-Schwartz Inequality. For proof

⟨x, y⟩2 ≤ ⟨x, x⟩⟨y, y⟩
expand everything of ⟨x − λy, x − λy⟩ which is going to give a quadratic equation in variable λ
⟨x − λy, x − λy⟩ = ⟨x, x − λy⟩ − λ⟨y, x − λy⟩
= ⟨x, x⟩ − λ⟨x, y⟩ − λ⟨y, x⟩ + λ2 ⟨y, y⟩
= ⟨x, x⟩ − 2λ⟨x, y⟩ + λ2 ⟨y, y⟩
Now unless x = λy we have ⟨x − λy, x − λy⟩ > 0 Hence the quadratic equation has no root therefore the
discriminant is greater than zero.
6
When field is C :
Modify the definition by X
⟨x, y⟩ = xi yi
i

Then we still have ⟨x, x⟩ ≥ 0

1.2 Open and Closed Ball

Definition 1.2.1: Open and Closed Ball in Normed Linear Space

An open Ball of radius r with center x in Normed Linear Space V is the set

{y ∈ V | ∥x − y∥ < r} = Br (x)

and Closed ball is the set

{y ∈ V | ∥x − y∥ ≤ r} = Br (x)

Now take Br (0) w.r.t ∥ · ∥1 , ∥ · ∥2 , ∥ · ∥∞ . Now imagine a sequence converging to origin. So if I

draw an ordinary circle around the origin then no matter how small the circle the points of the sequence
are eventually land inside the circle. If instead of that circle can same be said for diamond w.r.t norm 2. Then
i can take circle that is inside that diamond. Same is true for ∞−norm. Hence convergence with respect to all
norm 1 and norm 2 and even ∞ results for convergence.
Now there is no reason why we can not consider a norm on an infinite dimensional vector space. It will
work. Perhaps i can define only for some sequences where the morm converges.

Example 1.2.1
Suppose for set of all bounded infinite sequences a vector space because every number in a vector is less
than some number so if you add two vectors then add the bound and if you scale then scale the bound.
Now the ∞ norm works on that.
Now suppose you take all continuous real valued functions on closed interval [0, 1], such a function is
bounded and this is a vector space and we can define ∞−norm even for that because for all f in this space
attains its maximum value so just take that maximum value. Its an extremely infinite dimensional space.

Note:-
∞
R is the space of all sequences.

Question 2

Modify the above proof for field C

7
 Question 3

Show that the following are normed linear spaces.

(a) l∞ = Set of all bounded infinite sequences (x1 , x2 , · · · ) xi ∈ R with norm ∥x∥ = sup |xi |

(b) C[0, 1] = Set of all continuous functions [0, 1] → R with norm ∥f ∥ = sup |f (x)|
x∈[0,1]

1.3 Limit of a Sequence

Definition 1.3.1: Limit of Sequence in Normed Linear Space

A sequence {sn } in a normed linear space V converge to s means ∀ real number ϵ > 0 ∃ natural number
N such that for ∀ n > N ∥s − sn ∥ < ϵ

1.4 Continuity
Definition 1.4.1: Continuity in Normed Linear Space

Let S be a subset of V and f : S → W where V, W are normed linear space. f is continuous at v ∈ V

means ∀ ϵ > 0, ∃ δ > 0, st whenever ∥x − v∥ < δ for x ∈ S one has ∥f (x) − f (v)∥ < ϵ

Distance in a normed linear space for x, y ∈ V is

d(x, y) = ∥x, y∥

Hence properties of this d are

1 d(x, y) = 0 ⇐⇒ x = y

2 d(λx, λy) = |λ|d(x, y) for any scalar λ

3 d(u, v) + d(u, v) ≥ d(u, w)

8
Chapter 2

Metric Space

2.1 Definition
Definition 2.1.1: Metric Space X

A set X with a function d X × X → R≥0 such that

1 d(x, y) = 0 ⇐⇒ x = y

2 d(x, y) = d(y, x)

3 d(x, z) ≤ d(x, y) + d(y, z)

Notice that there is no homogeneity condition, and it does ot make sense as we don’t have a field. In fact
there is no notion of addition. But the condition 1 of norm has to be satisfied by this distance. Also we don’t
have a translational condition i.e. distance between x, y and distance between x + v, y + v has to be same. Hence
Note:-
A metric space need not be a vector space. So it doesn’t need a zero, or a notion of addition or scalar
multiplication.

If I take a metric space and take any subset of it. And those three conditions of distance functions are still
satisfied.
Note:-
Any subset of metric space is a metric space under the same distance function.

2.2 Open and Closed Ball and Set

Definition 2.2.1: Open Ball and Closed Ball in a Metric Space

An open ball of radius r with center c ∈ X in a metric space X is

Br (c) = {x ∈ X | d(c, x) < r}

and a closed ball is

Br (c) = {x ∈ X | d(c, x) ≤ r}

9
 Definition 2.2.2: Open Set and Closed Ball in a Metric Space

An open set in a metric space X is one of the form of union of some open balls and a closed set in a metric
space X is one of the form of X\some open sets

Note:-
We will do topology in Normed Linear Space (Mainly Rn and occasionally Cn )using the language of Metric
Space

Example 2.2.1 (Open Set and Close Set)

Open Set: • ϕS
• Br (x) (Any r > 0 will do)
x∈X
• Br (x) is open
Closed Set: • X, ϕ
• Br (x)
x−axis ∪ y−axis

Question 4

Is the set x−axis\{Origin} a closed set

Solution: We have to take its complement and check whether that set is a open set i.e. if it is a union of open
balls
Now this works well for points which are above or below the x−axis. But for origin no matter how small
the ball we take it willl have points from x−axis. Hence the set is not a closed set.
Question 5

Any continuous path in R2 is closed where path = f : [0, 1] → R2

Solution: This is true. To be proved later.

Analogous to: For continuous function f : [0, 1] → R, the image is a closed interval
Question 6

If i take X = x−axis ∪ y−axis then is it open

Solution: Yes because here the space is only the union of those two axis. So any ball would be like a cross or
line but it just as the metric space given to us. [It is open for this metric space but not open in R2 ]
Note:-
If S ⊂ X, then S itself has a collection of open sets of S by containing S as a metric space.

Definition 2.2.3: Neighborhood

For a point x in metric space X, a neighborhood of x is a set N such that x ∈an open set U ⊂ N
If N itself is open. then we say that N is an open neighborhood
N of x
U
x

10
 Theorem 2.2.1
If x ∈ open set V then ∃ δ > 0 such that Bδ (x) ⊂ V

Proof: By openness of V , x ∈ Br (u) ⊂ V

Bδ (x)
Br (u)
x
u

Given x ∈ Br (u) ⊂ V , we want δ > 0 such that x ∈ Bδ (x) ⊂ Br (u) ⊂ V . Let d = d(u, x). Choose δ such
that d + δ < r (e.g. δ < r−d
2 )
If y ∈ Bδ (x) we will be done by showing that d(u, y) < r but

d(u, y) ≤ d(u, x) + d(x, y) < d + δ < r

Note:-
S
V is open ⇐⇒ Br (x) (where r depends on x)
x∈V

Theorem 2.2.2
Let X be a metric space.
1. Union of open sets is open
2. Intersection of two open sets is open

Analogues to these as we are just taking complement of the open sets

1′ . Arbitrary intersection of closed sets is closed
2′ . Finite union of closed sets is closed.

S
Proof: 1. Let {Vα }α∈I be a collection of open sets where I is an index set. We want ti show Vα is open
S α∈I
in X. Since each Vα is open Vα = Brβ (cβ ) Then
β∈Jα

[ [ [
Vα = Brβ (cβ )
α∈I α∈I β∈Jα
[
= Brβ (cβ )
β∈⊔Jα

which is still a union of balls

2. The statement implies intersection of finite number of open sets is open. We can prove this by induction.
We will do by showing that for each x ∈ V1 ∩ V2 ∃ r > 0 s.t. Br (x) ⊂ V1 ∩ V2

11
 V2
V1

Br (x)

As x ∈ V1 ∃ r1 such that x ∈ Br1 (x) ⊂ V1 . Similarly x ∈ V2 ∃ r2 such that x ∈ Br2 (x) ⊂ V2 . Take
r = min{r1 , r2 }. Thus we have x ∈ Br (x) ⊂ V1 ∩ V2
The second part for closed sets are left as exercise

2.3 Topological Space

Definition 2.3.1: Topological Space

A topological space is a set X together with a collection of subsets of X (i.e. a subset of the power set of
X) that is closed under taking arbitrary unions and finite intersections. This collection is called a topology
on X

Note:-
S
Union means Sα = {x ∈ X | ∃ α s.t. x ∈ Sα }
α∈I T
Intersection means Sα = {x ∈ X | ∀α, x ∈ Sα }
α∈I

Question 7

Suppose i have a topological space X under given some topology. Is the entire set open ? And that the
empty set is open ?
S
Solution: If I = ϕ, Sα = {x ∈ X | ∃ α ∈ I s.t. x ∈ Sα } gives ϕ and
T α∈I
Sα = {x ∈ X | ∀α ∈ I, x ∈ Sα } gives X because ∀ α ∈ I condition is vacuously true for each x ∈ X.
α∈I

Note:-
Intersection of empty families are not defined in set theory. This brings a very important point. In a set
theory you have to have a universe. (Set theory have to avoid paradoxes, Russel Paradox) At the beginning
you construct a large enough universe and you taking subsets only from that universe. Notice all subsets we
are considering here are subsets of X and here we defined how we union and intersection mean. Though it
still this asks what our axioms of set theory. So you can change the part of the definition of topological space
like this “. . . with a collection of subsets of X including the empty set and the whole space. . . ”

(If you don’t like this as it is)

Note:-
If S is a subset of metric space X, then S is itself a metric space and as such open/closed sets as subsets of
metric space

12
 Question 8

Is there any connection between being open in X and being open in S (Similar question for closed)

Solution: Let x ∈ S. Now, Ball of radius r in S = S∩ Ball of radius r in X. Therefore

[
Open Set in S = Balls in S
[
= (Balls in X ∩ S)
[ 
= Balls in X ∩ S
= Open set X ∩ S
Part 2 is left as exercise
Corollary 2.3.1
If S ⊂ X is open in X then a subset T of S is open in S ⇐⇒ T is open in X

Corollary 2.3.2
If S ⊂ X is closed in X then a subset T of S is closed in S ⇐⇒ T is closed in X

Definition 2.3.2: Subspace of a Topological Space X

For any subset S of a topological space X, the collection S ∩ U , U open in X is called a subspace.

Question 9

Prove that subspace of a metric space X defines a topology on X

Wrong Concept 2.1

If x ∈ open V then there exists r > 0 such that x ∈ Br (x) ⊂ V

B
x

Idea: Why not we take r = inf{distance from x to boundary of ball B}.

Now we first have to ensure r > 0. Suppose that’s true.
Then we have to define boundary. What is boundary, We can give a reasonable definition (Boundary
has already a definition but we don’t know that for now). Let boundary of B = {x ∈ X | d(c, x) = δ}
Now this definition is not proper for our purpose.Because if we take union of all balls in V then we will
have lots of points as boundary but part of them should not be considered as boundary. Even if we take
this definition.
Then the big question comes/ We are taking a infimum of a certain set of real numbers. The very
first question arises is whether this set is nonempty. For example if we take B to be the metric space it

13
self we have no boundary.

Questions which come thorough this.

• Is there a meaningful way to define boundary
• Can we modify the idea

14
Chapter 3

Continuity in Metric Space

3.1 Limit Point and Closure

Definition 3.1.1: Limit Point

S ⊂ X is a metric space. We say that x ∈ X is a limit point of S if ∃ a sequence {sn } with all sn ∈ S \ {x}
such that sn → x (each sn is different from x)

Theorem 3.1.1
x is a limit point of S ⇐⇒ every neighborhood of x in X contains a point of S other than S.

Proof: If Part:
Let x be a limit point of S. Therefore take a sequence {sn } in S \ {x} with sn → x .
To prove what we want it is enough to show that Br (x) ∩ S contains a point other than x. As sn → x, ∃
N s.t. ∀ n > N d(x, sn ) < r i.e. sn ∈ Br (x). In particular sn ]inBr (x) ∩ (S \ {x})

Only If Part:
We need to produce a sequence {sn } ∈ S \ {x} with lim sn = x. Take sn ∈ B n1 (x) ∩ (S \ {x}) See that lim sn = x.
n→∞
This is essentially because n1 → 0.
Complete the rest of the proof.

Definition 3.1.2: Closure

Given a topological space X and S ⊂ X, the closure of the set S is S the smallest closed set containing S.

Theorem 3.1.2
S = Smallest closed set of X containing S = A
= S ∪ (limit points of S) = B
= {x ∈ X | x = lim for some sequence {sn } in S} = C
n→∞
= {x ∈ X | Every neighborhood of x intersects S} = D

Proof: A⊂D
Ac = (All open set V s.t. V ∩ S = ϕ)
S
Dc = {x ∈ X | ∃ open neighborhood of x, B s.t. B ∩ S = ϕ}
Clearly for all x ∈ Dc , x ∈ Ac . Hence Dc ⊂ Ac =⇒ A ⊂ D
15
D⊂B
Take x ∈ D. Suppose x ∈ / S. Now any neighborhood of x intersects S in a point hence it has to be a different
point from x since x ∈
/ S. Therefore x is a limit point of S. D ⊂ B

B⊂C
If x ∈ S then take a sequence

Question 10

What does it mean to be smallest closed set containing the set S here ?
T
Solution: All closed sets containing S is automatically closed and hence the smallest closed set containing S.
Proof: For proof of Theorem 3.1.2 notice A,B,C,D all contains S (obvious).
Note:-
We don’t need to show B,C,D are closed. We can also take the sets element wise and show each set is a subset
of the other. This may simplify our way of proof. (exercise)

Now see A and D completely deal with topology. A is about closed sets and D is about open sets. So A
and D close to each other. Now by the 3.1.1 we have equivalence of C and D. So we can prove like this

A ⇐⇒ D ⇐⇒ B & C

Left as exercise

Note:-
For these kind of proofs instead of looking for the most efficient way try to find a path that allows you to go
from anywhere to anywhere

3.2 Continuity
Definition 3.2.1: Continuity

f : X → Y function between metric spaces is continuous at a ∈ X if ∀ ϵ > 0 ∃ δ > 0 s.t.

d(x, a) < δ =⇒ d(f (a), f (x)) < ϵ

⇕ ⇕
x ∈ Bδ (x) =⇒ f (x) ∈ Bϵ (f (a))

That means f −1 (Any ball around f (a)) ⊃ Ball around a.

So f : X → Y is continuous at all points ⇐⇒ f −1 (Any ball intersecting the range) ⊃ A ball
Note:-
We can not say f −1 (Any ball) because because we need a ball that contains a point in the range

Theorem 3.2.1
f is continuous ⇐⇒ f −1 (Any open set in Y ) is open in X

16
Proof: If Part:-
 
It is enough to show f −1 (Any ball) is open on X because f −1 preserves unions f −1 f −1 (Vα )
S S

Vα =
α α
Let B is any open set (as its conceptually simpler to take open set here instead of a ball) in Y . Let
a ∈ f −1 (B). Hence we can say f (a) ∈ B. Since B is an open set we can say there is a ball
 Bϵ (f (a)) ⊂ B. Since f
is continuous ∃ δ such that f (x) ∈ Bϵ (f (a)) whenever x ∈ Bδ (a). Now f −1 (B) ⊃ f −1 Bϵ (f (a)) ⊃ Bδ (a) Hence

f −1 (B) is open.

Only If Part:-
 
Lets prove continuity ar a ∈ X. We are given that f −1 Bϵ (f (a)) is open and obviously contains a. Therefore


f −1 Bϵ f (a) contains a ball around a. Take δ = Radius of the ball.

Question 11

For a metric space X, show that S = {x ∈ X | lim sn = x} for some sequence {sn } in S.
n→∞

Question 12

For a function f : X → Y between metric spaces, show that the followings are equivalent.
1. f is continuous

2. f −1 (Open Set) is open

3. f −1 (Closed Set) is closed
4. f (S) = f (S)

5. xn → x =⇒ f (xn ) → f (x)
One or more of the above are wrong so check if they are true and if not then find the true statement.

Solution: 4 is wrong. How to correct and rest is left as exercise

Question 13

For f : X → Y any set map

(i) f −1 preserves unions, intersections, complements

(ii) Is there any condition on f under which f possesses the property above ?

Example 3.2.1 (Continuous Function)

1. Any constant function.

f g
2. X −
→Y −
→ Z f, g continuous =⇒ g · f is continuous
Inclusion
3. Is S ⊂ X then S −−−−−→ X is continuous
4. Projection Rn → R
(x1 ,x2 ,··· ,xn )7→xi

More generally for example R3 → R4

(x,y,z)7→(x,x,y,y)

17
 5. Map from metric space to euclidean space.
)
X → Rn f is continuous
⇐⇒
x 7→ (f1 (x), f2 (x), · · · , fn (x)) each fi is continuous

6. R × R → R : (x, y) 7→ x ± y, xy are continuous.

(
xn ± yn → x ± y
We need to prove xn → x and yn → y in R =⇒
xn yn → xy
1
R \ {0} → R : x 7→ x is continuous
7. sum and product of two continuous real valued function on X are continuous
f,g f,g +
f, g : X −−→ R continuous =⇒ X −−→ R × R −
→R
x 7−→ (f (x),g(x))

1
f : X → R =⇒ : X \ f −1 (0) → R is continuous
f | {z }
open set in X
−1
{0} is closed in R, so f (0) is closed in X by continuity of f
Therefore any polynomial in continuous real valued functions on X is continuous.

8. Special Case:
T
• Rn − → Rm linear map is continuous where (x1 , x2 , · · · , xn ) 7−→ (a11 x1 + · · · + a1n xn , a21 x1 +
· · · + a2n xn , · · · , am1 x1 + · · · + amn xn )
 
a11 a12 · · · a1n
 a21 a22 · · · a2n 
Matrix of T =  . .. .. 
 
 .. ..
. . . 
am1 am2 ··· amn
• Mn×n (R) → R is continuous
A 7−→ det(A)
1
: GLn (R) → R
det
Here Mn×n is a vector space of dimension n2 in which GLn (R) = {A | det(A) ̸= 0} is an
open set.
• GLn (R) → GLn (R) is continuous.
A7−→A−1

9. Any norm (f ) on Rn is uniformly continuous w.r.t usual topology on Rn i.e. f : Rn → R is continuous

w.r.t usual norms (∥ · ∥ = p0norm for p = 1, 2, ∞) on Rn (∥ · ∥) and R(| · |)

Theorem 3.2.2
Any norm (f ) on Rn is uniformly continuous w.r.t usual topology on Rn i.e. ∀ ϵ > 0 ∀ x, y ∈ Rn ∃ δ > 0
s.t. ∥x − y∥ < δ =⇒ |f (x) − f (y)| < ϵ

Proof: 
f (x) ≤ f (y) + f (x − y)

∥ |f (x) − f (y)| ≤ f (x − y)

f (y) ≤ f (x) + f (y − x)


yi ei where {ei } is the standard basis of Rn .

P P
Let x = xi ei and y =
X  X
f (x − y) = f (xi − yi )ei ≤ f ((xi − yi )ei ) = |xi − yi |f (ei )

18
 P
Notice |xi − yi | = ∥x − y∥1 . Let M = max{f (ei )} Then

|f (x) − f (y)| ≤ f (x − y) ≤ M ∥x − y∥1

ϵ
Thus ∥x − y∥ < M =⇒ |f (x) − f (y)| < ϵ

19
Chapter 4

Equivalence of Norms

We back to Normed Linear Space for a little while.

  p1
n p
P
In R , u = (u1 , u2 , · · · , un ) where each ui ∈ R. we have p−norm: ∥u∥p = |ui | where 1 ≤ p ≤ ∞.
i
2
Balls in R w.r.t. ∥ · ∥1 , ∥ · ∥2 , ∥ · ∥∞ .

Observe: A set V in R2 is
S
open w.r.t. ∥ · ∥1 ⇐⇒ V = Box in V centered box
u∈V
S
open w.r.t. ∥ · ∥2 ⇐⇒ V = Diamond in V centered box
u∈V
S
open w.r.t. ∥ · ∥∞ ⇐⇒ V = Circle in V centered box
u∈V

Definition 4.1: Equivalence of Norms

Suppose ∥ · ∥, ∥ · ∥′ are two norms in vector space V , We say that the two norms are equivalent if there
are constants α, β > 0 s.t.
α∥x∥′ ≤ ∥x∥ ≤ β∥x∥′

Example 4.0.1 (Norm Equivalence)

1. p = ∞ and p = 1
P
∥x∥∞ = max{|xi | | 1 ≤ i ≤ n} ≤ ∥x∥1 = |xi |
i
∥x∥∞ ≥ each |xi | =⇒ n∥x∥∞ ≥ ∥x∥1
Hence
1
∥x∥∞ | ≤ ∥x∥1 ≤ n∥x∥∞ and ∥x∥1 ≤ ∥x∥∞ ≤ ∥x∥1
n
20
 2. p = ∞ and p = 2 √
∥x∥∞ ≤ ∥x∥2 ≤ n∥x∥∞

Theorem 4.1
All norms on a finite dimensional vector space are equivalent

Proof: Proved in Theorem 5.2.7

Theorem 4.2
Suppose ∥ · ∥ and ∥ · ∥′ are equivalent on a vector space V . Then
(i) {xn } → x w.r.t. ∥ · ∥ ⇐⇒ {xn } → x w.r.t ∥ · ∥′

(ii) S ⊂ V is open w.r.t ∥ · ∥ ⇐⇒ S is open w.r.t ∥ · ∥′

Proof: For both proofs if we just prove one direction the we are done actually since we can just replace the
words to prove for opposite direction,

(i) If Part:-

Since ∥ · ∥, ∥ · ∥′ are equivalent we have ∃ α, β such that α∥x∥′ ≤ ∥x∥ ≤ β∥x∥′ . So if we show α ∥xn − x∥ <
∥xn − x∥ < αϵ we are done.
′
Let {xn } → x w.r.t ∥·∥ i.e. ∀ ϵ > 0 ∃ N s.t. ∀ n > N ∥xn − x∥ < αϵ. Hence we have α ∥xn − x∥ < αϵ.
′
Hence ∀ ϵ > 0 ∃ N such that ∀ n > N ∥xn − x∥ < ϵ

(ii) Only If Part:-

Br (x) and V ′ is open w.r.t ∥ · ∥′ ⇐⇒ Br′ (x)

S S
V is open w.r.t ∥ · ∥ ⇐⇒
x∈V x∈V
Now we have
Br (x) = {y | ∥y − x∥ < r} and Br′ (x) = {y | ∥y − x∥′ < s}
Hence by equivalence of the norms for any v

α∥v∥′ ≤ ∥v∥ ≤ β∥v∥′

Since ∥v∥ < r we have

r
∥v∥′ ≤ =⇒ B ′βr (x) ⊂ Br (x)
β

Corollary 4.1
p = 1 and p = ∞ on Rn (and Cn ) give the same topology as p = 2 norm

Corollary 4.2
Let xm be a square in Rn . xm = (xm1 , xm2 , · · · , xmn ). Then {xm } → x = (x1 , x2 , · · · , xn ) w.r.t ∥ · ∥2 ⇐⇒
{xmi } → xi in R for each i.

21
 Note:-
We can check this w.r.t ∥ · ∥∞
xm → x w.r.t ∥ · ∥∞ ⇐⇒ ∀ ϵ > 0 ∃ N s.t. ∀ m > N max{|xmi − xi | | 1 ≤ i ≤ n}
⇐⇒ each |xmi − xi | < ϵ ∀ i
⇐⇒ lim xmi = xi ∀ i
n→∞

22
Chapter 5

Compactness

5.1 Sequentially Compact

Definition 5.1.1: Sequentially Compact

Let (X, d) be a metric space. X is called sequentially compact if every sequence in X has a convergent
subsequence. (Often applied to a subset S of X)

Note:-
For S to be sequentially compact the limit of subsequence must be in S

Definition 5.1.2: Boundedness

A subset S of (X, d) is bounded if S ⊂ Br (x) for some x ∈ X and r > 0

Note:-
Boundedness depends on the metric but if two metrics are “equivalent” analogous to norms)

Theorem 5.1.1
A subset K of Rn is sequentially compact ⇐⇒ K is closed and bounded

Proof: Proof in steps

1. A closed interval [a, b] in R is sequentially compact

Proof: Given a sequence x1 , x2 , · · · in R in [a, b] we can extract a monotonic subsequence as follows:

We call xi to be a peak if xi > xj ∀ j > i. Now there are two cases. If number of peaks is infinite then
the next peak comes after the previous one so smaller than the previous one. So its a strictly decreasing
sequence. If number of peaks are finite then at some point we cant find a peak with this property that
means no matter which term i peak there is at least one term after that which is greater than or equal to
that term. y1 =a term after the last peak. and yi+1 =a term after yi such that yi+1 ≥ yi . Hence y1 , y2 , · · ·
is a weakly increasing sequence.
When {xn } contained in [a, b] by boundedness of the monotonic subsequence, it converges to its
sup/inf and the limit is in [a, b]

23
 2. [a1 , b1 ] × [a2 , b2 ] × · · · × [an , bn ] ⊂ Rn is sequentially compact (w.r.t p−norm for p = 1, 2, ∞.
Later for any norm)

Proof: Recall a sequence {xm } → x in Rn ⇐⇒ The sequence converges in each coordinate i.e. xmi → xi
Take a sequence in the given box. Extract a subsequence whose entries in 1st slot converge (necessarily
to xi in [a1 , b1 ] by step 1 From this sequence, extract a further subsequence whose entries in second slot
converge to x2 ∈ [a2 , b2 ]. Continue

3. Every closed subset of a sequentially compact set is sequentially compact

Proof: Exercise

This will show each closed and bounded subset of the Euclidean Space Rn is sequentially compact.
(because such a set will be contained in a box)

4. If K is sequentially compact then K is closed and bounded

Proof: If K is not closed then some limit point x of K will not be in K. Then there is a sequence {ym }
in K converges to x ∈
/ K violating sequential compactness of K.
If K is not bounded take {xm } ∈ K with ∥xm ∥ ≥ n then {xm } can not be convergent

Note:-
Step 4 works for any metric space. Then we need to have a ball instead of norm

Theorem 5.1.2
If K is a sequentially compact of a metric space X, then K is closed and bounded

Proof: Same argument as step 4 use xm such that d(xm , x) ≥ m

Question 14

If K is closed and bounded in (X, d) =⇒ K is sequentially compact

Solution: No. Any counter-example. Define a metric on real number which induces same topology as the normal
topology in such a way that there is a closed and bounded set that is not compact.
Question 15

1. If V , W are normed linear spaces can we define a norm on V × W ?

2. If V , W are metric spaces can we define a metric on V × W ?
3. If V , W are topological spaces can we define a topology on V × W ?

24
5.2 Open Cover and Compactness
Definition 5.2.1: Open Cover
S
Let {Vα }α∈I be a family of subsets of metric space X we say that {Vα }α∈I is a cover of X if Vα = X
α
and we say that {Vα }α∈I is an open cover if each Vα is open (in X)

Definition 5.2.2: Compact

X is called compact if each open cover of X has a finite subcover i.e. {Vα1 , Vα2 , · · · , Vαn } ⊂ {Vα }α∈I with
Vα1 ∪ Vα2 ∪ · · · ∪ Vαn = X

Note:-
1. This definition makes sense for any topological space X.
If X is a metric space then it is a fact that X is compact ⇐⇒ X is sequentially compact. This
is not true for general topological spaces. Both implications fail.

2. Reformulation of compactness for subset K of X in terms of open sets of X

K is compact ⇐⇒ Every cover of K bt open sets of K has a finite subcover.

As open sets of K are precisely (open sets of X) ∩ K. We have the following
K is compact ⇐⇒ For any family {Vα ∩ K}α∈I where Vα are open in X whose union is K,
there is a finite subcover. S
⇐⇒ For any family {Vα ∩ K}α∈I of open sets in X such that Vα ⊃ K,
α∈I
n
S
there must be a finite subfamily Vα1 , Vα2 , · · · , Vαn with Vαi ⊃ K
i=1

If i take this definition of compactness of a subset K of metric space X then K is compact as subset of
X ⇐⇒ K is compact as a subset of it itself

Theorem 5.2.1 Haine Borel Theorem

K ⊂ Rn is compact ⇐⇒ K is closed and boundeded

(w.r.t p = 1, 2 or ∞ norm as they are equivalent.)

Proof: Only If Part:-

Proof in steps

1 Closed interval [a, b] is compact in R. Proof: Theorem 5.2.4

2 Closed box [a1 , b1 ] × [a2 , b2 ] × · · · × [an , bn ] is compact in Rn . Proof: Theorem 5.2.6

3 A closed subset of a compact set is compact. Proof: Theorem 5.2.3

These steps would give the backward direction of Haine Borel Theorem i.e. suppose K is closed and bounded in
Rn =⇒ K ∈ [−M.M ]n =⇒ compact by 2

If Part:-
Bounded: First we have to show that K is compact =⇒ K is bounded an i.e. K ⊂ Br (x) in (X, d) for some
x ∈ X, r > 0

25
 Consider open cover {Bn (x)}n∈Z+ of X and hence of K. This must have a finite subcover Bn1 (x), Bn2 (x),
· · · , Bnk (x). Take r = max{n1 , n2 , · · · , nk } Hence

K is compact =⇒ K is closed

Closed: We will show that X \ K is open. Pick x ∈

/ K. Enough to construct an open neighborhood Ux ∋ x such
that Ux ∩ x = ϕ
Take z ∈ K. Let c = d(x, z) then
B 3c (x) ∩ B 3c (z) = ϕ by triangle inequality
∥ ∥
Wz Vz

K
Vz

z
x Wz

S
Now Vz ⊃ K. So {Vz } is an open cover of K. By compactness we have Vz1 ∪ Vz2 ∪ · · · ∪ Vzn ⊃ K. As
z∈K
Wz ∩ Vz = ϕ ∀z ∈ K. We have (Wz1 ∪ Wz2 ∪ · · · ∪ Wzn ) ∩K = ϕ
| {z }
Finite intersection of
open neighborhoods of x,
so call this Ux

Key fact that made this work: For x ̸= z in X, we could find open neighborhoods of V and W (of x and z
respectively) such that V ∩ W = ϕ. Topological spaces that satisfy this property are called Housdorff.
What we proved is the following

Theorem 5.2.2
For a Housdorff Topological space X any compact subset K is closed and bounded

Theorem 5.2.3 Haine Borel Theorem - If Part: Step 3

C is a closed subset of compact set X =⇒ C is compact.

S
Proof: Take any open cover {Vα }α∈I of C by open sets in X i.e Vα ⊃ C. Now {Vα }α∈I ∪ {X \ C} is an
α
open cover of x. We have a finite subcover by compactness of X. The same subcover (after dropping X \ C if
necessary) works for C.

Wrong Concept 5.1: Closed interval [a, b] is compact in R

Suppose {Vα }α∈I is an open cover of [a, b] by open sets in R.

Hence every one of the points in the interval is covered by one of the Vα . Hence there is some
interval contained in the Vα

( )
a b

So i could just ignore the Vα and say for each point in the interval we can get an open interval that is part
of a Vα . So how can i find a subcover. I could simply travel from one end to the other.
So i start with a so a must be contained in some open interval

26
 ( )
a b
δ1

Not only that i have covered up a small segment of the closed interval, upto a point, a+δ1 . Say [a, a+δ1 ) ⊂
V1 .
Let a + δ1 is contained in some open interval which is contained in V2 upto the point a + δ2

δ2
( () )
a b
δ1

Now continue.

What is wrong with this ?

We could have smaller and smaller intervals. For example length of first interval can be 13 , length
1
of second interval can be 19 , length of third interval can be 27 and so on. So its a geometric progression
and it will sum less than 1. So i just may not get there in finite number of steps.

Question 16

Suppose X is a topological space that is compact and 5.2 (Take x ro be a compact metric space if you like).
Prove that given disjoint compact subsets K and L, there are disjoint open sets U and V with K ⊂ U and
L ⊂ V (First do it for K = single point)

In the above exercise we could have replaced the word compact with another word which is closed because
X is given to be compact so any closed set will be compact and in a Housdorff space compact subset is also closed.
Note:-
1
Cauchy Sequence in Metric space need not converge. For example (0, 1) and take the sequence n. It wants to
converge to 0 but 0 is not there.

Theorem 5.2.4 Haine Borel Theorem - If Part: Step 1

[0, 1] is compact in R

Proof: Let {Vα }α∈I be a family of open sets in R covering [0, 1].
Let S = {a ∈ [0, 1] | [0, a] can be covered by a finite number of Vα ’s}. Our goal is to prove 1 ∈ S.
Let 0 ≤ x < y ≤ 1. So [0, x] ⊂ [0, y]. This y ∈ S =⇒ x ∈ S i.e x ∈ / S =⇒ y ∈ / S. Now S is nonempty
because 0 ∈ S and S is bounded. Let u = lub of S. Clearly 0 ≤ u ≤ 1. Hence it is enough to show u = 1 and
u ∈ S.
0 ∈ some open set Vα . Hence ∃ ϵ > 0 Bϵ (0) ⊂ Vα . Hence ∀ point x ∈ [0, ϵ) x ∈ S
For a ∈ [0, u), a ∈ S (otherwise a itself would be an upper bound for S). As {Vα }α∈I cover [0, 1], u ∈ Vβ .
So ∃ ϵ > 0 such that (u − ϵ, u + ϵ) ⊂ Vβ As u − ϵ ∈ S we have Vα1 sup Vα2 sup · · · Vαk ⊃ [0, u − ϵ] Then
Vαβ ∪ Vα1 ∪ Vα2 ∪ · · · Vαk ⊃ 0, u + 2ϵ . So u = 1 because otherwise some u + δ ∈ S contradicting that u is an
upper bound.

Question 17

Can the strategy from the last time be made to work ti actually extract a finite subcover of a given cover.

Theorem 5.2.5
f
Suppose X −
→ Y continuous and K ⊂ X is compact. Then f (K) is compact

27
Proof: Let {Vα }α∈I be an open cover of f (k) by open sets Vα of Y . So
!
[ [ [
−1
Vα ⊃ f (K) =⇒ f Vα = f −1 (Vα ) ⊃ f −1 (f (K)) ⊃ K
α α α

Thus f −1 (Vα ) α∈I is an open (because of continuity Theorem 3.2.1) cover of K.



Extract a finite subcover

f −1 (Vα1 ) ∪ f −1 (Vα2 ) ∪ · · · f −1 (Vαm ) ⊃ K
=⇒ f f −1 (Vα2 ) ∪ · · · f −1 (Vαm ) ⊃ f (K)


m
[
f f −1 (Vαi ) ⊃ f (K)


=⇒
i=1
m
As Vαi ⊃ f f −1 (Vαi ) we have

S
Vαi ⊃ f (K)
i=1

Question 18

f (Sequentially compact K) is sequentially compact

Theorem 5.2.6 Haine Borel Theorem - If Part: Step 2

K = [a1 , b1 ] × [a2 , b2 ] × · · · × [an , bn ] is compact in Rn

Proof: Induction on n. n = 1 we already proved in Theorem 5.2.4.Let F = {Vα }α∈I be a cover of K by

open sets in Rn . Fix u ∈ [a1 , b1 ] and consider {u} × [a2 , b2 ] × · · · × [an , bn ] Hence {u} × C is compact because
| {z }
=C is compact
by induction on n
Rn−1 → Rn which maps (y2 , y · · · , yn ) 7→ (u, y2 , · · · , yn ) or f (C) = {u} × C is continuous.
For each p = (u, y2 , · · · , yn ) in {u} × C pick an open neighborhood Vp ∈ F. Hence Vp ⊃ (x − ϵ, x + ϵ) ×
(y2 − ϵ, y2 + ϵ) × · · · × (yn − ϵ, yn + ϵ) for some ϵ = ϵp depending on p
| {z }
Wp

a1 u b1

By compactness of {u}×C, extract a finite subcover of the cover {Wp }. Hence Wp1 ∪Wp2 ∪×∪Wpk ⊃ {u}×C. Since
its a union of open sets we have in fact Wp1 ∪ Wp2 ∪ × ∪ Wpk ⊃ (u − ϵ, u + ϵ) × C where ϵ = min{ϵp1 , ϵp2 , · · · , ϵpk }.
Let Fu = {Vp1 , Vp2 < · · · , Vpk }. So
Vp1 ∪ Vp2 ∪ · · · ∪ Vpk ⊃ Wp1 ∪ Wp2 ∪ · · · ∪ Wpk ⊃ (u − ϵ, u + ϵ) × C
i.e. this finite subcover Fu cover not just the slice but a tube around it.
Now as u varies in [a1 , b1 ], (u − ϵu , u + ϵu ) gives an open cover. Extract a finite subcover (u1 − ϵu1 , u1 +
ϵu1 ), (u2 −ϵu2 , u2 +ϵu2 ), · · · , (ul +ϵul , ul +ϵul ) . Then Fu1 ∪Fu2 ∪· · ·∪Ful is a finite subcover of [a1 , b1 ]×C = K

28
 Question 19

Why the map Rn−1 → Rn which maps (y2 , y · · · , yn ) 7→ (u, y2 , · · · , yn ) or f (C) = {u} × C is continuous ?

Question 20

X, Y are topological spaces. K ⊂ X and

S Y ⊂ Y are compact subsets. Then K × L is compact subset of
X × Y where Open sets of X × Y are (Open set of X)×(Open set in Y )

Theorem 5.2.7
All norms on Rn are equivalent

Proof: Enough to show any norm f ∼ ∥ · ∥

i.e α∥u∥ ≤ f (u) ≤ β∥u∥∀ u

f (u)
i.e α ≤ ≤ β ∀ u∀ u ̸= 0
∥u∥
 
f (x) x x
Note that ∥x∥ =f ∥x∥ = f (u) where u = ∥x∥ , so ∥u∥ = 1. Hence it is enough to show that

α ≤ f (u) ≤ β

for any u with ∥u∥ = 1

Let S = {u | ∥u∥ = 1} is the unit sphere in Rn , which is closed and bounded
S is closed and bounded =⇒ S is compact
=⇒ f (S) is compact in R
=⇒ f (S) is closed and bounded in R
=⇒ f (S) has largest element in β and smallest
element α such that α ≤ f (S) ≤ β

29
Chapter 6

Differentiation

Derivative of f at a ∈ R is
f (a + h) − f (a)
f ′ (a) = lim
h→0 h
To take this limit f should be defined in some (a − ϵ, a + ϵ) i.e. f : neighborhood of a → R
f
Goal: Definition of f ′ (a) for a ∈(Some open U in Rm )−
→ Rn , a, h ∈ Rm
f (a + h) − f (a) makes sense in R but can’t divide by h, which is a vector in Rm . If m = 1 can use the
n
f
same definition. f : Open U in a R → Rn which maps a 7→ (f1 (a), f2 (a), · · · , fn (a)). If n = 1 i.e. Rm ⊃ U −
→R
we have partial derivatives.

Example 6.0.1 (Derivative of f : Rm → R)

f (x, y, z) = x4 sin(yz). Here

∂f
= 4x3 sin(yz)
∂x
f (x + h, y, z) − f (x, y, z)
= lim
h→0 h
Hence
∂f f (r + h, s, t) − f (r, s, t)
= lim
∂x p=(r,s,t) h→0 h
f (p + he1 ) − f (p)
= lim [p = re1 + se2 + te3 using standard basis e1 , e2 , e3 ]
h→0 h
Its a real number if the limit exists

6.1 Partial Derivatives

Definition 6.1.1: Partial Derivative of f : Rn ⊃ U → R

∂f
For f :(Open U in Rm )→ Rn , define “i−th partial derivative of f at a ∈ U ” to be Notation ∂xi ,
 a
∂f
∂xi (a), Di f (a) lim f (a+hehi )−f (a) (i = 1, 2, · · · , m)
h→a

Note that this limit (if exists) is in Rn .

)
If f = (f1 , f2 , · · · , fn ) (fi real
 
∂f ∂f1 ∂f2 ∂fn
(a) = (a), (a), · · · , (a)
values in function U → R) ∂xi ∂xi ∂xi ∂xi

30
 ∂fj
So for f : U → Rm , a ∈ U we get ∂xi where 1 ≤ j ≤ n and 1 ≤ i ≤ m. We can arrange these in a matrix of
dimensions n × m
∂f1 ∂f1
 
(a) · · · (a)
 ∂x1 ∂xm 
 . .. .. 
 . .
 . .


 ∂f ∂fn 
n
(a) · · · (a)
∂x1 ∂xm
Note:-
f ′ (a) can be defined as a linear map Rn → Rm
In the old situation f : R → R, f ′ (a) ∈ R is a 1 × 1 matrix, as such it encodes a linear map R →
′
R
x7→f (a)x

The tangent line can be though of as the graph of

y = x2 slope to linear map R → R
t7→10t
(1,1)

6.2 Differentiation
f (a+h)−f (a)−f ′ (a)h f
f ′ (a) is a number such that lim h = 0. Inspired by this for a ∈ U (Open in Rm ) −
→ Rn , we define
h→0
f ′ (a) is a linear map Rm → R such that
n

∥f (a + h) − f (a) − f ′ (a)h∥
lim = 0 in R
h→0 ∥h∥

(h is a small vector ∈ R)
Definition 6.2.1: Differentiation of f : Rm ⊃ U → Rn

U open set in Rm , f : U → Rn , a ∈ U given. We say that f is differentiable at a if there is a linear map

T : Rm → Rn such that
∥f (a + h) − f (a) − T (h)∥
lim =0
h→0 ∥h∥
i.e ∀ ϵ > 0, ∃ δ > 0 s.t.
∥f (a + h) − f (a) − T (h)∥
∥h∥ < δ =⇒ <ϵ
∥h∥
We call such a linear map T the derivative of f at a, denoted by f ′ (a), D(f (a))

Note that f ′ (a)h = Value of linear map f ′ (a) applied to a vector h

Note:-
If the above limit is 0 w.r.t any norm on RM (respectively Rn ) then the same limit is 0 w.r.t any other norm
on Rm (respectively Rn ) because all norms are equivalent

Theorem 6.2.1
f
Derivative is unique i.e. if a ∈ U (Open in Rm ) −
→ Rn , and

∥f (a + h) − f (a) − T (h)∥ ∥f (a + h) − f (a) − S(h)∥

lim = 0 = lim
h→0 ∥h∥ h→0 ∥h∥

then T = S i.e T (v) = S(v) ∀ v ∈ Rm

31
Proof: Let R = S − T . Want to show R(v) = 0 ∀ v ∈ Rm

∥R(h)∥ ∥S(h) − T (h)∥ ∥(f (a + h) − f (a) − T (h)) − (f (a + h) − f (a) − S(h))∥

= =
∥h∥ ∥h∥ ∥h∥
∥(f (a + h) − f (a) − T (h))∥ ∥(f (a + h) − f (a) − S(h))∥
≤ +
∥h∥ ∥h∥
∥R(h))∥
Taking lim , we get lim ∥h∥ = 0.
h→0 h→0
Fix any nonzero v, lim λv = 0. Take h = λv (λ ̸= 0)
λ→0

∥R(h)∥ |λ|∥R(v)∥ ∥R(v)∥

= =
∥h∥ |λ|∥v∥ ∥v∥

Hence
∥R(h)∥ ∥R(v)∥
0 = lim = lim =⇒ ∥R(v)∥ = 0 =⇒ Rv = 0
h→0 ∥h∥ h→0 ∥v∥

Question 21

If f : Rm → Rn is a linear map then what is f ′ : Rm → Rn

v7→Av

Solution: See that f ′ (a) = f . (Immediate from definition)

Question 22
g
→ Rn for some c ∈ Rn . Calculate g ′ (a).
For an affine map Rm −
v7→Av+c

Solution: g ′ (a) = the map h 7→ Ah

Theorem 6.2.2 Matrix of f ′ (a)

Prove that the matrix of f ′ (a) w.r.t standard basis of Rm and Rn is the Jacobian Matrix

Proof: jth column of matrix of T = T (ej ) ∈ Rn

∥f (a + λej ) − f (a) − T (λej )∥

lim by definition of T = f ′ (a)
λ→0 ∥λej ∥
(
∥f (a + λej ) − f (a) − λT (ej )∥ T (λej ) = λT (ej ) by linearity
= lim
λ→0 |λ| ∥λej ∥ = |λ|∥ej ∥ = |λ|
f (a + λej ) − f (a) λT (ej )
= lim −
λ→0 |λ| |λ|

Hence for λ > 0

f (a + λej ) − f (a)
lim − T (ej ) = 0
λ→0 |λ|

32
  
f1
 f2 
Let f =  . . Hence
 
 .. 
fn

f (a + λej ) − f (a)
T (ej ) = lim
λ→0 λ
   
f1 (a + λej ) f1 (a)  ∂f 
1
 f2 (a + λej )   f2 (a)  (a)
 −  ..   ∂xj
..
    

.   . 
  
 
 ∂f2
fn (a + λej ) fn (a)

 (a) 
= lim = ∂x
 j

λ→0 λ  . 

 .. 
 
 ∂fn 
(a)
∂xj

Matrix of f ′ (a)= Jacobian Matrix

∂f1 ∂f1
 
(a) · · · (a)
 ∂x1 ∂xm 
 . .. .. 
 . .
 . .


 ∂f ∂fn 
n
(a) · · · (a)
∂x1 ∂xm
∂fi
We have proved if f ′ (a) exists, then all partial derivatives ∂xj exists at x = a and make up the matrix of f ′ (a)
∂fi
If all ∂xj exists at x = a, does not imply f is differentiable at x = a?
Question 23
∂fi
Under Which conditions if all ∂xj exists at x = a, it implies that f is differentiable at x = a?

Theorem 6.2.3
If f ′ (a) exists then f is continuous at x = a

Proof: If f ′ (a) exists then f is continuous at x = a ⇐⇒ lim f (a + h) = f (a) ⇐⇒ lim ∥f (a + h) − f (a)∥ = 0

h→0 h→0

∥f (a + h) − f (a) + T (h) − T (h)∥ ≤ ∥f (a + h) − f (a) − T (h)∥ + ∥T (h)∥

∥f (a + h) − f (a) − T (h)∥
Now lim ∥h∥ = 0 · 0 = 0 and
h→0 ∥h∥
(
T is continuous (being linear) so
lim ∥T (h)∥ = 0 because
h→0 T (h) → T (0) = 0

33
Chapter 7

Examples on Multivariable
Differentiation

Example 7.1 (Example where all partial derivatives exist and function is continuous but f ′ does not
exists.)
( xy
√ 2 2 (x, y) ̸= (0, 0)
f (x, y) = x +y
0 (x, y) = (0, 0)

(i) Is f continuous at origin ?

∂f ∂f
(ii) Do , exist at origin ? elsewhere ?
∂x ∂y

Solution:

(i) Want |f (x, y) − f (0, 0)| → 0 as (x, y) → (0, 0)

r
xy x2 + y 2
p ≤ →0
x2 + y 2 2

as (x, y) → (0, 0)

(ii)
∂f y3 ∂f x3
= 3 = 3
∂x (x + y 2 ) 2
2 ∂y (x + y 2 ) 2
2

Now
∂f f (h, 0) − f (0, 0) 0−0
= lim = lim =0
∂x (0,0) h→0 h h→0 h
∂f
= 0. So if f ′ (0) exists then it will be the matrix 0
 
Similarly ∂y
0 . So it will be the zero operator
(0,0)  
1
=⇒ Dv f (origin) = 0 for any direction for any vector v. Let’s test for v =
1

f (0 + tv) − f (0, 0) f (t, t) t2

Dv f (origin) = lim = lim = lim √ ̸= 0
t→0 t t→0 t t→0 t 2t2

Thus f is not differentiable at origin. Therefore at least one of the partial derivatives must be discontinuous
at origin (here by symmetry both are discontinuous). ∂f ∂x = 0 at origin but = 1 at y−axis.

34
 Example 7.2 (Example where f ′ exists but not continuous)
(
x2 sin x1 x ̸= 0 p
Recall one-variable example g(x) = . Define f (x, y) = g( x2 + y 2 )
0 x=0

(i) Is f continuous ?
(ii) Is f differentiable ?
(iii) Is f ′ continuous at origin ?

Solution:
(i) Because f is composition of two continuous functions. f is continuous.

(ii) Need to check at origin only

Example 7.3
( 2
x y
x6 +y 2 (x, y) ̸= (0, 0)
f (x, y) =
0 (x, y) = (0, 0)

(i) Is f continuous at origin ?

(ii) Calculate the directional derivatives for unit vectors u = (cos θ, sin θ)
(iii) Is f differentiable at origin ?

Solution:
(i)
x5 1
f (x, x3 ) =
6
=
2x 2x
It has no limit as x → 0. Hence f is not continuous at origin.
(ii)

f (0 + hu) − f (0)
Du f (0) = lim
h→0 h
f (h cos θ, h sin θ)
= lim
h→0 h
1 h3 cos2 θ sin θ
= lim
h→0 h h6 cos6 θ + h2 sin2 θ

cos4 θ sin θ cos2 θ

= lim 4 = when sin θ ̸= 0
h→0 h cos6 θ + sin2 sin θ sin θ

When sin θ = 0, f = 0 on x−axis. So Du f (0) = 0 for θ = 0, π, . . . . SO Du f () exists for all u

(iii) If f ′ (0) exists then it’s matrix would be
 
0 . But then all directional derivatives would have to be zero
because Du f (a) = f ′ (a)v which is not possible

35
Chapter 8

Chain Rule of Differentiation and

Operator Norm

8.1 Operator Name

V, W are vector spaces. L(V, W ) = Set of linear maps V → W is a vector space via (A + B)(v) = A(v) + B(v)
and A(λv) = λA(v)
If V = Rm and Rn , dim(Rm , Rn ) = mn. We can identity L(Rm , Rn ) with n × m matrices.
∥A∥L(Rm ,Rn ) = ∥A∥ = sup ∥A(u)∥
∥u∥=1

This gives a norm because ∥A∥ ≥ 0 and ∥A∥ = 0 =⇒ A = 0 and ∥λA∥ = |λ|∥A∥. As (A + B)(u) = A(u) + B(u)
we have ∥(A + B)(u)∥ ≤
A(u)∥ + ∥B(u)∥ in W and hemce ∥A + B∥ ≤ ∥A∥ + ∥B∥.
Question 24

Why this is well defined ?

Solution: The set S = {u | ∥u∥ = 1} is closed and bounded in V , therefore compact. A being linear is
continuous. ∴ A(S) is a compact subset of W =⇒ A(S) is bounded.

Basic Properties:-
1. ∥Av∥ ≤ ∥A∥∥v∥ i.e. ∥Av∥W ≤ ∥A∥L ∥v∥V
v
Proof: If v = 0 then we are done. If v ̸= 0, u =so ∥u∥ = 1. Hence
∥v∥

∥Av∥
 
v
∥A∥ ≥ ∥Au∥ = A =
∥v∥ ∥v∥

2. ∥A(v)∥ ≤ M ∥v∥ for all v =⇒ ∥A∥ ≤ M in fact inf{M | ∥A(v)∥ ≤ M ∥v∥ ∀ v}

Proof: Suppose ∥A(v)∥ ≤ M ∥v∥ ∀ v. In particular ∀ v with ∥v∥ = 1. So ∥A(v)∥ ≤ M .

Rest exercise: If L < inf of the set show ∃ u of norm=1 with ∥A(v)∥ > L.
A B
3. U −
→V −
→ W linear maps between finite dimensional vector spaces then ∥BA∥ ≤ ∥B∥∥A∥

Proof: Take u with ∥u∥ = 1. Then

∥BA(u)∥ ≤ ∥B∥∥A(u)∥ ≤ ∥B∥∥A∥∥u∥ = ∥B∥∥A∥
Now take sup over u.

36
 Note:-
A, B 7→ BA is continuous because each slot of matrix of BA is obtained by adding/multi-
L(U,V )⊕L(V,W ) → L(U,W )
plying entries of A and B.

Question 25

Show An → A in L(U, V ), Bn → B in L(V, W ) then Bn An → BA in L(U, W )

8.2 Chain Rule

Theorem 8.2.1
Let
Rn

⊂
f g
Rm ⊃ U V Rk
∈

∈
a b

f is differentiable at a. and g is differentiable at b = f (a). Then g ◦ f is differentiable at a and

(g ◦ f )′ (a) = g ′ (f (a))f ′ (a)

| {z }
Multiplication
of matrices

Proof: 1-Variable Case

dz dz dy
dx = dy dx

dz ∆z ∆z ∆y
= lim = lim
dx ∆x→0 ∆x ∆x→0 ∆y ∆x
∆z ∆y
= lim lim
∆x→0 ∆y ∆x→0 ∆x
∆z ∆y
= lim lim [As ∆x → 0, ∆y → 0]
∆y→0 ∆y ∆x→0 ∆x

Multi Variable Case

f ′ (a) g ′ (f (a))
Note that Rm −−−→ Rn −−−−−→ Rk
If T = f ′ (a) then
∥f (a + h) − f (a) − T (h)∥
lim =0
h→0 ∥h∥
and S = g ′ (f (a)) = g ′ (b) then
∥g(b + k) − g(b) − S(k)∥
lim =0
h→0 ∥k∥
Let
∥α(h)∥
α(h) = f (a + h) − f (a) − T (h) ϵ(h) = → 0 as h → 0
∥h∥

 ∥β(k)∥ → 0 as k → 0


β(k) = g(b + k) − g(b) − S(k) η(k) = ∥k∥


0 when k = 0
37
η is continuous at k = 0. Now note that η : V − b → R because we ae always taking b + k for η. We want to show
that
∥g(f (a + h)) − g(f (a)) − ST (h)∥ ∥g(b + k) − g(b) − ST (h)∥
lim = 0 ⇐⇒ lim =0
h→0 ∥h∥ k→0 ∥h∥
where f (a + h) = b + k ⇐⇒ k = f (a + h) − f (a). We have taken a specific value of k depending on h. So now k
is a function of h. Hence T (h) = f (a + h) − f (a) − α(h) = k − α(h)
g(b + k) − g(b) − ST (h)
=g(b + k) − g(b) − S(k − α(h))
=g(b + k) − g(b) − S(k) + S(α(h))
Therefore
∥g(b + k) − g(b) − ST (h)∥ ∥g(b + k) − g(b) − S(k)∥ ∥S(α(h))∥
≤ +
∥h∥ ∥h∥ ∥h∥
want to bound each of these separately
∥S(α(h))∥ ∥α(h)∥
≤ ∥S∥ →0
∥h∥ ∥h∥
∥β(k)∥ ∥k∥
Now how to bound the first term. In the first term ∥h∥ = η(k) ∥h∥ . Now

k = T (h) + α(h)
=⇒ ∥k∥ ≤ ∥T (h)∥ + ∥α(h)∥
∥k∥ ∥T (h)∥ ∥α(h)∥ ∥T ∥∥h∥ ∥α(h)∥ ∥α(h)∥
=⇒ ≤ + ≤ + = ∥T ∥ +
∥h∥ ∥h∥ ∥h∥ ∥h∥ ∥h∥ ∥h∥
Hence
∥β(k)∥ ∥k∥ ∥α(h)∥
 
= η(k) ≤ η(k) ∥T ∥ +
∥k∥ ∥h∥ ∥h∥
∥α(h)∥
As h → 0 ∥T ∥ + ∥h∥ → ∥T ∥ + 0 which is finite. And as h → 0, k → 0 =⇒ η(k) → 0 because η is continuous
at 0.

8.3 Special Case of Chain Rule: When m = k = 1

γ g
→ Rn −
Open interval in R − → R. γ = parameterized curve in Rn
(g ◦ γ)′ (t) = g ′ (γ(t)) · γ ′ (t)
   ′ 
γ1 (t) γ1 (t)
γ n  ..  ′  .. 
R−
→ R maps t →  .  hence γ (t) =  . 
γ (t) γn′ (t)
 n
∂g ∂g

Now g ′ (y) = ··· . Hence
∂x1 y ∂xn y

 γ1′ (t)
 
∂g ∂g

 .. 
(g ◦ γ)′ (t) = ···  . 
∂x1 y ∂xn y
γn′ (t)
∂g
 
 ∂x1 y  γ1′ (t)
 

=  ...  ·  ... 
 
[Usual dot product of vectors in Rn ]
   
 
 ∂g  γn′ (t)
∂xn y

38
  ∂g

∂x1
y
..

Call   = ∇g(γ(t)) = Gradient of g at the point γ(t). Hence (g ◦ γ)′ (t) = ∇g(γ(t)) · γ ′ (t)
 
 . 
∂g
∂xn
y

Question 26

Fix u, v ∈ Rn and take parametrized curve γ(t) = u + tv. What does the above equation give at t = 0 (for
a given function g)

39
Chapter 9

Mean Value Theorem

We will use Euclidean norm on Rn and have Cauchy-Schwarz Inequality |v · w| ≤ ∥v∥∥w∥.

Theorem 9.1 1-Variable MVT

If f : [a, b] → R continuous and f ′ exists on (a, b), then ∃ c ∈ (a, b) s.t

f (a) − f (b) = f ′ (c)(b − a)

Proof: 1-variable MVT

⇑ Via Rolle’s Theorem, using f ′ (extremum) = 0
Extreme Value Theorem
⇑ [a, b] is compact, f is continuous =⇒ f ([a, b]) is compact in R =⇒ closed and bounded
Heine Borel Theorem

Question 27

First consider f : [a, b] → Rn continuous and f ′ exists on (a, b). Is there a c ∈ (a, b) s.t.
?
∥f (b) − f (a)∥ = ∥f ′ (c)∥(b − a)

Solution: No. For example f : [0, 2π] → R2 which maps t 7→ (sin t, cos t). Then f ′ (t) = (cos t, − sin t) and
∥f ′ (t)∥ = 1. f (2π) − f (0) = (0, 0)

Theorem 9.2 MVT of Real-Valued Functions

Let f be a continuous function [a, b] → Rn and f ′ (c) exists ∀ c ∈ (a, b). Then ∃ c ∈ (a, b) s.t

∥f (b) − f (a)∥ ≤ (b − a)∥f ′ (c)∥

(Here the norm is Euclidean norm. For the inequality which norm we take does matter.)

Proof: Clever use if 1−Variable MVT. We want to bound norm of f (b) − f (a) = z.
Idea: Dot with z and then use Cauchy Schwarz
f g=⟨−,z⟩
[a, b] Rn R
t f (t) ⟨f (t), z⟩

ϕ(t)

40
 n ′
Notice g : R → R where g maps x 7−→ ⟨x, z⟩ = x1 z1 + · · · + xn zn . Hence g is differentiable and g (x) =
z1 z2 · · · zn . Now we can apply M V T to ϕ.
ϕ(b) − ϕ(a) = (b − a)ϕ′ (c)

LHS = ϕ(b) = ϕ(a)

= ⟨f (b), z⟩ − ⟨f (a), z⟩
= ⟨f (b) − f (a), z⟩ = ⟨z, z⟩ = ∥z∥2
And
ϕ′ (c) = (g ◦ f )′ (c) = g ′ (f (c)) ◦ f ′ (c) = ⟨z, f ′ (c)⟩
Therefore
∥z∥2 = ⟨z, f ′ (c)⟩ ≤ (b − a)∥z∥∥f ′ (c)∥
Nothing to prove if ∥z∥ = 0 and else cancel ∥z∥ from both sides.

Theorem 9.3 General Multivariable MVT

f
→ Rn , f differentiable on U and ∥f ′ (x)∥ < M ∀ x ∈ U . Then ∀ a, b ∈ U
Let Rm ⊃ Convex Open U −

∥f (b) − f (a)∥ ≤ M ∥b − a∥

Proof: Given a, b ∈ U holds γ : [0, 1] → U which maps t → a + t(b − a) [This is valid by convexity]. Apply
MVT to
γ f
[0, 1] U Rn

g
We get ∈ (0, 1) such that ∥g(1) − g(0)∥ ≤ ∥g ′ (t)∥ i.e
∥f (b) − f (a)∥ ≤ ∥ f ′ (γ(t)) ◦ γ ′ (t) ∥ ≤ M ∥b − a∥
| {z } |{z}
Matrix Vector ↓
Multiplication Justify

Suppose U is convex in Rm , f : U → Rn and f ′ (a) = 0 ∀ a ∈ U . Then f = Constant because

∥f (b) − f (a)∥ ≤ 0∥b − a∥
∀ a, b ∈ U by MVT. What happens if U is open but not convex. If U is connected the conclusion is again true.
Definition 9.1: Connected Set in Rn

A set S in Rn is (path)connected if ∀ a, b ∈ S ∃ continuous function

γ : [0, 1] S
0 a
1 b

Theorem 9.4
If S is connected open set in Rn and f : S → Rn is differentiable on U with f ′ (a) = 0 ∀ a ∈ U then f (a) =
Constant.

Proof: ∀ x ∈ γ([0, 1]) find Br (x) ⊂ S. γ([0, 1]) is compact. So ∃ finite subcover of γ([0, 1]) by balls around
γ(t1 ), γ(t2 ), . . . , γ(tN ).
Order these alls so that a ∈ first ball and b ∈ last ball, any two consecutive balls overlap. This gives
piecewise linear path from a → b. Use MVT for each segment.

41
Chapter 10

Higher Derivatives

10.1 Class C 1 Functions

f
Open U in Rm − → Rn . D(f (a)) is a linear map Rm → Rn i.e D(f (a)) ∈ L(Rm , Rn ). If f is differentiable at
each a ∈ U , then we get a function Df : U → L(Rm , Rn ) which maps a 7−→ D(f (a)) = f ′ (a). We can ask about
continuity and differentiability of this map Df .
We want to consider C 1 (U ) functions which are all functions that are differentiable at each a ∈ U and Df
is continuous i.e C 1 (U ) = Set of continuously differentiable functions

Definition 10.1.1: C 1 Functions

f
→ Rn . Suppose f ′ (a) exists ∀ a ∈ U then we get a function
U open in Rm −

f: U L(Rm , Rn ) ⋍ Rmn
a f ′ (a)

which maps a 7−→ f ′ (a). We say f ∈ C 1 (U ), ”f is continuously differentiable” if f ′ (a) exists for each a
and f ′ is a continuous function

Theorem 10.1.1
∂fi
A function f : U Open in Rm → Rn is C 1 (U ) ⇐⇒ exists at each a ∈ U and are continuous
∂xj a
functions

Proof: If Part:-
 
f1 (a)
f (a + h) − f (a) − T h
→ Rn s.t f (a) =  ... , lim
f
1 a ∈ U Open in Rm − = 0Matrix of T w.r.t standard
 
h→0 ∥h∥
fn (a)
basis of Rm and Rn is
∂f1 ∂f1
 
· · ·

∂f ∂f
   ∂x1 a ∂xm a 
 .. .. 

T = ··· = . . ..
∂x1 a ∂xm a  . 

 ∂fn ∂fn 
···
∂x1 a ∂xm a
 
b1
 .. 
2 lim f (x) =  .  ⇐⇒ lim fI (x) = bi for each i = 1, 2, . . . , n
x→v x→v
bn
42
By 1 and 2 the proof of forward direction is obvious

Only If Part:-
If we prove that f ′ (a) exists for each a ∈ U then f ′ is automatically continuous because by 1 the matrix of
∂fi
f ′ (a) must be the Jacobian Matrix and we are given that all entries of this matrix namely the functions are
∂xj
continuous so apply 2
Another
  reduction: We may assume that n = 1 because this case in general, it follows immediately that
f1
for f =  ... ,
 

fn
 
f1 (a)  
′  ..  ∂f ∂f
f (a) =  .  = ···
∂x1 a ∂xm a
fn (a)
Hence      ′ 
f1 (a + h) f1 (a) f1 (a)h
f (a + h) − f (a) − T h 1  ..   ..   .. 
= .  −  .  −  . 
∥h∥ ∥h∥

fn (a + h) fn (a) fn′ (a)h
lim of this = 0 because in each slot the limits is 0 by n = 1 case which we have assumed, and will prove now.
h→0

Note:-
∂fi
Proof of the fact that in case of n = 1 if (j = 1, 2, . . . , m) are continuous functions U → R then f ′ (a)
∂xj
exists for each a ∈ U
h
∂f ∂f
i f (a + h) − f (a) − T h
We want to show f ′ (a) = ∂x · · · ∂x i.e lim = 0. We want to bound the
1
a m
a h→0 ∥h∥
 T  T
numerator. Fix a = a1 · · · am . Let h = h1 · · · hm . Now choose r > 0 such that Br (a) ⊂ U and
restrict h such that ∥h∥ < r.  
∂f  
* ∂x1
a
h1 +
X ∂f  ..   .. 
 
Th = hj =  . ,  . 
j
∂xj a  
∂f hm
∂xm
| {z a }
∥
q

And f (a + h) − f (a) = f (a1 , h1 , . . . , am + hm ) − f (a1 , . . . , an )

Idea: Bound this in terms of partial derivatives using the mean value theorem (ordinary 1-variable version,
∂f
which is applicable because each ∂x j
is continuous)

f (a + h) − f (a) = f (a1 + h1 , a2 , . . . , am ) − f (a)

+ f (a1 + h1 , a2 + h2 , . . . , am ) − f (a1 + h1 , a2 , . . . , am )
.. ..
. .
+ f (a1 + h1 , a2 + h2 , . . . , am + hm ) − f (a1 + h1 , a2 + h2 , . . . , am−1 + hm−1 , am )
 ∂f 
 
* ∂x1 v
1 
h1 +
MVT ∂f ∂f ∂f . . 

= h1 + h2 + · · · + hm = 
 .. ,  .. 
 
∂x1 v1 ∂x2 v2 ∂xm vm  
∂f hm
∂xm
vm
| {z }
∥
p

43
 (a1 + h1 , a2 + h2 , a3 + h3 )

(a1 + h1 , a2 + h2 , a3 + v3 )

(a1 + h1 , a2 + h2 , a3 )
(a1 + h1 , a2 + v2 , a3 )
(a1 + h1 , a2 , a3 )
(a1 , a2 , a3 ) (a1 + v1 , a2 , a3 )

Notice that v1 , v2 , . . . , vm are functions of h. Putting together we get

 ∂f ∂f

∂x1 − ∂x 1
v1 a
* +
f (a + h) − f (a) − T h 1 
.

= h, 
 .. 
∥h∥ ∥h∥

 
∂f ∂f
∂xm − ∂x 1
vm a
1
= ∥h∥∥p − q∥ = ∥p − q∥
∥h∥
Showing lim ∥p − q∥ = 0 is enough to complete the proof
h→0

∂f ∂f
lim ∥p − q∥ = 0 ⇐= p − q → 0 as h → 0 ⇐= − → 0 as h → 0
h→0 ∂xi vi ∂xi a

∂f
which is true because ∂xi is a continuous function. More formally choose ∥h∥ < δ s.t

∂f ∂f ϵ
(b) − (a) < ∀ b ∈ Bδ (a)
∂xi ∂x1 m
That ensures ∥p − q∥ < ϵ by triangle inequality.
Question 28

xy 2
(
xx2 +y 4 (x, y) ̸= (0, 0)
f (x, y) =
0 else
∂f ∂f
1. Calculate all directional derivatives in particular ∂x , ∂y

2. Does f ′ (a, b) exists at all a, b ∈ R

3. Is f continuous everywhere

10.2 Higher Derivatives and Class C k functions

Now we want to define class C k of functions for k ≥ 0. [Like in 1-Variable case. Usefull for Taylor’s theorem].
Now second derivative of f : U → Rn at a ∈ U ⊂ Rm = derivative at a of f ′ : U → L(Rm , Rn ) ⋍ Rmn .
′
∂fpq
∴ f ′′ (a) or D2 f (a) : Rm → L(Rm , Rn ). Matrix of f ′′ (a) has mn rows and m columns and equals to where
∂xj
p = 1, . . . , n, q = 1, . . . , m and j = 1, . . . , m
Question 29

∂ ∂
Do and commute ?
∂xl ∂xk

Solution: No but actually yes under some good conditions. We discussed here
44
 If f ′′ (a) exists for all a ∈ U , then we get a function f ′′ or D2 f : U → L(Rm , L(Rm , Rn )) which maps
a 7→ f (a). Dimension of the RHS is m2 n. Now we can ask about continuity and differentiability of f ′′
′′

Definition 10.2.1: C k Functions

Open U ⊂ Rm , f : U → Rn . Then
f is C 0 if f is continuous
f is C 1 if f ′ (a) exists ∀ a ∈ U and f ′ is continuous
f is C 2 if f ′′ (a) exists ∀ a ∈ U and f ′′ is continuous
.. .. ..
. . .
f is C k if f (k) (a) exists ∀ a ∈ U and f (k) is continuous

Note:-
How to understand L(V, L(U, W )) where U, V, W are vector spaces. Just set theoretically

Maps( A , Maps( B , C )) ⋍ Maps( A × B , C )

↓ ↓
f (a) : B C ϕ ϕ(a, b)
b ϕ(a, b)
Under this dictionary, maps in L(V, L(U, W )) must correspond to some special kind of maps V × U → W .

Hence we can say L(Rm , L(Rm , Rn )) is equivalent to the space of maps Rm × Rm → Rn which is space of
bilinear maps from Rm × Rm to Rn
∂fi
Component functions of f ′ : U → L(Rm , Rn ) are precisely ∂x j
where i = 1, . . . , n and j = 1, . . . , m. So
  
′′ ∂ ∂
m m n
matrix of f (a) w.r.t standard basis of R and L(R , R ) will consist of numbers fi (a). If f ′′ (a)
∂xk ∂xj
exists then these are generated to exist. If f ′′ (a) exists at each a ∈ U then we have the function

f ′′ : U L(Rm , L(Rm , Rn )) ⋍ Space of bilinear maps Rm × Rm → Rn

a f ′′ (a)

definition ∂ ∂
f ′′ is C 2 (U ) ⇐⇒ f ′′ is continuous on U ⇐⇒ fi are continuous functions U → R
∂xk ∂xj

Theorem 10.2.1
Let U be open in R2 and f : U → R. (a, b) ∈ R and U ⊃ Q(h, k) = [a, a + h] × [b, b + k]. Define

∆(h, k) = f (a + h, b + k) − f (a, b + k) − f (a + h, b) + f (a, b)

= [f (a + h, b + k) − f (a + h, b)] − [f (a, b + k) − f (a, b)]

Then ∃ (s, t) ∈ interior of the rectangle Q(h, k) such that

 
∂ ∂
∆(f, Q) = hk f (s, t) := D21 f (s, t)
∂y ∂x

Proof. U (x) = f (x, b + k) − f (x, b). Hence

∆(f, Q) = U (a + h) − U (a)

45
By M V T we have s ∈ (a, a + h) such that
 
∂f ∂f
∆(f, Q) = hU ′ (s) = h (s, b + k) − (s, b)
∂x ∂x

Apply M V T again and we get t ∈ (b, b + k) such that

 
∂f ∂f ∂ ∂
(s, b + k) − (s, b) = k f (s, t)
∂x ∂x ∂y ∂x

And hence  
∂ ∂
∆(f, Q) = hk f (s, t)
∂y ∂x

Theorem 10.2.2
∂f ∂f ∂ ∂ ∂ ∂
Suppose for f , ∂x , ∂y , ∂y ∂x f exist everywhere and D21 f = ∂y ∂x f is continuous at (a, b). Then
∂ ∂
D12 f (a, b) = ∂x ∂y f exists and
a,b
D21 f (a, b) = D12 f (a, b)

Proof. Let ϵ > 0 then continuity of D21 means that ∃ δ > 0 such that ∀ h, k with max(|h|, |k|) < δ

|D21 f (x, y) − D21 f (a, b)| < ϵ

∀ x, y ∈ Q(h, k)
Take h, k as above and use the Theorem 10.2.1 to find (s, t) such that ∆(f, Q) = hkD21 f (s, t). So

f (a + h, b + k) − f (a + h, b) f (a, b + k) − f (a, b)
 
∆(f, Q) 1
− D21 f (a, b) < ϵ i.e. − − D21 f (a, b) < ϵ
hk h k k

Take limits as k → 0  
1
∂f ∂f
f (a + h, b) − f (a, b) − D21 f (a, b) < ϵ
h
∂y ∂y
 
As we take limit h → 0 the quantity h1 ∂f ∂f
∂y f (a + h, b) − ∂y f (a, b) actually exists and is equal to D21 f (a, b) i.e.
 
∂ ∂
∂x ∂y f (a, b) = D12 f (a, b) exists and is equal to D21 f (a, b)

Corollary 10.2.1
If f is C 2 (U ) then D21 f = D12 f at each point of U

Theorem 10.2.3
Let U ⊂ Rm and f : U → Rn a C k map i.e. k−th total derivative f (k) exists and is continuous on U then

Di1 i2 ···ik f = Diσ(1) iσ(2) ···iσ(k) f

e.g D24714 f = D42417 (f )

Proof. May take m = 1 and work with component real valued functions for k > 2 keep all but two variables
fixed and use earlier result for requisite partial derivative of f . Any permutation can be realized as a sequence of
transpositions

46
Chapter 11

Multivariable Taylor Theorem

In one variable
h h2 hn−1 hn
f (a + h) = f (a) + f ′ (a) + f ′′ (a) + · · · + f (n−1) (a) + f (n) (c)
1! 2! (n − 1)! n!

for a c between a, a + h.

Theorem 11.1 Multivariable Taylor Theorem

Let U ⊂ Rn open and f : U → Rm a C m map (m ≥ 1). Given a ∈ U , for any h in some neighborhood W
of origin, O we have W + a ⊂ U

2 hm−1 hm
f (a + h) = f (a) + f ′ (a) 1!
h
+ f ′′ (a) h2! + · · · + f (m−1) (a) + f (m) (c)
(m − 1)! m!

h1
Σ terms like Dij f (a)hi hj
 h2 

 
D1 f (a) D2 f (a) · · · Dn f (a)  .  2!
 .. 
hn

Hence
m−1
X X (D1s1 · · · Dnsn f )(a) s1 s2
f (a + h) = h1 h2 · · · hsnn + r(h)
s1 !s2 ! · · · sn !
k=0 s1 +s2 +···+sn =k
remainder term

where r(h) is of the form

X (D1s1 · · · Dnsn f )(a + θh) s1 s2
h1 h2 · · · hsnn
s1 +s2 +···+sn =m
s 1 !s2 ! · · · sn !

where θ ∈ (0, 1)

Note:-
r(h)
∥h∥m−1 → 0 as h → 0

In one-variable f : [a, b] → R. Then f (0) , f (1) , . . . , f (m−1) exists in [a, b] and f (m) exists in (a, b). Suppose
s, t ∈ [a, b]. Then there exists θ exactly between s and t such that

f (m−1) (s) f (m) (θ)

f (t) = f (s) + f ′ (s)(t − s) + · · · + (t − s)m−1 + (t − s)m−1
(m − 1)! (m)!
| {z }
p(t)
47
. Then

p(s) = f (s), p′ (s) = f ′ (s), p′′ (s) = f ′′ (s), . . . , p(m−1) (s) = f (m−1) (s) and p(m) (x) = 0 identically

So for g(x) = f (x) − p(x)

g(s) = g ′ (s) = · · · = g (m−1) (s) = 0
.
Idea: Use M V T on g, g ′ , . . . , g (m−1) on [s, t]
If g(t) = 0 then with g(s) = 0 we get (by Rolle’s theorem) θ1 between s and t such that g ′ (θ1 ) = 0.
Now g ′ (θ1 ) = 0 and g ′ (s) = 0 =⇒ we get θ2 between θ1 and s such that g ′′ (θ2 ) = 0. Now g ′′ (θ2 ) = 0 and
g ′′ (s) = 0 =⇒ we get θ3 between θ2 and s such that g ′′′ (θ3 ) = 0 and so on.. till we get θm with g (m) (θm ) = 0.
Take θ = θm .
But is g(t) = 0 ? g(t) = f (t) − p(t) need not be zero.
Idea: We can adjust g by constant M (x − s)m without affecting g(s) = g ′ (s) = · · · = g (m−1) (s) = 0 and we also
want to apply the Rolle’s theorem. Adjust constant M to make g(t) = 0
New g(x) = f (x) − p(x) − M (x − s)m such that g(t) = f (t) − p(t) − M (t − s)m = 0. Hence

f (t) − p(t)
M=
(t − s)m

We get g (m) (θ) = f (m) (θ) − 0 − m!M = 0. S

f (m) (θ)
M=
m!
Equate these two expressions for M and solve for f (θ) to get the result.
Question 30

Carry out proof of multivariable taylor’ theorem following the strategy sketched in the class, specially
dn
using the chain rule to calculate dtn f (a + th)

Question 31

In ‘some sense’, the one-variable Taylor’s Theorem for f (a + th) stays valid in multivariable case.

f
It is enough to proof for m = 1. We have a ∈ U ⊆ Rn −
→ R, f is C m . Then there is a neighborhood W of
origin in Rn such that for any h ∈ W we have a + h ∈ U and

2 hm−1
f (a + h) = f (a) + f ′ (a) 1!
h
+ f ′′ (a) h2! + · · · + f (m−1) (a) + r(h)
(m − 1)!

f (m) (a+θh) m
where r(h) = m! h for some θ ∈ (0, 1) but need to make sense of this.
Proof. Use one-variable taylor’s theorem for the composite
f
[0, 1] a+W ⊂U R

t a + th f (a + th) = g(t)

g is C m because the map t 7→ a + th is C ∞ Hence

m−1
X g (k) (0) g (m) (θ)
f (a + h) = g(1) = +
k! m!
k=0

48
for some θ ∈ (0, 1). Thus we will be done by showing
X k!
g (k) (t) = Ds1 · · · Dnsn f (a + th)hs11 hs22 · · · hsnn
s1 !s2 ! · · · sn ! 1
s1 +s2 +···+sn =k
X
= Di1 · · · Dik f (a + th)hi1 hi2 · · · hin
1≤i1 ,...,ik ≤n

Using chain rule for k = 1

d
g ′ (t) = f (a + th)
dt
d
= f ′ (a + th) (a + th)
dt
= f ′ (a + th)h
Xn
= Di f (a + th)hi
i=1

For k = 2
n n
d ′ d X X d
g ′′ (t) + g (t) = Di f (a + th)hi = Di f (a + th)hi
dt dt i=1 i=1
dt
X n
n X X
= Dj Di f (a + th)hi hj = Dj Di f (a + th)hi
i=1 j=1 1≤i,j≤n

Continue like this

Addendum to Taylor’s Formula: Bounding the error term

For a ∈ U ⊆ Rn and f of class C m from U to R we know that for h ∈ some ball B around origin, we have
a + B ⊂ U and
m−1
X X (D1s1 · · · Dnsn f )(a) s1 s2
f (a + h) = h1 h2 · · · hsnn + r(h)
s1 !s2 ! · · · sn !
k=0 s1 +s2 +···+sn =k

where r(h) is of the form

X (D1s1 · · · Dnsn f )(a + θh) s1 s2
h1 h2 · · · hsnn
s1 +s2 +···+sn =m
s !s
1 2 ! · · · sn !

where θ ∈ (0, 1)
Now because a + B is compact and D1s1 · · · Dnsn f is continuous on U , we can find a constant c such that
n
P
for any s1 , . . . , sn with =m
i=)
D1s1 · · · Dnsn f (a + x)
<c
s1 !s2 ! · · · sn !
for each h ∈ B Also |hi | ≤ ∥h∥. Therefore
X
|r(h)| < c∥h∥m = k∥h∥m
s1 +···+sn =m

r(h)
and therefore ∥h∥m−1 → 0 as h → 0

49
Chapter 12

Maximum and Minimum of

Multivariable Functions

For a C 3 function (in a neighborhood of a in R), by Taylor’s Theorem

 
some point
1 1
f (a + h) = f (a) + f ′ (a)h + f ′′ (a)h2 + f ′′′  between  h3
2 6
a and a + h
| {z }
Remainder term r(h)
r(h)
h2
→0 as h→0

Suppose f ′ (a) = 0 “a is a critical point of f ”. Then

f (a + h) − f (a) 1 r(h)
= f ′′ (a) + 2
h2 2 h
r(h)
If f ′′ (a) > 0 then f has a local minimum at a because choose δ > 0 such that |h| < δ, h2 < 12 f ′′ (a). Then
RHS > 0 ∀ h such that |h| < δ and so for h ∈ (−δ, δ), f (a + h) > f (a) i.e. f (a) is minimum value of f in the
neighborhood (a − δ, a + δ). Similarly f ′′ (a) < 0 then f has a local maximum at a.
We want to find an analogy of this for multivariable case
f : (open U in Rn ) → R a C 3 function. Then for h ∈ some open neighborhood W of origin, a + h ∈ U
 
some point
1 1
f (a + h) = f (a) + f ′ (a)h + f ′′ (a)(h, h) + f ′′′  between  (h, h, h)
2 6
a and a + h
| {z }
Remainder term r(h)
r(h)
h2
→0 as h→0
 
h1
  1 X
= f (a) + D1 · · · Dn  ...  +

Di Dj f (a)hi hj + r(h)
2 i,j
hn
   
h1 h1
 .  1 
h1 · · · hn [Di Dj f (a)]  ...  + r(h)
 
= f (a) + D1 . . . Dn  ..  +
 
2
hn hn
1
= f (a) + ∇f (a) · h + hT [Di Dj f (a)] h + r(h)
2 | {z }
Hessian Matrix
of f at a

50
 Definition 12.1: Hessian Matrix of f

f is C 1 ⇐⇒ ∂f are not continuous on U
n
Let f : (open U in R ) → R such that ∂xi So components of
f ′′ exists at a
f ′′ are Di Dj f (a). Hessian of f at a = Square matrix [Di Dj f (a)]

When f is C 2 , Hessian matrix is Symmetric Matrix

Definition 12.2: Critical Point
f
→ R. a ∈ U is called critical point if f ′ (a) = 0 ⇐⇒ ∇f (a) = 0
Let f be a C 1 function, Open U in Rn −

If f has local maximum at a, then along any line through a the same must be hold, so all directional
derivative =0 at a.
Definition 12.3: Non-degenerate Point

If f is C 2 then a critical point a is called non-degenerate if the Hessian, Hf (a) is non-singular i.e.
det(Hf (a)) ̸= 0

Claim 12.1
Symmetric Matrix A is positive (semi)definite ⇐⇒ ∀ nonzero vector x ∈ Rn , xT Ax > 0 (resp. ≥ 0)

Proof. If Part:
P
x = ci vi . Where vi is the eigen-basis. Then
i

!T   !T  
X X X X X
xT Ax = ci vi A c j vj  = ci vi  λ j c j vj  = λi c2i > 0 [viT vj = δij ]
i j i j i

Only If Part:
Use xT Ax > 0 for x = vi eigenvector < 0, viT Avi = vi λi vi = λi

Note:-
Determinant of positive definite matrix > 0 and Determinant of negative definite matrix has sign (−1)n

Theorem 12.1
Let f : (open U in Rn ) → R. Suppose f has a local maximum or minimum at a then

1 If f ′ (a) exists then f ′ (a) = 0 i.e. a is a critical point.

2 Suppose in addition to that f ′′ (a) exists then if f has local maximum at a, then f ′′ (a) ≤ 0 and if f
has local minimum at a, then f ′′ (a) ≥ 0

Proof. 1 For n = 1 let we have local minimum at a. Then for small |h|

f (a+h)−f (a)
)
h ≥0 for h > 0
f (a+h)−f (a)
Thus imply respectively that f ′ (a) must be ≥ 0 and ≤ 0
h ≤0 for h < 0

51
 For n > 1 use n = 1 in every direction i.e. for function f |a+tv for t ∈ open interval to conclude Dv f (a) = 0
∀ directions. So f ′ (a) = 0

2 For n = 1
f ′ (a + h) − f ′ (a) f ′ (a + h)
f ′′ (a) = lim = lim
h→0 h h→0 h
Observation: If f has local maximum at a then for 0 < |h| < δ, f (a + h) ≥ f (a). So by M V T there is k
between 0 and h such that
f (a + h) − f (a)
= f ′ (a + k)
h
f ′ (a+k)
Using the observation f ′′ (a) = lim h ≥0
h→0

P f |a+tv ∀ direction
For n > 1Papplying this to each vectors v we get all Dv2 f (a) ≥ 0. In terms of
2
P
Hessian let v = ci ei =⇒ Dv f = ci Di f =⇒ D f (a) = i,j cj ci Dj Di f (a) in a neighborhood of a.
 
c1
 .. 
Dv2 f (a) = c1
 
··· cn Hf (a)  . 
cn

Theorem 12.2
If f : (open U in Rn ) → R is a C 3 function and a is a non-generate critical point of f then

f has a local minimum at a ⇐⇒ H is positive definite

⇐⇒ All eigenvalues of H are positive
f has a local maximum at a ⇐⇒ H is negative definite
⇐⇒ All eigenvalues of H are negative
f has saddle-point otherwise H is indefinite

Proof. If Part:
We already proved the if direction in Theorem 12.1

Only If Part:
By Taylor’s theorem
*0 1 T
f ′ (a)x
f (a + x) − f (a) =   + x Hx + r(x)
2
r(x) T
with as ∥x∥ → 0, ∥x∥ 2 → 0. Let’s assume that H is positive definite. So far x ̸= 0 and x Hx > 0. The function
T
x → x Hx is continuous, so on the compact set {u | ∥u∥ = 1} it is bounded and achieves its infimum µ. So µ > 0
So T 
xT Hx
 
x x
≥ µ ∀ x ̸= 0 =⇒ H
∥x∥2 ∥x∥ ∥x∥
r(x) |r(x)| µ
Since ∥x∥ 2 → 0 as ∥x∥ → 0, we can find δ > 0 such that ∥x∥2 < 2 when ∥x∥ < δ. Thus for ∥x∥ < δ we have
f (a + x) − f (a) ≥ 0 i.e. f has a local minimum at a

52
 Definition 12.4: Saddle Point

At a nondegenrate critical point a, H has both

a positive eigenvalue, say λ1 with eigen vector u1
a negative eigenvalue, say λ2 with eigen vector u2

This means Du2 1 f (a) > 0, so in the u1 direction f has local minimum and Du2 2 f (a) < 0, so in the u2
direction f has local maximum

4
2
0
−2
2
−4
−2 0
−1
0
1
2 −2

Example 12.1
Many times functions are C ∞ whenever defined so all of the above applies.
• f (x, y) = c, constant. All derivatives are zero, H is zero.

• f (x, y) = ax + by + c linear, (a, b) ̸= (0, 0). No critical points.

• f (x, y) = quadratic.
General case (x1 , x2 , . . . , xn ) = x ∈ Rn
n
X X n
X
Φ(x) = aii x2i + 2aij xi xj + p i xi + r
i=1 1≤i<j≤n i=1

= xT Ax + px + r
    
a11 ··· a1n x1 x1
  . .. ..   ..  + p  . 
= x1 ··· xn  .. . .  .  1 ··· pn  ..  + r [where aij = aji ]
an1 ··· ann xn xn
Pn
Hence Di Φ(x) = j=1 aij xj + pi , DΦ(x) = 2Ax + p. Critical points: x such that 2Axp = 0
If 2A = H is nonsingular then there is an unique critical point, namely x = −H −1 p. Then this point
is local minimum is H is positive definite, local maximum id H is negative definite and saddle point
otherwise

53
Chapter 13

Examples of Functions and Analyze

Critical Points

Graph of Φ(x) = Φ(x1 , . . . , xn ) is in Rn+1 . We can visualize it in Rn by drawing level sets, namely plot
Φ(x1 , . . . , xn ) = c for various values of constant c in R

Examples
1 f (x, y) = x2

−2

−4

−6
−6 −4 −2 0 2 4 6

2 f (x, y) = x2 + y 2 . Level Sets = Circles centered at (0, 0)

54
 3 f (x, y) = x2 − y 2 . Level Sets c = 0 =⇒ x = ±y, c = 1 =⇒ x2 − y 2 = 1, c = −1 =⇒ x2 − y 2 = −1

c = −1

c=1

4 f (x, y) = xy
 
x+y u2 −v 2 0 1
u= √ ,v = x−y
√ . Then x = u+v
√ ,y = u−v
√ and f (x, y) = . Here A = 1
. Hence eigenvectors
2 2 2 2 2 2 1 0
   
1 1
are and
1 −1

C = −2

C=0 x

We should understand graphs of ‘Quadratic Hypersurfaces’ Φ(x) = 0, where Φ(x) is a quadratic polynomial
in n variables.
‘Standard Form’ is λ1 x22 +λ2 x22 +· · ·+λn x2n + Constant. We will see that by a shift of origin and orthogonal
change of coordinates, we can express any general quadratic Φ to the Standard Form
1 Getting Rid of Linear Part

λ1 x21 + λ2 x22 + · · · + λn x2n + p1 x1 + · · · + pn xn + constant

pi
=λ1 (x1 − a1 )2 + · · · + λn (xn − an )2 + another constant [−2λi ai = pi =⇒ ai = − , assuming λi ̸= 0]
2λi

2 In general we express x in terms of new basis consisting of orthonormal eigenvectors of A.

Nationalizing a matrix A, Γ−1 AΓ = D-diagonal matrix where columns of Γ = eigen basis corresponding to
matrix A. Here Γ is orthogonal matrix ΓΓT = ΓT Γ = I and we have ΓT AΓ = D =⇒ A = ΓDΓT . Now

Φ(x) = xT Ax + pX + r
55
 Let x∗ = coordinate vector of x in terms of new basis consisting of columns of Γ

x∗ = Γ−1 x = ΓT x we use this to formulate Φ

= (xT Γ)D(ΓT x) + pΓ(ΓT x) + r = Φ(x)
= x∗ T Dx∗ + pΓx∗ + r = Ψ(x∗ )
↓ ↓
standard linear
form form

Use step 1 to eliminate the linear term

Now we will look into some more examples.

1 f (x, y) = x2 − xy + y 2

− 21
   
1 1 −1
A= and H =
− 12 1 −1 2
H is positive definite because diagonal entries are positive and determinant = 3 > 0. So the unique critical
point (0, 0) is a local minima

Note:-
  (
a c a, b > 0
2 × 2 symmetric matrix is positive definite ⇐⇒
c b ab − c2 > 0

 T    
x x x
2 Φ(x) = 2x2 + 3y 2 − 4xy − 12x − 14y + 21 = A +p +r
y y y
   
2 −2 4 −4  
A= and H = and p = −12 14
−2 3 −4 6
H is positive definite as diagonal entries are positive and determinant = 8 > 0. The critical point is the
solution of the equation
        
x −12 4 −4 x −12
H =− ⇐⇒ =−
y 13 −4 6 y 14

Hence x = 2, y = −1. Therefore minimum value Φ(2, −1) = 2

Note:-
Another way: Complete the squares

Φ(x) = 2(x − 2)2 + 4(y + 1)2 − 4(x − 2)(y + 1) + 2

= 2u2 + 3v 2 − 4uv + 2

3 f (x, y) = x3 + y 3 − 3x − 3y
 2 
3x − 3
f ′ (x, y) = 3x2 − 3 3y 2 − 3 ,
 
∇f =
3y 2 − 3
(
3x2 − 3 = 0
Critical points are (x, y) such that f ′ (x, y) = 0 i.e. . There are 4 critical points = (±1, ±1)
3y 2 − 3 = 0
 
6x 0
Hessian H =
0 6x

(1, 1, ) → local min, (−1, −1) → local max, (±1, ∓1) → saddle points

56
 Note:-
For x − y 2 + 3x − 3y there are no critical points
3

57
Chapter 14

Constrained Optimizations and

Lagrange Multipliers

Example: Optimize f (x, y)y 2 − x2 subject to the constraint h(x, y) = x2 + y 2 = 1

In other words we want to find extrema of

f |M where M is the level curve for h at level
1 i.e. M = h−1 (1)
Form the way level sets of f interact
with M , here we see that we have maximum
at (0, ±1) and minimum at (±1, 0)
It also appears that the constrained
graph and the level curve of the objective
function f are tangential to each other
↓
What it means
We will define Tangent Space to a
level set of a C 1 function at a point p on M

14.1 Tangent Space

Definition 14.1.1: Tangent Space

Tangent Space to a hypersurface M = f −1 (c) in Rn where f : (Open U ⊂ Rn ) → R is a C 1 function and

c ∈ R at a point p ∈ M is a subspace of Rn defined to be

Tp M = ker(f ′ (p)) = {v ∈ Rn | f ′ (p)(v) = 0} = {v ∈ Rn | ∇f (p) · v = 0}

Geometric tangent space considering to our mental image = Tp M + p = Shift Tp M by vector p. Likewise
define Normal Space to be the set of vectors orthogonal to Tp M i.e. Tp M ⊥

58
 Eg. f (x, y) = y − x2, M = f−1 (0). p = (3, 9)
 ∈ M.
Here f ′ (p) = −2x 1 (3,9) = −6 1
 
x
7→ −6x + y
y

Tp M = ker(f ′ (p)) = {(x, y) | y = 6x} = red line.

Np M = line y = − 61 x = blue line
Geometric tangent space = Tp M + p and
geometric normal space = Np M + p

14.2 Lagrange Multiplier

Let U be open in Rn . f : U → R objective function and h : U → R constraint function. Want to find extrema of
f restricted to the level set M = {x ∈ U | h(x) = c} = h−1 (c) for c ∈ R
f |M has local maxima at p ∈ M means for some W ⊂ U , f (p) ≥ f (x) ∀ x ∈ W ∩ M

Definition 14.2.1: C 1 Path and Velocity Vector

A C 1 path centered at p ∈ U in U ⊂ Rn is a C 1 map γ : (−ε, ε) → U where 0 7→ p. We call γ ′ (0) =

velocity of γ at 0

Theorem 14.2.1 Lagrange Multiplier

Let U be open in Rn . f : U → R, h : U → R. Let f, h are C 1 functions. Let M = h−1 (c). If h′ (p) ̸= 0 and
f |M has a local extrema at p ∈ M then ∃!λ ∈ R such that

∇f (p) = λ∇h(p)

Proof. Consider paths on level set M = h−1 (c) i.e.

γ : (−ε, ε) M = h−1 (c)

∩
U h R
Then h = γ(t) = c ∀ t ∈ (−ε, ε). Hence by Chain Rule

h′ (p)γ ′ (0) = ∇h(p) · γ ′ (0) = 0

i.e. {velocity vectors of all paths γ on M centered at p} ⊂ Tp M

Key Fact: When h′ (p) ̸= 0 we have equality! Proof of this fact uses Implicit Function Theorem
Now let’s recall the objective function f and recall that p is assured to be a local max/min. If γ is a C 1
curve on M then in particular f |image(γ) also has a max/min at p. Therefore

0 = (f ◦ γ)′ (o) = ∇f (p) · γ ′ (0)

i.e. ∇f is orthogonal to velocity vectors to all curves centered at p.

By claim ∇f (p) ⊥ Tp M , we already say ∇h(p) ⊥ Tp M . We know ∇h(p) ̸= 0 by assumption. Hence ∃!λ
such that ∇f (p) = λ∇h(p)

14.3 Some Examples for Applications

(i) f (x, y) = y 2 − h2 and h(x, y) = x2 + y 2 , c = 1. Therefore M = h−1 (1) = Unit Circle

59
  
a
Suppose p = is an extremum of f |M
b
       
−2x −2a 2x 2a
∇f (p) = = ∇h(p) = =
2y (a,b) 2b 2y (a,b) 2b

We know that ∃!λ ∈ R such that    

−2a 2a
=λ
2b 2b
This is not possible unless one of a, b os 0. Therefore

a=0 =⇒ b = ±1 and λ = 1
b=0 =⇒ a = ±1 and λ = −1

(ii) f (x, y) = y is subject to constraint h(x, y) = y − g(x) = 0 where g : R → R is some C 1 function. This is
equivalent to finding extrema of y = g(x) as in school
 
a
Suppose p = gives an extremum
b
   ′ 
0 −g (a)
∇f (p) = = λ∇h(p) = λ
1 1

i.e. 1 = λ =⇒ 0 = −λg ′ (a) =⇒ g ′ (a) = 0 as expected

(iii) f (x, y) = x2 subject to h(x, y) = y = 0
   
2x 0
∇f = = λ∇h = =⇒ λ = 0, x = 0
0 1

Note:-
If we instead take h(x, y) = y 2 , then we get (, xy) = (0, 0) but λ arbitrary

x2 y2
(iv) f (x, y) = xy subject to h(x, y) = 9 + 4 =1
   2x 
y
∇f = = λ∇h = λ y9
x 2
Therefore
2x y x2 y2
y= λ, x= λ, + =1
9 2 9 4
λ = ±3. Find extrema. As constraint = ellipse, a compact set, evaluating f as candidates is enough to find
max and min.
(v) Find the points on the sphere x2 + y 2 + z 2 = 9 closest/furthest from (a, b, c) → arbitrary point in R3
f (x, y, z) = (x − a)2 + (y − b)2 + (z − c)2 and h(x, y, z) = x2 + y 2 + z 2 = 9. Complete this and see that
geometrically obvious solution emerge

Next we will prove Inverse Function Theorem and Implicit Function Theorem and come back to justify the
claim. In fact we will then be able to prove the general version of Lagrange Multiplier Method i.e. with multiple
constraints

14.4 Generalized Lagrange Multiplier

Theorem 14.4.1 Generalized Lagrange Multiplier

U open ⊂ Rn = Rd+m want to find extrema of objective function f : U → R subject to constraint h = c

60
 for a C 1 function: U → Rm where c ∈ Rm i.e. we want to find extreme of f |M =h1 (c)

Key Assumption: ∀ x ∈ M , h′ (x) is surjective i.e. h′ (x) : Rd+m → Rm . (So ker(h′ (x)) has dim d. Recall we
called ker(h′ (x)) = Tx M )
Suppose f |M has a local extremum at p ∈ M Then ∃! real numbers λ1 , λ2 , . . . , λm such that

∇f (p) = λ1 ∇h1 (p) + · · · + λn ∇hn (p)

 T
where h(p) = h1 (p) · · · hm (p) ∈ Rm
Proof. we will show that
1 ∇f (p) ⊥ Tp M

2 Any vector ⊥ Tp M is a linear combination of ∇hi (p)

These are the steps.

1 Let v ∈ Tp M = ker(f ′ (p)). By HW4 Problem v can be represented by some curve based at p i.e. we can
find a C 1 curve γ : (−ε, ε) → M ⊂ U where 0 7→ p such that γ ′ (0) = v.
γ f
As we have an extremum of f |M at p it is also an extreme point for (−ε, ε) −
→M −
→ R. So by 1-Variable
Calculus (f ◦ γ)′ (0) = 0 i.e. f ′ (p)γ ′ (0) = 0 i.e. ∇f (p) · v = 0
2 For every curve γ as above h ◦ γ = constant. Therefore h′ (p)γ ′ (0) = 0 i.e. ∇h′ (p) · v = 0
 ∂h1 ∂h1
···
∂x1 (p) ∂xn (p) ∇h1 (p)T
  
′  .. .. ..  =  ..   T
h (p) =  . . .   .  = ∇h1 (p) · · · ∇hm (p)
∂hm ∂hm T
∂x1 (p) · · · ∂xn (p)
∇h m (p)

So
∇h1 (p) · v = 0, . . . , ∇hm (p) · v = 0
Therefore ∇hi (p) ⊥ Tp M . Everything is in Rn = Rm+d . ∴ (Tp M )⊥ has ∇h1 (p), . . . , ∇hm (p) as a
| {z } | {z }
m linearly dim n−m
independent =d
vectors
basis. i.e. 2 is proved

61
Chapter 15

Inverse Function Theorem

Definition 15.1: Homeomorphism

A bijective continuous function whose inverse is also continuous is called homeomorphism

Theorem 15.1 Inverse Function Theorem

Suppose U be an open set in Rn . f : U → Rn be a C 1 function. f ′ (a) = A is invertible . Then
1 f is injective in some neighborhood of a
f
2 There are open sets V ⊂ U, a ∈ V and W ⊂ Rn , f (a) ∈ W such that f is a bijection V −
→
− W whose
←
g
inverse, g is also continuous i.e. a local homeomorphism i.e. at the given point a there exists a
neighborhood at which f is homeomorphism
3 f −1 is also differentiable on W i.e. for any f (u) ∈ W

Df − (f (u)) = Df (u)−1

Note:-
1. Crucial that dim U and target are the same

2. There are appropriate versions of the theorem when f ′ (a) is injective / surjective / arbitrary (when f ′ (a)
is surjective it is the Implicit Function Theorem) those versions can be proved using the theorem

Proof. • n = 1 is easy. Directly using M V T

• We may assume that a = 0 and f (a) = 0 (replace f by f (u + a) − f (a)) and f ′ (a) = Identity (replace f by
f ′ (a)−1 f (a)) check that the result for given f follows easily from result for this normalized f
Normalization makes formulation / calculation in the proof a bit simpler but may assume a bit the
natural main ideas, so we won’t normalize.
1 Injectivity of f on a ball B around a of small radius ε. We will choose ε later.
Best linear approximation for f (x) near a is f (x) + f ′ (a)(x − a). If f were = this function, the theorem is
easy so let’s examine the remainder
r(x) = f (x) − f (a) − f ′ (a)(x − a)
r′ (x) = f ′ (x) − f ′ (a)
We can make f ′ (x) − f ′ (a) small in some ball B around a by continuity of f ′ at a.
r(x1 ) − r(x2 ) = f (x1 ) − f (x2 ) − f ′ (a)(x1 − x2 )

Choose a good open ball B centered at a with all of the following properties:
62
 (i) Ensure that ∀ x ∈ B ∥f ′ (x) − f ′ (a)∥ < ε
f det
→ L(Rn ) −−→ R is continuous at a and det f ′ (a) ̸= 0 so can choose B such that ∀ x ∈ B, det f ′ (x) ̸= 0
(ii) U −
and hence f (x) is invertible.
(iii) Shrink B further if necessary to ensure B ⊂ U (useful later to minimize a continuous function on this
compact set.)

U
∂B
f (∂B)

f −1 (W ) = V B
W f (B)
r
a f (a)

r < 12 δ

∀ x ∈ B , f ′ (x) is invertible. ∥r′ (x)∥ = ∥f ′ (x) − f ′ (a)∥ < ε. By M V T applied on r(x) on the convex set B,
we get for any x1 , x2 ∈ B

∥f (x1 ) − f (x2 ) − f ′ (a)(x1 − x2 )∥ = ∥r(x1 ) − r(x2 )∥ ≤ ε∥x1 − x2 ∥

Now recall ∥p − q∥ ≥ ∥p∥ − ∥q∥. Hence

∥f (x1 ) − f (x2 ) − f ′ (a)(x1 − x2 )∥ ≥ ∥f ′ (a)(x1 − x2 )∥ − ∥f (x1 ) − f (x2 )∥

Needed
↓
Upshot: ∥f (x1 ) − f (x2 )∥ ≥ ∥f ′ (a)(x1 − x2 )∥ − ε∥x1 − x2 ∥ ≥ · · ·

Note:-
At this point if we had normalized f ′ (a) = Identity then we would have gotten

∥f (x1 ) − f (x2 )∥ ≥ (1 − ε)∥x1 − x2 ∥

Taking ε < 1 fives the injectivity of f

In our case we need to find lower bound on ∥f ′ (a)(x1 − x2 )∥. Minimize {∥f ′ (a)u∥ | ∥u∥ = 1}. f ′ (a)
is continuous and the set of all unit vectors is compact. This set has a minimum, minimum=m > 0 as it is
invertible so f ′ (non zero vector) ̸= 0.
Now take ε < m and then in the resulting ball B we have

∥f (x1 ) − f (x2 )∥ ≥ (m − ε)∥x1 − x2 ∥ (15.1)

f
This gives the injectivity of f on B. So we have the bijection B −
←−
−−−−
−→
− f (B). (15.1) is saying that any
f −1 =g
y1 = f (x1 ), y2 = f (x2 ) in f (B), i.e. g(y1 ) = x1 , g(y2 ) = x2
1
∥g(y1 ) − g(y2 )∥ ≤ ∥y1 − y2 ∥
m−ε
i.e. g is uniformly continuous.

63
2 We have bijection of f between B and f (B), V is supposed to be open but we have taken open ball, so its
open. Inverse of f (g) is continuous so what left is f (B) open
To show that f is a local Homeomorphism it is enough to find an open ball W around f (a) with
f
W ⊂ f (B). Then we simply take V = f −1 (W ) which is open by continuity of f and clearly V −
→
− W are
←
g
bijections just restrict f, g from B, f (B) respectively.
How to construct W ? What radius to take around W ? Stay away from f (∂B). δ = min{∥f (x) −
f (a)∥ | x ∈ ∂B} > 0. Choose radius of W to be 12 δ. We will be done if we show W ⊂ f (B) i.e. given any
c ∈ W ∃ x∗ ∈ B such that f (x∗ ) = c (x∗ is necessarily unique, by injectivity).
(
B → R≥0
Idea: Consider the differentiable function
x 7→ ∥f (x) − c∥2
Note that ∥c − any point on f (∂B)∥ > r by triangle inequality where as ∥c − f (a)∥ < r (as c is inside
W = ball of radius r < 21 δ around f (a)). Hence ∥f (x) − c∥2 will take its minimum value at some point say
x∗ ∈ B. Now f = (f1 , f2 , . . . , fn ) and c = (c1 , . . . , cn )
n
X
µ(x) = ∥f (x) − c∥2 = (fi (x) − ci )2
i=1

Derivative of this function is 0 at x∗

f µ
B Rn R

x f (x) = y ∥y − c∥2

Hence
µ′ (f (x∗ )) ◦ f ′ (x∗ ) = 0
| {z }

2(f1 (x∗ ) − c1 ) · · · 2(fn (x∗ ) − cn ) (Matrix of f ′ (x∗ ) – Invertible)

 

Therefore 2(f1 (x∗ ) − c1 ) · · · 2(fn (x∗ ) − cn ) must be 0 i.e. fi (x∗ ) = ci i.e. f (x∗ ) = c. So we
 

showed that each c ∈ W is in the image of f . Now take V = f −1 (W ) and we have the Homeomorphism.

3 Differentiability of f −1 = g at any point y ∈ W .

f
x g
y
add h add k
f
x+h g
y+k ∈W

Take small k ∈ Rn and let h = g(y + k) − g(y) and k = f (x + h) − f (x). Each of h and k determines
the other uniquely. In particular h ̸= 0 ⇐⇒ k ̸= 0 (by bijectivity). h → 0 ⇐⇒ k → 0 (by continuity of
f and g). α(h) = f (x + h) − f (x) − T h = k − T h where T = f ′ (x). Then we have ∥α(h)∥
∥h∥ → 0 as ∥h∥ → 0.
We want to show g ′ (y) = T −1

64
 ∥β(k)∥
Let β(k) = g(y + k) − g(y) − T −1 k = h − T −1 k. We will show that as k → 0, ∥k∥ →0

∥β(k)∥ ∥h − T −1 k∥ ∥T −1 (T h − k)∥ ∥T −1 ∥
= = ≤ ∥T h − k∥
∥k∥ ∥k∥ ∥k∥ ∥k∥
∥T −1 ∥
= ∥α(h)∥
∥k∥
∥T −1 ∥ ∥α(h)∥
= ∥h∥
∥k∥ ∥h∥
∥h∥ ∥α(h)∥
= ∥T −1 ∥
∥k∥ ∥h∥
∥h∥ 1 ∥α(h)∥
We know by ∥k∥ < m−ε by (15.1). ∥h∥ → 0 as k → 0 because then h → 0.

Note:-
• For another proof of surjectivity onto W , see Rudin’s use of contraction property
• There is a more general result which assumed only invertibility of f ′ (x) for x ∈ U but not continuity
of f ′ everywhere. (See exposition on Terence Tao’s Blog: https://terrytao.wordpress.com/tag/
inverse-function-theorem/)
• f need not be globally invertible!
Example = See Problem 17 from Rudin
f (x, y) = (ex cos y, ex sin y). Then
 x
−ex sin y det

′ e cos y
f (x, y) = x −−→ (ex )2 (cos2 x + sin2 y) = e2x > 0
e sin y ex cos y

Thus f is locally invertible everywhere with C 1 inverse.

• f is not globally one-one f (x, y) = f (x, y + 2π). Do the rest.

Corollary 15.1
If f is a C 1 map from open U in Rn to Rn and f ′ (x) is invertible ∀ x ∈ U then
1 f is an open map

2 f is locally invertible with each such inverse a C 1 dunction (because matrix of (f −1 )′ = inverse of
matrix of f ′ and entries of A−1 = 1
det A (polynomials in entries of A) in particular A → A−1 is
continuous)

65
Chapter 16

Implicit Function Theorem

Notation: For n > m let n = m + d. Write points of Rn = Rd × Rm as (x, y) where x ∈ Rd , y ∈ Rm

Theorem 16.1 Implicit Function Theorem

Let U open in Rd+m . Φ : U → Rm is a C 1 map such that Φ′ (p) is surjective (which means columns of the
m×(d+m) matrix of Φ′ (p) span Rm ). WLOG suppose the last m columns of Φ′ (p) are linearly independent
∂Φ  
and hence span Rm i.e. the m × m matrix “ = Dd+1 Φ(p) · · · Dd+m Φ(p) ” is invertible. Then
∂y p
f
1. ∃ a neighorhood W of a in )Rd and a unique C 1 map W − → Rm such that
f (a) = b, (x, f (x)) ∈ U ∀ x ∈ W and
i.e. f is an implicit solution to the equation Φ(x, y) = c
Φ(x, f (x)) = c ∀ x ∈ W

2. One can calculate f ′ (x) by “Implicit Differentiation”

To understand this, first examine two cases:

 
x
• When Φ is a linear map given by a matrix A. Here we are solving the equation A =c
y

∂Φ
• d = m = 1 i.e. n = 2 Φ(x, y) = x2 + y 2 − 1, solving Φ(x, y) = 0 = c. When ̸= 0 we can locally
∂y p=(a,b)
solve for y in terms of x near p. h i
∂Φ ∂Φ
 
DΦ = ∂x ∂y = 2a 2b
(a,b)

2b = 0 at (±1, 0)
Proof. We will choose W later. Define
ψ
U Rd+m

(x, y) (x, Φ(x, y))

 
I O
Note ψ ′ has the matrix ∂Φ ∂Φ . This is nonsingular in a neighborhood of p. So by Inverse Function Theorem
∂x ∂y
1
ψ is invertible with C inverse in a neighborhood V of p

V ψ(V )
(a, b) (a, c)
(x, y) (x, Φ(x, y))
(u, α(u, v)) (u, v)

66
Definition of α(u, v) defined on ψ(V ). This tells us α(a, c) = b. Whenever Φ(x, y) = c i.e.

Φ ψ −1
(x, y) −
→ (x, c) −−−→ (x, α(x, c)) = (x, y)

i.e. y = α(x, c) and Φ(x, α(x, c)) = c

So we are forced to define f (x) = α(x, c). But what should be the domain of this function f i.e. what
should we take W to be.
 
open ball W
(a, c) ∈ ψ(V ) is open ⊃ × {c}
around a in Rd

Now for any x ∈ W we know (x, c) ∈ ψ(V ) i.e. (x, α(x, c)) ∈ V so we define f : W → Rm where
f (x) = α(x, c) and we have derived the function. Now ϕ−1 is C 1 and α is component of ϕ−1 so all components
of ϕ−1 is also C 1 . hence f is C 1
Uniqueness of f is not true in general for arbitrary W . Φ(x, y) = x2 + y 2 , c = 1. In W = W1 ⊔ W2
(√
1 − x2 x ∈ W1 [is forced]
f (x) = √ √
1 − x2 or − 1 − x2 x ∈ W2

. We have choice for f on W2 .

If W is connected, f will be unique. Eg. take W to be a ball. Suppose g is another solution to
Φ(x, y) = c i.e. Φ(x, g(x)) = c for x ∈ W and g(a) = b. Then consider the set S = {x ∈ W | f (x) = g(x)}. Show
that this set is both closed (easy S = (f − g)−1 (0)) and open.
Calculate derivative of f using the fact that ψ ◦ ψ −1 = Identity and Chain Rule.

Example 16.0.1 (Application of Implicit Function Theorem)

   
a x
(i) Linear map Φ : Rd+m → Rm given by matrix A. Given A = c. Want to solve A =c
b y
A = [P | Q] where P is m × d and Q is m × m and Q is invertible. i.e.
   
x −1 x
[P | Q] = c ⇐⇒ [Q P | I] = Q−1 c ⇐⇒ Q−1 P x + y = Q−1 c
y y
⇐⇒ y = Q−1 c − Q−1 P x

∂Φ
(ii) We can solve for y in terms of x near any (a, b) on the unit circle when ̸= 0. [This is mate
∂y (a,b)
when b ̸= 0 i.e. at all points except (±1, 0)].
 
DΦ|(a,b) = 2a 2b

We can see directly

√ )
when b > 0 y= 1 − x2
√ near (a,b) in fact ∀ x ∈ (−1, 1)
when b < 0 y = − 1 − x2

∂Φ
Similarly we can solve for x in terms of y when = 2a ̸= 0 This is true when a ̸= 0
∂x (a,b)

67
Remark: Implicit Function Theorem gives a sufficient condition to be able to locally solve a system of linear
equations
Φ1 (x1 , . . . , xd , y1 , . . . , ym ) = c1 


 for yi ’s in terms of xi ’s
Φ2 (x1 , . . . , xd , y1 , . . . , ym ) = c1 


.. .. .. locally near a given solution

. . . 


 y = b and x = a

Φm (x1 , . . . , xd , y1 , . . . , ym ) = c1

Note:-
The condition of invertibility of submatrix of Φ is not necessary. Eg. Φ(x, y) = y − x3 near (0, 0)

∂Φ
DΦ|(0,0) = −3x2
   
1 (0,0)
= 0, 1 (0, 0) = 0
∂x
√
but still we can solve for x in terms of y: x = 3 y

68
Chapter 17

Complex Differentiation

f
Suppose U open in C = R2 , U −
→ C a differentiable map i.e. Df as an R linear operator R2 → R2 is defined.
Note:-
There is one thing that makes C differ from R2 i.e. C forms a field.

Definition 17.1: Complex Differentiation

f : U → C is called complex differentiable at z0 ∈ U if

f (z0 + h) − f (z0 )
lim exists
h→0 h
Thus the limit equals to f ′ (z0 ). h is a complex number and this is a division in the field C. ‘h → 0’ means
∥h∥ → 0

Definition 17.2: Holomorphic Function

We say that f is holomorphic on U if f ′ (z) exists ∀ z ∈ U . Then f ′ : U → C is the derivative of f where

z 7→ f ′ (z)

Theorem 17.1
f is holomorphic ⇐⇒ so is f ′

Cauchy Riemann Conditions

Suppose f ′ (z0 ) exists where z0 = a + ib ∈ open U ⊂ C. f (z) = u + iv where u : U → R and v : U → R. First
take h = t ∈ R.
f (a + t + ib) − f (a + ib)
f ′ (z0 ) = lim
t→0 t
u(a + t, b) − u(a, b) v(a + t, b) − v(a, b)
= lim + i lim
t→0 t t→0 t
∂u ∂v
= +i (17.1)
∂x z0 ∂x z0

69
Now take h = it, t ∈ R

f (a + ib + it) − f (a + ib)
f ′ (z0 ) = lim
t→0 it
u(a, b + t) − u(a, b) v(a, b + t) − v(a, b)
= lim + i lim
t→0
" it #
t→0 it
1 ∂u ∂v
= +i
i ∂y z0 ∂y z0

∂v ∂u
= −i (17.2)
∂y z0 ∂y z0

Equating (17.1) and (17.2) f is complex differentiable at z0 and

∂u ∂v ∂v ∂u
= =−
∂x z0 ∂y z0 ∂x z0 ∂y z0

In fact more is true.

Claim 17.1
f
For open U ⊂ C if U −
→ C is complex differentiable at z0 ∈ U then f is also real differentiable as a function
U → R2 and its Jacobin is (letting f = (u, v), u, v : U → R)
" # " # " #
∂u ∂u ∂u ∂u ∂v ∂v
∂x ∂y ∂x ∂y ∂y − ∂x
Jf (z0 ) = ∂v ∂v = = ∂v
∂x ∂y − ∂u
∂y
∂u
∂x ∂x
∂v
∂y
z0 z0 z0

70