Lecture Notes On Modelling PDF

Lecture Notes for
Computational Methods in Chemical

Engineering (CL 701)
Sachin C. Patwardhan,
Department of Chemical Engineering,
Indian Institute of Technology, Bombay,
Mumbai 400076, India.
Contents
Chapter 1. Mathematical Modeling and Simulation 1

1. Introduction 1
2. Mechanistic Models and Abstract Equation Forms 3
3. Summary 19
4. Appendix: Basic Concepts of Partial Differential Equations 19
Chapter 2. Fundamentals of Functional Analysis 23

1. Introduction 23
2. Vector Spaces 26
3. Normed Linear Spaces and Banach Spaces 29
4. Inner Product Spaces and Hilbert Spaces 33
5. Problem Formulation, Transformation and Convergence 42
6. Summary 51
7. Exercise 51
Chapter 3. Linear Algebraic Equations and Related Numerical Schemes 55

1. Solution of Ax = b and Fundamental Spaces of A 55
2. Direct Solution Techniques 60
3. Solutions of Sparse Linear Systems 61
4. Iterative Solution Techniques for solving Ax = b 73
5. Well Conditioned and Ill-Conditioned Problems 88
6. Solving Nonlinear Algebraic Equations 97
7. Solutions of ODE-BVP and PDEs by Orthogonal Collocation 108
8. Summary 115
9. Appendix 115
10. Exercise 119
Chapter 4. ODE-IVPs and Related Numerical Schemes 127

1. Motivation 127
2. Analytical Solutions of Multivariable Linear ODE-IVP 135
3. Numerical Methods for the solution of ODE-IVP 141
4. Stability Analysis and Selection of Integration Interval 156
5. Summary 161
iii
iv CONTENTS
6. Exercise 162
Chapter 5. Optimization and Related Numerical Schemes 165

1. Introduction 165
2. Principles of Optimization 165
3. Model Parameter Estimation 171
4. Multivariate Linear Regression 179
5. Projections: Geometric Interpretation of Linear Regression 182
6. Statistical Interpretations of Linear Regression 189
7. Simple Polynomial Models and Hilbert Matrices 199
8. Nonlinear in Parameter Models 203
9. Unconstrained Nonlinear Programming 207
10. Numerical Methods Based on Optimization Formulation 222
11. Summary 238
12. Exercise 238
Bibliography 243
CHAPTER 1
Mathematical Modeling and Simulation
1. Introduction
A modern chemical plant consists of interconnected units such as heat
exchangers, reactors, distillation columns, mixers etc. with high degree of inte-
gration to achieve energy efficiency. Design and operation of such complex plants
is a challenging problem. Mathematical modeling and simulation is a cost ef-
fective method of designing or understanding behavior of these chemical plants
when compared to study through experiments. Mathematical modeling can-
not substitute experimentation, however, it can be effectively used to plan the
experiments or creating scenarios under different operating conditions. Thus,
best approach to solving most chemical engineering problems involves judicious
combination of mathematical modeling and carefully planned experiments.
To begin with, let us look at types of problems that can arise in context of
modeling and simulation. Consider a typical small chemical plant consisting of
a reactor and a distillation column, which is used to separate the product as
overhead (see Figure 1). The reactants, which are separated as bottom product
of the distillation column, are recycled to the reactor. We can identify following
problems
• Process Design problem
Given: Desired product composition, raw material composition and avail-
ability.
• To Find: Raw material flow rates, reactor volume and operating con-
ditions (temperature, pressure etc.), distillation column configuration
(feed locations and product draws), reboiler,condenser sizes and oper-
ating conditions (recycle and reflux flows, steam flow rate, operating
temperatures and pressure etc.)
• Process Retrofitting: Improvements in the existing set-up or oper-
ating conditions
Plant may have been designed for certain production capacity and
assuming certain raw material quality. We are often required to assess
whether
1
2 1. MATHEMATICAL MODELING AND SIMULATION
— Is it possible to operate the plant at a different production rate?

— What is the effect of changes in raw material quality?
— Is it possible to make alternate arrangement of flows to reduce
energy consumption?
• Dynamic behavior and operability analysis: Any plant is de-
signed by assuming certain ideal composition of raw material quality,
temperature and operating temperatures and pressures of utilities. In
practice, however, it is impossible to maintain all the operating condi-
tions exactly at the nominal design conditions. Changes in atmospheric
conditions of fluctuations in steam header pressure, cooling water tem-
perature, feed quality fluctuations, fouling of catalysts, scaling of heat
transfer surfaces etc. keep perturbing the plant from the ideal oper-
ating condition. Thus, it becomes necessary to understand transient
behavior of the system in order to
— reject of effects of disturbances on the key operating variables such
as product quality
— achieve transition from one operating point to an economically
profitable operating point.
— carry out safety and hazard analysis
In order to solve process design or retrofitting problems, mathematical mod-

els are developed for each unit operation starting from first principles. Such
mechanistic (or first principles) models in Chemical Engineering are combina-
tion of mass, energy and momentum balances together with associated rate
equations, equilibrium relation and equations of state.
• Mass balances: overall, component.

• Rate equations: mass, heat and momentum transfer rates (constitu-
tive equations.), rate of chemical reactions
• Equilibrium principles : physical( between phases) and chemical
(reaction rate equilibrium).
• Equations of state: primarily for problems involving gases.
From mathematical viewpoint, these models can be classified into two broad
classes
• Distributed parameter model: These models capture the relationship

between the variables involved as functions of time and space.
• Lumped parameter models: These models lump all spatial variation
and all the variables involved are treated as functions time alone.
2. MECHANISTIC MODELS AND ABSTRACT EQUATION FORMS 3
Figure 1. Typical processing plant: Schematic daigram
The above two classes of models together with the various scenarios under
consideration give rise to different types of equation forms such as linear / non-
linear algebraic equations, ordinary differential equations or partial differential
equations. In order to provide motivation for studying these different kinds of
equation forms, we present examples of different models in chemical engineering
and derive abstract equation forms in the following section.
2. Mechanistic Models and Abstract Equation Forms

2.1. Linear Algebraic Equations. Plant wide or section wide mass
balances are carried out at design stage or later during operation for keeping
material audit. These models are typical examples of systems of simultaneous
linear algebraic equations..
Figure 2
Example 1. Recovery of acetone from air -acetone mixture is achieved us-

ing an absorber and a flash separator (Figure 2). A model for this system is
developed under following conditions
• All acetone is absorbed in water
• Air entering the absorber contains no water vapor
• Air leaving the absorber contains 3 mass % water vapor
The flash separator acts as a single equilibrium stage such that acetone mass
fraction in vapor and liquid leaving the flash separator is related by relation
(2.1) y = 20.5x
where y mass fraction of the acetone in the vapor stream and x mass fraction
of the acetone in the liquid stream. Operating conditions of the process are as
follows
• Air in flow: 600 lb /hr with 8 mass % acetone
• Water flow rate: 500 lb/hr
It is required that the waste water should have acetone content of 3 mass
% and we are required to determine concentration of the acetone in the vapor
stream and flow rates of the product streams.
Mass Balance:
(2.2) Air : 0.92Ai = 0.97Ao

(2.3) Acetone : 0.08 Ai = 0.03 L + y V
(2.4) W ater : W = 0.03 Ao + (1 − y)V + 0.97L
(2.5) Design requirement : x = 0.03
Figure 3. Flash Drum: Schematic Diagram
Equilibrium Relation:
(2.6) y = 20.5 x
(2.7) ⇒ y = 20.5 × 0.03 = 0.615
Substituting for all the known values and rearranging, we have
⎡ ⎤⎡ ⎤ ⎡ ⎤
0.97 0 0 Ao 0.92 × 600
⎢ ⎥⎢ ⎥ ⎢ ⎥
(2.8) ⎣ 0 0.03 0.615 ⎦ ⎣ L ⎦ = ⎣ 0.08 × 600 ⎦
0.03 0.385 0.97 V 500
The above model is a typical example of system of linear algebraic equations,
which have to be solved simultaneously. The above equation can be represented
in abstract form set of linear algebraic equations
(2.9) Ax = b
where x and b are a (n × 1) vectors (i.e. x, b ∈ Rn ) and A is a (n × n) matrix.
2.2. Nonlinear Algebraic Equations. Consider a stream of two

components A and B at a high pressure Pf and temperature Tf as shown in
Figure 3. If the Pf is greater than the bubble point pressure at Tf , no vapor will
be present. The liquid stream passes through a restriction (valve) and is flashed
in the drum, i.e. pressure is reduced from Pf to P . This abrupt expansion
takes place under constant enthalpy. If the pressure P in the flash drum is less
than the bubble point pressure of the liquid feed at Tf , the liquid will partially
Figure 4
vaporize and two phases at the equilibrium with each other will be present in
the flash drum. The equilibrium relationships are
• Temperature of the liquid phase = temperature of the vapor phase.
• Pressure of the liquid phase = pressure of the vapor phase.
• Chemical potential of the i0 th component in the liquid phase = Chem-
ical potential of the i0 th component in the vapor phase
Example 2. Consider flash vaporization unit shown in Figure 4. A hydro-

carbon mixture containing 25 mole % of n butane, 45 mole %of n -hexane is
to be separated in a simple flash vaporization process operated at 10 atm. and
2700 F. The equilibrium k- values at this composition are
Component zi ki
n-butane 0.25 2.13
n-pentane 0.45 1.10
n-hexane 0.30 0.59
Let xi represent mole fraction of the component i in liquid phase and yi
represent mole fraction of the component i in vapor phase. Model equations for
the flash vaporizer are
• Equilibrium relationships
(2.10) ki = yi /xi (i = 1, 2, 3)
• Overall mass balance
(2.11) F =L+V
• Component balance
(2.12) zi ∗ F = xi ∗ L + yi ∗ V (i = 1, 2, 3)
(2.13) = xi ∗ L + ki ∗ xi ∗ V
X
(2.14) xi = 1
Note that this results in a set of simultaneous 5 nonlinear

algebraic equations in 5 unknowns Equations (2.11-2.14) can be written
in abstract form as follows
(2.15) f1 (x1 , x2 , x3 , L, V ) = 0
(2.16) f2 (x1 , x2 , x3 , L, V ) = 0
............................ = 0
(2.17) f5 (x1 , x2 , x3 , L, V ) = 0
which represent coupled nonlinear algebraic equations. These equations

have to be solved simultaneously to find solution vector
h iT
(2.18) x = x1 x2 x3 L V
The above 5 equations can also be further simplified as follows

∙ µ ¶ ¸
V
xi = zi / 1 + (ki − 1)
F
P
Using xi = 1,we have
X zi
(2.19) f (V /F ) = −1=0
1 + (V /F ) (ki − 1)
In general, we encounter n nonlinear algebraic equations in n variables,
which have to be solved simultaneously. These can be expressed in the following
abstract form
(2.20) f1 (x1 , x2 , x3, ......xn ) = 0

(2.21) f2 (x1 , x2 , x3, ......xn ) = 0
............................ = 0
(2.22) fn (x1 , x2 , x3, ......xn ) = 0
Using vector notation, we can write
(2.23) F (x) = 0 ; x ∈ Rn
h iT
x = x1 x2 ... xn
where 0 represents n × 1 zero vector. Here F (x) ∈ Rn represents n dimensional

function vector defined as
h iT
(2.24) F (x) = f1 (x) f2 (x) ... fn (x)
2.3. Optimization Based Formulations. Variety of modeling and de-

sign problems in chemical engineering are formulated as optimization problems.
Example 3. Consider a simple reaction
A→B
modelled using the following reaction rate equation

−E
(2.25) −ra = −dCa /dt = ko (Ca )n exp( )
RT
carried out in a batch reactor. It is desired to find the kinetic parameters ko , E
and n from the experimental data. The following data is collected from batch
experiments in a reactor at different temperatures
Reaction Rate Concentration Temperature

−ra1 Ca1 T1
−ra2 Ca2 T2
.... .... ....
−raN CaN TN
Substituting these values in the rate equation will give rise to N equations in
three unknowns, which is an overdetermined set equations. Due to experimental
errors in the measurements of temperature and reaction rate, it may not be
possible to find a set of values of {ko , E, n}such that the reaction rate equation is
satisfied at all the data points. However one can decide to select {V o, E, n}such
that the quantity
XN ∙ ¸2
n −E
(2.26) Φ= −rai − ko (Cai ) exp( )
i=1
RTi
Suppose we use −raie to denote the estimated reaction rate
n −E
(2.27) −raie = ko Cai exp( )
R ∗ Ti
then, the problem is to choose parameters {ko , E, n} such that the sum of the
square of errors between the measured and estimated rates is minimum, i.e.
Min X
N
(2.28) Φ(ko , E, n) = [−rai − (−raie )]2
ko , E, n i=1
Example 4. Cooling water is to be allocated to three distillation columns.

Up to 8 million liters per day are available, and any amount up to this limit
may be use. The costs of supplying water to each equipment are
Equip. 1: f1 = |1 − D1 | − 1 for 0 ≤ D1 ≤ 2
= 0 (otherwise)
Equip. 2: f2 = − exp( −1
2
(D2 − 5)2 ) for 0 ≤ D2 ≤ ∞
Equip. 2: f2 = D32 − 6D3 + 8 for 0 ≤ D3 ≤ 4
P
Minimize Φ = fi to find D1 , D2 , and D3
Note that this is an example of a typical multi-dimensional optimization

problem, which can be expressed in abstract form
Min
(2.29) Φ(x)
x
where x ∈ Rn and f (x) : Rn → R is a scalar objective function. A general
problem of this type may include constraints on x or functions of x.
2.4. ODE - Initial Value Problem (ODE-IVP). For most of the

processing systems of interest to the chemical engineer, there are three funda-
mental quantities :mass, energy and momentum. These quantities are can be
characterized by variables such as density, concentration, temperature, pressure
and flow rate. These characterizing variables are called as state of the processing
system. The equations that relate the state variables (dependent variables) to
the independent variables are derived from application of conservation principle
on the fundamental quantities and are called the state equations.
Let quantity S denote any one of the fundamental quantities
• Total mass
• Mass of the individual components
• Total energy.
• Momentum
Then, the principles of the conservation of the quantity S states that:
[Accumulation of S within a system] = [Flow of S in the system]
time period time period
- [Flow of S out of the system]
time period
Figure 5. General lumped parameter system
+ [Amount of S generated within the system]

time period
-[Amount of S consumed by the system]
time period
Figure 5 shows schematic diagram of a general system and its interaction
with external world. Typical dynamic model equations are as follows
Total Mass Balance
X X
d(ρV ) = ρi Fi − ρj Fj
i:inlet j:outlet
Mass Balance of the component A
dna d(Ca V ) X X
(2.30) = = Cai Fi − Caj Fi + or − rV
dt dt
Total energy Balance
dE d(U + K + P ) X X dH
(2.31) = = ρi Fi hi − ρj Fj hj ± Q ± WS '
dt dt dt
i : inlet j : outlet
where
ρ: density of the material in the system
ρi : density of the material in the i’th inlet stream

ρj : density of the material in the j’th outlet stream
V : Total volume of the system
Fi: Volumetric flow rate of the i’th inlet stream
Fj: Volumetric flow rate of the j’th outlet stream
na : number of moles of the component A in the system
CA : Molal concentration ( moles /volume)of A in the system
CAi : Molal concentration ( moles /volume)of A in the i’th inlet stream
CAj : Molal concentration ( moles /volume)of A in the j’th outlet stream
r: reaction rate per unit volume of the component A in the system.
hi: specific enthalpy of the material in the i’th inlet stream
hi: specific enthalpy of the material in the j’th outlet stream
U, K, P : internal, kinetic and potential energies of the system, respectively.
Q : Amount of the heat exchanged between the system and the sur-
roundings per unit time
WS : Shaft work exchanged between the system and its surroundings.
By convention, a quantity is considered positive if it flows in and negative
if it flows out. The state equations with the associated variables constitute the
’lumped parameter mathematical model’ of a process, which yields the dynamic
or static behavior of the process. The application of the conservation principle
stated above will yield a set of differential equations with the fundamental quan-
tities as the dependent variables and time as independent variable. The solution
of the differential equations will determine how the state variables change with
time i.e., it will determine the dynamic behavior of the process. The process is
said to be at the steady state if the state variables do not change with time. In
this case, the rate of accumulation of the fundamental quantity S is zero and
the resulting balance yields a set of algebraic equations
Example 5. Stirred Tank Heater (STH) System (Figure 6): Total

momentum of the system remains constant and will not be considered. Total
mass balance: Total mass in the tank at any time t = ρV = ρAh where A
represents cross sectional area.
d (ρAh)
(2.32) = ρFi − ρF
dt
Assuming that the density is independent of the temperature,
dh
(2.33) A = Fi − F
dt
Hot Water Flow Cold Water Flow
CV-3 CV-1
FC FT
LT LC
Flow Setpoint
Valve
Position Level
CV-2 Setpoint
Steam Flow
TT
Figure 6. Stitted Tank Heater (STH) System
Now, flow out due to the gravity is also a function of height

√
F =k h
Thus,
dh √
(2.34) A+ k h = Fi
dt
Total energy of liquid in the tank is given by
E =U +k+P
However, since tank does not move

dk dP dE dU
= =0; =
dt dt dt dt
For liquid systems
dU dH
(2.35) ≈
dt dt
where H is total enthalpy of the liquid in the tank.
(2.36) H = ρV Cp (T − Tref ) = ρAhCp (T − Tref )
Tref represents reference temperature where the specific enthalpy of the liquid is
assumed to be zero. Now, using the energy conservation principle
d (ρAhCp (T − Tref ))
(2.37) = ρFi Cp (Ti − Tref ) − ρF Cp (T − Tref ) + Q
dt
where Q is the amount of heat supplied by the steam per unit time. Assuming
Tref = 0, we have
d(hT ) Q
(2.38) A = Fi Ti − F T +
dt ρCp
d(hT ) dT dh
A = Ah + AT
dt dt dt
dT
= Ah + T (Fi − F )
dt
Q
= Fi Ti − F T +
ρCp
Or
dT Q
Ah = Fi (Ti − T ) +
dt ρCp
Summarizing modelling steps
dh 1 1 √
(2.39) = (Fi − F ) = (Fi − k h)
dt A A
dT Fi Q
(2.40) = (Ti − T ) +
dt Ah AhρCp
The associated variables can be classified as
• state(or dependent) variables : h, T
• Input (or independent) variables :Ti , Fi , Q
• Parameters: A, ρ, Cp
Steady state behavior can be computed by solving following two equations
dh √
(2.41) = Fi − k h = 0
dt
dT Fi Q
(2.42) = (Ti − T ) + =0
dt Ah ρCp
Once we choose independent variables Fi = F i , Ti = T i and Q = Q, the steady
state h = h and T = T can be computed by simultaneously solving nonlinear
algebraic equations (2.41-2.42).
The system will be disturbed from the steady state if the input variables
suddenly change value at t = 0. Consider following two situations in which we
need to investigate transient behavior of the above process
• Ti decreases by 10% from its steady state value T i at t = 0. Liquid level
remains at the same steady state value as Ti does not influence the total
mass in tank. The temperature T in the tank will start decreasing with
time (see Figure 7). How T (t) changes with time is determined by the
Figure 7. System response to step change in Ti
Figure 8. System reponse for step change in Fi
solution of the equation (2.39) using the initial as condition T (0) = T ,

the steady state value of T .
• Fi is decreased by 10% from its steady state value F i : Since Fi appears

in both the dynamic equations, the temperature and the liquid level will
start changing simultaneously and the dynamics will be governed by
simultaneous solution of coupled nonlinear differential equations (2.39-
2.40) starting with initial conditions T (0) = T , h(0) = h.
Figures 8 show schematic diagrams of the process responses for step change
in Fi .
It is also possible to investigate response of the system for more complex
inputs, such as
Ti (t) = T i + ∆Ti sin(ωt)
where above function captures daily variation of cooling water inlet tempera-
ture. In each case, the transient behavior T (t) and h(t) is computed by solving
the system of ODEs subject to given initial conditions and time variation of
independent inputs (i.e. forcing functions).
The model we considered above did not contain variation of the variables
with respect to space. Such models are called as ’Lumped parameter models’
and are described by ordinary differential equations of the form
dx1
(2.43) = f1 [x1 (t), x2 (t), ..., xn (t), u1 (t), .., um (t)]
dt
............................................
dxn
(2.44) = fn [x1 (t), x2 (t), ..., xn (t), u1 (t), .., um (t)]
dt
x1 (0) = x1 , ....xn (0) = xn (Initial conditions)
where {xi (t)} denote the state (or dependent) variables and {ui (t)} denote inde-
pendent inputs (or forcing functions) specified for t ≥ 0. Using vector notation,
we can write the above set of ODEs in more compact form
dx
(2.45) = F (x, u)
dt
(2.46) x(0) = x0
where
(2.47) x(t) = [x1 (t)......xn (t)]T ∈ Rn

(2.48) u(t) = [u1 (t)......un (t)]T ∈ Rm
(2.49) F (x, u) = [f1 (x, u)........fn (x, u)]T ∈ Rn
and u(t) is a forcing function vector defined over t ≥ 0.

• Steady State Simulation Problem: If we fix independent inputs to some
constant value, say u(t) = u for t ≥= 0,then we can find a steady state
solution x = x corresponding to these constant inputs by simultane-
ously solving n nonlinear algebraic equations
(2.50) F (x, u) = 0
obtained by setting dx/dt = 0 where 0 represents n × 1 zero vector.

• Dynamic Simulation Problem: Given input trajectories
(2.51) u(t) = [u1 (t) u2 (t)......um (t)]T
as a function of time for t ≥= 0 and with the initial state x(0), integrate
dx
(2.52) = F (x, u(t))
dt
over interval 0 ≤ t ≤ tf to determine state trajectories
(2.53) x(t) = [x1 (t) x2 (t)..........xn (t)]T

Figure 9. Shell and tube heat exchanger
Since u(t) is a known function of time, we re-state the above problem

as
dx
(2.54) = Fu (x, t) ; x(0) = x0
dt
Fu (x, t) (= F (x, u(t))) denotes F () with the given u(t).
2.5. PDEs and ODE-Boundary value Problems. Most of the systems

encountered in chemical engineering are distributed parameter systems. Even
though behavior of some of these systems can be adequately represented by
lumped parameter models, such simplifying assumptions may fail to provide
accurate picture of system behavior in many situations and variations of vari-
ables along time and space have to be considered while modeling. This typically
result in a set of partial differential equations.
Example 6. Consider the double pipe heat exchanger in which a liquid flow-
ing in the inner tube is heated by steam flowing countercurrently around the tube
(Figure 10). The temperature in the pipe changes not only with time but also
along the axial direction z. While developing the model, it is assumed that the
temperature does not change along the radius of the pipe. Consequently , we have
only two independent variables, i.e. z and t. To perform the energy balance,we
consider an element of length ∆z as shown in the figure. For this element, over
a period of time ∆t
(2.55) ρCp A∆z[(T )t+Λt − (T )t ] = ρCp V A(T )z ∆t − ρCp V A(T )z+∆z ∆t
(2.56) +Q∆t(πD∆z)
This equation can be explained as
[accumulation of the enthalpy during the time period ∆t]
= [flow in of the enthalpy during ∆t] - [flow out of the enthalpy during ∆t]
[enthalpy transferred from steam to the liquid through wall during ∆t]
where
Q: amount of heat transferred from the steam to the liquid per unit time
and per unit heat transfer area.
A: cross section area of the inner tube.
V : average velocity of the liquid(assumed constant).
D: external diameter of the inner tube.
Dividing both the sides by (∆z∆t) and taking limit as ∆t → 0 and ∆z → 0,
we have
∂T (z, t) ∂T (z, t)
(2.57) ρCp A = −ρCp V A + πDQ
∂t ∂z
(2.58) Q = U[Tst − T ]
Boundary conditions:
T (t, z = 0) = T1 fort ≥ 0
Initial condition
(2.59) T (t = 0, z) = T0 (0, z)
Steady State Simulation: Find T (z) given T (z = 0) = T1 when ∂T /∂t = 0,

i.e. solve for
∂T
(2.60) ρCp V A = πDQ = πDQU (Tst − T )
∂z
(2.61) T (0) = T1
This results in a ODE-IVP, which can be solved to obtain steady state profiles
T (z) for specified heat load and liquid velocity.
Dynamic Simulation
∂T ∂T
(2.62) ρCp A = −ρCp V A + πDQ
∂t ∂z
with
(2.63) T (t, 0) = T1 at z = 0 and t º 0 : Boundary condition

(2.64) T (0, z) = T0 (z) : Initial temperature profile
This results in a Partial Differential Equation (PDE) model for the distributed
parameter system.
Example 7. Now, let us consider the situation where the some hot liquid
is used on the shell side to heat the tube side fluid (see Figure 10). The model
Figure 10. Double Pipe Heat Exchanger
equations for this case can be stated as

∂Tt (z, t) ∂Tt (z, t)
(2.65) ρt Cpt At = −ρt Cpt Vt At + πDQ(z, t)
∂t ∂z
∂Ts (z, t) ∂Ts (z, t)
(2.66) ρs Cpt As = ρs Cps Vs As − πDQ(z, t)
∂t ∂z
(2.67) Q(z, t) = U[Ts (z, t) − Tt (z, t)]
where subscript t denotes tube side and subscript s denotes shell side. The initial
and boundary conditions become
(2.68) Tt (t, 0) = Tt0 at z = 0 and t º 0 : Boundary condition

(2.69) T (0, z) = Tt0 (z) : Initial temperature profile
(2.70) Ts (t, 1) = Ts1 at z = 1 and t º 0 : Boundary condition

(2.71) T (0, z) = Ts0 (z) : Initial temperature profile
These are coupled PDEs and have to be solved simultaneously to understand the
transient behavior. The steady state problem can be stated as
dTt (z, t)
(2.72) ρt Cpt Vt At = πDU[Ts (z) − Tt (z)]
dz
dTs (z, t)
(2.73) ρs Cps Vs As = πDU[Ts (z) − Tt (z)]
dz
(2.74) Tt (0) = Tt0 at z = 0
(2.75) Ts (1) = Ts1 at z = 1
Equations (2.72-2.73) represent coupled ordinary differential equations. The

need to compute steady state profiles for the counter-current double pipe heat
exchanger results in a boundary value problem (ODE-BVP) as one variable is
specified at z = 0 while the other is specified at at z = 1.
4. APPENDIX: BASIC CONCEPTS OF PARTIAL DIFFERENTIAL EQUATIONS 19
Typical partial differential equations we come across in engineering applica-

tions are of the form
∂u ∂2u
(2.76) ∇2 u = a + b 2 + cu + f (x1 , x2 , x3 , t)
∂t ∂t
subject to appropriate boundary conditions and initial conditions. This PDE is
solved in a three dimensional region V , which can be bounded or unbounded.
The boundary of V is denoted by S. On the spatial surface S, we have boundary
conditions of the form
(2.77) b) .∇u + β(s, t)u = h(s, t)

(α(s, t) n
b is the outward normal direction to S and s represents spatial coordinate

where n
along S. We can classify the PDEs as follows
• Elliptic: a = b = 0
• Parabolic: a 6= 0, b = 0
• Hyperbolic: b > 0
3. Summary
These lecture notes introduces various basic forms of equations that
appear in steady state and dynamic models of simple unit operations. Following
generic forms or problem formulations have been identified
• Linear algebraic equations
• Nonlinear algebraic equations
• Unconstrained optimization
• Ordinary Differential Equations : Initial Value Problem (ODE-IVP)
• Ordinary Differential Equations : Boundary Value Problem (ODE-
BVP)
• Partial Differential Equations (PDEs)
Methods for dealing with numerical solutions of these generic forms / for-
mulations will be discussed in the later parts.
4. Appendix: Basic Concepts of Partial Differential Equations

Definition 1. Order of PDE: Order of a PDE is highest order of deriv-
ative occurring in PDE.
Definition 2. Degree of PDE: Power to which highest order derivative

is raised.
Example 8. Consider PDE

(4.1) ∂u/∂t + (d2 u/dz 2 )n = u3
Here the Oredr = 2 and Degree = n.

Solutions of PDEs are sought such that it is satisfied in the domain and
on the boundary conditions are satisfied. A problem is said to be well
posed when the solution uniquely determined and it is sufficient smooth and
differentiable function of the independent variables. The boundary conditions
have to be consistent with one another in order for a problem to be well posed.
This implies that at the points common to boundaries, the conditions should
not violet each other.
A linear PDE can be classified as:
• Homogeneous equations:Differential equation that does not contain any
terms other than dependent variables and their derivatives.
(4.2) ∂u/∂t = ∂u2 /∂x2
∂ 2 u/∂x2 + ∂ 2 u/∂y 2 = 0
• Non homogeneous equations: Contain terms other than dependent vari-
ables
(4.3) ∂u/∂t = ∂ 2 u/∂x2 + sin x
(4.4) ∂ 2 u/∂x2 + ∂ 2 u/∂y 2 = sin x sin y

The boundary conditions can be similarly homogeneous or non homogeneous
depending on whether they contain terms independent of dependent variables.
The PDEs typically encountered in engineering applications are 2nd order

PDEs (reaction-diffusion systems, heat transfer, fluid-flow etc.)
Classification of 2nd order PDEs:
Consider a 2nd order PDE in n independent variables (x1 , x2 , x3 , x4 ) = (x, y, z, t).

This can be written as
X4 X 4
∂2u
(4.5) aij = f [∂u/∂x1 , ......∂u/∂x4 , , u, x1 , ........., x4 ]
i=1 j=1
∂xi ∂xj
4. APPENDIX: BASIC CONCEPTS OF PARTIAL DIFFERENTIAL EQUATIONS 21
aij are assumed to be independent of 0 u0 and its derivative. They can be functions
of (xi ). aij can always be written as aij = aji for i 6= j as
∂2u ∂2u
(4.6) =
∂xi ∂xj ∂xj ∂xi
Thus, aij are elements of a real symmetric matrix A. Obviously A has real eigen
values. The PDE is called
• Elliptic: if all eigenvalues are +ve or-ve.
• Hyperbolic: if some eigenvalues are +ve and rest are -ve.
• Parabolic: if at-least one eigen value is zero.
The classification is global if aij are independent of xi , else it is local. El-

liptic Problems typically arise while studying steady-state behavior of diffusive
systems. Parabolic or hyperbolic problems typically arise when studying tran-
sient behavior of diffusive systems.
CHAPTER 2
Fundamentals of Functional Analysis
1. Introduction
When we begin to use concept of vectors in formulating mathematical
models for engineering systems, we tend to associate the concept of a vector
space with the three dimensional coordinate space. The three dimensional space
we are familiar with can be looked upon as a set of objects called vectors, which
satisfy certain generic properties. While working with mathematical modeling,
however, we need to deal with variety of such sets containing such objects.
It is possible to ’distill’ essential properties satisfied by vectors in the three
dimensional vector space and develop a more general concept of a vector space,
which is collection of objects satisfying these properties. Such a generalization
can provide a unified view of problem formulations and the solution techniques.
Generalization of the concept of vector and vector space to any general set
other than collection of vectors in three dimensions is not sufficient. In order
to work with these sets of generalized vectors, we need various algebraic and
geometric structures on these sets, such as norm of a vector, angle between two
vectors or convergence of a sequence of vectors. To understand why these struc-
tures are necessary, consider the fundamental equation that arises in numerical
analysis
(1.1) F (x) = 0
where x is a vector and F (.) represents some linear or nonlinear operator, which
when operates on x yields the zero vector 0. In order to generate a numerical
approximation to the solution of equation (6.2), this equation is further trans-
formed to formulate an iteration sequence as follows
£ ¤
(1.2) x(k+1) = G x(k) ; k = 0, 1, 2, ......
© ª
where x(k) : k = 0, 1, 2, ...... is sequence of vectors in vector space under con-
sideration. The iteration equation is formulated in such a way that the solution
x∗ of equation (1.2), i.e.
x∗ = G [x∗ ]
23
24 2. FUNDAMENTALS OF FUNCTIONAL ANALYSIS
is also a solution of equation (6.2). Here we consider two well known examples
of such formulation.
Example 9. Iterative schemes for solving single variable nonlinear

algebraic equation: Consider one variable nonlinear equation
f (x) = x3 + 7x2 + x sin(x) − ex = 0
This equation can be rearranged to formulate an iteration sequence as

(k)
£ (k) ¤3 £ (k) ¤2
exp(x ) − x − 7 x
(1.3) x(k+1) = (k)
sin(x )
Alternatively, using Newton-Raphson method for single variable nonlinear equa-
tions, iteration sequence can be formulated as
f (x(k) )
x(k+1) = x(k) −
[df (x(k) )/dx]
£ (k) ¤3 £ ¤2
x + 7 x(k) + x(k) sin(x(k) ) − exp(x(k) )
(1.4) = x(k) − 2
3 [x(k) ] + 14x(k) + sin(x(k) ) + x(k) cos(x(k) ) − exp(x(k) )
Both the equations (1.3) and (1.4) are of the form given by equation (1.2).
Example 10. Solving coupled nonlinear algebraic equations by suc-

cessive substitution
Consider three coupled nonlinear algebraic equations
(1.5) ⎡ ⎤ ⎡ ⎤
f1 (x, y, z, w) xy − xz − 2w ⎡ ⎤
⎢ f (x, y, z, w) ⎥ ⎢ y 2 x + zwy − 1 ⎥ 0
⎢ 2 ⎥ ⎢ ⎥ ⎢ ⎥
F (x, y, z, w) = ⎢ ⎥=⎢ ⎥=⎣ 0 ⎦
⎣ f3 (x, y, z, w) ⎦ ⎣ z sin(y) − z sin(x) − ywx ⎦
0
f4 (x, y, z, w) wyz − cos(x)
which have to be solved simultaneously.. One possible way to transform the
above set of nonlinear algebraic equations to the form given by equation (1.2) is
as follows as
(1.6) ⎡ (k+1) ⎤ ⎡ ⎤
x 2w(k) /(y (k) − z (k) )
⎢ y (k+1) ⎥ ⎢ 1/ ¡y (k) x(k) + z (k) w(k) ¢ ⎥ £ ¤
(k+1) ⎢ ⎥ ⎢ ⎥
x ≡ ⎢ (k+1) ⎥ = ⎢ (k) (k) (k) ¡ (k) (k)
¢ ⎥ ≡ G x(k)
⎣ z ⎦ ⎣ y x w / sin(y ) − sin(x ) ⎦
(k+1)
¡ ¢
w cos(x(k) )/ y (k) z (k)
Example 11. Picard’s Iteration: Consider single variable ODE-IVP
dx
(1.7) F [x(z)] = − f (x, z) = 0 ; x(0) = x0 ; 0 ≤ z ≤ 1
dz
1. INTRODUCTION 25
which can be solved by formulating the Picard’s iteration scheme as follows
Zz
£ ¤
(1.8) x(k+1) (z) = x0 + f x(k) (q), q dq
0
Note that this procedure generates a sequence of functions x(k+1) (z) over the
interval 0 ≤ z ≤ 1 starting from an initial guess solution x(0) (z). If we can
treat a function x(k) (z) over the interval 0 ≤ z ≤ 1 as a vector, then it is easy
to see that the equation (1.8) is of the form given by equation (1.2).
Fundamental concern in such formulations is, whether the sequence of vec-

tors
© (k) ª
x : k = 0, 1, 2, ...
converges to the solution x∗ . Another important question that naturally arises

while numerically computing the sequences of vectors in all the above examples
is that when do we terminate these iteration sequences. In order to answer such
questions while working with general vector spaces, we have to define concepts
such as norm of a vector in such spaces and use it to generalize the notion of
convergence of a sequence.
A branch of mathematics called functional analysis deals with generalization
of geometric concepts, such as length of a vector, convergence, orthogonality etc.
used in the three dimensional vector spaces to more general finite and infinite
dimensional vector spaces. Understanding fundamentals of functional analysis
helps understand basis of various seemingly different numerical methods. In this
part of the lecture notes we introduce some concepts from functional analysis
and linear algebra, which help develop a unified view of all the numerical meth-
ods. In the next section, we review fundamentals of functional analysis, which
will be necessary for developing a good understanding of numerical analysis.
Detailed treatment of these topics can be found in Luenberger [11] and Kreyzig
[7].
A word of advice before we begin to study these grand generalizations. While
dealing with the generic notion of vector spaces, it is difficult to visualize shapes
as we can do in the three dimensional vector space. However, if you know the
concepts from three dimensional space well, then you can develop an under-
standing of the corresponding concept in any general space. It is enough to
know the Euclidean geometry well in the three dimensions.
2. Vector Spaces
Associated with every vector space is a set of scalars F (also called as
scalar field or coefficient field) used to define scalar multiplication on the space.
In functional analysis, the scalars will be always taken to be the set of real
numbers (R) or complex numbers (C).
Definition 3. (Vector Space): A vector space X is a set of elements

called vectors and scalar field F together with two operations . The first opera-
tion is called addition which associates with any two vectors x, y ∈ X a vector
x + y ∈ X , the sum of x and y. The second operation is called scalar multipli-
cation, which associates with any vector x ∈ X and any scalar α a vector αx (a
scalar multiple of x by α). The set X and the operations of addition and scalar
multiplication are assumed to satisfy the fallowing axioms for any x, y, z ∈ X ·
(1) x + y = y + x (commutative law)

(2) x + (y + z) = (x + y) + z (associative law)
(3) There exists a null vector 0 such that x + 0 = x for all x ∈ X
(4) α(x + y) = αx + αy
(5) (α + β) x = αx + βx (4,5 are distributive laws)
(6) αβ (x) = α (βx)
(7) αx = 0 when α = 0.
αx = x when α = 1.
(8) For convenience −1x is defined as −x and called as negative of a vector
we have
x + (−x) = 0
In short, given any vectors x, y ∈ X and any scalars α, β ∈ R,we can write
that αx + βy ∈ X when X is a linear vector space.
Example 12. (X ≡ Rn , F ≡ R) : n− dimensional real coordinate space. A

typical element x ∈ X can be expressed as
h iT
(2.1) x = x1 x2 ..... xn
where xi denotes the i’th element of the vector.
Example 13. (X ≡ C n , F ≡ C) : n− dimensional complex coordinate

space.
Example 14. (X ≡ Rn , F ≡ C) : It may be noted that this combination of

set X and scalar field F does not form a vector space. For any x ∈ X and any
α ∈ C the vector αx ∈X.
/
2. VECTOR SPACES 27
Example 15. (X ≡ l∞ , F ≡ R) : Set of all infinite sequence of real numbers.

A typical vector x of this space has form x = (ζ 1 , ζ 2 , ........., ζ k , ........).
Example 16. (X ≡ C[a, b], F ≡ R) : Set of all continuous functions over

an interval [a, b] forms a vector space. We write x = y if x(t) = y(t) for all
t ∈ [a, b] The null vector 0 in this space is a function which is zero every where
on [a, b] ,i.e
f (t) = 0 for all t ∈ [a, b]
If x and y are vectors from this space and α is real scalar then (x + y)(t) =
x(t) + y(t) and (αx)(t) = αx(t) are also elements of C[a, b].
¡ ¢
Example 17. X ≡ C (n) [a, b], F ≡ R : Set of all continuous and n times
differentiable functions over an interval [a, b] forms a vector space.
Example 18. X ≡ set of all real valued polynomial functions defined on

interval [a, b] together with F ≡ R forms a vector space.
Example 19. The set of all functions f (t) for which

Zb
|f (t)|p dt < ∞
a
is a linear space Lp .
Example 20. (X ≡ Rm × Rn , F ≡ R) : Set of all m × n matrices with real

elements. Note that a vector in this space is a m × n matrix and the null vector
corresponds to m × n null matrix. It is easy to see that, if A, B ∈ X, then
αA + βB ∈ X and X is a linear vector space.
Definition 4. (Subspace): A non-empty subset M of a vector space X

is called subspace of X if every vector αx + βy is in M wherever x and y are
both in M. Every subspace always contains the null vector, I.e. the origin of the
space x
Example 21. Subspaces

(1) Two dimensional plane passing through origin of three dimensional co-
ordinate space. (Note that a plane which does not pass through the
origin is not a sub-space.)
(2) A line passing through origin of Rn
(3) The set of all real valued n’th order polynomial functions defined on
interval [a, b] is a subspace of C[a, b].
Thus, the fundamental property of objects (elements) in a vector space is

that they can be constructed by simply adding other elements in the space.
This property is formally defined as follows.
Definition 5. (Linear Combination): A linear combination of vectors
x , x(2)
(1)
, , .......x
(m)
in a vector space is of the form α1 x(1) + α2 x(2) + .............. +
αm x(m) where (α1 , ...αm ) are scalars.
Note that we are dealing with set of vectors
© (k) ª
(2.2) x : k = 1, 2, ......m.
. Thus, if X = Rn and x(k) ∈ Rn represents k’th vector in the set, then it is a
vector with n components i.e.
h iT
(2.3) x(k) = x1(k) x2(k) .... xn(k)
Similarly, if X = l∞ and x(k) ∈ l∞ represents k’th vector in the set, then x(k)
represents a sequence of infinite components
h iT
(k) (k) (k)
(2.4) x = x1 .... xi ......
Definition 6. (Span of Set of Vectors): Let S be a subset of vector
space X. The set generated by all possible linear combinations of elements of S
is called as span of S and denoted as [S]. Span of S is a subspace of X.
Definition 7. (Linear Dependence): A vector x is said to be linearly
dependent up on a set S of vectors if x can be expressed as a linear combination
of vectors from S. Alternatively, x is linearly dependent upon S if x belongs to
span of S, i.e. x ∈ [S]. A vector is said to be linearly independent of set S, if
it not linearly dependent on S . A necessary and sufficient condition for the set
of vectors x(1) , x(2) , .....x(m) to be linearly independent is that expression
X m
(2.5) αi x(i) = 0
i=1
implies that αi = 0 for all i = 1, 2......m.

Definition 8. (Basis): A finite set S of linearly independent vectors is
said to be basis for space X if S generates X i.e. X = [S]
A vector space having finite basis (spanned by set of vectors with finite
number of elements) is said to be finite dimensional. All other vector spaces
are said to be infinite dimensional. We characterize a finite dimensional space
by number of elements in a basis. Any two basis for a finite dimensional vector
space contain the same number of elements.
3. NORMED LINEAR SPACES AND BANACH SPACES 29
Example 22. Basis of a vector space

h iT
(1) Let S = {v} where v = 1 2 3 4 5 and let us define span of
S as [S] = αv where α ∈ R represents a scalar. Here, [S] is one
dimensional vector space and subspace of R5
© ª
(2) Let S = v(1) , v(2) where
⎡ ⎤ ⎡ ⎤
1 5
⎢ 2 ⎥ ⎢ 4 ⎥
⎢ ⎥ ⎢ ⎥
⎢ ⎥ ⎢ ⎥
(2.6) v(1) = ⎢ 3 ⎥ ; v(2) = ⎢ 3 ⎥
⎢ ⎥ ⎢ ⎥
⎣ 4 ⎦ ⎣ 2 ⎦
5 1
Here span of S (i.e. [S]) is two dimensional subspace of R5 .
(3) Consider set of nth order polynomials on interval [0, 1]. A possible basis
for this space is
(2.7) p(1) (z) = 1; p(2) (z) = z; p(3) (z) = z 2 , ...., p(n+1) (z) = z n
Any vector p(t) from this pace can be expressed as
(2.8) p(z) = α0 p(1) (z) + α1 p(2) (z) + ......... + αn p(n+1) (z)

= α0 + α1 z + .......... + αn z n
Note that [S] in this case is (n + 1) dimensional subspace of C[a, b].
(4) Consider set of continuous functions over interval, i.e. C[−π, π]. A
well known basis for this space is
(2.9) x(0) (z) = 1; x(1) (z) = cos(z); x(2) (z) = sin(z),
(2.10) x(3) (z) = cos(2z), x(4) (z) = sin(2z), ........
It can be shown that C[−π, π] is an infinite dimensional vector space.
3. Normed Linear Spaces and Banach Spaces

In three dimensional space, we use lengths to compare any two vectors.
Generalization of the concept of length of a vector in three dimensional vector
space to an arbitrary vector space is achieved by defining a scalar valued function
called norm of a vector.
Definition 9. (Normed Linear Vector Space): A normed linear vector

space is a vector space X on which there is defined a real valued function which
maps each element x ∈ X into a real number kxkcalled norm of x. The norm
satisfies the fallowing axioms.
(1) kxk ≥ 0 for all x ∈ X ; kxk = 0 if and only if x =0 (zero vector)

(2) kx + yk ≤ kxk + kykfor each x, y ∈ X. (triangle inequality).
(3) kαxk = |α| . kxk for all scalars α and each x ∈ X
The above definition of norm is an abstraction of usual concept of length of
a vector in three dimensional vector space.
Example 23. Vector norms:

P
N
(1) (Rn , k.k1 ) :Euclidean space Rn with 1-norm: kxk1 = |xi |
i=1
(2) (Rn , k.k2 ) :Euclidean space Rn with 2-norm:
"N # 12
X
kxk2 = (xi )2
i=1
(Rn , k.k∞´) :Euclidean space Rn with ∞−norm: kxk∞ = max |xi |

(3) ³
(4) Rn , k.kp :Euclidean space Rn with p-norm,
"N # p1
X
(3.1) kxkp = |xi |p
i=1
, where p is a positive integer

(5) n-dimensional complex space (C n ) with p-norm,
"N # p1
X
(3.2) kxkp = |xi |p
i=1
, where p is a positive integer

(6) Space of infinite sequences (l∞ )with p-norm: An element in this space,
say x ∈ l∞ , is an infinite sequence of numbers
(3.3) x = {x1, x2 , ........, xn , ........}
such that p-norm is bounded

"∞ # p1
X
(3.4) kxkp = |xi |p < ∞
i=1
for every x ∈ l∞ , where p is an integer.
Example 24. (C[a, b], kx(t)k∞ ) : The normed linear space C[a, b] together
with infinite norm
max
(3.5) kx(t)k∞ = |x(t)|
a≤t≤b
3. NORMED LINEAR SPACES AND BANACH SPACES 31
It is easy to see that kx(t)k∞ defined above qualifies to be a norm

(3.6) max |x(t) + y(t)| ≤ max[|x(t)| + |y(t)|] ≤ max |x(t)| + max |y(t)|
(3.7) max |αx(t)| = max |α| |x(t)| = |α| max |x(t)|

Other types of norms, which can be defined on the set of continuous functions
over [a, b] are as follows
Zb
(3.8) kx(t)k1 = |x(t)| dt
a
⎡b ⎤ 12
Z
(3.9) kx(t)k2 = ⎣ |x(t)|2 dt⎦
a
Once we have defined norm of a vector in a vector space, we can proceed

to generalize the concept of convergence of sequence of vectors. Concept of
convergence is central to all iterative numerical methods.
Definition 10. (Convergence): In a normed linear space an infinite se-

© ª
quence of vectors x(k) : k = 1, 2, ....... is said to converge to a vector x∗ if the
©° ° ª
sequence °x∗ − x(k) ° , k = 1, 2, ... of real numbers converges to zero. In this
case we write x(k) → x∗ .
© ª
In particular, a sequence x(k) in Rn converges if and only if each component
of the vector sequence converges. If a sequence converges, then its limit is
unique.
© ª
Definition 11. (Cauchy sequence): A sequence x(k) in normed linear
° °
space is said to be a Cauchy sequence if °x(n) − x(m) ° → 0 as n, m → ∞.i.e.
° °
given an ε > 0 there exists an integer N such that °x(n) − x(m) ° < ε for all
n, m ≥ N
Example 25. Convergent sequences: Consider the sequence of vectors

represented as
⎡ ⎤ ⎡ ⎤
1 + (0.2)k 1
⎢ −1 + (0.9)k ⎥ ⎢ −1 ⎥
⎢ ⎥ ⎢ ⎥
(3.10) x(k) = ⎢ ¡ ¢ ⎥→⎢ ⎥
⎣ 3/ 1 + (−0.5)k ⎦ ⎣ 3 ⎦
(0.8)k 0
with respect to any p-norm defined on R4 . It can be shown that it is a Cauchy
sequence. Note that each element of the vector converges to a limit in this case.
When we are working in Rn or C n ,all convergent sequences are Cauchy

sequences and vice versa. However, all Cauchy sequences in a general vector
space need not be convergent. Cauchy sequences in some vector spaces exhibit
such strange behavior and this motivates the concept of completeness of a vector
space.
Definition 12. (Banach Space): A normed linear space X is said to be

complete if every Cauchy sequence has a limit in X. A complete normed linear
space is called Banach space.
Examples of Banach spaces are

(Rn , k.k1 ) , (Rn , k.k2 ) , (Rn , k.k∞ )
(Cn , k.k1 ) , (Cn , k.k2 ) , (l∞ , k.k1 ) , (l∞ , k.k2 )
etc. Concept of Banach spaces can be better understood if we consider an
example of a vector space where a Cauchy sequence is not convergent, i.e. the
space under consideration is an incomplete normed linear space. Note that,
even if we find one Cauchy sequence in this space which does not converge, it
is sufficient to prove that the space is not complete.
Example 26. Let X = (Q, k.k1 ) i.e. set of rational numbers (Q) with scalar
field also as the set of rational numbers (Q) and norm defined as
(3.11) kxk1 = |x|
A vector in this space is a rational number. In this space, we can construct
Cauchy sequences which do not converge to a rational numbers (or rather they
converge to irrational numbers). For example, the well known Cauchy sequence
x(1) = 1/1
x(2) = 1/1 + 1/(2!)
.........
x(n) = 1/1 + 1/(2!) + ..... + 1/(n!)
converges to e, which is an irrational number. Similarly, consider sequence
x(n+1) = 4 − (1/x(n) )
Starting from initial point x(0) = 1, we can generate the sequence of rational
numbers
3/1, 11/3, 41/11, ....
√
which converges to 2 + 3 as n → ∞.Thus, limits of the above sequences is
outside the space X and the space is incomplete.
4. INNER PRODUCT SPACES AND HILBERT SPACES 33
Example 27. Consider sequence of functions in the space of twice differen-

tiable continuous functions C (2) (−∞, ∞)
1 1
f (k) (t) = + tan−1 (kt)
2 π
defined in interval −∞ < t < ∞, for all integers k. The range of the func-
tion is (0,1). As k → ∞, the sequence of continuous function converges to a
discontinuous function
u(∗) (t) = 0 −∞<t<0

= 1 0<t<∞
Example 28. Let X = (C[0, 1], k.k1 ) i.e. space of continuous function on
[0, 1] with one norm defined on it i.e.
Z1
(3.12) kx(t)k1 = |x(t)| dt
0
and let us define a sequence [11]

⎧ ⎫
⎪
⎨ 0 (0 ≤ t ≤ ( 12 − n1 ) ⎪
⎬
(n)
(3.13) x (t) = n(t − 12 ) + 1 ( 12 − n1 ) ≤ t ≤ 12 )
⎪
⎩ ⎪
⎭
1 (t ≥ 12 )
Each member is a continuous function and the sequence is Cauchy as

¯ ¯
° (n) ° ¯ ¯
(3.14) °x − x(m) ° = 1 ¯ 1 − 1 ¯ → 0
2 n m¯
¯
However, as can be observed from Figure1, the sequence does not converge to a
continuous function.
The concepts of convergence, Cauchy sequences and completeness of space

assume importance in the analysis of iterative numerical techniques. Any iter-
ative numerical method generates a sequence of vectors and we have to assess
whether the sequence is Cauchy to terminate the iterations.
4. Inner Product Spaces and Hilbert Spaces

Concept of norm explained in the last section generalizes notion of length of
a vector in three dimensional Euclidean space. Another important concept in
three dimensional space is angle between any two vectors. Given any two unit
Figure 1. Sequence of continuous functions
vectors in R3 , say x
b and yb,the angle between these two vectors is defined using
inner (or dot) product of two vectors as
µ ¶T
T x y
(4.1) cos(θ) = (b x) y b=
kxk2 kyk2
(4.2) = x b1 yb1 + x
b2 yb2 + x
b3 yb3
The fact that cosine of angle between any two unit vectors is always less than
one can be stated as
(4.3) |cos(θ)| = |hb bi| ≤ 1
x, y
Moreover, vectors x and y are called orthogonal if (x)T y = 0. Orthogonality
is probably the most useful concept while working in three dimensional Euclid-
ean space. Inner product spaces and Hilbert spaces generalize these simple
geometrical concepts in three dimensional Euclidean space to higher or infinite
dimensional spaces.
Definition 13. (Inner Product Space): An inner product space is a

linear vector space X together with an inner product defined on X × X. Corre-
sponding to each pair of vectors x, y ∈ X the inner product hx, yi of x and y is
a scalar. The inner product satisfies following axioms.
(1) hx, yi = hy, xi (complex conjugate)

(2) hx + y, zi = hx, zi + hy, zi
(3) hλx, yi = λ hx, yi

hx, λyi = λ hx, yi
(4) hx, xi ≥ 0 and hx, xi = 0 if and only if x=0
Axioms 2 and 3 imply that the inner product is linear in first entry. The
1
quantity hx, xi 2 is a candidate function for defining norm on the inner product
space. Axioms 1 and 3 imply that kαxk = |α| kxk and axiom 4 implies that
p
kxk > 0 for x 6= 0. If we show that hx, xisatisfies triangle inequality, then
p
hx, xi defines a norm on space X . We first prove Cauchy-Schwarz inequality,
p
which is generalization of equation (4.3), and proceed to show that hx, xi
p
defines the well known 2-norm on X, i.e. kxk2 = hx, xi.
Lemma 1. (Cauchey- Schwarz Inequality): Let X denote an inner prod-

uct space. For all x, y ∈ X ,the following inequality holds
(4.4) |hx, yi| ≤ [hx, xi]1/2 [hy, yi]1/2
The equality holds if and only if x = λy or y = 0
Proof: If y = 0, the equality holds trivially so we assume y 6= 0. Then, for

all scalars λ,we have
(4.5) 0 ≤ hx − λy, x − λyi = hx, xi − λ hx, yi − λ hy, xi + |λ|2 hy, yi

hy, xi
In particular, if we choose λ = , then, using axiom 1 in the definition of
hy, yi
inner product, we have
hy, xi hx, yi
(4.6) λ= =
hy, yi hy, yi
2 hx, yi hy, xi
(4.7) ⇒ −λ hx, yi − λ hy, xi = −
hy, yi
2 hx, yi hx, yi 2 |hx, yi|2
(4.8) = − =−
hy, yi hy, yi
|hx, yi|2
(4.9) ⇒ 0 ≤ hx, xi −
hy, yi
p
or | hx, yi| ≤ hx, xi hy, yi
The triangle inequality can be can be established easily using the Cauchy-
Schwarz inequality as follows
(4.10) hx + y, x + yi = hx, xi + hx, yi + hy, xi + hy, yi .

(4.11) ≤ hx, xi + 2 |hx, yi| + hy, yi
p
(4.12) ≤ hx, xi + 2 hx, xi hy, yi + hy, yi
p p p
(4.13) hx + y, x + yi ≤ hx, xi + hy, yi
p
Thus, the candidate function hx, xi satisfies all the properties necessary to
define a norm, i.e.
p p
(4.14) hx, xi ≥ 0 ∀ x ∈ X and hx, xi = 0 iff x = 0
p p
(4.15) hαx, αxi = |α| hx, xi
p p p
(4.16) hx + y, x + yi ≤ hx, xi + hy, yi (Triangle inequality)
p
Thus, the function hx, xi indeed defines a norm on the inner product space
X. In fact the inner product defines the well known 2-norm on X, i.e.
p
(4.17) kxk2 = hx, xi
and the triangle inequality can be stated as
(4.18) kx + yk22 ≤ kxk22 + 2 kxk2 . kyk2 + kyk22 . = [kxk2 + kyk2 ]2
(4.19) or kx + yk2 ≤ kxk2 + kyk2
Definition 14. (Angle) The angle θ between any two vectors in an inner
product space is defined by
∙ ¸
−1 hx, yi
(4.20) θ = cos
kxk2 kyk2
Example 29. Inner Product Spaces
(1) Space X ≡ Rn with x = (ξ 1, ξ 2, ξ 3, .......ξ n, ) and y = (η 1 , η 2 ........η n )
X
n
T
(4.21) hx, yi = x y = ξ i η i
i=1
X
n
(4.22) hx, xi = (ξ i )2 = kxk22
i=1
n
(2) Space X ≡ R with x = (ξ 1, ξ 2, ξ 3, .......ξ n, ) and y = (η 1 , η 2 ........η n )
(4.23) hx, yiW = xT W y
where W is a positive definite matrix. The corresponding 2-norm is
p √
defined as kxkW,2 = hx, xiW = xT W x
(3) Space X ≡ C n with x = (ξ 1, ξ 2, ξ 3, .......ξ n, ) and y = (η 1 , η 2 ........η n )

X
n
(4.24) hx, yi = ξ i ηi
i=1
X
n X
n
(4.25) hx, xi = ξiξi = |ξ i |2 = kxk22
i=1 i=1
(4) Space X ≡ L2 [a, b] of real valued square integrable functions on [a, b]

with inner product
Zb
(4.26) hx, yi = x(t)y(t)dt
a
is an inner product space and denoted as L2 [a, b]. Well known examples
of spaces of this type are the set of continuous functions on [−π, π] or
[0, 2π],which we consider while developing Fourier series expansions of
continuous functions on these intervals using sin(nπ) and cos(nπ) as
basis functions.
(5) Space of polynomial functions on [a, b]with inner product
Zb
(4.27) hx, yi = x(t)y(t)dt
a
is a inner product space. This is a subspace of L2 [a, b].

(6) Space of complex valued square integrable functions on [a, b] with inner
product
Zb
(4.28) hx, yi = x(t)y(t)dt
a
is an inner product space.
Definition 15. (Hilbert Space): A complete inner product space is called

as an Hilbert space.
Definition 16. (Orthogonal Vectors): In a inner product space X two

vector x, y ∈ X are said to be orthogonal if hx, yi = 0.We symbolize this by
x⊥y. A vector x is said to be orthogonal to a set S (written as x⊥s) if x⊥z for
each z ∈ S.
Just as orthogonality has many consequences in plane geometry, it has many

implications in any inner-product space [11]. The Pythagoras theorem, which
is probably the most important result the plane geometry, is true in any inner
product space.
Lemma 2. If x⊥y in an inner product space then kx + yk22 = kxk22 + kyk22

.
Proof: kx + yk22 = hx + y, x + yi = kxk22 + kyk22 + hx, yi + hy, xi .
Definition 17. (Orthogonal Set): A set of vectors S in an inner product

space X is said to be an orthogonal set if x⊥y for each x, y ∈ S and x 6= y.
The set is said to be orthonormal if, in addition each vector in the set has norm
equal to unity.
Note that an orthogonal set of nonzero vectors is linearly independent set.

We often prefer to work with an orthonormal basis as any vector can be uniquely
represented in terms of components along the orthonormal directions. Common
examples of such orthonormal basis are (a) unit vectors along coordinate direc-
tions in Rn (b) sin(nt) and cos(nt) functions in C[0, 2π].
4.1. Gram-Schmidt procedure. Given any linearly independent set in

an inner product space, it is possible to construct an orthonormal set. This
procedure is called Gram-Schmidt procedure. Consider a linearly independent
© ª
set of vectors x(i) ; i = 1, 2, 3.....n in a inner product space we define e(1) as
(1) x(1)
(4.29) e = (1)
kx k2
We form unit vector e(2) in two steps.
®
(4.30) z(2) = x(2) − x(2) , e(1) e(1)
®
where x(2) , e(1) is component of x(2) along e(1) .
z(2)
(4.31) e(2) =
kz(2) k2
.By direct calculation it can be verified that e(1) ⊥e(2) . The remaining orthonor-
mal vectors e(i) are defined by induction. The vector z(k) is formed according
to the equation
X
k−1
(k) (i) ® (i)
(k) (k)
(4.32) z =x − x , e .e
i=1
and
z(k)
(4.33) e(k) = ; k = 1, 2, .........n
kz(k) k2
Again it can be verified by direct computation that z(k) ⊥e(i) for all i < k.
Example 30. Gram-Schmidt Procedure in R3 : Consider X = R3 with

hx, yi = xT y. Given a set of three linearly independent vectors in R3
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
1 1 2
(1) ⎢ ⎥ (2) ⎢ ⎥ (3) ⎢ ⎥
(4.34) x = ⎣ 0 ⎦; x = ⎣ 0 ⎦; x = ⎣ 1 ⎦
1 0 0
we want to construct and orthonormal set. Applying Gram Schmidt procedure,
⎡ 1 ⎤
√
(1) x(1) ⎢ 2 ⎥
(4.35) e = (1) . = ⎣ 0 ⎦
kx k2 1 √
2
®
(4.36) z(2) = x(2) − x(2) , e(1) .e(1)
⎡ ⎤ ⎡ 1 ⎤ ⎡ ⎤
1
1 √
⎢ ⎥ 1 ⎢ 2 ⎥ ⎢ 2 ⎥
= ⎣ 0 ⎦− √ ⎣ 0 ⎦=⎣ 0 ⎦
2 √1
0 2
− 12
⎡ ⎤
√1
z(2) ⎢ 2
⎥
(4.37) e(2) = (2)
.=⎣ 0 ⎦
kz k2 1
− √2
® ®
z(3) = x(3) − x(3) , e(1) .e(1) − x(3) , e(2) .e(2)
⎡ ⎤ ⎡ 1 ⎤ ⎡ 1 ⎤ ⎡ ⎤
2 √ √ 0
⎢ ⎥ √ ⎢ 2
⎥ √ ⎢ 2
⎥ ⎢ ⎥
(4.38) = ⎣ 1 ⎦ − 2⎣ 0 ⎦ − 2⎣ 0 ⎦=⎣ 1 ⎦
0 √1 − √12 0
2
z(3) h iT
e(3) = . = 0 1 0
kz(3) k2
Note that the vectors in the orthonormal set will depend on the definition of
inner product. Suppose we define the inner product as follows
(4.39) hx, yiW = xT W y
⎡ ⎤
2 −1 1
⎢ ⎥
W = ⎣ −1 2 −1 ⎦
1 −1 2
° ° √
where W is a positive definite matrix. Then, length of °x(1) °W,2 = 6 and the
e(1) becomes
unit vector b
⎡ 1 ⎤
√
x(1) ⎢ 6 ⎥
(4.40) e(1)
b = (1) .=⎣ 0 ⎦
kx kW,2 1 √
6
The remaining two orthonormal vectors have to be computed using the inner
product defined by equation 4.39.
Example 31. Gram-Schmidt Procedure in C[a,b]: Let X represent set

of continuous functions on interval −1 ≤ t ≤ 1 with inner product defined as
Z1
(4.41) hx(t), y(t)i = x(t)y(t)dt
−1
Given a set of four linearly independent vectors
(4.42) x(1) (t) = 1; x(2) (t) = t; x(3) (t) = t2 ; x(4) (t) = t3
we intend to generate an orthonormal set. Applying Gram-Schmidt procedure

x(1) (t) 1
(4.43) e(1) (t) = (1)
=√
kx (t)k 2
Z1
(1)
®
(2) t
(4.44) e (t), x (t) = dt = 0
2
−1
®
(4.45) z(2) (t) = t − x(2) , e(1) .e(1) = t = x(2) (t)
z(2)
(4.46) e(2) =
kz(2) k
Z1 ∙ 3 ¸1
° (2) °2 t 2
(4.47) °z (t)° = 2
t dt = =
3 −1 3
−1
r
° (2) ° 2
(4.48) °z (t)° =
3
r
3
(4.49) e(2) (t) = .t
2
® ®
z(3) (t) = x(3) (t) − x(3) (t), e(1) (t) .e(1) (t) − x(3) (t), e(2) (t) .e(2) (t)
⎛1 ⎞ ⎛r 1 ⎞
Z Z
1 3 3 ⎠ (2)
= t2 − ⎝ t2 dt⎠ e(1) (t) − ⎝ t dt e (t)
2 2
−1 −1
1 1
(4.50) = t2 − − 0 = t2 −
3 3
z(3) (t)
(4.51) e(3) (t) =
kz(3) (t)k
Z1 µ ¶2
° °2 ® 1
(4.52) where °z(3) (t)° = z(3) (t), z(3) (t) = 2
t − dt
3
−1
Z1 µ ¶ ∙ 5 ¸1
4 2 2 1 t 2t3 t
= t − t + dt = − +
3 9 5 9 9 −1
−1
2 4 2 18 − 10 8
= − + = =
3 9 9 45 45
r r
° (3) ° 8 2 2
(4.53) °z (t)° = =
45 3 5
The orthonormal polynomials constructed above are well known Legandre poly-
nomials. It turns out that
r
2n + 1
(4.54) en (t) = pn (t) ; (n = 0, 1, 2.......)
2
where
(−1)n dn ©¡ ¢ ª
2 n
(4.55) Pn (t) = n 1 − t
2 n! dtn
are Legandre polynomials. It can be shown that this set of polynomials forms a
orthonormal basis for the set of continuous functions on [-1,1].
Example 32. Gram-Schmidt Procedure in other Spaces

(1) Shifted Legandre polynomials: X = C[0, 1] and inner product de-
fined as
Z1
(4.56) hx(t), y(t)i = x(t)y(t)dt
0
(2) Hermite Polynomials: X ≡ L2 (−∞, ∞), i.e. space of continuous

functions over (−∞, ∞) with 2 norm defined on it and
Z∞
(4.57) hx(t), y(t)i = x(t)y(t)dt
−∞
Apply Gram-Schmidt to the following set of vectors in L2 (−∞, ∞)

(3) Laguerre Polynomials: X ≡ L2 (0, ∞), i.e. space of continuous
functions over (0, ∞) with 2 norm defined on it and
Z∞
(4.58) hx(t), y(t)i = x(t)y(t)dt
0
Apply Gram-Schmidt to the following set of vectors in L2 (0, ∞)

t
(4.59) x(1) (t) = exp(− ) ; x(2) (t) = tx(1) (t) ;
2
(4.60) x (t) = t x (t) ; ......x(k) (t) = tk−1 x(1) (t) ; ....
(3) 2 (1)
The first few Laguerre polynomials are as follows

L0 (t) = 1 ; L1 (t) = 1 − t ; L2 (t) = 1 − 2t + (1/2)t2 ....
5. Problem Formulation, Transformation and Convergence

5.1. One Fundamental Problem with Multiple Forms. Using the
generalized concepts of vectors and vector spaces discussed above, we can look
at mathematical models in engineering as transformations, which map a subset
of vectors from one vector space to a subset in another space.
Definition 18. (Transformation):Let X and Y be linear spaces and let

M be subset of X. A rule which associates with every element x ∈ M to an
element y ∈ Y is said to be transformation from X to Y with domain M. If
y corresponds to x under the transformation we write y = F(x) where F(.) is
called an operator.
The set of all elements for which and operator F is defined is called as
domain of F and set of all elements generated by transforming elements in
domain by F are called as range of F. If for every y ∈ Y , there is utmost one
x ∈ M for which F(x) = y , then F(.) is said to be one to one. If for every
y ∈ Y there is at least one x ∈ M, then F is said to map M onto Y.
Definition 19. (Linear Transformations): A transformation F map-

ping a vector space X into a vector space Y is said to be linear if for every
x(1) , x(2) ∈ X and all scalars α, β we have
5. PROBLEM FORMULATION, TRANSFORMATION AND CONVERGENCE 43
(5.1) F(αx(1) + βx(2) ) = αF(x(1) ) + βF(x(2) ).
Note that any transformation that does not satisfy above definition is not a
linear transformation.
Definition 20. (Continuous Transformation): A transformation F :

© ª
X → Y is continuous at print x(0) ∈ X if and only if x(n) → x(0) implies
¡ ¢
F (x(n) ) → F x(0) . If F (.) is continuous at each x(0) ∈ X, then we say that
the function is a continuous function.
Example 33. Operators

(1) Consider transformation
(5.2) y = Ax
where y ∈ Rm and x ∈ Rn and A ∈ Rm × Rn . Whether this mapping

in onto Rn depends on the rank of the matrix. It is easy to check that
A is a linear operator. However, contrary to the common perception,
the transformation
(5.3) y = Ax + b
does not satisfy equation (2.1) and does not qualify as a linear trans-
formation.
(2) d/dt(.) is an operator from the space of continuously differentiable func-
tions to the space of continuous function.
R1
(3) The operator 0 [.]dt maps space of integrable functions into R.
A large number of problems arising in applied mathematics can be stated

as follows [9]: Solve equation
(5.4) y =F(x)
where x ∈ X, y ∈ Y are linear vector spaces and operator F : X → Y. In

engineering parlance, x, y and F represent input, output and model, respec-
tively. Linz [9] proposes following broad classification of problems encountered
in computational mathematics
• Direct Problems: Given operator F and x, find y.In this case, we
are trying to compute output of a given system given input. The com-
putation of definite integrals is an example of this type.
• Inverse Problems:Given operator F and y, find x. In this case we

are looking for input which generates observed output. Solving system
of simultaneous (linear / nonlinear) algebraic equations, ordinary and
partial differential equations and integral equations are examples of this
category
• Identification problems:Given operator x and y, find F. In this
case, we try to find the laws governing systems from the knowledge of
relation between in the inputs and outputs.
The direct problems can be treated relatively easily. Inverse problems and
identification problems are relatively difficult to solve and form the central
theme of the numerical analysis.
Once we understand the generalized concepts of vector spaces, norms, oper-
ators etc., we can see that the various inverse problems under consideration are
not fundamentally different. For example,
• Linear algebraic equations : X ≡ Rn
(5.5) Ax = b
can be rearranged as
(5.6) F (x) = Ax − b =0
• Nonlinear algebraic equations: X ≡ Rn
(5.7) F (x) = 0
h iT
F (x) = f1 (x) f2 (x) ... fn (x)
n×1
• ODE—IVP: X ≡ C n [0, ∞)
⎡ ⎤ " #
dx(t)
(5.8) ⎣ dt − F (x(t), t) ⎦ = 0
x(0) − x0 0
can be expressed in an abstract form as
(5.9) F [x(t)] = 0 ; x(t) ∈ C n [0, ∞)
• ODE-BVP: X ≡ C (2) [0, 1]
(5.10) Ψ[d2 u/dz 2 , du/dz, u, z] = 0 ; (0 < z < 1)
Boundary Conditions
(5.11) f1 [du/dz, u, z] = 0 at z = 0
(5.12) f2 [du/dz, u, z] = 0 at z = 1
which can be written in abstract form
F [u(t)] = 0 ; u(t) ∈ C (2) [0, 1]
Here, the operator F [u(t)] consists of the differential operator Ψ(.)
together with the boundary conditions and C (2) [0, 1] represents set of
twice differentiable continuous functions.
As evident from the above abstract representations, all the problems can be
reduced to one fundamental equation of the form
(5.13) F(x) = 0
where x represents a vector in the space under consideration. It is one funda-
mental problem, which assumes different forms in different context and different
vector spaces. Viewing these problems in a unified framework facilitates better
understanding of the problem formulations and the solution techniques.
5.2. Numerical Solution. Given any general problem of the form (5.13),
the approach to compute a solution to this problem typically consists of two
steps
• Problem Transformation: In this step, the original problem is trans-
formed into a one of the known standard forms, for which numerical
tools are available.
• Computation of Numerical Solution: Use a standard tool or a
combination of standard tools to construct a numerical solution to the
transformed problem. The three most commonly used tools are (a)
linear algebraic equation solvers (b) ODE-IVPs solvers (c) numerical
optimization. A schematic diagram of the generic solution procedure
is presented in Figure (2). In the sub-section that follows, we will de-
scribe two most commonly used tools for problem transformation. The
tools used for constructing solutions and their applications in different
context will form the theme for the rest of the book.
5.3. Tools for Problem Transformation. As stated earlier, given a sys-

tem of equations (5.13), the main concern in numerical analysis is to come up
with an iteration scheme of the form
£ ¤
(5.14) x(k+1) = G x(k) ; k = 0, 1, 2, ......
Original
Problem
Transformation
Taylor Series / Weierstrass
Theorem
Tool 2:
Solutions
Tool 1: of ODE-
Solutions Transformed Problem in IVP
of Linear Standard Format
Algebraic
Equations Tool 3:
Numerical Numerical
Solution Optimization
Figure 2
where in such a way that the solution x∗ of equation (6.5), i.e.

(5.15) x∗ = G [x∗ ]
satisfies
(5.16) F(x∗ ) = 0
Key to such transformations is approximation of continuous functions using
polynomials. The following two important results from functional analysis play
pivotal role in the development of many iterative numerical schemes. The Tay-
lor series expansion helps us develop local linear or quadratic approximations of
a function vector in the neighborhood of a point in a vector space. The Weier-
strass approximation theorem provides basis for approximating any arbitrary
continuous function over a finite interval by polynomial approximation. These
two results are stated here without giving proofs.
5.3.1. Local approximation by Taylor series expansion. Any scalar function
f (x) : R → R, which is continuously differentiable n times at x = x, the Taylor
series expansion of this function in the neighborhood the point x = x can be
expressed as
∙ ¸ ∙ ¸
∂f (x) 1 ∂ 2 f (x)
(5.17) f (x) = f (x) + δx + 2
(δx)2 + ...
∂x 2! ∂x
∙ n ¸
1 ∂ f (x)
(5.18) .... + . (δx)n + Rn (x, δx)
n! ∂xn
1 ∂ n+1 f (x + λδx)
Rn (x, δx) = n+1
(δx)n+1 ; (0 < λ < 1)
(n + 1)! ∂x
While developing numerical methods, we require more general Taylor series

expansions for multi-dimensional cases. We consider following two multidimen-
sional cases
• Case A: Scalar Function f (x) : Rn → R
(5.19) f (x) = f (x) + [∇f (x)]T δx

1 £ ¤
(5.20) + δxT ∇2 f (x) δx + R3 (x, δx)
∙ 2! ¸
∂f (x)
∇f (x) =
∂x
∙ ¸T
∂f ∂f ∂f
(5.21) = ......
∂x1 ∂x2 ∂xn x=x
∙ 2 ¸
∂ f (x)
∇2 f (x) =
∂x2
⎡ 2 ⎤
∂ f ∂2f ∂ 2f
⎢ ∂x21 ......
⎢ ∂x1 ∂x2 ∂x1 ∂xn ⎥ ⎥
⎢ ∂ f 2 2
∂ f ∂ 2f ⎥
⎢ ...... ⎥
(5.22) = ⎢
⎢ ∂x2 ∂x1 ∂x2
2
∂x2 ∂xn ⎥ ⎥
⎢ ...... ...... ...... ...... ⎥
⎢ ⎥
⎣ ∂2f ∂2f ∂2f ⎦
......
∂xn ∂x1 ∂xn ∂x2 ∂x2n x=x
1 X n X n Xn 3
∂ f (x + λδx)
R3 (x, δx) = δxi δxj δxk ; (0 < λ < 1)
3! i=1 j=1 k=1 ∂xi ∂xj ∂xk
Note that the gradient ∇f (x) is an n×1 vector and the Hessian ∇2 f (x)
is an n × n matrix..
Example 34. Consider the function vector f (x) : R2 → R
f (x) = x21 + x22 + e(x1 +x2 )

h iT
which can be approximated in the neighborhood of x = 1 1 using the
Taylor series expansion as
∙ ¸
∂f1 ∂f1
f (x) = f (x) + δx
∂x1 ∂x2 x=x
⎡ 2 ⎤
∂ f ∂ 2f
⎢ ∂x2 ∂x1 ∂x2 ⎥
(5.23) + [δx]T ⎢
⎣ ∂ f
1
2
⎥
⎦ δx+R3 (x, δx)
∂2f
∂x2 ∂x1 ∂x22 x=x
" #
h i x −1
1
= 2(1 + e2 ) + (2 + e2 ) (2 + e2 )
x2 − 1
" #T " #" #
x1 − 1 (2 + e2 ) e2 x1 − 1
(5.24) + + R3 (x, δx)
x2 − 1 e2 (2 + e2 ) x2 − 1
• Case B: Function vector F (x) : Rn → Rn

∙ ¸
∂F (x)
(5.25) F (x) = F (x) + δx + R2 (x, δx)
∂x
⎡ ⎤
∂f1 ∂f1 ∂f1
......
⎢ ∂x1 ∂x2 ∂xn ⎥
∙ ¸ ⎢ ∂f2 ∂f2 ∂f2 ⎥
∂F (x) ⎢ ...... ⎥
⎢ ⎥
= ⎢ ∂x1 ∂x2 ∂xn ⎥
∂x ⎢ ⎥
⎢ ...... ...... ...... ...... ⎥
⎣ ∂f ∂f ∂fn ⎦
n n
......
∂x1 ∂x2 ∂xn x=x
h i
Here matrix ∂F∂x(x) is called as Jackobian.
Example 35. Consider the function vector F (x) ∈ R2

" # " #
f1 (x) x21 + x22 + 2x1 x2
F (x) = =
f2 (x) x1 x2 e(x1 +x2 )
h iT
which can be approximated in the neighborhood of x = 1 1 using the
Taylor series expansion as
⎡ ⎤
" # ∂f1 ∂f1
f1 (x) ⎢ ∂x2 ⎥
F (x) = + ⎣ ∂x
∂f
1
∂f2 ⎦ δx + R2 (x, δx)
f2 (x) 2
∂x1 ∂x2 x=x
" # " #" #
4 4 4 x1 − 1
= + + R2 (x, δx)
e2 2e2 2e2 x2 − 1
A classic application of this theorem in the numerical analysis is Newton-

Raphson method for solving set of simultaneous nonlinear algebraic equations.
Consider set of n nonlinear equations
(5.26) fi (x) = 0 ; i = 1, ...., n

(5.27) or F (x) = 0
which have to be solved simultaneously. Suppose x∗ is a solution such that

F (x∗ ) = 0. If each function fi (x) is continuously differentiable, then, in the
neighborhood of x∗ we can approximate its behavior by Taylor series, as
∙ ¸
£ (k) ¡ ∗ (k) ¢¤ ∂F £ (k) ¤
∗
(5.28) F(x ) = F x + x −x ∼ (k)
= F (x ) + ∆x =0
∂x x=x(k)
Solving above linear equation yields the iteration sequence
∙∙ ¸ ¸−1
(k+1) (k) ∂F £ ¤
(5.29) x =x − F x(k)
∂x x=x(k)
We will discuss this approach in greater detail in the next chapter.
5.3.2. Polynomial approximation over an interval. Given an arbitrary
continuous function over an interval, can we approximate it with another ”simple”
function with arbitrary degree of accuracy? This question assumes significant
importance while developing many numerical methods. In fact, this question can
be posed in any general vector space. We often use such simple approximations
while performing computations. The classic examples of such approximations
are use of a rational number to approximate an irrational number (e.g. 22/7 is
used in place of π or series expansion of number e) or polynomial approxima-
tion of a continuous function. This subsections discusses rationale behind such
approximations.
Definition 21. (Dense Set) A set D is said to be dense in a normed space

X, if for each element x ∈X and every ε > 0, there exists an element d ∈D
such that kx − dk < ε.
Thus, if set D is dense in X, then there are points of D arbitrary close to

any element of X. Given any x ∈X , a sequence can be constructed in D which
converges to x. Classic example of such a dense set is the set of rational numbers
in the real line. Another dense set, which is widely used for approximations, is
the set of polynomials. This set is dense in C[a, b] and any continuous func-
tion f (t) ∈ C[a, b] can be approximated by a polynomial function p(t) with an
arbitrary degree of accuracy
Theorem 1. (Weierstrass Approximation Theorem): Consider space

C[a, b], the set of all continuous functions over interval [a, b], together with
∞−norm defined on it as
max
(5.30) kf (t)k = |f (t)|
t ∈ [a, b]
Given any ε > 0, for every f (t) ∈ C[a, b] there exists a polynomial p(t) such that
|f (t) − p(t)| < ε for all t ∈ [a, b].
This fundamental result forms the basis of many numerical techniques.
Example 36. We routinely approximate unknown continuous functions us-

ing polynomial approximation. For example, specific heat of a pure substance at
constant pressure (Cp ) is approximated as a polynomial function of temperature
(5.31) Cp ' a + bT + cT 2
(5.32) or Cp ' a + bT + cT 2 + dT 3
While developing such empirical correlations over some temperature interval
[T1, T2 ] , it is sufficient to know that (Cp ) is some continuous function of tem-
perature over this interval. Even though we do not know exact form of this
functional relationship, we can invoke Weierstrass theorem and approximate it
as a polynomial function.
Example 37. Consider a first order ODE-IVP

dy
(5.33) = −5y + 2y 3 ; y(0) = y0
dz
and we want to generate profile y ∗ (z) over interval [0, 1]. It is clear that the
true solution to the problem y ∗ (z) ∈ C (1) [0, 1], i.e. the set of once differentiable
continuous functions over interval [0, 1]. Suppose the true y ∗ (z) is difficult to
generate analytically and we want to compute an approximate solution, say y(z),
for the ODE-IVP under consideration. Applying Weierstrass theorem, we can
choose a polynomial approximation to y ∗ (z) as
(5.34) y(z) = a0 + a1 z + a2 z 2 + a3 z 3
Substituting (5.34) in ODE, we get
(5.35) a1 + 2a2 z + 3a3 z 2 = −5(a0 + a1 z + a2 z 2 + a3 z 3 )
+2(a0 + a1 z + a2 z 2 + a3 z 3 )3
Now, using the initial condition, we have
y(0) = y0 ⇒ a0 = y0
7. EXERCISE 51
In order to estimate the remaining three coefficients, we force residual R(z)

obtained by rearranging equation (5.35)
R(z) = 5a0 + (2a2 + 5a1 )z + (3a3 + 5a2 )z 2

(5.36) +5a3 z 3 − 2(a0 + a1 z + a2 z 2 + a3 z 3 )3
zero at two intermediate points, say z = z1 and z = z2 , and at z = 1, i.e.
R(z1 ) = 0 ; R(z2 ) = 0 ; R(1) = 0 ;
This gives nonlinear equations in three unknowns, which can be solved simul-
taneously to compute a1, a2 and a3 . In effect, the Weierstrass theorem has been
used to convert an ODE-IVP to a set of nonlinear algebraic equations.
6. Summary
In this chapter, we review important concepts from functional analysis
and linear algebra, which form the basis of synthesis and analysis the numerical
methods. We begin with the concept of a general vector space and define various
algebraic and geometric structures like norm and inner product. We also inter-
pret the notion of orthogonality in a general inner product space and develop
Gram-Schmidt process, which can generate an orthonormal set from a linearly
independent set. We later introduce two important results from analysis, namely
the Taylor’s theorem and Weierstrass approximation theorems, which play piv-
otal role in formulation of iteration schemes . We then proceed to develop theory
for analyzing convergence of linear and nonlinear iterative schemes using eigen
value analysis and contraction mapping principle, respectively. In the end, we
establish necessary and sufficient conditions for optimality of a scalar valued
function, which form basis of the optimization based numerical approaches.
7. Exercise
(1) While solving problems using a digital computer, arithmetic operations
can be performed only with a limited precision due to finite word length.
Consider the vector space X ≡ R and discuss which of the laws of
algebra (associative, distributive, commutative) are not satisfied for
the floating point arithmetic in a digital computer.
(2) Show that the solution of the differential equation
d2 x
+x=0
dt2
is a linear space. What is the dimension of this space?
(3) Show that functions 1, exp(t), exp(2t), exp(3t) are linearly independent
over any interval [a,b].
(4) Does the set of functions of the form
f (t) = 1/(a + bt)
constitute a linear vector space?

(5) Give an example of a function which is in L1 [0, 1] but not in L2 [0, 1].
(6) Decide linear dependence or independence of
(a) (1,1,2), (1,2,1), (3,1,1)
¡ ¢ ¡ ¢ ¡ ¢ ¡ ¢
(b) x(1) − x(2) , x(2) − x(3) , x(3) − x(4) , x(4) − x(1) for any x(1) , x(2) , x(3) , x(4)
(c) (1,1,0), (1,0,0), (0,1,1), (x,y,z) for any scalars x,y,z
(7) Describe geometrically the subspaces of R3 spanned by following sets
(a) (0,0,0), (0,1,0), (0,2,0)
(b) (0,0,1), (0,1,1), (0,2,0)
(c) all six of these vectors
(d) set of all vectors with positive components
(8) Consider the space X of all n × n matrices. Find a basis for this vector
space and show that set of all lower triangular n × n matrices forms a
subspace of X.
(9) Determine which of the following definitions are valid as definitions for
norms in C(2) [a, b]
(a) max |x(t)| + max |x0 (t)|
(b) max |x0 (t)|
(c) |x(a)| + max |x0 (t)|
(d) |x(a)| max |x(t)|
(10) In a normed linear space X the set of all vectors x ∈X such that
kx−xk ≤ 1 is called unit ball centered at x.
(a) Sketch unit balls in R2 when 1, 2 and ∞ norms are used.
(b) Sketch unit ball in C[0,1] when maximum norm is used.
(c) Can you draw picture of unit ball in L2 [0, 1]?
(11) Two norms k.ka and k.kb are said to be equivalent if there exists two
positive constants c1 and c2 ,independent of x, such that
c1 kxka ≤ kxkb ≤ c2 kxka
Show that in Rn the 2 norm (Euclidean norm) and ∞−norm (maximum

norm) are equivalent.
(12) How that
|kxk − kyk| ≤ kx − yk
7. EXERCISE 53
(13) A norm k.ka is said to be stronger than another norm k.kb if
lim ° (k) ° ° (k) °

°x ° = 0 ⇒ lim °x ° = 0
a b
k→∞ k→∞
but not vice versa. For C[0,1], show that the maximum norm is stronger
than 2 norm.
(14) Show that function kxk2,W : Rn → R defined as
√
kxk2,W = xT W x
defines a norm on when W is a positive definite matrix.

(15) Show that function hx, yi : Rn × Rn → R defined as
hx, yi = xT W y
defines an inner product on when W is a positive definite matrix. The

√
corresponding 2-norm is defined as kxk2,W = xT W x.
(16) Consider X = R3 with hx, yi = xT W y. Given a set of three linearly
independent vectors in R3
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
1 3 1
(1) ⎢ ⎥ (2) ⎢ ⎥ (3) ⎢ ⎥
x = ⎣ 2 ⎦; x = ⎣ 2 ⎦; x = ⎣ 2 ⎦
1 1 3
we want to construct and orthonormal set. Applying Gram Schmidt
procedure,
hx, yiW = xT W y
⎡ ⎤
2 −1 1
⎢ ⎥
W = ⎣ −1 2 −1 ⎦
1 −1 2
(17) Gram-Schmidt Procedure in C[a,b]: Let X represent set of con-
tinuous functions on interval 0 ≤ t ≤ 1 with inner product defined
as
Z1
hx(t), y(t)i = w(t)x(t)y(t)dt
0
Given a set of four linearly independent vectors
x(1) (t) = 1; x(2) (t) = t; x(3) (t) = t2 ;
find orthonormal set of vectors if (a) w(t) = 1 (Shifted Legandre Poly-

nomials) (b) w(t) = t(1 − t) (Jacobi polynomials).
(18) Show that in C[a,b] with maximum norm, we cannot define an inner
product hx, yi such that hx, xi1/2 = kxk∞ . In other words, show that
in C[a, b] the following function
max
hf (t),g(t)i = |x(t)y(t)|
t
cannot define an inner product.
(19) In C (1) [a, b] is
Zb
hx, yi = x0 (t)y0 (t)dt + x(a)y(a)
a
an inner product?
(20) Show that in C (1) [a, b] is
Zb
hx, yi = w(t)x(t)y(t)dt
a
with w(t) > 0 defines an inner product.

(21) Show that parallelogram law holds in any inner product space.
kx + yk2 + kx − yk2 = 2 kxk2 + 2 kyk2
Does it hold in C[a,b] with maximum norm?
(22) The triangle inequality asserts that, for any two vectors x and y belonging
to an inner product space
kx + yk2 ≤ ||y||2 +||x||2
After squaring both the sides and expanding,reduce this to Schwarz in-
equality. Under what condition Schwarz inequality becomes an equal-
ity?
(23) Show that operator d/dx(.) : C (1) [a, b] → C[a, b] is onto C[a, b] but not
one to one.
(24) If L is a linear operator, show that L(0) = 0.
(25) If L is a linear operator, show that range of L is a linear vector space.
Show by example that this is not necessarily true for nonlinear operator.
CHAPTER 3
Linear Algebraic Equations and Related Numerical

Schemes
1. Solution of Ax = b and Fundamental Spaces of A

The central problem of linear algebra is solution of linear equations of
type
(1.1) a11 x1 + a12 x2 + ........... + a1n xn = b1

(1.2) a21 x1 + a22 x2 + ........... + a2n xn = b2
(1.3) .................................................... = ....
(1.4) am1 x1 + am2 x2 + .......... + amn xn = bm
which can be expressed in vector notation
(1.5) Ax = b
⎡ ⎤
a11 . . a1n
⎢ . . . . ⎥
⎢ ⎥
(1.6) A=⎢ ⎥
⎣ . . . . ⎦
am1 . . amn
x ∈ Rn ; b ∈ Rm and A ∈ Rm × Rn . Here m represents number of equations

while n represents number of unknowns. Three possible situations arise while
developing mathematical models
• Case (m = n) : system may have a unique solution / no solution /
multiple solutions depending on rank of matrix A and vector b
• Case (m > n) : system may have no solution or may have multiple
solutions
• Case (m < n) : multiple solution
In these lecture notes, we are interested in the first case , i.e. m = n,
particularly when the number of equations are large.
Before we discuss the numerical methods to solve equation (1.5), let us
briefly review some geometric concepts associated with this equation. Consider
55
56 3. LINEAR ALGEBRAIC EQUATIONS AND RELATED NUMERICAL SCHEMES
Figure 1
the following system of equations

" # " #
2 −1 1
(1.7) x=
1 1 5
There are two ways of interpreting the above matrix vector equation geometri-
cally.
• Row picture : If we consider two equations separately as
" #T " #
2 x
(1.8) 2x − y = =1
−1 y
" #T " #
1 x
(1.9) x+y = =5
1 y
then, each one is a line in x-y plane and solving this set of equations
simultaneously can be interpreted as finding the point of their intersec-
tion (see Figure 1 (a)).
• Column picture : We can interpret the equation as linear combina-
tion of column vectors, i.e. as vector addition
" # " # " #
2 −1 1
(1.10) x1 + x2 =
1 1 5
Thus, the system of simultaneous equations can be looked upon as one

vector equation i.e. addition or linear combination of two vectors (see
Figure 1 (b)).
1. SOLUTION OF Ax = b AND FUNDAMENTAL SPACES OF A 57
Now consider the following set of equations

" #" # " #
1 1 x1 2
(1.11) =
2 2 x2 5
In row picture, this is clearly an inconsistent case (0 = 1) and has no solution
as the row vectors are linearly dependent. In column picture, no scalar multiple
of
v = [1 2]T
can found such that αv = [2 5]T . Thus, in a singular case
Row picture fails ⇐⇒ Column picture fails
i.e. if two lines fail to meet in the row picture then vector b cannot be expressed
as linear combination of column vectors in the column picture.
Now, consider a general system of linear equations Ax = b where A is an
n × n matrix.
Row picture : Let A be represented as
⎡ ¡ ¢T ⎤
r(1)
⎢ ¡ (2) ¢T ⎥
⎢ r ⎥
(1.12) A = ⎢ ⎢ .... ⎥
⎥
⎣ ¡ ¢ ⎦
T
r(n)
¡ ¢T
where r(i) represents i’th row of matrix A. Then Ax = b can be written
as n equations
⎡ ¡ ¢T ⎤ ⎡ ⎤
r(1) x b1
⎢ ¡ (2) ¢T ⎥ ⎢ ⎥
⎢ r x ⎥ ⎢ b2 ⎥
⎢ ⎥
(1.13) ⎢ ⎥=⎢ ⎥
⎣ ¡ ....¢ ⎦ ⎣ .... ⎦
T
r(n) x bn
¡ ¢T
Each of these equations r(i) x = bi represents a hyperplane in Rn (i.e. line
in R2 ,plane in R3 and so on). Solution of Ax = b is the point x at which
all these hyperplanes intersect (if at all they intersect in one point).
Column picture : Let A be represented as A = [ c(1) c(2) .........c(n) ] where
c(i) represents ith column of A. Then we can look at Ax = b as one vector
equation
(1.14) x1 c(1) + x2 c(2) + .............. + xn c(n) = b
Components of solution vector x tells us how to combine column vectors to
obtain vector b.In singular case, the n hyperplanes have no point in common
or equivalently the n column vectors are not linearly independent. Thus, both
these geometric interpretations are consistent with each other.
Now, to understand behavior of solutions of type (1.5), we can define four
fundamental spaces associated with matrix A
Definition 22. (Column Space): The space spanned by column vectors

of matrix A is defined as column space of the matrix and denoted as R(A).
Definition 23. (Row Space): The space spanned by row vectors of matrix
A is called as row space of matrix A and denoted as R(AT ).
Definition 24. (Null space): The set of all vectors x such that Ax = 0̄
is called as null space of matrix A and denoted as N(A).
A non-zero null space is obtained only when columns of A are linearly de-
pendent. If columns of A are linearly independent, then N(A) ≡ {0̄}.
Definition 25. (Left Null Space) :The set of all vectors y such that
A y = 0̄ is called as null space of matrix A and denoted as N(AT ).
T
A non-zero left null space is obtained only when rows of A are linearly
dependent. If rows of A are linearly independent, then N(AT ) ≡ {0̄}.
The following fundamental result, which relates dimensions of row and col-
umn spaces with the rank of a matrix, holds true for any m × n matrix A.
Theorem 2. (Fundamental Theorem of Linear Algebra): Given a

m × n matrix A
dim[R(A) ] = Number of linearly independent columns of A = rank(A)

dim[N(A)] = n − rank(A)
dim[R(AT ) ] = Number of linearly independent rows of A = rank(A)

dim[N(AT ) ] = m − rank(A)
In other words, number of linearly independent columns of A equals number of

linearly independent rows of A.
Note that
• When matrix A operates on vector x ∈ R(AT ) (i.e. a vector belonging
to row space of A) it produces a vector Ax ∈ R(A) (i.e. a vector in
column space of A)
1. SOLUTION OF Ax = b AND FUNDAMENTAL SPACES OF A 59
• The system of equations Ax = b can be solved if and only if b belongs

to the column space of A. i.e., b ∈ R(A). the solution is unique only if
N(A) ≡ {0̄}. If N(A) 6= {0̄} i.e. if columns of A are linearly dependent
and b ∈ R(A), then we can find infinite solutions to Ax = b.
Example 38. Consider the following set of equations

" #" # " #
1 1 x1 2
(1.15) =
2 2 x2 4
h iT h iT
It is easy to see that x1 x2 = 1 1 is a solution
" # " # " #
1 1 2
(1.16) (1) + (1) =
2 2 4
as the vector on R.H.S. belongs to R(A). But, this is not the only solution.
We can write
" # " # " #
1 1 2
(1.17) 3 + (−1) =
2 2 4
h iT h iT
This implies that x1 x2 = 3 −1 is also a solution to the above
problem. Why does this happen and how can we characterize all possible solu-
tions to this problem? To answer this question, let us find null space of ma-
trix A. In this particular case, by simple visual inspection, we can find that
h iT h iT
x1 x2 = 1 −1 is a vector belonging to the null space of A.
" # " # " #
1 1 0
(1.18) (1) + (−1) =
2 2 0
h iT
In fact, null space of A can be written as N(A) = α 1 −1 for any real
scalar α. Thus,
" #" # " #
1 1 α 0
(1.19) =
2 2 −α 0
h iT h iT
This implies that, if x1 x2 = 1 1 is a solution to (1.15), then any
vector
" # " #
1 1
(1.20) x= +α
1 −1
is also a solution to (1.15).

Thus, if we add any vector from the null space of A to the solution of (1.5),
then we get another solution to equation (1.5). If N(A) ≡ {0̄} and a solution
exists, i.e. b ∈ R(A), then the solution is unique. If N(A) 6= {0̄} and b ∈ R(A),
then there are infinite solutions to equation (1.5).
Methods for solving linear algebraic equations can be categorized as (a)
direct or Gaussian elimination based schemes and (b) iterative schemes. In the
sections that follow, we discuss these techniques in detail.
2. Direct Solution Techniques

There are several methods which directly solve equation (1.5). Prominent
among these are such as Cramer’s rule, Gaussian elimination, Gauss-Jordan
method and LU decomposition. We assume that you have some exposure to
these method in earlier courses on engineering mathematics. Let ϕ denote the
number of divisions and multiplications required for generating solution by a
particular method. We first compare various methods on the basic of ϕ.[6]
• Cramers Rule:
(2.1) ϕ(estimated) = (n − 1)(n + 1)(n!) + n ∼
= n2 ∗ n!
For a problem of size n = 100 we have ϕ ∼= 10162 and the time estimate
for solving this problem on DEC1090 is approximately 10149 years.
• Gaussian Elimination and Backward Sweep: By maximal pivot-
ing and row operations, we first reduce the system (1.5) to
(2.2) b
Ux = b
where U is a upper triangular matrix and then use backward sweep to

solve (2.2) for x.For this scheme, we have
n3 + 3n2 − n ∼ n3
(2.3) ϕ= =
3 3
For n = 100 we have ϕ ∼ 5
= 3.3 ∗ 10 .
• LU-Decomposition: LU decomposition is used when equation (1.5)
is to be solved for several different values of vector b, i.e.,
(2.4) Ax(k) = b(k) ; k = 1, 2, ......N

The sequence of computations is as follows
(2.5) A = LU (Solved only once)

(2.6) Ly(k) = b(k) ; k = 1, 2, ......N
(2.7) U x(k) = y(k) ; k = 1, 2, ......N
3. SOLUTIONS OF SPARSE LINEAR SYSTEMS 61
For N different b vectors,

n3 − n
(2.8) ϕ= + Nn2
3
• Gauss_Jordon Elimination: In this case, we start with [A : I : b]
and by sequence row operations we reduce it to [I : A−1 : x] i.e.
Sequence of
£ ¤ £ ¤
(2.9) A : I : b(k) → I : A−1 : x(k)
Row Operations
For this scheme, we have
n3 + (N − 1)n2
(2.10) ϕ=
2
Thus, Cramer’s rule is certainly not suitable for numerical computations.
The later three methods require significantly smaller number of multiplication
and division operations when compared to Cramer’s rule and are best suited
for computing numerical solutions moderately large (n ≈ 1000) systems. When
number of equations is significantly large (n ≈ 10000), even the Gaussian elim-
ination and related methods can turn out to be computationally expensive and
we have to look for alternative schemes that can solve (1.5) in smaller number
of steps. When matrices have some simple structure (few non-zero elements
and large number of zero elements), direct methods tailored to exploit spar-
sity and perform efficient numerical computations. Also, iterative methods give
some hope as an approximate solution x can be calculated quickly using these
techniques. In these lecture notes we describe direct methods for sparse lin-
ear systems and iterative computing schemes, which are useful when system
dimension is large.
3. Solutions of Sparse Linear Systems

A system of Linear equations given by (1.5) is called Sparse if only a rela-
tively small number of its matrix elements (aij ) are nonzero. The sparse patterns
that frequently occur are
• Tridiagonal
• Bond Diagonal with bond width M
• Block Diagonal
It is wasteful to apply general linear algebra methods on these problems.
Special methods are evolved for solving such sparse systems that achieve consid-
erable reduction in computation time and memory space requirements. In this
section, we first provide motivation for looking at sparse matrix forms. Later,
some of the sparse matrix algorithms are discussed in detail. This is meant to
be a brief introduction to sparse matrix computations and the treatment of the
topic is, by no means, exhaustive.
3.1. Origin of Sparse Linear Systems.

3.1.1. Solutions ODE-BVP using finite difference method. Consider the fol-
lowing general form of 2nd order ODE-BVP problem frequently encountered in
engineering problems
ODE:
(3.1) Ψ[d2 y/dz 2 , dy/dz, y, z] = 0 ; (0 < z < 1)
Boundary Conditions
(3.2) f1 [dy/dz, y, z] = 0 at z = 0
(3.3) f2 [dy/dz, y, z] = 0 at z = 1
Let y ∗ (z) ∈ C (2) [0, 1] denote true solution to the above ODE-BVP. Depending
on the nature of operator Ψ,it may or may not be possible to find the true
solution to the problem. In the present case, however, we are interested in
finding an approximate numerical solution, say y(z), to the above ODE-BVP.
The basic idea in finite difference approach is to convert the ODE-BVP to
a set of linear or nonlinear algebraic equations using Taylor series expansion as
basis. In order to achieve this, the domain 0 ≤ z ≤ 1 is divided into (n + 1)
equidistant grid points z0 , z1 ......, zn located such that
zi = i(∆z) = i/(n) for i = 0, 1, ......n
Let the value of y at location zi be defined as yi = y(zi ). Using the Taylor Series
expansion yi+1 = y(zi+1 ) = y(zi + ∆z) can be written as
(3.4)
(2) (3)
yi+1 = yi + (dy/dz)i (∆z) + (1/2!)yi (∆z)2 + (1/3!) yi (∆z)3 + ..................
Where
(k)
(3.5) yi = (dk y/dz k )z=zi
So similarly we can write yi−1 = y(zi−1 ) = y(zi − ∆z) as

(3.6)
(2) (3)
yi−1 = yi − (dy/dz)i (∆zi ) + (1/2!) yi (∆z)2 − (1/3!) yi (∆z)3 + ..................
0
From equations (3.4) and (3.6) we can arrive at several expressions for yi−1 .
From equation (3.4) we get [6]
(yi+1 − yi ) h (2) i
(3.7) (dy/dz)i = − yi (∆ z/2) + ....
∆z
From equation (3.6) we get
(yi − yi−1 ) h (2) i
(3.8) (dy/dz)i = + yi (∆ z/2) − .......
∆z
Combining equations (3.4) and (3.6) we get
(yi+1 − yi−1 ) h (3) i
(3.9) (dy/dz)i = − yi (∆ z 2 /3!) + ........
2(∆z)
The first two formulae are accurate to O(∆z) while the last one is accurate
to O[(∆z)2 ] and so is more commonly used. Equation (3.7) and (3.8) can be
(2)
combined to give an equation for second derivatives yi at location i:
(2) (yi+1 − 2yi + yi−1 ) h (4) 2

i
(3.10) yi = − 2yi (∆ z /4!) + ......
(∆z)2
Note that approximations (3.9) and (3.10) both are of order O[∆z)2 ]. One usu-
ally avoids using formulae having different accuracies for the same independent
variable [6]. These equations can be used to reduce the ODE-BVP to set of
algebraic equations as follows:
• Step 1 : Force residual Ri at each internal grid point to zero,i.e.,
∙ ¸
(yi+1 − 2yi + yi−1 ) (yi+1 − yi−1 )
(3.11) Ri = Ψ , , yi , zi = 0
(∆z)2 2(∆z)
(3.12) i = 1, 2, 3....................n − 1.
This gives (n−1) equations in (n + 1) unknowns. ODE-BVP is satisfied

exactly at internal grid points.
• Step 2: Use B.C. to generate remaining equations
— Approach 1: Use one-sided derivatives only at the boundary
points, i.e.,
(y1 − y0 )
(3.13) f1 [ , y0 , 0] = 0
(∆z)
(yn − yn−1 )
(3.14) f2 [ , y0 , 0] = 0
(∆z)
This gives remaining two equations.
— Approach 2:
(y1 − y−1 )
(3.15) f1 [ , y0 , 0] = 0
(2∆z)
yn+1 − yn−1
(3.16) f2 [ , y0 , 0] = 0
(∆z)
This approach introduces two more variables y−1 and yn+1 at hy-
pothetical grid points. Thus we have n + 3 variables and n + 1
equations, two more algebraic equations can be generated by set-
ting residual at zero at boundary points,i.e., at z0 and zn ,i.e.,
R0 = 0 and Rn = 0
This results in (n + 3) equations in (n + 3) unknowns.
Remark 1. Forcing residual Ri = 0 at internal grid points implies choosing

the values of yi such that the ODE-BVP is satisfied exactly at the internal grid
points. Obviously larger the value of N, the closer is the numerical solution
expected to be near exact solution. Note that Ri = 0 at internal grid points does
not imply yi equals yi∗ (exact solution) at that point, i.e. y(zi ) 6= y ∗ (zi ).
Example 39. Consider steady state heat transfer/conduction in a slab of

thickness L, in which energy is generated at a constant rate of q W/m3 . The
boundary at z = 0 is maintained at a constant temperature T0 ,while the boundary
at z = L dissipates heat by convection with a heat transfer coefficient h into the
ambient temperature at T∞ . The mathematical formulation of the conduction
problem is given as
(3.17) kd2 T /dz 2 + q = 0
0<z<L
(3.18) B.C. : T (0) = T0 at z = 0

∙ ¸
dT
(3.19) k = h [T∞ − T (L)]
dz z=L
(Note that this problem can be solved analytically.) Dividing the region 0 ≤ z ≤
L into n equal subregions and setting residuals zero at the internal grid points,
we have
(3.20) (Ti−1 − 2Ti + 2Ti+1 )/(∆z)2 + q/k = 0
i = 1, 2, .......(.n − 1).
Or
(3.21) (Ti−1 − 2Ti + Ti+1 ) = −(∆z)2 q/k
Note that at i = 1 we have
(3.22) (T0 − 2T1 + T2 ) = −(∆z)2 q/k
Using B.C. (3.18) we have
(3.23) −2T1 + T2 = −(∆z)2 q/k − T0
Using one sided derivative at z = 1and using B.C. (3.19)
(3.24) k(Tn − Tn−1 )/(∆z) = h((T∞ − Tn )
or
(3.25) Tn − Tn−1 = h∆z(T∞ − Tn )/k
(3.26) Tn (1 + h∆z/k) − Tn−1 = h∆zT∞ /k
Rearranging the equations in matrix form

(3.27)
⎡ ⎤⎡ ⎤ ⎡ ⎤
−2 1 0 0....................... T1 −(∆z)2 q/k − T0
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ 1 −2 1 0....................... ⎥⎢ T2 ⎥ ⎢ −(∆z)2 q/k ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ 0 1 −2 1...................... ⎥⎢ T3 ⎥ ⎢ . ⎥
⎢ ⎥⎢ ⎥=⎢ ⎥
⎢ . . . ........................ ⎥⎢ . ⎥ ⎢ . ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎣ . . . ........ − 2.......1..... ⎦⎣ . ⎦ ⎣ . ⎦
0 0 0 .... − 1.. (1 + h∆z/k) Tn −h(∆z)T∞ /k
we get a sparse tridiagonal matrix.
3.1.2. Solution of PDE using Finite Difference Method [6]. Consider elliptic
PDEs described by
∇2 u = cu + f (x, y, z)
∂2u ∂2u ∂2u
or + + = cu + f (x, y, z)
∂x2 ∂y 2 ∂z 2
which is solved on a 3 dimensional bounded region V with boundary S. The
boundary conditions on spatial surface S are given as
hα(s)n,∇ui +β(s)u = h(s)

These equations can be transformed into a set of linear (or nonlinear) algebraic
equations by using Taylor’s theorem and approximating
µ 2 ¶
∂ u (ui+1,j,k − 2ui,j,k + ui−1,j,k )
2
=
∂x ijk (∆x)2
µ 2 ¶
∂ u (ui,j+1,k − 2ui,j,k + ui,j−1,k )
2
=
∂y ijk (∆y)2
µ 2 ¶
∂ u (ui,j,k+1 − 2ui,j,k + ui,j,k−1 )
2
=
∂z ijk (∆z)2
and so on.
Example 40. Laplace equation represents a prototype for steady state dif-
fusion processes. For example 2-dimensional Laplace equation
(3.28) ∂ 2 T /∂x2 + ∂ 2 T /∂y 2 = 0
represents a description of 2-dimensional steady state heat conduction in a solid

, where T is temperature and x, y are space coordinates. Equations similar to
this arise in many problems of fluid mechanics, heat transfer and mass transfer.
In the present case , we consider conduction in a rectangular plate of dimension
Lx × Ly. The boundary conditions are as follows:
(3.29) x = 0 : T = T1 ; x = Lx : T = T3
(3.30) y = 0 : T = T2 ; y = Ly : T = T4
Construct the 2 -dimensional grid with (nx + 1) equispaced grid lines parallel to
y axis and (ny + 1) equispaced grid lines parallel to x axis. The temperature T
at (i, j) th grid point is denoted as Tij = T (xi, yWe force the residual to be zero
at each internal grid point to obtain the following set of equations:
(3.31) (Ti+1,j − 2Ti ,j +Ti−1,j )/(∆x)2 + (Ti,j+1 − 2Ti ,j +Ti,j−1 )/(∆y)2 = 0
for (i = 1, 2, .......nx − 1) and ( j = 1, 2, ........ny − 1). As the boundary tempera-

tures are known, number of variables exactly equals no of interior nodes. Note
that regardless of the size of the system, each equation contains not more than
5 unknowns, resulting in a sparse linear algebraic system. Consider the special
case when
∆x = ∆y
For this case the above equations can be written as
(3.32) Ti−1,j + Ti,j−1 − 4Ti,j + Ti,j+1 + Ti+1,j = 0

for (i = 1, 2, .......nx − 1) and (j = 1, 2, ........ny − 1)

Using boundary conditions we have
(3.33) Ti,0 = T2 ; Ti,ny = T4

i = 0, 1, ..........nx
(3.34) T0,j = T1 ; Tnx ,j = T3
j = 0, 1, .........ny
Now we define
(3.35) x = [T11 T12 .............T1,ny −1, .........., Tnx −1,1. .............Tnx −1,ny −1 ]T
And rearrange the above set of equations in form of Ax = b, then A turns out
to be a large sparse matrix. Even for 10 internal grid lines in each direction we
would get a 100 × 100 sparse matrix associated with 100 variables.
3.1.3. Cubic Spline Interpolation [6]. Suppose we are given n + 1 values

{y0 , y1 , ....yn } some dependent variable y(z) corresponding to values {z0 , z1 , ....zn }
of independent variable z over some finite interval. We have to find a continuous
function f (z) that passes through each of these points. Invoking Weierstarss
theorem, we propose a n0 th degree polynomial function
y = f (z) = α0 + α1 z + α2 z 2 + ... + αn z n
Note that values of y and x are exactly known and this is a problem of finding
an n’th degree interpolation polynomial that passes through all the points. The
coefficients of this polynomial can be easily found by solving equation
⎡ ⎤⎡ ⎤ ⎡ ⎤
1 z0 ... (z0 )n α0 y0
⎢ 1 z ... (z )2 ⎥ ⎢ α ⎥ ⎢ y ⎥
⎢ 1 1 ⎥⎢ 1 ⎥ ⎢ 1 ⎥
(3.36) ⎢ ⎥⎢ ⎥ = ⎢ ⎥
⎣ ... ... ... ..... ⎦ ⎣ ..... ⎦ ⎣ .... ⎦
1 zn ... (zn )2 αn yn
(3.37) or Aα = y
If matrix A is nonsingular, coefficient vector can be simply computed as α =A−1 y.

The matrix A appearing in the above equation is known as Vandermonde
matrix. Larger matrices of this type tend to become numerically ill-conditioned.
As a consequence, if the number of data points is large, fitting a large order
polynomial can result in a polynomial which exhibits unexpected oscillatory
behavior. In order to avoid such oscillations and difficulties due to matrix ill
conditioning, the data is divided into sub-intervals and a lower order (say cubic)
spline approximation is developed on each interval. For example, n cubic splines

fitting n + 1 data points can be expressed as
(3.38) p0 (z) = α0,0 + α1,0 (z − z0 ) + α2,0 (z − z0 )2 + α3,0 (z − z0 )3

(3.39) (z0 ≤ z ≤ z1 )
(3.40) p1 (z) = α0,1 + α1,1 (z − z1 ) + α2,1 (z − z1 )2 + α3,1 (z − z1 )3
(3.41) (z1 ≤ z ≤ z2 )
....... = ..................................................
pn−1 (z) = α0,n−1 + α1,n−1 (z − zn−1 )
(3.42) +α2,n−1 (z − zn−1 )2 + α3,n−1 (z − zn−1 )3
(3.43) (zn−1 ≤ z ≤ zn )
There are total 4n unknown coefficients {α0,0 , α1,0 .......α3,n−1 } to be determined.

In order to ensure continuity and smoothness of the approximation, the following
conditions are imposed
(3.44) pi (zi ) = yi ; i = 0, 1, 2, ..., n − 1

(3.45) pn−1 (zn ) = yn
(3.46) pi (zi+1 ) = pi+1 (zi+1 ) ; i = 0, 1, ....n − 2
dpi (zi+1 ) dpi+1 (zi+1 )
(3.47) = ; i = 0, 1, ....n − 2
dz dz
d2 pi (zi+1 ) d2 pi+1 (zi+1 )
(3.48) = ; i = 0, 1, ....n − 2
dz 2 dz 2
which result in 4n − 2 conditions. Two additional conditions are imposed at the
boundary points
d2 p0 (z0 ) d2 pn−1 (zn )
(3.49) = =0
dz 2 dz 2
which are referred to as free boundary conditions. If the first derivatives at the
boundary points are known,
dp0 (z0 ) d2 pn−1 (zn )
(3.50) = d0 ; = dn
dz dz 2
then we get the clamped boundary conditions.
Using constraints (3.44-3.48), together with free boundary conditions, we
get
(3.51) α0,i = yi ; ( i = 0, 1, 2, ..., n − 1)
(3.52) α0,n−1 + α1,n−1 (∆zn−1 ) + α2,n−1 (∆zn−1 )2 + α3,n−1 (∆zn−1 )3 = yn

(3.53) α0,i + α1,i (∆zi ) + α2,i (∆zi )2 + α3,i (∆zi )3 = α0,i+1

(3.54) α1,i + 2α2,i (∆zi ) + 3α3,i (∆zi )2 = α1,i+1
α2,i + 3α3,i (∆zi ) = α2,i+1
for( i = 0, 1, 2, ..., n − 2 )
(3.55) α2,0 = 0
(3.56) α2,n−1 + 3α3,n−1 (∆zn−1 ) = 0
Eliminating α3,i using

α2,i+1 − α2,i
(3.57) α3,i = for ( i = 0, 1, 2, ..., n − 2 )
3 (∆zi )
−α2,n−1
(3.58) α3,n−1 =
3 (∆zi )
and eliminating α1,n−1 using equation 3.52
1 ∆zi
(3.59) α1,i = (α0,i+1 − α0,i ) − (2α2,i + α2,i+1 )
∆zi 3
for ( i = 0, 1, 2, ..., n − 2 )
yn − α0,n−1
(3.60) α1,n−1 = − (∆zn−1 ) α2,n−1 − α3,n−1 (∆zn−1 )2
∆zn−1
we get only {α1,i : i = 0, 1, ...n − 1} as unknowns
(3.61) α2,0 = 0
(3.62) (∆zi−1 ) α2,i−1 + 2(∆zi + ∆zi−1 )α2,i + (∆zi−1 ) α2,i+1 = bi
for ( i = 1, 2, ..., n − 2 )
where
3(α0,i+1 − a0,1 ) 3(α0,i − a0,i−1 ) 3(yi+1 − y1 ) 3(yi − yi−1 )
bi = − = −
∆zi ∆zi−1 ∆zi ∆zi−1
for ( i = 1, 2, ..., n − 2 )
1 2
(3.63) (∆zn−2 ) α2,n−2 + (∆zn−2 + ∆zn−1 )α2,n−1 = bn
3 3
µ ¶
yn 1 1 yn−2
(3.64) bn = − + yn−1 +
∆zn−1 ∆zn−1 ∆zn−2 ∆zn−2
Defining vector α2 as
h iT
α2 = α2,0 α2,1 ....... α2,n
the above set of n equations can be rearranged as

(3.65) Aα2 = b
where A is a (n + 1) × (n + 1) and Y is (n + 1) vector. Elements of A and b
can be obtained from equations (3.61-3.63). Note that matrix A will be a near
tridiagonal matrix.
3.2. Algorithms for Solving Sparse Linear Systems [5].

3.2.1. Thomas Algorithm for Tridiagonal and Block Tridiagonal Matrices.
Consider system of equation given by following equation
⎡ ⎤⎡ ⎤ ⎡ ⎤
b1 c1 0 ... ... ... ... 0 x1 d1
⎢ a2 b2 c2 0 ... ... ... 0 ⎥⎢ x2 ⎥ ⎢ d2 ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ 0 a3 b3 c3 ... ... ... 0 ⎥⎢ x3 ⎥ ⎢ .... ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ 0 0 a4 b4 c4 . ... ... ... ⎥⎢ .... ⎥ ⎢ .... ⎥
(3.66) ⎢ ⎥⎢ ⎥ =⎢ ⎥
⎢ ... ... .. ... ... ... ... ... ⎥⎢ .... ⎥ ⎢ .... ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ ... ... ... ... ... ... cn−2 0 ⎥⎢ .... ⎥ ⎢ .... ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎣ ... ... ... ... ... an−1 bn−1 cn−1 ⎦⎣ .... ⎦ ⎣ .... ⎦
0 0 0 0 ... .0 an bn xn dn
where matrix A is a tridiagonal matrix.
Step 1:Triangularization: Forward sweep with normalization
(3.67) γ 1 = c1 /b1
ck
(3.68) γk = ; k = 2, 3, ....(n − 1)
bk − ak γ k−1
(3.69) β 1 = d1 /b1
(dk − ak β k−1 )
(3.70) βk = ; k = 2, 3, ....n
(bk − ak γ k−1 )
This sequence of operations finally results in the following system of equations
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
1 γ 1 0 .... 0 x1 β1
⎢ 0 1 γ .... 0 ⎥ ⎢ x ⎥ ⎢ β ⎥
⎢ 2 ⎥ ⎢ 2 ⎥ ⎢ 2 ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎢ ... 0 1 .... . ⎥ ⎢ .... ⎥ = ⎢ . ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎣ .... .... .... .... γ n−1 ⎦ ⎣ .... ⎦ ⎣ . ⎦
0 0 ... .... 1 xn βn
Step 2: Backward sweep leads to solution vector
xn = β n
(3.71) xk = β k − γ k xk+1
(3.72) k = (n − 1), .(n − 2), ......, 1
Total no of multiplications and divisions
ϕ = 5n − 8
Which is far smaller than the n3 /3 operations(approximately) for Gaussian

elimination and backward sweep required for dense matrices.
Block Thomas Algorithm: Consider block triangular system of the form
⎡ ⎤⎡ ⎤ ⎡ ⎤
B1 C1 [0]. ..... .... [0]. x(1) d(1)
⎢ A2 B2 C2 .... ..... ..... ⎥⎢ x(2) ⎥ ⎢ d(2) ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
(3.73) ⎢ .... ..... ..... ..... ..... [0]. ⎥⎢ . ⎥=⎢ . ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎣ ..... ..... ..... ..... .Bn−1 Cn−1 ⎦⎣ . ⎦ ⎣ . ⎦
[0] ..... ..... [0] An Bn x(n) d(n)
where Ai , Bi and Ci are matrices and xi and di represent vectors of appro-

priate dimensions. Thomas algorithm can be developed for such systems in a
analogous manner.
• Step 1:Block Triangularization
Γ1 = [B1 ]−1 C1
(3.74) Γk = [Bk − Ak Γk−1 ]−1 Ck ; k = 2, 3, ....(n − 1)
(3.75) β(1) = [B1 ]−1 d(1)
β(k) = [Bk − Ak Γk−1 ]−1 (d(k) − Ak β(k−1) ) ; k = 2, 3, ....n
• Step 2: Backward sweep
(3.76) x(n) = β(n)
(3.77) x(k) = β(k) − Γk x(k+1)

(3.78) k = (n − 1), .(n − 2), ......, 1
3.2.2. Triangular and Block Triangular Matrices. A triangular matrix is a

sparse matrix with zero-valued elements above the diagonal,i.e.,
⎡ ⎤
l11 0 . . 0
⎢ l ⎥
⎢ 12 l22 . . 0 ⎥
⎢ ⎥
L=⎢ . . . . . ⎥
⎢ ⎥
⎣ . . . . . ⎦
l n1 . . . l nn
To solve a system Lx = b, the following algorithm is used
(3.79) x1 = b1 /l11
P
i−1
[bi − lij xj ]
j=1
(3.80) xi = ; i = 2, 3, .....n
lii
The operational count ϕ i.e., the number of multiplications and divisions, for
this elimination process is
(3.81) ϕ = n(n + 1)/2

which is considerably smaller than the Gaussian elimination..
In some applications we encounter equations with a block triangular matri-
ces. For example,
⎡ ⎤ ⎡ (1) ⎤ ⎡ (1) ⎤
A11 [0] .... [0] η b
⎢ A ⎥ ⎢ (2) ⎥ ⎢ ⎥
⎢ 12 A22 .... [0] ⎥ ⎢ η ⎥ ⎢ b(2) ⎥
⎢ ⎥ ⎢ ⎥=⎢ ⎥
⎣ ..... . .... ..... ⎦ ⎣ .... ⎦ ⎣ .... ⎦
An1 An2 .... Ann η (n) b(n)
Where Aij are m × m sub-matrices while η (i) ∈ Rm and b(i) ∈ Rm are sub-
vectors for i = 1, 2, ..n. The solution of this type of systems is completely
analogous to that of lower triangular systems,except that sub-matrices and sub-
vectors are used in place of scalers. The block counterpart for lower triangular
system is
(3.82) η (i) = (A11 )−1 b(1)
X
i−1
(i) −1 (i)
(3.83) η = (Aii ) [b − (Aij η (i) )] ; i = 2, 3, .....n
j=1
The above form does not imply that the inverse (Aii )−1 should be compared
explicitly. For example we can find η (1) by Gaussian elimination to solve the
system A11 η (1) = b(1)
4. ITERATIVE SOLUTION TECHNIQUES FOR SOLVING Ax = b 73
3.2.3. Solution of a Large System By Partitioning. If matrix A is equation

(1.5) is very large, then we can partition matrix A and vector b as
" #" # " #
A11 A12 x(1) b(1)
Ax = =
A21 A22 x(2) b(2)
where A11 is a (m × m) square matrix. this results in two equations
(3.84) A11 x(1) + A12 x(2) = b(1)
(3.85) A21 x(1) + A22 x(2) = b(2)
which can be solved sequentially as follows
(3.86) x(1) = [A11 ]−1 [b(1) − A12 x(2) ]
(3.87) A21 [A11 ]−1 [b(1) − A12 x(2) ] + A22 x(2) = b(2)
£ ¤
(3.88) A22 − A21 [A11 ]−1 A12 x(2) = b(2) − (A21 A−1
11 )b
(1)
or
£ ¤−1 £ (2) ¤
x(2) = A22 − A21 [A11 ]−1 A12 b − (A21 A−1
11 )b
(1)
It is also possible to work with higher number of partitions equal to say 9, 16

.... and solve the given system.
4. Iterative Solution Techniques for solving Ax = b

By this approach, we starts from any initial guess, say x(0) ,and generate
an improved estimate x(k+1) from previous approximation x(k) . This sequence
° ° ° °
is terminated when some norm of the residue °r(k) ° = °Ax(k) − b° becomes
sufficiently small. Such a method can be developed by splitting matrix A. If A
is expressed as
(4.1) A=S−T
then, equation (1.5) can be written as
Sx = T x + b
Thus, starting from a guess solution

(0)
(4.2) x(0) = [x1 ........x(0)
n ]
we can generate a sequence of approximate vectors as follows
(4.3) x(k+1) = S −1 [T x(k) + b] ; (k = 0, 1, 2, .....)

Requirements on S and T matrices are as follows [15] : matrix A should be

decomposed into A = S − T such that
• Matrix S should be easily invertible
© ª
• Sequence x(k) : k = 0, 1, 2, .... should converge to x∗ where x∗ is the
solution of Ax = b.
At the k’th iteration step, we can write
¡ ¢k h¡ ¢k−1 ¡ −1 ¢k−2 i
(4.4) x(k) = S −1 T x(0) + S −1 T + S T + ... + S −1 T + I S −1 b
If we select (S −1 T ) such that

lim ¡ −1 ¢k
(4.5) S T = [0]
k→∞
where [0] represents null matrix, then, using identity
£ ¡ ¢¤−1 ¡ ¢ ¡ ¢k−1 ¡ −1 ¢k
I − S −1 T = I + S −1 T + .... + S −1 T + S T + ...
we can write
£ ¡ ¢¤−1 −1
x(k) → I − S −1 T S b = [S − T ]−1 b =A−1 b
for large k. The above expression clearly explains how the iteration sequence
generates a numerical approximation to A−1 b, provided condition (4.5) is sat-
isfied.
Let D, L and U be diagonal, strictly lower triangular and strictly upper
triangular parts of A, i.e.,
(4.6) A=L+D+U
There are three popular iterative formulations [15]
• Jacobi Method:
(4.7) S = D
(4.8) T = −(L + U)
• Gauss-Seidel Method
(4.9) S = L+D
(4.10) T = −U
• Relaxation Method:
µ ¶
1
(4.11) S =L+ D
ω
µ ¶
1−ω
(4.12) T = D−U
ω
where 0 < ω < 2

The above formulations in vector-matrix notation, though helps in analyzing
these algorithms, is not suitable for developing computer programs. We will
derive computationally efficient algorithms and convergence criteria for each of
these iteration schemes in the following subsections.
4.1. Jacobi-Method. In order to understand the rationale behind formu-

lation of Jacobi iterations, consider the first equation in the set of equations
Ax = b, i.e.,
(4.13) a11 x1 + a12 x2 + ......... + a1n xn = b1
Rearranging this equation, we can arrive at a iterative formula for computing
(k+1)
x1 , as
(k+1) (k)
(4.14) x1 = [b1 − a12 x2 ....... − a1n xn(k) ]/a11
Similarly, using second equation from Ax = b, we can derive
(k+1) (k) (k)
(4.15) x2 = [b2 − a21 x1 − a23 x3 ........ − a2n xn(k) ]/a22
and, in general, using ith row
(k+1) (k) (k) (k)
(4.16) xi = [b2 − ai1 x1 ...... − ai,i−1 xi−1 − ai,i+1 xi+1 ..... − ai,n x(k)
n ]/aii
Arranging the above equations in the matrix form yields

(4.17) x(k+1) = −D−1 (L + U)x(k) + D−1 b
(4.18) Dx(k+1) = −(L + U)x(k) + b
In the above derivation, it is implicitly assumed that aii 6= 0. If this is not the
case, simple row exchange is often sufficient to satisfy this condition. Suppose
we denote residue vector r as
(4.19) r(k) = b − Ax(k)
(k)
X
n
(k)
(4.20) i.e. ri = bi − aij xj
j=1
then, the standard termination criterion is

° (k) °
°r °
<ε
kbk
where ε is an arbitrarily small number (such as 10−6 or 10−8 ). Another standard
termination criterion can be
° (k) °
°x − x(k+1) °
<ε
kx(k+1) k
This condition is practically equivalent to the previous condition. Equation

(4.16) is more suitable from the view point of programming than equation (4.3)
and the algorithm can be stated as follows:
Jacobi Algorithm
INITIALIZE : b, A, x(0) , kmax , ε
k=0
δ = 100 ∗ ε
WHILE [(δ > ε) AND (k < kmax )]
FOR i = 1 : n n
P (0)
ri = bi − aij xj
j=1
(1) (0)
xi = xi + (ri /aii )
END FOR
δ = krk / kbk
k =k+1
x(0) = x(1)
END WHILE
4.2. Gauss-Seidel Method. When matrix A is large, there is a practical

difficulty with the Jacobi method. It required to store all components of x(k) in
the computer memory (as a separate variables) until calculations of x(k+1) is
(k+1)
over. The Gauss-Seidel method overcomes this difficulty by using xi imme-
(k+1)
diately in the next equation while computing xi+1 .This modification leads to
the following set of equations
(k+1) (k) (k)

(4.21) x1 = [b1 − a12 x2 − a13 x3 ......a1n xn(k) ]/a11
n o n o
(k+1) (k+1) (k)
(4.22) x2 = [b2 − a21 x1 − a23 x3 + ..... + a2n xn(k) ]/a22
n o n o
(k+1) (k+1) (k+1) (k)
(4.23) x3 = [b3 − a31 x1 + a32 x2 − a34 x4 + ..... + a3n xn(k) ]/a33
...... = .............................................................
(k+1) (k+1)
(4.24) xn(k+1) = [bn − an1 x1 ......an,n−1 xn−1 ]/ann
Again it is implicitly assumed that pivots aii are non zero. Now in the above
(k+1)
set of equations, if we move all terms involving xi from R.H.S. to L.H.S. ,
we get
⎡ ⎤⎡ (k+1) ⎤
a11 0 0 . x1
⎢ a21 a22 . . ⎥⎢ . ⎥
⎢ ⎥⎢ ⎥
(4.25) ⎢ ⎥⎢ ⎥
⎣ . . . . ⎦⎣ . ⎦
(k+1)
an1 . . ann xn
⎡ ⎤⎡ (k) ⎤ ⎡ ⎤
0 −a12 . . −a1n x1 b1
⎢ . . . . . ⎥⎢ . ⎥ ⎢ . ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
(4.26) = ⎢ ⎥⎢ ⎥+⎢ ⎥
⎣ . . . .. −an−1,n ⎦⎣ . ⎦ ⎣ . ⎦
(k)
0 . . . 0 xn bn
(4.27) or (L + D)x(k+1) = −Ux(k) + b
Thus, we have
S = L+D
T = −U
and, using vector-matrix notation, the Gauss-Seidel method can be stated as

£ ¤
(4.28) or x(k+1) = (L + D)−1 −Ux(k) + b
Gauss-Seidel Algorithm
INITIALIZE :b, A, x, kmax , ε
k=0
δ = 100 ∗ ε
FOR i = 1 : n n
P
ri = bi − aij xj
j=1
xi = xi + (ri /aii )
END FOR
Tc = krk / kbk
k =k+1
END WHILE
4.3. Relaxation Method. Suppose we have a starting value say y,

of a quantity and we wish to approach a target value y ∗ by some method. Let
application of the method change the value from y to yb. If yb is between y and
ye, which is even closer to y ∗ , then we can approach ye faster by magnifying the
change (b y − y) [15]. In order to achieve this, we need to apply a magnifying
factor ω > 1 and get
(4.29) ye − y = ω (b
y − y)
(4.30) or ye = ω yb + (1 − ω) y
This amplification process is an extrapolation and is an example of over-

relaxation. If the intermediate value yb tends to overshoot target y ∗ , then
we may have to use ω < 1 ; this is called under-relaxation.
Application of over-relaxation to Gauss-Seidel method leads to the fol-
lowing set of equations
(k+1) (k) (k+1) (k)
(4.31) x̃i = xi + ω[xi − xi ]
i = 1, 2, .....n
(k+1)
where xi are generated by Gauss-Seidel method, i.e.,
µ ¶" X
i−1 Xn
#
(k+1) 1 (k+1) (k)
(4.32) xi = bi − aij xj − aij xj
aii j=1 j=i+1
i = 1, 2, .....n
With some algebraic manipulations, the above equations can be rearranged in

vector matrix form as follows
(4.33) (D + ωL)x(k+1) = [(1 − ω)D − ωU]x(k) + ω b
Relaxation Algorithm
INITIALIZE : b, A, x, kmax , ε, ω
k=0
δ = 100 ∗ ε
FOR i = 1 : n n
P
qi = bi − aij xj
j=1
zi = xi + (qi /aii )
xi = ωzi + (1 − ω)xi
END FOR
r = b−Ax
Tc = krk / kbk
k =k+1
END WHILE
4.4. Convergence of Iterative Methods [15, 5]. In order to solve equa-

tion (1.5), we have formulated an iterative scheme
¡ ¢
(4.34) x(k+1) = S −1 T x(k) + S −1 b
Let the true solution equation (1.5) be
¡ ¢
(4.35) x∗ = S −1 T x∗ + S −1 b
Defining error vector
(4.36) e(k) = x(k) − x∗
and subtracting equation (4.35) from equation (4.34), we get

¡ ¢
(4.37) e(k+1) = S −1 T e(k)
Thus, if we start with some e(0) , then after k iterations we have

¡ ¢
(4.38) e(1) = S −1 T e(0)
¡ ¢
(4.39) e(2) = S −2 T 2 e(1) = [S −1 T ]2 e(0)
(4.40) ....... = ..........
(4.41) e(k) = [S −1 T ]k e(0)
The convergence of the iterative scheme is assured if
lim
(4.42) e(k) = 0
k→∞
lim
(4.43) i.e. [S −1 T ]k e(0) = 0
k→∞
for any initial guess vector e(0) . It mat be noted that equation (4.37) is a linear
difference equation of form
(4.44) z(k+1) = Bz(k)
subject to initial condition z(0) .Here z ∈ Rn and B is a n×n matrix. In the next
sub-section, we analyze behavior of the solutions of linear difference equations
of type (4.44). We then proceed with applying these general results to the
specific problem at hand. i.e. convergence of iteration schemes for solving
linear algebraic equations.
4.4.1. Eigenvalue Analysis. To begin with, let us consider scalar linear iter-
ation scheme
(4.45) z(k+1) = bz(k)
where z(k) ∈ R and b is a real scalar. It can be seen that
(4.46) z(k) = (b)k z(0) → 0 as k → ∞
if and only if |b| < 1.To generalize this notation to a multidimensional case,
consider equation of type (4.44) where z(k) ∈ Rn . Taking motivation from the
scalar case, we propose a solution to equation (4.44) of type
(4.47) z(k) = λk v
where λ is a scalar and v ∈ Rn is a vector. Substituting equation (4.47) in

equation (6.54), we get
(4.48) λk+1 v = B(λk v)

(4.49) or λk (λI − B) v = 0
Since we are interested in a non-trivial solution, the above equation can be

reduced to
(4.50) (λI − B) v = 0
where v 6= 0. Note that the above set of equations has n equations in (n +

1) unknowns (λ and n elements of vector v). Moreover, these equations are non-
linear. Thus, we need to generate an additional equation to be able to solve the
above set exactly. Now, the above equation can hold only when the columns of
matrix (λI − B) are linearly dependent and v belongs to null space of (λI − B) .
If columns of matrix (λI − B) are linearly dependent, matrix (λI − B) is sin-
gular and we have
(4.51) det (λI − B) = 0
Note that equation (4.51) is nothing but the characteristic polynomial of matrix
A and its roots are called eigenvalues of matrix A. For each eigenvalue λi we
can find the corresponding eigen vector v(i) such that
(4.52) Bv(i) = λi v(i)
Thus, we get n fundamental solutions of the form (λi )k v(i) to equation (6.54)
and a general solution to equation (6.54) can be expressed as linear combination
of these fundamental solutions
(4.53) z(k) = α1 (λ1 )k v(1) + α2 (λ2 )k v(2) + ..... + αn (λn )k v(n)
Now, at k = 0 this solution must satisfy the condition
(4.54) z(0) = α1 v(1) + (α2 )k v(2) + ..... + αn v(n)

h ih iT
(4.55) = v(1) v(2) .... v(n) α1 α2 .... αn
(4.56) = Ψα
where Ψ is a n × n matrix with eigenvectors as columns and α is a n × 1 vector

of n coefficients. Let us consider the special case when the eigenvectors are
linearly independent. Then, we can express α as
(4.57) α =Ψ−1 z(0)
Behavior of equation (4.53) can be analyzed as k → ∞.Contribution due to the

i’th fundamental solution (λi )k v(i) → 0 if and only if |λi | < 1.Thus, z(k) → 0 as
k → ∞ if and only if
(4.58) |λi | < 1 for i = 1, 2, ....n
If we define spectral radius of matrix A as
max
(4.59) ρ(B) = |λi |
i
then, the condition for convergence of iteration equation (6.54) can be stated as
(4.60) ρ(B) < 1
Equation (4.53) can be further simplified as

⎡ ⎤⎡ ⎤
(λ1 )k 0 ..... 0 α1
h i⎢ 0 (λ2 )k 0 ... ⎥⎢ α2 ⎥
(k) (1) (2) (n) ⎢ ⎥⎢ ⎥
(4.61)
z = v v .... v ⎢ ⎥⎢ ⎥
⎣ .... .... ..... .... ⎦⎣ ... ⎦
0 .... 0 (λn )k αn
⎡ ⎤
(λ1 )k 0 ..... 0
⎢ 0 k
(λ2 ) 0 ... ⎥
⎢ ⎥ −1 (0)
(4.62) = Ψ ⎢ ⎥Ψ z = Ψ (Λ)k Ψ−1 z(0)
⎣ .... .... ..... .... ⎦
k
0 .... 0 (λn )
where Λ is the diagonal matrix

⎡ ⎤
λ1 0 ..... 0
⎢ 0 λ2 0 ... ⎥
⎢ ⎥
(4.63) Λ=⎢ ⎥
⎣ .... .... ..... .... ⎦
0 .... 0 λn
Now, consider set of n equations
(4.64) Bv(i) = λi v(i) for (i = 1, 2, ....n)
which can be rearranged as

h i
Ψ = (1) (2) (n)
v v .... v
⎡ ⎤
λ1 0 ..... 0
⎢ 0 λ2 0 ... ⎥
⎢ ⎥
(4.65) BΨ = ⎢ ⎥Ψ
⎣ .... .... ..... .... ⎦
0 .... 0 λn
(4.66) or B = ΨΛΨ−1
Using above identity, it can be shown that

¡ ¢k
(4.67) Bk = ΨΛΨ−1 = Ψ (Λ)k Ψ−1
and the solution of equation (6.54) reduces to
(4.68) z(k) = Bk z(0)
and z(k) → 0 as k → ∞ if and only if ρ(B) < 1. The largest magnitude

eigen value, i.e., ρ(B) will eventually dominate and determine the rate at which
z(k) → 0̄. The result proved in this section can be summarized as follows:
© ª
Theorem 3. A sequence of vectors z(k) : k = 0, 1, , 2, .... generated by the
iteration scheme
z(k+1) = Bz(k)
where z ∈Rn and B ∈ Rn × Rn , starting from any arbitrary initial condition
z(0) will converge to limit z∗ = 0̄ if and only if
ρ(B) < 1
Note that computation of eigenvalues is a computationally intensive task.

The following theorem helps derive a sufficient condition for convergence of
linear iterative equations.
Theorem 4. For a n × n matrix B, the following inequality holds for any

induced matrix norm
(4.69) ρ(B) ≤ kBk
Proof: Let λ be eigen value of B and v be the corresponding eigenvector. Then,
we can write
(4.70) kBvk = kλvk = |λ| kvk
By definition of induced matrix norm
(4.71) kBzk ≤ kBk kzk
Thus, |λ| ≤ kBk for any z ∈ Rn . This inequality is true for all λ and this
implies
(4.72) ρ(B) ≤ kBk
Using above theorem, a sufficient condition for convergence of iterative scheme
can be derived as
(4.73) kBk < 1
as ρ(B) ≤ kBk < 1 ⇒ ρ(B) < 1.
The above sufficient condition is more useful from the viewpoint of compu-
tations as kBk1 and kBk∞ can be computed quite easily. On the other hand,
the spectral radius of a large matrix can be comparatively difficult to compute.
4.4.2. Convergence Criteria for Iteration Schemes. The criterion for conver-
gence of iteration equation (4.37) can be derived using results derived above.
The necessary and sufficient condition for convergence of (4.37) can be stated
as
ρ(S −1 T ) < 1
i.e. the spectral radius of matrix S −1 T should be less than one.
The necessary and sufficient condition for convergence stated above requires
computation of eigen values of S −1 T, which is a computationally demanding
task when the matrix dimension is large. If for a large dimensional matrix,
we could check this condition before starting iterations, then we might as well
solve the problem by a direct method rather than using iterative approach to
save computations. Thus, there is a need to derive some alternate criteria
for convergence, which can be checked easily before starting iterations. For
example, using Theorem 3.3, we can obtain sufficient conditions for convergence
° −1 ° ° °
°S T ° < 1 OR °S −1 T ° < 1
1 ∞
which are significantly easy to evaluate than the spectral radius. Also, if the
matrix A has some special properties, such as diagonal dominance or symme-
try and positive definiteness, then we can derive easily computable criteria by
exploiting these properties.
Definition 26. A matrix A is called diagonally dominant if

X
n
(4.74) |aij | < |aii | for i = 1, 2., ...n
j=1(j6=i)
Theorem 5. [5] A sufficient condition for the convergence of Jacobi and

Gauss-Seidel methods is that the matrix A of linear system Ax = b is diagonally
dominant.
Proof: See Appendix.
Theorem 6. [4] The Gauss-Seidel iterations converge if matrix A an sym-

metric and positive definite.
Proof: See Appendix.
Theorem 7. [15] For an arbitrary matrix A, the necessary condition for

the convergence of relaxation method is 0 < ω < 2.
Proof: The relaxation iteration equation can be given as
£ ¤
(4.75) x(k+1) = (D + ωL)−1 [(1 − ω)D − ωU]x(k) + ω b
Defining
(4.76) Bω = (D + ωL)−1 [(1 − ω)D − ωU]
£ ¤
(4.77) det(Bω ) = det (D + ωL)−1 det [(1 − ω)D − ωU]
Now, using the fact that the determinant of a triangular matrix is equal to
multiplication of its diagonal elements, we have
£ ¤
(4.78) det(Bω ) = det D−1 det [(1 − ω)D] = (1 − ω)n
Using the result that product of eigenvalues of Bω is equal to determinant of
Bω ,we have
(4.79) λ1 λ2 ...λn = (1 − ω)n
where λi (i = 1, 2...n) denote eigenvalues of Bω .
(4.80) |λ1 λ2 ...λn | = |λ1 | |λ2 | .... |λn | = |(1 − ω)n |
It is assumed that iterations converge. Now, convergence criterion requires
(4.81) λi (Bω ) < 1 for i = 1, 2, ...n
(4.82) ⇒ |λ1 | |λ2 | .... |λn | < 1

(4.83) ⇒ |(1 − ω)n | < 1
This is possible only if
(4.84) 0<ω<2
The optimal or the best choice of the ω is the one that makes spectral radius
ρ(Bω )smallest possible and gives fastest rate of convergence.
Theorem 8. [5] A sufficient condition for the convergence of relaxation

methods when matrix A of linear system Ax = b is strictly diagonally dominant
is that 0 < ω ≤ 1.
Proof: Left to reader as an exercise.
Theorem 9. [5]For an symmetric and positive definite matrix A, the relax-

ation method converges if and only if 0 < ω < 2.
The theorems 3 and 6 guarantees convergence of Gauss-Seidel method or

relaxation method when matrix A is symmetric and positive definite. Now, what
do we do if matrix A in Ax = b is not symmetric and positive definite? Can we
transform the problem to another equivalent problem such that conditions for
above theorem are satisfied? Well, if matrix A is non-singular, we can multiply
both the sides of the equation by AT and transform the original problem as
¡ T ¢ ¡ ¢
(4.85) A A x = AT b
¡ ¢
The resulting matrix AT A is always symmetric and positive definite as
¡ ¢
(4.86) xT AT A x = (Ax)T (Ax) > 0 for any x 6= 0
Now, for the transformed problem, we are guaranteed convergence if we use
Gauss-Seidel method. Thus, instead of working with the original system of
equations Ax = b, it is always better to formulate the iteration scheme for the
transformed problem (4.85).
Example 41. Consider system Ax = b where

" #
2 −1
(4.87) A=
−1 2
For Jacobi method
" #
0 1/2
(4.88) S −1 T =
1/2 0
(4.89) ρ(S −1 T ) = 1/2
Thus, the error norm at each iteration is reduced by factor of 0.5

For Gauss-Seidel method
" #
0 1/2
(4.90) S −1 T =
0 1/4
(4.91) ρ(S −1 T ) = 1/4
Thus, the error norm at each iteration is reduced by factor of 1/4. This implies
that, for the example under consideration
(4.92) 1 Gauss Seidel iteration ≡ 2 Jacobi iterations
For relaxation method,

" #−1 " #
2 0 2(1 − ω) ω
(4.93) S −1 T =
−ω 2 0 2(1 − ω)
⎡ ⎤
(1 − ω) (ω/2)
(4.94) = ⎣ ω2 ⎦
(ω/2)(1 − ω) (1 − ω + )
4
−1 2
(4.95) λ1 λ2 = det(S T ) = (1 − ω)
(4.96) λ1 + λ2 = trace(S −1 T )
ω2
(4.97) = 2 − 2ω +
4
Now, if we plot ρ(S −1 T ) v/s ω, then it is observed that λ1 = λ2 at ω = ω opt .From
equation (4.95), it follows that
(4.98) λ1 = λ2 = ωopt − 1
at optimum ω.Now,
(4.99) λ1 + λ2 = 2(ω opt − 1)

ω 2opt
(4.100) = 2 − 2ωopt +
4
√
(4.101) ⇒ ω opt = 4(2 − 3) ∼
= 1.07
(4.102) ⇒ ρ(S −1 T ) = λ1 = λ2 ∼
= 0.07
This is a major reduction in spectral radius when compared to Gauss-Seidel

method. Thus, the error norm at each iteration is reduced by factor of 1/16
(∼
= 0.07) if we choose ω = ωopt .
Example 42. Consider system Ax = b where

⎡ ⎤ ⎡ ⎤
4 5 9 1
⎢ ⎥ ⎢ ⎥
(4.103) A=⎣ 7 1 6 ⎦ ; b =⎣ 1 ⎦
5 2 9 1
If we use Gauss-Seidel method to solve for x, the iterations do not converge as
⎡ ⎤−1 ⎡ ⎤
4 0 0 0 −5 −9
−1 ⎢ ⎥ ⎢ ⎥
(4.104) S T = ⎣ 7 1 0 ⎦ ⎣ 0 0 −6 ⎦
5 2 9 0 0 0
(4.105) ρ(S −1 T ) = 5.9 > 1
Now, let us modify the problem by pre-multiplying Ax = b by AT on both the

¡ ¢ ¡ ¢
sides, i.e. the modified problem is AT A x = AT b . The modified problem
becomes
⎡ ⎤ ⎡ ⎤
90 37 123 16
⎢ ⎥ ⎢ ⎥
(4.106) AT A = ⎣ 37 36 69 ⎦ ; AT b = ⎣ 8 ⎦
123 69 198 24
The matrix AT A is symmetric and positive definite and, according to Theorem

3, the iterations should converge if Gauss-Seidel method is used. For the trans-
formed problem, we have
⎡ ⎤−1 ⎡ ⎤
90 0 0 0 −37 −123
⎢ ⎥ ⎢ ⎥
(4.107) S −1 T = ⎣ 37 36 0 ⎦ ⎣ 0 0 −69 ⎦
123 69 198 0 0 0
(4.108) ρ(S −1 T ) = 0.96 < 1
and within 220 iterations (termination criterion 1 × 10−5 ), we get following

solution
⎡ ⎤
0.0937
⎢ ⎥
(4.109) x = ⎣ 0.0312 ⎦
0.0521
which is close to the solution
⎡ ⎤
0.0937
⎢ ⎥
(4.110) x∗ = ⎣ 0.0313 ⎦
0.0521
computed as x∗ = A−1 b.
From example 3, we can clearly see that the rate of convergence depends on
ρ(S −1 T ). From analysis of some simple problems, we can generate the following
table [5]
Method Convergence Rate No. of iterations

Jacobi O(1/2n2 ) O(2n2 )
Gauss_Seidel O(1/n2 ) O(n2 )
Relaxation with optimal ω O(2/n) O(n/2)
5. Well Conditioned and Ill-Conditioned Problems

One of the important issue in computing solutions of large dimensional linear
system of equations is the round-off errors caused by the computer. Some ma-
trices are well conditioned and the computations proceed smoothly while some
are inherently ill conditioned, which imposes limitations on how accurately the
system of equations can be solved using any computer or solution technique.
Before we discuss solution techniques for (1.5), we introduce measures for as-
sessing whether a given system of linear algebraic equations is inherently ill
conditioned or well conditioned.
Normally any computer keeps a fixed number of significant digits. For ex-
ample, consider a computer that keeps only first three significant digits. Then,
adding
0.234 + 0.00231 → 0.236
results in loss of smaller digits in the smaller number. When a computer can
commits millions of such errors in a complex computation, the question is,
how do these individual errors contribute to the final error in computing the
solution? As mentioned earlier, the set of all m × n matrices together with real
scalars defines a linear vector space. Suppose we solve for Ax = b using LU
decomposition, the elimination algorithm actually produce approximate factors
L0 and U 0 . Thus, we end up solving the problem with a wrong matrix, i.e.
(5.1) A + δA = L0 U 0
instead of right matrix A = LU. In fact, due to round off errors inherent in any
computation using computer, we actually end up solving the equation
(5.2) (A + δA)(x+δx) = b+δb
The question is, how serious are the errors δx in solution x, due to round off
errors in matrix A and vector b? Can these errors be avoided by rearranging
5. WELL CONDITIONED AND ILL-CONDITIONED PROBLEMS 89
computations or are the computations inherent ill-conditioned? In order to an-

swer these questions, we need to develop some quantitative measure for matrix
conditioning.
The following section provides motivation for developing a quantitative mea-
sure for matrix conditioning. In order to develop such a index, we need to define
the concept of norm of a m×n matrix. The formal definition of matrix norm and
method of computing 2-norm of a matrix is explained in the later sub-sections.
Concept of condition number of a matrix is introduced next.
5.1. Motivation for looking at Matrix Conditioning. In many

situations, if the system of equations under consideration is numerically well
conditioned, then it is possible to deal with the menace of round off errors by
re-arranging the computations. If the system of equations is inherently an ill
conditioned system, then the rearrangement trick does not help. Let us try
and understand this by considering to simple examples and a computer that
keeps only three significant digits.
Consider the system (System-1)
" #" # " #
0.0001 1 x1 1
(5.3) =
1 1 x2 2
If we proceed with Guassian elimination without maximal pivoting , then the
first elimination step yields
" #" # " #
0.0001 1 x1 1
(5.4) =
0 −9999 x2 −9998
and with back substitution this results in
(5.5) x2 = 0.999899
which will be rounded off to
(5.6) x2 = 1
in our computer which keeps only three significant digits. The solution then
becomes
h iT h i
(5.7) x1 x2 = 0.0 1
However, using maximal pivoting strategy the equations can be rearranged as

" #" # " #
1 1 x1 2
(5.8) =
0.0001 1 x2 1
and the Guassian elimination yields

" #" # " #
1 1 x1 2
(5.9) =
0 0.9999 x2 0.9998
and again due to three digit round off in our computer, the solution becomes
h iT h i
x1 x2 = 1 1
Thus, A is a well conditioned numerically only thing that can cause calculation
blunders is wrong pivoting strategy. If maximum pivoting is used then natural
resistance to round off the errors is no longer compromised.
Now, in order to understand difficulties with ill conditioned systems, con-
sider another system (System-2)
" #" # " #
1 1 x1 2
(5.10) =
1 1.0001 x2 2
By Guassian elimination
" #" # " # " # " #
1 1 x1 2 x1 2
(5.11) = =⇒ =
0 0.0001 x2 0 x2 0
If we change R.H.S. of the system 2 by a small amount
" #" # " #
1 1 x1 2
(5.12) =
1 1.0001 x2 2.0001
" #" # " # " # " #
1 1 x1 2 x1 1
(5.13) = =⇒ =
0 0.0001 x2 0.0001 x2 1
Note that change in the fifth digit of second element of vector b was amplified
to change in the first digit of the solution.
Thus, matrix A in the system 2 is ill conditioned as it is near singular.
Hence, no numerical method can avoid sensitivity of the system 2 to small
permutations. The ill conditioning can be shifted from one plane to another
but it cannot be eliminated.
5.2. Induced Matrix Norms. We have already mentioned that set of all
m × n matrices with real entries (or complex entries) can be viewed a linear
vector space. In this section, we introduce the concept of induced norm of
a matrix, which plays a vital role in the numerical analysis. A norm of a
matrix can be interpreted as amplification power of the matrix. To develop a
numerical measure for ill conditioning of a matrix, we first have to quantify this
amplification power of the matrix.
Definition 27. (Induced Matrix Norm): The induced norm of a m × n

matrix A is defined as mapping from Rm × Rn → R+ such that
Max kAxk
(5.14) kAk =
x 6= 0 kxk
In other words, kAk bounds the amplification power of the matrix i.e.
(5.15) kAxk ≤ kAk kxk for all x ∈ Rn
The equality holds for at least one non zero vector x ∈ Rn . An alternate way
of defining matrix norm is as follows
Max
(5.16) kAk = kAxk
x 6= 0, kxk ≤ 1
The following conditions are satisfied for any matrices A, B ∈ Rm × Rn
(1) kAk > 0 if A 6= [0] and k[0]k = 0
(2) kαAk = |α|.kAk
(3) kA + Bk ≤ kAk + kBk
(4) kABk ≤ kAk kBk
The induced norms, i.e. norm of matrix induced by vector norms on Rm
and Rn , can be interpreted as maximum gain or amplification factor of the
matrix.
Computation of Matrix Norms
Consider 2-norm of a matrix, which can be defined as
||Ax||2
(5.17) ||A||2 = max
x6= 0 ||x||2
Squaring both sides

||Ax||22
(5.18) ||A||22 = max
x6= 0 ||x||22
(Ax)T (Ax)
= max
x6= 0 (xT x)
xT Bx
= max T
x6= 0 (x x)
Where B = AT A is a symmetric and positive definite matrix. Positive definite-

ness of matrix B implies
−
(5.19) xT Bx > 0 if x 6= 0 and xT Bx = 0 if and only if x = 0
Obviously, if A is nonsingular
(5.20) xT Bx = xT AT Ax = (Ax)T (Ax) > 0 if x 6= 0
Now, a positive definite symmetric matrix can be diagonalized as
(5.21) B = ΨΛΨT
Where Ψ is matrix with eigen vectors as columns and Λ is the diagonal matrix
with eigenvalues of B (= AT A) on the diagonal. Note that in this case Ψ is
unitary matrix ,i.e.,
(5.22) ΨΨT = I or ΨT = Ψ−1
and eigenvectors are orthogonal. Using the fact that Ψ is unitary, we can write
(5.23) xT x = xT ΨΨT x = yT y
xT Bx yT Λy
(5.24) or T
= T
where y = ΨT x
(x x) (y y)
Suppose eigenvalues λi of AT A are numbered such that
(5.25) 0 ≤ λ1 ≤ λ2 ≤ .................. ≤ λn
Then
yT Λy (λ1 y12 + ................ + λn yn2 )
(5.26) = ≤ λn
(yT y) (y12 + ................. + yn2 )
This implies
yT Λy xT Bx xT (AT A)x
(5.27) = = ≤ λn
(yT y) (xT x) (xT x)
The equality holds only at the corresponding eigenvector of AT A, i.e.,
£ (n) ¤T T £ (n) ¤T
v (A A)v(n) v λn v(n)
(5.28) T
= T
= λn
[v(n) ] v(n) [v(n) ] v(n)
Thus,
(5.29) ||A||22 = max ||Ax||2 /||x||2 = λmax (AT A)

x6= 0
or
(5.30) ||A||2 = [λmax (AT A)]1/2
where λmax (AT A) denotes maximum magnitude eigenvalue or spectral radius of

AT A. Other commonly used matrix norms are
• 1-norm: Maximum over column sums
" n #
max X
(5.31) ||A||1 = |aij |
1 ≤ j ≤ n i=1
• ∞−norm: Maximum over row sums

" n #
max X
(5.32) ||A||∞ = |aij |
1 ≤ i ≤ n j=1
Remark 2. There are other matrix norms, such as Frobenious norm, which
are not induced matrix norms. Frobenious norm is defined as
" n n #1/2
XX
||A||F = |aij |2
i=1 j=1
5.3. Condition Number: A Measure to Quantify Matrix Ill-conditioning.

Consider system of equations given as Ax = b. We examine two situations: (a)
errors in representation of vector b and (b) errors in representation of matrix
A.
5.3.1. Case: Perturbation in vector b. Consider the case when there is a
change in b to b + δb. Such an error might come from experimental data or
from round off error. Such a perturbation causes a change in solution from x
to x + δx.
(5.33) A(x + δx) = b + δb
By subtracting Ax = b from the above equation we have :
(5.34) Aδx = δb
To develop a measure for conditioning of a matrix we compare relative change/error
in solution,i.e. ||δx|| / ||x|| to relative change in b ,i.e. ||δb|| / ||b||. Now
(5.35) δx = A−1 δb ⇒ ||δx|| ≤ ||A−1 || ||δb||
Also,
(5.36) Ax = b ⇒ ||b|| = ||Ax|| ≤ ||A|| ||x||
Combining the above two inequalities, we can write
(5.37) ||δx|| ||b|| ≤ ||A−1 || ||A|| ||x|| ||δb||
||δx|| ||δb||
(5.38) ⇒ ≤ (||A−1 || ||A||)
||x|| ||b||
The above inequality holds for every b and δb vector. The number
(5.39) C(A) = ||A−1 || ||A||
is called as condition number of matrix A. Thus the condition number
||δx||/||x||
(5.40) ≤ C(A) = ||A−1 || ||A||
||δb||/||b||
gives an upper bound on the possible amplification of errors in b while comput-

ing the solution.
5.3.2. Case: Perturbation in matrix A. Suppose ,instead of solving for Ax =
b due to truncation errors we end up solving
(5.41) (A + δA)(x + δx) = b
Then by subtracting Ax = b from the above equation we obtain
(5.42) Aδx + δA(x + δx) = 0
or
(5.43) δx = −A−1 δA(x + δx)
Taking norm on both the sides, we have
(5.44) ||δx|| = ||A−1 δA(x + δx)||
(5.45) or ||δx|| ≤ ||A−1 || ||δA|| ||x + δx|
(5.46) ||δx||/||x + δx|| ≤ (||A−1 || ||A||) ||δA||/||A||
(5.47) ||δx||/||x + δx||/||δA||/||A|| ≤ C(A) = ||A−1 || ||A||
Again,the condition number gives an upper bound on % change in solution to
% error A. In simple terms, condition number of a matrix tells us how serious
is the error in solution of Ax = b due to the truncation or round off errors in a
computer. These inequalities mean that round off error comes from two sources
• Inherent or natural sensitivity of the problem,which is measured by
C(A)
• Actual errors δb or δA.
It has been shown that the maximum pivoting strategy is adequate to keep
(δA) in control so that the whole burden of round off errors is carried by the
condition number C(A). If condition number is high (>1000), the system is ill
conditioned and is more sensitive to round off errors. If condition number is
low (<100) system is well conditioned and you should check your algorithm for
possible source of errors.
5.3.3. Computations of condition number. Let λn denote the largest magni-
tude eigenvalue of matrix A and λ1 denote the smallest magnitude eigen value
of A. Then,
(5.48) ||A||22 = ρ(AT A) = λn
Also,
£ ¤
(5.49) ||A−1 ||22 = ρ[(A−1 )T A−1 ] = ρ (AAT )−1
This follows from identity
(A−1 A)T = I
AT (A−1 )T = I
(5.50) (AT )−1 = (A−1 )T
Now, if λ is eigenvalue of AT A and v is the corresponding eigenvector, then
(5.51) (AT A)v = λv

(5.52) AAT (Av) = λ(Av)
λ is also eigenvalue of AAT . Thus, we can write

£ ¤ £ ¤
(5.53) ||A−1 ||22 = ρ (AAT )−1 = ρ (AT A)−1
Also, since AAT is a symmetric positive definite matrix, we can diagonalize it

as
(5.54) AT A = ΨΛΨT
£ ¤
⇒ (AT A)−1 = [ΨΛΨT ]−1 = (ΨT )−1 Λ−1 Ψ−1 = ΨΛ−1 ΨT
as Ψ is a unitary matrix. Thus, if λ is eigen value of AT A then 1/λ is eigen value

of (AT A)−1 . If λ1 smallest eigenvalue of AT A then 1/λ1 is largest magnitude
eigenvalue of AT A
(5.55) ⇒ ρ[(AT A)−1 ] = 1/ λ1
Thus, the condition number of matrix A can be computed using 2-norm as
(5.56) C(A) = ||A|| ||A−1 || = (λn /λ1 )1/2
where λn and λ1 are largest and smallest magnitude eigenvalues of AT A.
Example 43. Consider matrix

⎡ ⎤
1 2 3
⎢ ⎥
(5.57) A=⎣ 4 5 6 ⎦
7 8 9
This ordinary looking matrix is near singular with eigen values (computed using
MATLAB)
(5.58) λ1 = 16.117 ; λ2 = −1.1168 ; λ3 = −1.0307 × 10−15

has the condition number of C(A) = 3.8131e + 016. If we attempt to compute

inverse of this matrix using MATLAB (which has arguably one of the best linear
equations solvers) we get following result
⎡ ⎤
−0.4504 0.9007 −0.4504
⎢ ⎥
(5.59) A−1 = 1016 × ⎣ 0.9007 −1.8014 0.9007 ⎦
−0.4504 0.9007 −0.4504
with a warning: ’ Matrix is close to singular or badly scaled. Results may be

inaccurate.’ The difficulties in computing inverse of this matrix are apparent if
we further compute product A × A−1 , which yields
⎡ ⎤
0 −4 0
⎢ ⎥
(5.60) A × A−1 = ⎣ 0 8 0 ⎦
4 0 0
On the other hand, consider matrix

⎡ ⎤
1 2 1
⎢ ⎥
(5.61) B = 10−17 × ⎣ 2 1 2 ⎦
1 1 3
with eigen values
(5.62) λ1 = 4.5616 × 10−17 ; λ2 = −1 × 10−17 ; λ3 = 4.3845 × 10−18
Looking at small magnitude eigenvalues (near singularity), we may anticipate

trouble in computations. However, the condition number of this matrix is C(A) =
16.741. If we proceed to compute of B −1 and product B × B −1 using MATLAB,
we get
⎡ ⎤
0.5 1.5 −2.5
⎢ ⎥
(5.63) B = 1017 × ⎣ 0.5 −0.5 0.5 ⎦
−0.5 −0.5 1/5
and B ∗ B −1 = I, i.e. identity matrix. Thus, it is important to realize that

each system of linear equations has a inherent character, which can be quanti-
fied using the condition number of the associated matrix. The best of the linear
equation solvers cannot overcome the computational difficulties posed inherent
ill conditioning of a matrix. As a consequence, when such ill conditioned matri-
ces are encountered, the results obtained using any computer or any solver are
unreliable.
6. SOLVING NONLINEAR ALGEBRAIC EQUATIONS 97
6. Solving Nonlinear Algebraic Equations

Consider set of n nonlinear equations
(6.1) fi (x) = 0 ; i = 1, ...., n

(6.2) or F (x) = 0
which have to be solved simultaneously. Among the various approaches

available for solving these equations, the method of successive substitution and
Newton Raphson method are based on successive solutions of subproblems in-
volving linear algebraic equations of type (1.5).
6.1. Successive Substitution [5]. In many situations, equation (6.2) can

be rearranged as
(6.3) Ax = G(x)
h iT
(6.4) G(x) = g1 (x) g2 (x) .... gn (x)
such that the solution of equation (6.3) is also solution of equation (6.2). The
nonlinear Equation (6.3) can be used to formulate iteration sequence of the form
£ ¤
(6.5) Ax(k+1) =G x(k)
£ ¤
Given a guess x(k) ,the R.H.S. is a fixed vector, say b(k) = G x(k) , and compu-
tation of the next guess x(k+1) essentially involves solving the linear algebraic
equation
Ax(k+1) = b(k)
at each iteration. Thus, the set of nonlinear algebraic equations is solved by for-
mulating a sequence of linear sub-problems. Computationally efficient method
of solving such sequence of linear problems would be to use LU decomposition
of matrix A.
A special case of interest is when matrix A = I in equation (6.5).In this
case, if the set of equations given by (6.3) can be rearranged as
(6.6) xi = gi (x) ; (i = 1, ...........n)
then method of successive substitution can be arranged as

• Jacobi-Iterations
(k+1)
xi = gi [x(k) ] ; ( i = 1, ......, n)
• Gauss Seidel Iterations

(k+1) (k+1) (k+1) (k)
(6.7) xi = gi [x1 , ........xi−1 , xi , ............xn(k) ]
i = 1, .........., n
• Relaxation Method
(k+1) (k) (k)
xi = xi + ω[gi (xk ) − xi ]
A popular method of this type is Wegstein iterations. Given initial
guess vector x(0)
(1)
xi = gi (x(0) ) ; ( i = 1, ............., n)
£ ¤
(k) gi (x(k) ) − gi (x(k−1) )
si = h i
(k) (k−1)
xi − xi
(k)
(k) si
(6.8) ωi =h i
(k)
1 − si
(k+1) (k) (k)
xi = gi (x(k) ) + ωi [xi − gi (x(k) )]
i = 1, ........., n
The iterations can be terminated when
° (k+1) °
(6.9) °x − x(k) ° < ε
Example 44. Consider the ODE-BVP describing steady state conditions in

a tubular reactor with axial mixing (TRAM) in which an irreversible 2nd order
reaction is carried out.
1 d2 C dC
(6.10) − − DaC 2 = 0 (0 ≤ z ≤ 1)
P e dz 2 dz
dC
(6.11) = P e(C − 1) at z = 0;
dz
dC
(6.12) = 0 at z = 1;
dz
Using method of finite difference, we get following set of n+1 nonlinear algebraic
equations
1 Ci+1 − 2Ci + Ci−1 Ci+1 − Ci−1
(6.13) 2 − = DaCi2
Pe (∆z) 2 (∆z)
(6.14) or αCi+1 − βCi + αCi−1 = DaCi2
i = 1, 2, ...n − 1
where
µ ¶ µ ¶
1 1 2
(6.15) α= 2 + ; β= ;
(∆z) P e 2 (∆z) P e (∆z)2
C1 − C0
(6.16) = P e(C0 − 1)
∆z
Cn − Cn−1
(6.17) = 0
∆z
the above set of nonlinear algebraic equations can be arranged as
(6.18)
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
−(1 + ∆zP e) 1 0. ..... .... 0 C0 −P e (∆z)
⎢ α −β α .... ..... ..... ⎥ ⎢ C1 ⎥ ⎢ DaC12 ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎢ .... ..... ..... ..... ..... 0. ⎥ ⎢ . ⎥=⎢ ..... ⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎣ ..... ..... ..... ..... −β α ⎦ ⎣ . ⎦ ⎣ DaCn−12 ⎦
0 ..... ..... ... −1 1 Cn 0
If we define
h iT
(6.19) x= C0 C1 ... Cn
then, equation (6.18) is of the form Ax = G(x).
6.2. Newton Raphson Method. For a general set of simultaneous equa-

tions F (x) =0, it may not be always possible to transform to form Ax = G(x) by
simple rearrangement of equations. Even when it is possible, the iterations may
not converge. When the function vector F (x) is once differentiable, Newton-
Raphson method provides a way to transform an arbitrary set of equations
F (x) =0 to form Ax = G(x) using Taylor series expansion.
The main idea behind the Newton-Raphson method is solving the set of non-
linear algebraic equations (6.2) by formulating a sequence of linear subproblems
of type
(6.20) A(k) ∆x(k) = b(k)

(6.21) x(k+1) = x(k) + 4x(k) ; k = 0, 1, 2, ....
© ª
in such a way that sequence x(k) : k = 0, 1, 2, .... converges to solution of
equation (6.2). Suppose x∗ is a solution such that F (x∗ ) = 0. If each function
fi (x) is continuously differentiable, then, in the neighborhood of x∗ we can

approximate its behavior by Taylor series, as
£ ¡ ¢¤
(6.22) F(x∗ ) = F x(k) + x∗ −x(k)
∙ ¸
(k) ∂F £ (k) ¤
(6.23) = F (x ) + ∆x + ....
∂x x=x(k)
Since x(k) is assumed to be close to x∗ , second and higher order terms can be
neglected. Defining
∙ ¸
(k) ∂F
(6.24) J =
∂x x=x(k)
(6.25) F (k) = F (x(k) )
we can solve for
(6.26) F (x∗ ) =
e F (k) + J (k) 4 x(k) = 0
Now 4x(k) can be interpreted as the error committed in approximating x∗ by
x(k) . We can obtain an improved approximations x(k+1) of x∗ as
(6.27) J (k) 4 x(k) = −F (k)
(6.28) x(k+1) = x(k) + 4x(k)
Alternatively, iterations can be formulated by solving
£ (k)T (k) ¤
(6.29) J J 4 x(k) = −J (k)T F (k)
(6.30) x(k+1) = x(k) + 4x(k)
£ ¤
where J (k)T J (k) is symmetric and positive definite matrix. Iterations can be
terminated when the following convergence criteria is satisfied
||F (x(k+1) )|| < ε
Often Newton Raphson method finds a large step 4x(k) such that approximation
of the function vector by linear term in Taylor series is not valid in interval
£ (k) (k) ¤
x , x + ∆x(k) . In order to alleviate this problem,we find a corrected x(k+1)
as
(6.31) x(k+1) = x(k) + λ(k) 4 x(k)
where 0 < λ(k) ≤ 1 is chosen such that
(6.32) ||F (x(k+1) )|| < ||F (x(k) )||
In practical problems this algorithm converges faster than the one that uses
λ(k) = 1.
Note that the Newton-Raphson iteration equation can be expressed in form

(6.3) as follows
(6.33) x(k+1) = x(k) − λ[(∂F/∂x)(k) ]−1 F (x(k) ) = G(x(k) )
i.e.
G(x) = x − λ[∂F/∂x]−1 F (x)
It is easy to see that at the solution x = x∗ of F (x), i.e. when F (x∗ ) = 0, the
iteration equation 6.33 has a fixed point x∗ = G(x∗ ).
Modified Newton Raphson Algorithm:
INITIALIZE: x(0) , ε1 , ε2 ,α, k, kmax, δ 2
° °
δ 1 = °F (0) °
WHILE [(δ 1 > ε1 ) AND (δ 2 > ε2 ) AND (k < kmax )]
Solve
£ ¤
J (k) 4 x(k) = −F (k) (OR J (k)T J (k) 4 x(k) = −J (k)T F (k) )
λ(0) = 1, j = 0
x(k+1) = x(k) + λ(0) 4 x(k)
° °
δ 1 = °F (k+1) °
£ ° °¤
WHILE δ 1 > °F (k) )°
λ(j) = αλ(j−1)
x(k+1) = x(k) + λ(j) 4 x(k)
° °
δ 1 = °F (k+1) °
END WHILE
δ 2= ||x(k+1) − x(k) || / ||x(k+1) ||
k =k+1
END WHILE
6.2.1. Quasi-Newton Method with Broyden Update. A major difficulty with
Newton Raphson method is that it requires calculation of Jacobian at each iter-
ation. The quasi-Newton methods try to overcome this difficulty by generating
approximate successive Jacobians using function vectors evaluated at previous
iterations. While moving from iteration k to (k + 1), if ||x(k+1) − x(k) || is not
too large, then it can be argued that J (k+1) is ”close” to J (k) . Under such
situation ,we can use the following rank-one update of the Jacobian
(6.34) J (k+1) = J (k) + y(k) [z(k) ]T
Here, y(k) and z(k) are two vectors that depend on x(k) , x(k+1) , F (k) and F (k+1) .
To arrive at the update formula, consider Jacobian J (k) that produces step 4x(k)
as
(6.35) J (k) 4 x(k) = −F (k)
(6.36) x(k+1) = x(k) + 4x(k)
Step x(k+1) predicts a function change
(6.37) ∆F (k) = F (k+1) − F (k)
We impose the following two conditions to obtain estimate of J (k+1) .

(1) In the direction perpendicular to 4x(k) , our knowledge about F is
maintained by new Jacobian estimate J (k+1) . This means for a vector,
say r , if [∆x(k) ]T r = 0, then
(6.38) J (k) r = J (k+1) r
In other words, both J (k) and J (k+1) will predict some change in direc-
tion perpendicular to ∆x(k) .
(2) J (k+1) predicts for 4x(k) , the same ∆F (k) in linear expansion, i.e.,
(6.39) F (k+1) = F (k) − J (k+1) 4 x(k)
or
(6.40) J (k+1) 4 x(k) = ∆F (k)
Now, for vector r perpendicular to4x(k) , we have
(6.41) J (k+1) r = J (k) r + y (k) [z(k) ]T r
As
(6.42) J (k+1) r = J (k) r
We have
(6.43) y (k) [z(k) ]T r = 0
Since 4x(k) is perpendicular to r, we can choose z(k) = 4x(k) . Substituting this

choice of z(k) in equation (6.34) and post multiplying equation (6.34) by 4x(k) ,
we get
(6.44) J (k+1) 4 x(k) = J (k) 4 x(k) + y (k) [4x(k) ]T 4 x(k)
Using equation (3), we have
(6.45) ∆F (k) = J (k) 4 x(k) + y (k) [4x(k) ]T 4 x(k)
which yields
£ ¤
(k) ∆F (k) − J (k) 4 x(k)
(6.46) y =
[[4x(k) ]T 4 x(k) ]
Thus, the Broyden’s update formula for the Jacobian is

[∆F (k) − J (k) 4 x(k) ][4x(k) ]T
(6.47) J (k+1) = J (k) +
[[4x(k) ]T 4 x(k) ]
This can be further simplified as
(6.48) ∆F (k) − J (k) 4 x(k) = F (k+1) − (F (k) + J (k) 4 x(k) ) = F (k+1)
Thus, Jackobian can be updated as
1 £ (k+1) ¤
(6.49) J (k+1) = J (k) + (k) T (k)
F [4x(k) ]T
[[4x ] 4 x ]
Broyden’s update derived by an alternate approach yields following formula [8]
1 £ (k) ¤ (k) T (k)
(6.50) J (k+1) = J (k) − (k) T (k) J ∆F (k)
− p(k)
[p ] J
[[p ] J ∆F (k) ]
6.3. Convergence of Iteration Schemes. Either by successive sub-
stitution approach or Newton -Raphson method, we generate an iteration se-
quence
(6.51) x(k+1) = G(x(k) )
which has a fixed point
(6.52) x∗ = G(x∗ )
at solution of F (x∗ ) = 0.Once we formulate an iteration scheme of the form
(6.51), we start from some initial guess solution, say x(0) , and generates a se-
© ª
quence of vectors x(k) ∈ Rn : k = 1, 2, 3, ... . The iterations are terminated
° ° © ª
when °x(k+1) − x(k) ° ≤ ε where ε is a small number, provided x(k) forms a
convergent sequence. The main concern while dealing with such equations is
whether this sequence of vector converges to some limit, say x∗ . In mathematical
terms, we want to know conditions under which the iterative sequences start-
ing from arbitrary initial guesses will converge to a fixed point x∗ of equation
(6.5) such that
(6.53) x∗ = G [x∗ ]
When the equation (6.5) is linear, i.e. it can be expressed in the form
(6.54) x(k+1) = Bx(k)
where B is n×n matrix, then necessary and sufficient conditions for convergence
can be derived using eigenvalue analysis as shown earlier. When F [x] is a
nonlinear, contraction mapping theorem is used to derive sufficient conditions
for convergence. We discuss both these cases in the following subsections.
6.3.1. Contraction Mapping Principle. Contraction mapping theorem devel-

ops sufficient conditions for convergence of general nonlinear iterative equation
(6.5). Unlike the eigenvalue analysis used for analysis of its linear counterpart,
this theorem is more useful for developing understanding of convergence process.
Consider general nonlinear iteration equation (6.5) which defines a mapping
from a Banach space X into itself.
Definition 28. (Contraction Mapping): An operator G : X → X given
by equation (6.5), mapping a Banach space X into itself, is called a contraction
mapping of closed ball
© ° ° ª
U (x(0) , r) = x ∈ X : °x − x(0) ° ≤ r
if there exists a real number θ (0 ≤ θ < 1) such that
° ¡ (1) ¢ ¡ ¢° ° °
°G x − G x(2) ° ≤ θ °x(1) − x(2) °
for all x(1) , x(2) ∈ U (x(0) , r). The quantity θ is called contraction constant of G
on U(x(0) , r).
When the map F (.) is differentiable, an exact characterization of the con-
traction property can be developed.
Lemma 3. Let the operator G(.) on a Banach space X be differentiable in
U(x(0) , r). Operator G(.) is a contraction of U(x(0) , r) if and only if
° °
° ∂G °
° ° (0)
° ∂x ° ≤ θ < 1 for every x ∈ U (x , r)
where k.k is any induced operator norm.
The contraction mapping theorem is stated next. Here, x(0) refers to the
initial guess vector in the iteration process given by equation (6.5).
Theorem 10. [14, 9] If G(.) maps U (x(0) , r) into itself and G(.) is a con-
traction mapping on the set with contraction constant θ, for
r ≥ r0
1 ° £ ¤ °
°G x(0) − x(0) °
r0 =
1−θ
then:
(1) G(.) has a fixed point x∗ in U(x(0) , r0 ) such that x∗ = G(x∗ )
(2) x∗ is unique in U(x(0) , r)
£ ¤
(3) The sequence x(k) generated by equation x(k+1) = G x(k) converges to
x∗ with
° (k) ° ° °
°x − x∗ ° ≤ θk °x(0) − x∗ °
(4) Furthermore, the sequence x(k) generated by equation

£ ¤
x(k+1) = G x(k) starting from any initial guess x(0) ∈ U(x(0) , r0 )
converges to x∗ with
° (k) ° ° °
°x − x∗ ° ≤ θk °x(0) − x∗ °
The proof of this theorem can be found in Rall [14] and Linz [9].
Example 45. [9] Consider simultaneous nonlinear equations

1 1
(6.55) z + y2 =
4 16
1 1
(6.56) sin(z) + y =
3 2
We can form an iteration sequence
1 1 ¡ (k) ¢2
(6.57) z (k+1) = − y
16 4
1 1
(6.58) y (k+1) = − sin(z (k) )
2 3
Using ∞−norm In the unit ball U(x(0) = 0, 1) in the neighborhood of origin, we
have
° ¡ (i) ¢ ¡ ¢°
°G x − G x(j) °
∞
µ ¯ ¶
¡
1 ¯ (i) 2¢ ¡ ¢¯ ¯
(j) 2 ¯ 1 ¯ (i)
¯
(j) ¯
(6.59) = max ¯ y − y ¯ , sin(x ) − sin(x )
4 3
µ ¶
1 ¯¯ (i) ¯ ¯
(j) ¯ 1 ¯ (i)
¯
(j) ¯
(6.60) ≤ max y −y , x −x
4 3
1°°x(i) − x(j) °
°
(6.61) ≤ ∞
2
Thus, G(.) is a contraction map with θ = 1/2 and the system of equation has a
unique solution in the unit ball U(x(0) = 0, 1) i.e. −1 ≤ x ≤ 1 and −1 ≤ y ≤ 1.
The iteration sequence converges to the solution.
Example 46. [9] Consider system

(6.62) x − 2y 2 = −1
(6.63) 3x2 − y = 2
which has a solution (1,1). The iterative method

¡ ¢2
(6.64) x(k+1) = 2 y (k) − 1
¡ ¢2
(6.65) y (k+1) = 3 x(k) − 2
is not a contraction mapping near (1,1) and the iterations do not converge even if
we start from a value close to the solution. On the other hand, the rearrangement
q
(6.66) x(k + 1) = (y (k) + 2)/3
q
(k+1)
(6.67) y = (x(k) + 1)/2
is a contraction mapping and solution converges if the starting guess is close to

the solution.
6.3.2. Convergence Criteria for Iteration Schemes. Defining error
(6.68) e(k+1) = x(k+1) − x∗ = G(x(k) ) − G(x∗ )
and using Taylor series expansion, we can write
(6.69) G(x∗ ) = G[x(k) − (x(k) − x∗ )]

∙ ¸
(k) ∂G
(6.70) ' G(x ) − (x(k) − x∗ )
∂x x=x(k)
Substituting in (6.68)
∙ ¸
(k+1) ∂G
(6.71) e = e(k)
∂x x=x(k)
where
e(k) = x(k) − x∗
and using definition of induced matrix norm, we can write
°∙ ¸ °
||e(k+1) || ° ° ∂G °
°
(6.72) < ° ∂x °
||e(k) || x=x(k)
It is easy to see that the successive errors will reduce in magnitude if the fol-
lowing condition is satisfied at each iteration i.e.
°∙ ¸ °
° ∂G °
(6.73) ° ° < 1 for k = 1, 2, ....
° ∂x (k)
°
x=x
Applying contraction mapping theorem, a sufficient condition for conver-

gence of iterations in the neighborhood x∗ can be stated as
°∙ ¸°
° ∂G °
(6.74) ° °
° ∂x ° ≤ θ1 < 1
1
or °∙ ¸°
° ∂G °
° °
° ∂x ° ≤ θ∞ < 1
∞
Note that this is only a sufficient conditions. If the condition is not satisfied,
then the iteration scheme may or may not converge. Also, note that introduction
of step length parameter λ(k) in Newton-Raphson step as
(6.75) x(k+1) = x(k) + λ(k) ∆x(k)
° ° ° °
such that °F (k+1) ° < °F (k) )° ensures that G(x) is a contraction map and
ensures convergence.
6.4. Condition Number of Nonlinear Set of Equations. Concept of
condition number can be easily extended to analyze numerical conditioning of
set on nonlinear algebraic equations. Consider nonlinear algebraic equations of
the form
(6.76) F (x, u) = 0 ; x ∈Rn , u ∈Rm
where F is n × 1 function vector and u is a set of known parameters or indepen-
dent variables on which the solution depends. The condition number measures
the worst possible effect on the solution x caused by small perturbation in u.
Let δx represent the perturbation in the solution caused by perturbation δu,i.e.
(6.77) F (x+δx, u+δu) = 0
Then the condition number of the system of equations is defined as
sup ||δx||/||x||
(6.78) C(x) =
δu ||δu||/||u||
||δx||/||x||
(6.79) ⇒ ≤ C(x)
||δu||/||u||
If the solution does not depend continuously on u, then the C(x) becomes
infinity and such systems are called as (numerically) unstable systems. Systems
with large condition numbers are more susceptible to computational errors.
Example 47. [1] Consider equation

(6.80) x − eu = 0
Then,
eu+δu − eu
(6.81) δx/x = = eδu − 1
ey
¯ ¯
sup ¯¯ eδu − 1 ¯¯
(6.82) C(x) = u
δu ¯ δu ¯
For small δu, we have eδu = 1 + δu and
C(x) = |u|
7. Solutions of ODE-BVP and PDEs by Orthogonal Collocation

In this section, we demonstrate how combination of Weierstrass theo-
rem and Newton-Raphson method can be used to solve ODE boundary value
problem and PDEs. Application of Weierstrass theorem facilitates conversion of
ODE-BVP and certain class of PDEs to a set of nonlinear algebraic equations,
which can be solved using Newton-Raphson method. Effectively, the ODE-BVP
/ PDE is solved by forming a sequence of linear algebraic sub-problems.
Consider ODE-BVP described by
(7.1) Ψ[d2 y/dz 2 , dy/dz, y, z] = 0 ; z ∈ (0, 1)
(7.2) f1 [dy/dz, y, z] = 0 at z = 0
(7.3) f2 [dy/dz, y, z] = 0 at z = 1
The true solution to problem is a function, say y ∗ (z) ∈ C(2) [0, 1], which be-
longs to the set of twice differentiable continuous functions. According to the
Weierstrass theorem, any continuous function over an interval can be approxi-
mated with arbitrary accuracy using a polynomial function of appropriate de-
gree. Thus, we assume an (approximate) n’th order polynomial solution to
ODE-BVP of the form
(7.4) y(z) = θ0 p0 (z) + ....... + θn+1 pn+1 (z)
{pi (z); i = 0, 1, 2, .....n + 1} should be linearly independent vectors in C (2) [0, 1].
A straightforward choice of such linearly independent vectors is
(7.5) p0 (z) = 1, p1 (z) = z, ......pn (z) = z n
Alternatively, orthogonal polynomials can be used such as

• Shifted Legandre polynomials, which are orthonormal with respect
to
Z 1
(7.6) hpi (z), pj (z)i = pi (z)pj (z)dz
0
7. SOLUTIONS OF ODE-BVP AND PDES BY ORTHOGONAL COLLOCATION 109
• Jacobi polynomials, which are orthonormal with respect to

Z1
(7.7) hpi (z), pj (z)i = W (z)pi (z)pj (z)dz
0
(7.8) W (z) = z (1 − z)β

α
where α and β are constants.

The next step is to convert the ODE-BVP to a set of nonlinear algebraic
equations. We first select n internal collocation (grid) points at z = zi , (i =
1, 2, ...n) in the interval [0, 1]. These collocation points need not be equispaced.
It has been shown that, if these collocation points are chosen at roots of n’th or-
der orthogonal polynomial, then the error |y ∗ (z) − y(z)| is evenly distributed in
the entire domain of z. For example, one possibility is to choose the orthogonal
collocation points at the roots of shifted Legandre polynomials.
Order (n) Roots

1 0.5
2 0.21132, 0.78868
3 0.1127, 0.5, 0.8873
4 0.9305,0.6703, 0.3297, 0.0695
5 0.9543, 0.7662, 0.5034, 0.2286, 0.0475
6 0.9698, 0.8221, 0.6262, 0.3792, 0.1681, 0.0346
7 0.9740, 0.8667, 0.7151, 0.4853, 0.3076, 0.1246, 0.0267
In fact, the name orthogonal collocation can be attributed to the choice the
collocation points at roots of orthogonal polynomials. After selecting the loca-
tion of collocation points, the approximate solution (4.7) is used to convert the
ODE-BVP together with the BCs into a set of nonlinear algebraic equations by
setting residuals at n collocation (grid) points and the two boundary points
equal to zero.
Approach 1
Let us denote
(7.9) yi (θ) = y(zi ) = θ0 p0 (zi ) + ....... + θn+1 pn+1 (zi )
(7.10) yi0 (θ) = [dy/dz]z=zi = θ0 p00 (zi ) + .......................θ n+1 p0n+1 (zi )
£ ¤
(7.11) yi00 (θ) = d2 y/dz 2 z=zi = θ0 p000 (zi ) + .......................θ n+1 p00n+1 (zi )
where
(7.12) θ = [θ0 ...........θ n+1 ]T
0
Substitute for y, y and y 00 in equation (3.1) and enforcing residual to be equal
to zero at the grid points, we get
(7.13) Ψ[yi00 (θ), yi0 (θ), yi (θ), zi ] = 0
i = 1, ......n
Similarly, enforcing residuals at boundary points equal to zero yields
(7.14) f1 [y00 (θ), y0 (θ), 0] = 0
(7.15) f2 [[y10 (θ), y1 (θ), 1] = 0

Thus, we have n+2 nonlinear equations in n +2 unknowns, which can be solved
simultaneously using Newton-Raphson method for estimating θ.
Approach 2
The above approach is not convenient from computational viewpoint. Note
that we have to select an initial guess for vector θ to start the Newton-Raphson
method. Now, unlike linear algebraic equations, a set of nonlinear equations can
have multiple solutions and the solution we reach by applying Newton Raphson
method is governed by the choice of the initial guess vector θ(0) . Instead of
working with θ as unknowns, this approach works with yi as unknowns. It is
much easy to generate initial guess for yi using knowledge of underlying physics
of the problem under consideration. In order to see how this is done, consider
n + 2 equations
(7.16) y0 = θ0 p0 (0) + ....... + θn pn+1 (0)
(7.17) yi = θ0 p0 (zi ) + ....... + θn pn+1 (zi )
i = 1, ......n
(7.18) yn+1 = θ0 p0 (1) + ....... + θn pn+1 (1)

which can be rearranged as
⎡ ⎤⎡ ⎤ ⎡ ⎤
p0 (0) p1 (0) .... pn+1 (0) θ0 y0
⎢ p (z ) p (z ) .... pn+1 (z1 ) ⎥⎢ θ1 ⎥ ⎢ y1 ⎥
⎢ 0 1 1 1 ⎥⎢ ⎥ ⎢ ⎥
(7.19) ⎢ ⎥⎢ ⎥ = ⎢ ⎥
⎣ .... .... .... .... ⎦⎣ ... ⎦ ⎣ ... ⎦
p0 (1) p1 (1) .... pn+1 (1) θn+1 yn+1
(7.20) or Mθ = y
where the matrix M is computed at the internal collocation points and the
boundary points. Using equation (7.20), we can write
(7.21) θ = M −1 y = Ry
Using equation (7.21), we can express vector of first derivatives as
⎡ 0 ⎤ ⎡ 0 0 0 ⎤⎡ ⎤
y0 p0 (0) p1 (0) .... pn+1 (0) θ0
⎢ y0 ⎥ ⎢ p0 (z ) p0 (z ) .... p0 (z ) ⎥ ⎢ θ ⎥
⎢ 1 ⎥ ⎢ 1 1 1 n+1 1 ⎥ ⎢ 1 ⎥
(7.22) ⎢ ⎥ = ⎢ 0 ⎥⎢ ⎥
⎣ ... ⎦ ⎣ .... .... .... .... ⎦ ⎣ ... ⎦
0 0 0 0
yn+1 p0 (1) p1 (1) .... pn+1 (1) θn+1
(7.23) = Nθ = [NR] y =S y
If we express the matrix S as
⎡ £ ¤T ⎤
s(0)
⎢ £ (1) ¤T ⎥
⎢ s ⎥
(7.24) S=⎢
⎢ ....
⎥
⎥
⎣ £ ¤T ⎦
s(n+1)
where s(i) represents (i + 1)’th row vector of matrix S, then

0 £ ¤T
(7.25) yi = s(i) y
Similarly,
⎡ 00 ⎤ ⎡ 00 00 00 ⎤⎡ ⎤
y0 p0 (0) p1 (0) .... pn+1 (0) θ0
⎢ y1
00 ⎥ ⎢ 00
p0 (z1 )
00
p1 (z1 ) ....
00
pn+1 (z1 ) ⎥⎢ θ1 ⎥
⎢ ⎥ ⎢ ⎥⎢ ⎥
(7.26) ⎢ ⎥ = ⎢ ⎥⎢ ⎥
⎣ ... ⎦ ⎣ .... .... .... .... ⎦⎣ ... ⎦
00 00 00 00
yn p0 (1) p1 (1) .... pn+1 (1) θn+1
(7.27) = Qθ = [QR] y = T y
and
00 £ ¤T
(7.28) yi = t(i) y
£ ¤
where t(i) represents (i + 1)’th row vector of matrix T. Using these transfor-
mations, the equation (7.13-7.15) can be written as
h£ ¤T £ ¤T i
(7.29) Ψ s(i) y, t(i) y, yi , zi = 0
i = 1, ......n
h£ ¤T i
(7.30) f1 s(0) y, y0 , 0 = 0
h£ ¤T i
(7.31) f2 s(n+1) y, yn+1 , 1 = 0
which can to be solved simultaneously for unknown vector y using Newton-

Raphson method. In particular, if we choose
(7.32) p0 (z) = 1, p1 (z) = z, ......pn+1 (z) = z n+1
then, matrices S and T can be computed using

⎡ ⎤
1 0 .... 0
⎢ 1 z1 .... (z1 )n+1 ⎥
⎢ ⎥
(7.33) M = ⎢ ⎥
⎣ .... .... .... .... ⎦
1 1 .... 1
⎡ ⎤
0 1 .... 0
⎢ 0 1 .... (n + 1)(z1 )n ⎥
⎢ ⎥
(7.34) N = ⎢ ⎥
⎣ .... .... .... .... ⎦
0 1 .... (n + 1)
⎡ ⎤
0 0 2 6z0 .... 0
⎢ 0 0 2 6z1 .... n(n + 1)(z1 )n−1 ⎥
⎢ ⎥
(7.35) Q = ⎢ ⎥
⎣ .... .... .... .... .... .... ⎦
0 0 2 6zn .... n(n + 1)
where zi are collocation points.
Example 48. [6] Consider the ODE-BVP describing steady state conditions
in a tubular reactor with axial mixing (TRAM) in which an irreversible 2nd
order reaction is carried out
1 d2 C dC
(7.36) 2
− − DaC 2 = 0 (0 ≤ z ≤ 1)
P e dz dz
dC
(7.37) = P e(C − 1) at z = 0;
dz
dC
(7.38) = 0 at z = 1;
dz
Using method of orthogonal collocation with n = 3 and defining
h iT
(7.39) C = C0 C1 ... C4
we get following set of five nonlinear algebraic equations

1 h£ (i) ¤T i h¡ (i) ¢T i
(7.40) t C − s C − DaCi2 = 0
Pe
i = 1, 2, 3
h£ ¤T i
(7.41) t(0) C − P e(C0 − 1) = 0
h£ ¤T i
(7.42) t(4) C = 0
where the matrices
⎡ ⎤
−13 14.79 −2.67 1.88 −1
⎢ −5.32 3.87 2.07 −1.29 0.68 ⎥
⎢ ⎥
⎢ ⎥
(7.43) S=⎢ 1.5 −3.23 0 3.23 −1.5 ⎥
⎢ ⎥
⎣ −0.68 1.29 −2.07 −3.87 5.32 ⎦
1 −1.88 2.67 −14.79 13
⎡ ⎤
84 −122.06 58.67 −44.60 24
⎢ 53.24 −73.33 26.27 −13.33 6.67 ⎥
⎢ ⎥
⎢ ⎥
(7.44) T =⎢ −6 16.67 −21.33 16.67 −6 ⎥
⎢ ⎥
⎣ 6.76 −13.33 26.67 −73.33 53.24 ⎦
24 −44.60 58.67 −122.06 84
Example 49. [6] Consider the 2-dimensional Laplace equation
(7.45) ∂ 2 u/∂x2 + ∂ 2 u/∂y 2 = f (x, y)
0<x<1 ; 0<y<1
where u(x, y) represents the dimensionless temperature distribution in a furnace
and x, y are space coordinates. The boundary conditions are as follows:
(7.46) x = 0 : u = u∗ ; x = 1 : u = u∗
(7.47) y = 0 : u = u∗ ; y = 1 : k(∂u/∂y) = h(u∞ − u(x, 1))
Using nx internal grid lines parallel to y axis and ny grid lines parallel to y-
asix, we get nx × ny internal collocation points. Corresponding to the chosen
collocation points, we can compute matrices (Sx , Tx ) and (Sy , Ty ) using equations
(7.23) and (7.27). Using these matrices, the PDE can be transformed as
£ (i) ¤T (i) £ (j) ¤T (j)
(7.48) tx U + ty U = f (xi , yj )
i = 1, 2, ...nx ; j = 1, 2, ...ny
h i
(7.49) U (i) = u0,i u1,i ... unx+1,i
h i
(7.50) U (j) = uj,o uj,1 ... uj,ny+1
At the boundaries, we have
(7.51) u0,j = u∗ ; (j = 0, 1, ...nx+1 )

(7.52) u1,j = u∗ ; (j = 0, 1, ...nx+1 )
(7.53) ui,0 = u∗ ; (i = 0, 1, ...ny+1 )
£ y +1) ¤T (i)
(7.54) k s(n
y U = h(u∞ − u(xi , 1))
(i = 1, ...ny )
The above procedure yields (nx + 1) × (ny + 1) nonlinear equations in (nx + 1) ×

(ny + 1) unknowns, which can be solved simultaneously using Newton-Raphson
method.
Remark 3. Are the two methods presented above, i.e. finite difference
and collocation methods, doing something fundamentally different? Suppose
we choose n0 th order polynomial (4.7), we are essentially approximating the
true solution vector y ∗ (z) ∈ C(2) [0, 1] by another vector (i.e. the polynomial
function) in (n + 2) dimensional subspace of C(2) [0, 1]. If we choose n inter-
nal grid points by finite difference approach, then we are essentially finding a
vector y in Rn+2 that approximates y ∗ (z). In fact, if we compare the Approach
2 presented above and the finite difference method, the similarities are more
apparent as the underlying (n + 2) dimensional subspace used in approxima-
tions become identical. Let us compare the following two cases (a) finite dif-
ference method with 3 internal grid points (b) collocation with 3 internal grid
points on the basis of expressions used for approximating the first and second
order derivatives computed at one of the grid points. For the sake of compar-
ison, we have taken equi-spaced grid points for collocation method instead of
taking them at the roots of 3’rd order orthogonal polynomial. Thus, for both
collocation and finite difference method, the grid (or collocation) points are at
{z0 = 0, z1 = 1/4, z2 = 1/2, z3 =h 3/4, z4 = 1} and wei want to estimate the ap-
proximate solution vector y = y0 y1 y2 y3 y4 in both the cases. Let us
compare expressions for derivative at z = z2 used in both the approaches.
Finite Difference
(y3 − y1 )
(7.55) (dy/dz)2 = = 2y3 − 2y1 ; ∆z = 1/4
2(∆z)
(y3 − 2y2 + y1 )
(7.56) (d2 y/dz 2 )2 = = 16y3 − 32y2 + 16y1
(∆z)2
9. APPENDIX 115
Collocation
(7.57) (dy/dz)2 = 0.33y0 − 2.67y1 + 2.67y3 − 0.33y4

(7.58) (d2 y/dz 2 )2 = −1.33y0 + 21.33y1 − 40y2 + 21.33y3 − 1.33y4
It becomes clear from the above expressions that the essential difference between
the two approaches is the way the derivatives at any grid (or collocation) point
is approximated. The finite difference method takes only immediate neighboring
points for approximating the derivatives while the collocation method finds deriv-
atives as weighted sum of all the collocation (grid) points. As a consequence,
the approximate solutions generated by these approaches will be different.
8. Summary
In these lecture notes, we have developed methods for efficiently solv-
ing large dimensional linear algebraic equations. To begin with, we introduce
induced matrix norms and use them to understand matrix ill conditioning and
susceptibility of matrices to round of errors. The direct methods for dealing
with sparse matrices are discussed next. Iterative solution schemes and their
convergence characteristics are discussed in the subsequent section. We then
present techniques for solving nonlinear algebraic equations, which are based
on successive solutions of linear algebraic sub-problem. In the last section, we
have discussed the orthogonal collocation technique, which converts ODE-BVPs
or certain class of PDEs into a set of nonlinear algebraic equations, which can
be solved using Newton-Raphson method.
9. Appendix
9.1. Proof of Theorem 2 [4]: For Jacobi method,
(9.1) S −1 T = −D−1 [L + U]
⎡ a12 a1n ⎤
0 − ..... −
⎢ a11 a11 ⎥
⎢ a12 ⎥
⎢ − 0 ..... .... ⎥
⎢ ⎥
(9.2) = ⎢ a22 an−1,n ⎥
⎢ ..... ..... .... − ⎥
⎢ an−1,n−1 ⎥
⎣ a12 ⎦
− ..... ..... 0
ann
As matrix A is diagonally dominant, we have

Xn
(9.3) |aij | < |aii | for i = 1, 2., ...n
j=1(j6=i)
Xn ¯ ¯
¯ aij ¯
(9.4) ⇒ ¯ ¯ < 1 for i = 1, 2., ...n
¯ aii ¯
j=1(j6=i)
⎡ ⎤
X n ¯ ¯
° −1 ° max ⎣ ¯ aij ¯
(9.5) °S T ° = ¯ ¯⎦ < 1
∞
i ¯ aii ¯
j=1(j6=i)
Thus, Jacobi iteration converges if A is diagonally dominant.

For Gauss Seidel iterations, the iteration equation for i’th component of the
vector is given as
µ ¶" Xi−1 Xn
#
(k+1) 1 (k+1) (k)
(9.6) xi = bi − aij xj − aij xj
aii j=1 j=i+1
Let x∗ denote the true solution of Ax = b. Then, we have

µ ¶" X
i−1 Xn
#
1
(9.7) x∗i = bi − aij x∗j − aij x∗j
aii j=1 j=i+1
Subtracting (9.7) from (9.6), we have

µ ¶ "X i−1 ³ ´ Xn ³ ´
#
(k+1) 1 (k+1) (k)
(9.8) xi − x∗i = aij x∗j − xj + aij x∗j − xj
aii j=1 j=i+1
or
" i−1 ¯ ¯ n ¯ ¯ ¯³ #
¯ ¯ X ¯ aij ¯ ¯¯³ ´¯ X ¯ a ¯ ´¯
¯ (k+1) ¯ ¯ ¯ ¯ x∗j − x(k+1) ¯ ¯
¯ ¯ ¯ x∗j − x
ij (k) ¯
(9.9) ¯xi − x∗i ¯ ≤ ¯ aii ¯ j ¯+ ¯ aii ¯ j ¯
j=1 j=i+1
Since
° ∗ ° ¯³ ´¯
°x − x(k) ° = max ¯¯ x∗j − xj(k) ¯¯
∞
j
we can write
¯ ¯ ° ° ° °
¯ (k+1) ¯
(9.10) ¯xi − x∗i ¯ ≤ pi °x∗ − x(k+1) °∞ + qi °x∗ − x(k) °∞
where
i−1 ¯
X ¯ Xn ¯ ¯
¯ aij ¯ ¯ aij ¯
(9.11) pi = ¯ ¯ ¯ ¯
¯ aii ¯ ; qi = ¯ aii ¯
j=1 j=i+1
Let s be value of index i for which

¯ (k+1)
¯
¯
∗¯ max ¯¯³ ∗ ´¯
(k+1) ¯
(9.12) xs − xs = ¯ xj − xj ¯
j
9. APPENDIX 117
Then, assuming i = s in inequality (9.10), we get

° ∗ ° ° ° ° °
(9.13) °x − x(k+1) ° ≤ pi °x∗ − x(k+1) ° + qi °x∗ − x(k) °
∞ ∞ ∞
or
° ∗ ° qs ° °
(9.14) °x − x(k+1) ° ≤ °x∗ − x(k) °
∞ 1 − ps ∞
Let
max qj
(9.15) µ=
j 1 − pj
° ∗ ° qs ° °
(9.16) °x − x(k+1) ° ≤ °x∗ − x(k) °
∞ 1 − ps ∞
then it follows that

° ∗ ° ° °
(9.17) °x − x(k+1) ° ≤ µ °x∗ − x(k) °
∞ ∞
Now, as matrix A is diagonally dominant, we have

(9.18) 0 < pi < 1 and 0 < qi < 1
X
n ¯ ¯
¯ aij ¯
(9.19) 0 < pi + qi = ¯ ¯<1
¯ aii ¯
j=1(j6=i)
Let
⎡ ⎤
Xn ¯ ¯
max ⎣ ¯ aij ¯
(9.20) β= ¯ ¯⎦
i ¯ aii ¯
j=1(j6=i)
Then, we have
(9.21) pi + qi ≤ β < 1
It follows that
(9.22) qi ≤ β − pi
and
qi β − pi β − pi β
(9.23) µ= ≤ ≤ =β<1
1 − pi 1 − pi 1 − pi
Thus, it follows from inequality (9.17) that
° ∗ ° ° °
(9.24) °x − x(k) ° ≤ µk °x∗ − x(0) °
∞ ∞
i.e. the iteration scheme is a contraction map and

lim
(9.25) x(k) = x∗
k→∞
9.2. Proof of Theorem 3. For Gauss-Seidel method, when matrix A is

symmetric, we have
(9.26) S −1 T = (L + D)−1 (−U) = −(L + D)−1 (LT )
Now, let e represent unit eigenvector of matrix S −1 T corresponding to eigen-
value λ, i.e.
(9.27) −(L + D)−1 (LT )e = λe
(9.28) or LT e = −λ(L + D)e
Taking inner product of both sides with e,we have
T ®
(9.29) L e, e = −λ h(L + D)e, ei
T ®
L e, e he,Lei
(9.30) λ = − =−
hDe, ei + hLe, ei hDe, ei + hLe, ei
Defining
(9.31) α = hLe, ei = he,Lei
X n
(9.32) σ = hDe, ei = aii (ei )2 > 0
i=1
we have
¯ ¯
α ¯ α ¯
(9.33) λ=− ⇒ |λ| = ¯¯ ¯
α+σ α + σ¯
Note that σ > 0 follows from the fact that trace of matrix A, is positive as
eigenvalues of A are positive. Using positive definiteness of matrix A, we have
®
(9.34) hAe, ei = hLe, ei + hDe, ei + LT e, e
(9.35) = σ + 2α > 0
This implies
(9.36) −α < (σ + α)
Since σ > 0, we can say that
(9.37) α < (σ + α)
i.e.
(9.38) |α| < (σ + α)
This implies
¯ ¯
¯ α ¯
(9.39) |λ| = ¯¯ ¯<1
α + σ¯
10. EXERCISE 119
10. Exercise
(1) A polynomial
y = a0 + a1 x + a2 x2 + a3 x3
passes through point (3, 2), (4, 3), (5, 4) and (6, 6) in an x-y coordi-
nate system. Setup the system of equations and solve it for coefficients
a0 to a3 by Gaussian elimination. The matrix in this example is (4 X
4 ) Vandermonde matrix. Larger matrices of this type tend to become
ill-conditioned.
(2) Solve using Gauss Jordan method.
u + v + w = −2
3u + 3v − w = 6
u − v + w = −1
to obtain A−1 . What coefficient of v in the third equation, in place
of present −1 , would make it impossible to proceed and force the
elimination to break down?
(3) Decide whether vector b belongs to column space spanned by x(1), x(2) , ....
(a) x(1) = (1, 1, 0); x(2) = (2, 2, 1); x(3) = (0, 2, 0); b = (3, 4, 5)
(b) x(1) = (1, 2, 0); x(2) = (2, 5, 0); x(3) = (0, 0, 2); x(4) = (0, 0, 0); any
b
(4) Find dimension and construct a basis for the four fundamental sub-
spaces associated with each of the matrices.
" # " #
0 1 4 0 0 1 4 0
A1 = ; U2 =
0 2 8 0 0 0 0 0
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
0 1 0 1 2 0 1 1 2 0 1
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
A2 = ⎣ 0 0 1 ⎦ ; A3 = ⎣ 0 1 1 0 ⎦ ; U1 = ⎣ 0 1 1 0 ⎦
0 0 0 1 2 0 1 1 2 0 1
(5) Find a non -zero vector x∗ orthogonal to all rows of
⎡ ⎤
1 2 1
⎢ ⎥
A=⎣ 2 4 3 ⎦
3 6 4
(In other words, find Null space of matrix A.) If such a vector exits,
can you claim that the matrix is singular? Using above A matrix find
one possible solution x for Ax = b when b = [ 4 9 13 ]T . Show that
if vector x is a solution of the system Ax = b, then (x+αx∗ ) is also a

solution for any scalar α, i.e.
A(x + αx∗ ) = b
Also, find dimensions of row space and column space of A.

(6) If product of two matrices yields null matrix, i.e. AB = [0], show that
column space of B is contained in null space of A and the row space of
A is in the left null space of B.
(7) Why there is no matrix whose row space and null space both contain
the vector h i T
x= 1 1 1
(8) Find a 1 × 3 matrix whose null space consists of all vectors in R3 such
that x1 + 2x2 + 4x3 = 0. Find a 3 × 3 matrix with same null space.
(9) If V is a subspace spanned by
⎡ ⎤ ⎡ ⎤ ⎡ ⎤
1 1 1
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎣ 1 ⎦;⎣ 2 ⎦;⎣ 5 ⎦
0 0 0
find matrix A that has V as its row space and matrix B that has V as
its null space.
(10) Find basis for each of subspaces and rank of matrix A
(a)
⎡ ⎤ ⎡ ⎤⎡ ⎤
0 1 2 3 4 1 0 0 0 1 2 3 4
⎢ ⎥ ⎢ ⎥⎢ ⎥
A = ⎣ 0 1 2 4 6 ⎦ = LU = ⎣ 1 1 0 ⎦ ⎣ 0 0 0 1 2 ⎦
0 0 0 1 2 0 1 1 0 0 0 0 0
(b)
⎡ ⎤⎡ ⎤
1 0 0 0 1 2 0 1 2 1
⎢ 2 1 0 0 ⎥⎢ 0 0 2 2 0 0 ⎥
⎢ ⎥⎢ ⎥
A=⎢ ⎥⎢ ⎥
⎣ 2 1 1 0 ⎦⎣ 0 0 0 0 0 1 ⎦
3 2 4 1 0 0 0 0 0 0
(11) Consider application of finite difference method to solving ODE-BVP
with non-equidistant grid point i.e.
∆zi = zi+1 − zi ; i = 0, 1, 2, ....
Derive expressions for the first and second derivatives

∙ ¸
(2) 2 yi+1− yi yi− yi − 1 1 (3)
yi = − − yi (∆zi − ∆zi−1 ) + ....
∆zi + ∆zi−1 ∆zi ∆zi−1 3
10. EXERCISE 121
(12) In several fluid mechanics problems, one needs to solve for ∇4 Ψ =

0 where Ψ is the stream function. Solving these equations numerically
requires computation of higher derivatives. Derive following expressions
for
d3 y yi+2 − 2yi+1 + yi − 2yi−1 + yi−2
3
= 3 + O (∆z)2
dz 2 (∆z)
d4 y yi+2 − 4yi+1 + 6yi − 4yi−1 + yi−2
4
= 4 + O (∆z)2
dz 2 (∆z)
(13) Consider the solution of the linear system
x + 0.9y + 0.1z = 1
0.4x + y + 0.4z = 0
0.8x + 0.1y + z = 0
by iteration scheme
⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎡ ⎤
x(k+1) 1 0 0.9 0.1 x(k)
⎢ (k+1) ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥
⎣ y ⎦ = ⎣ 0 ⎦ − ⎣ 0.4 0 0.4 ⎦ ⎣ y (k) ⎦
z (k+1) 0 0.8 0.1 0 z (k)
Show that this iteration scheme converges to the unique solution for
arbitrary starting guess.
(14) The true solution Ax = b is slightly different from the elimination
solution to LU x0 = b; A − LU misses zero matrix because of round off
errors. One possibility is to do everything in double precision, but a
better and faster way is iterative refinement: Compute only one vector
r = b − Ax0 in double precision, solve LUy = r,and add correction y to
x0 to generate an improved solution x1 = x0 + y.
Problem:Multiply x1 = x0 + y, by LU , write the result as splitting
Sx1 = T x0 + b, and explain why T is extremely small. This single step
bring almost close to x.
(15) Consider system
⎡ ⎤ ⎡ ⎤
1 1/2 1/3 1
⎢ ⎥ ⎢ ⎥
A = ⎣ 1/2 1/3 1/4 ⎦ ; b = ⎣ −1 ⎦
1/3 1/4 1/5 1
where A is Hilbert matrix with aij = 1/(i + j − 1), which is severely
ill- conditioned. Solve using
(a) Gauss-Jordan elimination
(b) exact computations
(c) rounding off each number to 3 figures.

Perform 4 iterations each by
(a) Jacobi method
(b) Gauss- Seidel method
(c) Successive over-relaxation method with ω = 1.5
h iT
Use initial guess x(0) = 1 1 1 and compare in each case how
close to the x(4) is to the exact solution. (Use 2-norm for comparison).
Analyze the convergence properties of the above three iterative
processes using eigenvalues of the matrix (S −1 T ) in each case. Which
iteration will converge to the true solution?
(16) The Jacobi iteration for a general 2 by 2 matrix has
" # " #
a b a 0
A= ; D=
c d 0 d
If A is symmetric positive definite, find the eigenvalues of J = S −1 T =

D−1 (D − A) and show
that Jacobi iterations converge.
(17) It is desired to solve Ax = b using Jacobi and Gauss-Seidel iteration
scheme where
⎡ ⎤
⎡ ⎤ ⎡ ⎤ −7 1 −2 3
4 2 1 1 2 −2 ⎢ ⎥
⎢ ⎥ ⎢ ⎥ ⎢ 1 8 1 3 ⎥
A=⎣ 1 5 3 ⎦ ; A=⎣ 1 1 1 ⎦ ; A=⎢ ⎥
⎣ −2 1 −5 1 ⎦
2 4 7 2 2 1
1 0 −1 −3
Will the Jacobi and Gauss-Seidel the iterations converge? Justify

your answer. (Hint: Check for diagonal dominance before proceeding
to compute eigenvalues).
(18) Given matrix
⎡ ⎤
0 1 0 1
1⎢
⎢ 1 0 1 0 ⎥
⎥
J= ⎢ ⎥
2⎣ 0 1 0 1 ⎦
1 0 1 0
find powers J 2 , J 3 by direct multiplications. For which matrix A is this

a Jacobi matrix J = I − D−1 A ? Find eigenvalues of J.
10. EXERCISE 123
(19) The tridiagonal n × n matrix A that appears when finite difference

method is used to solve second order PDE / ODE-BVP and the corre-
sponding Jacobi matrix are as follows
⎡ ⎤ ⎡ ⎤
2 −1 0 ... 0 0 1 0 ... 0
⎢ −1 2 −1 ... 0 ⎥ ⎢ 1 0 1 ... 0 ⎥
⎢ ⎥ 1⎢ ⎥
⎢ ⎥ ⎢ ⎥
A = ⎢ ... ... ... ... ... ⎥ ; J = ⎢ ... ... ... ... ... ⎥
⎢ ⎥ 2⎢ ⎥
⎣ 0 ... −1 2 −1 ⎦ ⎣ 0 ... 1 0 1 ⎦
0 ... 0 1 2 0 ... 0 1 0
Show that the vector
h iT
x = sin(πh) sin(2πh) ... sin(nh)
satisfies Jx = λx with eigenvalue λ = cos(πh). Here, h = 1/(n + 1)

and hence sin [(n + 1)πh] = 0.
Note: The other eigenvectors replace π by 2π, 3π,...., nπ. The other
eigenvalues
are cos(2πh), cos(3πh),...... , cos(nπh) all smaller than cos(πh) < 1.
(20) Consider the following system
" #
1 1
A=
1 1+ε
Obtain A−1 , det(A) and also solve for [x1 x2 ]T . Obtain numerical
values for ε = 0.01, 0.001 and 0.0001. See how sensitive is the solution
to change in ε.
(21) If A is orthogonal matrix, show that ||A|| = 1 and also c(A) = 1.
Orthogonal matrices and their multiples(αA) are the only perfectly
conditioned matrices. " #
2 −1
(22) For the positive definite matrix, A = , compute ||A−1 || =
−1 2
1/λ1 ; and c(A) = λ2 /λ1. Find the right side b and a perturbation vector
δb such that the error is worst possible, i.e. find vector x such that
|| δx||/||x|| = c|| δ b ||/||b||
(23) Find a vector x orthogonal to row space, and a vector y orthogonal to

column space, of
⎡ ⎤
1 2 1
⎢ ⎥
A= ⎣ 2 4 3 ⎦
3 6 4
(24) Show that vector x − y is orthogonal to vector x + y if and only if

||x|| = ||y||.
(25) For a positive definite matrix A, the Cholensky decomposition is A =
L D LT = RRT where R = LD1/2 . Show that the condition number
of R is square root of condition number of A. It follows that Gaussian
elimination needs no row exchanges for a positive definite matrix; the
condition number does not deteriorate, since c(A) = c(RT )c(R).
(26) Show that for a positive definite symmetric matrix, the condition num-
ber can be obtained as
c(A) = λmax (A)/λmin (A)

(27) If A is an orthonormal(unitary) matrix (i.e. AT A = I), show that
||A|| = 1 and also c(A) = 1. Orthogonal matrices and their multiples
(αA)are he only perfectly conditioned matrices.
(28) Show that λmax (i.e. maximum magnitude eigen value of a matrix) or
even max |λi |,is not a satisfactory norm of a matrix, by finding a 2x2
counter example to
λmax (A + B) ≤ λmax (A) + λmax (B)
and to
λmax (AB) ≤ λmax (A) λmax (B)
(29) Prove the following inequalities/ identities
kA + Bk ≤ kAk + kBk
kABk ≤ kAk kBk

C(AB) ≤ C(A)C(B)
° °
kAk2 = °AT °2
(30) If A is an orthonormal (unitary) matrix, show that kAk2 = 1 and also
condition number C(A) = 1. Note that orthogonal matrices and their
multiples are the only perfectly conditioned matrices.
(31) Show that for a positive definite symmetric matrix, the condition num-
ber can be obtained as C(A) = λmax (A)/λmin (A).
(32) Show that λmax , or even max |λi | , is not a satisfactory norm of a
matrix, by finding a 2 × 2 counter examples to following inequalities
λmax (A + B) ≤ λmax (A) + λmax (B)
λmax (AB) ≤ λmax (A)λmax (B)

10. EXERCISE 125
(33) For the positive definite matrix

" #
2 −1
A=
−1 2
compute the condition number C(A) and find the right hand side b
of equation Ax = b and perturbation δb such that the error is worst
possible, i.e.
kδxk kδbk
= C(A)
kxk kbk
(34) Consider the following two sets of nonlinear equations
(a) Set 1:
x21 + x22 − 4 = 0
x21 − x22 − 1.5 = 0
h iT
(0)
x = 1 1
(b) Set 2 (Banana Function):
10(−x21 + x2 ) = 0
x1 − 1 = 0
h iT
x(0) = −1 1
(c) Set 3:
2x = sin [(x + y)/2]
2y = cos [(x − y)/2]
h iT
(0)
x = 10 −10
(d) Set 4:
sin(x) + y 2 + ln(z) = 7
3x + 2y − z 3 = −1
x+y+z = 5
h iT
x(0) = 0 2 2
(a) Perform 3 iterations using (a) successive substitution (b) New-
ton Raphson (c) Newton Raphson with Broyden’s update and compare
progress of x(k) towards the true solution in each case.
(b) Check whether the sufficient condition for convergence is satis-
fied at each iteration by computing infinite norms of the Jacobians.
(35) Consider following set of nonlinear algebraic equations Perform 3 iter-

ations using
(a) Set 2 (Banana Function):
10(−x21 + x2 ) = 0
x1 − 1 = 0
h iT
x(0) = −1 1
(36) The following coupled differential equations characterize a system
d2 u
+ 2uev = f1 (u, v, z) ; u(0) = 0; u(1) = 5
dz 2
d2 v
+ 5uv = f2 (u, v, z) ; v(0) = 1; v(1) = 2
dz 2
Obtain sets of nonlinear algebraic equations using (a) finite differ-
ence method with 2 internal grid points and (b) orthogonal collocation
with 2 collocation points. Also, arrange these equations as Ax = G(x)
so that method of successive substitution can be used for formulating
iterative scheme.
CHAPTER 4
ODE-IVPs and Related Numerical Schemes
1. Motivation
In these lecture notes, we undertake the study of solution techniques for
multivariable and coupled ODE-IVPs. The numerical techniques for solving
ODE-IVPs form basis for a number of numerical schemes and are used for
solving variety of problems such as
• Dynamic simulation of lumped parameter systems
• Solution of ODE-BVP
• Solving Parabolic / Hyperbolic PDEs
• Solving simultaneous nonlinear algebraic equation
and so on. In order to provide motivation for studying the numerical meth-
ods for solving ODE-IVPs, we first formulate numerical schemes for the above
problems in the following subsections.
1.1. Dynamic behavior of lumped parameter systems.
Example 50. Three isothermal CSTRs in series
Consider three isothermal CSTRs in series in which a first order liquid phase
reaction of the form
(1.1) A −→ B
is carried out. It is assumed that volume and liquid density remains constant
in each tank and
dCA1
(1.2) V1 = F (CA0 − CA1 ) − kV1 CA1
dt
dCA2
(1.3) V2 = F (CA1 − CA2 ) − kV2 CA2
dt
dCA3
(1.4) V3 = F (CA2 − CA3 ) − kV3 CA3
dt
127
128 4. ODE-IVPS AND RELATED NUMERICAL SCHEMES
Defining τ = F/V, we can re arrange the above set of equations as

⎡ ⎤ ⎡ ⎤⎡ ⎤
dCA1
−(k + 1/τ 1 ) 0 0 CA1
⎢ dt ⎥ ⎢ ⎥⎢ ⎥
(1.5)⎣ dCdtA2 ⎦ = ⎣ 1/τ −(k + 1/τ 2 ) 0 ⎦ ⎣ CA2 ⎦
dCA3
dt
0 1/τ −(k + 1/τ 3 ) CA3
⎡ ⎤
1/τ 1
⎢ ⎥
(1.6) + ⎣ 0 ⎦ CA0
0
x = [CA1 , CA2 , CA3 ]T

(1.7) dx
= Ax + BCA0
dt
where matrices A and B are defined in the above equation. Now, suppose
initially CA0 = C̄A0 , till t = 0, and, for t ≥ 0, CA0 was changed to CA0 = 0.
Then we are required to solve
dx
(1.8) = Ax; x = x(0) at t = 0
dt
and generate trajectories x(t) ( i.e. CA1 (t), CA2 (t) and CA3 (t)) over interval
[0, tf ]. This is a typical problem of dynamic simulation of lumped parameter
system.
Example 51. Continuous Fermenter
Consider a continuously operated fermenter described by the following set

of ODEs
dX
(1.9) = F1 (X, S, P, D, Sf ) = −DX + µX
dt
dS 1
(1.10) = F2 (X, S, P, D, Sf ) = D(Sf − S) − µX
dt YX/S
dP
(1.11) = F3 (X, S, P, D, Sf ) = −DP + (αµ + β)X
dt
where X represents effluent cell-mass or biomass concentration, S represents
substrate concentration and P denotes product concentration. It is assumed
that product concentration (S) and the cell-mass concentration (X) are mea-
sured process outputs while dilution rate (D) and the feed substrate concen-
tration (Sf ) are process inputs which can be manipulated. Model parameter
µ represents the specific growth rate, YX/S represents the cell-mass yield, α and
1. MOTIVATION 129
β are the yield parameters for the product. The specific growth rate model is
allowed to exhibit both substrate and product inhibition:
P
µm (1 −)S
Pm
(1.12) µ=
S2
Km + S +
Ki
where µm represents maximum specific growth rate, Pm represents product sat-
uration constant, Km substrate saturation constant and the Ki represents sub-
strate inhibition constant. Defining state and input vectors as
h iT h iT
(1.13) x= X S P ; u = D Sf
the above equation can be represented as

dx
(1.14) = F (x, u)
dt
A typical problem dynamic simulation problem is to find trajectories of product,
biomass and substrate concentrations over an interval [0, tf ], given their initial
values and dilution rate D(t) and feed substrate concentration Sf as a function
of time over [0, tf ].
In abstract terms, the dynamic simulation problem can be states as follows.
Given time trajectories of independent variables {u(t) : 0 ≤ t ≤ tf },and initial
state, x(0), of the system, obtain state trajectories {x(t) : 0 ≤ t ≤ tf } by
integrating
dx
(1.15) = F [x(t), u(t)] ; x = x(0) at t = 0
dt
where x ∈ Rn represents dependent or state variables and u ∈ Rm denote in-
dependent inputs. As the independent variable trajectories are known a-priori
while solving ODE-IVP, the problem can be looked at as n-ODE’s in n variables
with variable coefficients. Thus, the above problem can be re-stated as
dx
(1.16) = Fu (x, t) ; x(0) = x0
dt
In other words, a forced dynamic systems can be looked upon as unforced sys-
tems with variable parameters.
1.2. Solving nonlinear algebraic equations using method of homo-

topy. Consider the problem of solving simultaneous nonlinear algebraic equa-
tion of the form
(1.17) F (x) = 0
where x ∈ Rm and F (x) is a m × 1 function vector. Introducing a parameter λ

such that(0 ≤ λ ≤ 1), we define
(1.18) F (x(λ)) = (1 − λ)F (x(0) )
where x(0) represents some arbitrary initial guess. Obviously, at λ = 1 we have
F (x) = 0. Differentiating w.r.t. λ and rearranging, we get
dx ∂F −1
(1.19) = −[ ] F (x(0) )
dλ ∂x
(1.20) x(0) = x(0)
Integrating above ODE-IVPs from λ = 0 to λ = 1 produces the solution of
nonlinear the equations at λ = 1.
1.3. Solutions of Parabolic / Hyperbolic PDE’s.

1.3.1. Finite Difference Method. Use of finite difference method to solve par-
abolic or hyperbolic equations with finite spatial boundaries results in set of
coupled linear / nonlinear ODE-IVPs.
•
Example 52. Consider the ODE-BVP describing steady state con-

ditions in a tubular reactor with axial mixing (TRAM) in which an
irreversible 2nd order reaction is carried out.
∂C 1 ∂ 2 C ∂C
(1.21) = − − DaC 2 (0 ≤ z ≤ 1)
∂t P e ∂z 2 ∂z
(1.22) t = 0 : c(z, 0) = 0
∂C(0, t)
(1.23) = P e (C(0, t) − 1) at z = 0;
∂z
∂C(1, t)
(1.24) = 0 at z = 1;
∂z
Using finite difference method along the spatial coordinate z with m − 1
internal grid points, we have
µ ¶
dCi (t) 1 Ci+1 (t) − 2Ci (t) + Ci−1 (t)
=
dt Pe (∆z)2
µ ¶
Ci+1 (t) − Ci−1 (t)
(1.25) − − Da [Ci (t)]2
2 (∆z)
i = 1, 2, ....m − 1
C1 (t) − C0 (t)
(1.26) = P e (C0 (t) − 1)
∆z
Cm+1 (t) − Cm (t)
(1.27) = 0
∆z
1. MOTIVATION 131
The above set of ODEs, together with initial conditions,
(1.28) C1 (0) = C2 (0) = ..... = Cm+1 (0) = 0
defines an ODE-IVP of type (1.16).
Example 53. Consider the 2-dimensional unsteady state heat transfer prob-
lem
∂T ∂2T ∂2T
(1.29) = α[ 2 + 2 ]
∂t ∂x ∂y
(1.30) t = 0 : T = F (x, y)
(1.31) x = 0 : T (0, y, t) = T ∗ ; x = 1 : T (1, y, t) = T ∗
dT (x, 1, t)
(1.32) y = 0 : T (x, 0, t) = T ∗ ; y = 1 : k = h(T∞ − T (x, 1, t))
dy
Example 54. where T (x, y, t) is the temperature at locations (x, y) at time

t and α is the thermal diffusivity. By finite difference approach, we construct a
2-dimensional grid with nx + 1 equispaced grid lines parallel to the y-axis and
ny + 1 grid lines parallel to the x-axis. The temperature T at the (i, j)’th grid
point is given by
(1.33) Tij (t) = T (xi , yi , t)
Now, we force the residual to zero at each internal grid point to generate a set
of coupled ODE-IVP’s as
dTij α α
(1.34) = 2
[Ti+1,j − 2Ti,j + Ti−1,j ] + [Ti,j+1 − 2Ti,j + Ti,j−1 ]
dt (∆x) (∆y)2
i = 1, 2, ....nx − 1 and j = 1, 2, ....ny − 1
The values of Tij at boundary points corresponding to x = 0, x = 1, and

y = 0 can be easily written using boundary conditions.
dTi,ny (t)
(1.35) k = h(T∞ − Ti,ny (t))
dy
i = 1, 2, ....nx − 1
1.3.2. Orthogonal Collocation Method.
Example 55. Consider the ODE-BVP describing steady state conditions in

a tubular reactor with axial mixing (TRAM) given in the above section. Using
method of orthogonal collocation with m internal collocation points, we get
dCi (t) 1 h£ (i) ¤T i h¡ (i) ¢T i
= t C − s C − DaCi2
dt Pe
i = 1, 2, 3, ...m
h£ ¤T i
t(0) C = P e(C0 (t) − 1)
h£ ¤T i
t(m+1) C = 0
£ ¤ ¡ ¢
where the matrices t(i) and s(i) represent row vectors of matrices T and S, as
defined in the Section 5 (example 6) of lecture notes on linear algebraic equations
and related numerical scheme.. Here, Ci (t) represents concentration at the i’th
collocation point. The above set of ODEs, together with initial conditions,
C1 (0) = C2 (0) = ..... = Cm+1 (0) = 0
defines an ODE-IVP of type (1.16).
1.4. Solution of ODE-BVP: Shooting Method. By this approach, we

reduce the 2nd or higher order ODE-BVP to a set of first order ODE’s.
Example 56. the ODE-BVP describing tubular reactor with axial mixing
(TRAM) in which an irreversible 2nd order reaction is carried out is given as
1 d2 C dC
(1.36) − − DaC 2 = 0 (0 ≤ z ≤ 1)
P e dz 2 dz
dC
= P e(C − 1) at z = 0;
(1.37) dz
dC
=0 at z = 1;
dz
where C is the dimensionless concentration, z is axial position, P e is the Peclet
number for mass transfer and Da is the Damkohler number. Now, defining new
state variables
dC
(1.38) x1 = C and x2 =
dz
1. MOTIVATION 133
we can transform the above ODE’s as

dx1
(1.39) = x2
dz
dx2
(1.40) = (P e)x2 + (Da.P e)x21
dz
(1.41) x2 = P e(x1 − 1) at z=0
(1.42) x2 = 0 at z=1
Shooting method attempts to solve this problem by converting it into and ODE-
IVP problem as follows
• Step 1: Assume the ’missing’ initial condition x1 (0) at z = 0.

• Step 2: Integrate (shoot) the ODEs from z = 0 to z = 1 as if it is an
ODE-IVP using any standard numerical integration method for solving
ODE-IVP
• Step 3: Check whether all the specified boundary conditions are satis-
fied at z = 1.
If not, use method such as Newton - Raphson or successive substi-
tution to correct the guess value at z = 0 and go to step 1.
If the BC at z = 1 is satisfied, terminate the iterations.
For TRAM problem, this method can be applied as follows:
• Assume x1 (0) = s ⇒ x2 (0) = P e(s − 1)
• Integrate the two ODE’s simultaneously using any standard integration
method from z = 0 to z = 1.
• The check to be used comes from the given B.C.
(1.43) f (s) = x2 (1) = 0

• The value of s can be changed from iteration to iteration by the secant
method.
∙ ¸
(k+1) (k) (k) s(k) − s(k−1)
(1.44) s = s − f [s ]
f [s(k) ] − f [s(k−1) ]
Alternatively, we can use Newton-Raphson method for generating
∙ ¸
(k+1) (k) f [s(k) ]
(1.45) s =s −
[df/ds]s=s(k)
The derivative [df /ds]s=s(k) can be computed by simultaneously integrating sen-
sitivity equations. Given a set of the first order nonlinear equations
dx
(1.46) = F (x) ; x(0) = x0 ; x ∈ Rn
dz
and F (x) is a n × 1 vector, the associated sensitivity equations are defined as

∙ ¸
dΦ(z) ∂F
(1.47) = Φ(z) ; Φ(0) = I
dz ∂x
where
∙ ¸
∂x(z)
(1.48) Φ(z) =
∂x0
represents the n × n sensitivity of solution vector x(z) with respect to the initial
conditions and I denotes identity matrix. In the present case, the sensitivity
equations are
" #
dΦ 0 1
(1.49) = Φ(z)
dz 2Da Pe x1 Pe
⎡ ⎤
∂x1 (z) ∂x1 (z)
⎢ ∂x1 (0) ∂x2 (0) ⎥
(1.50) Φ(z) = ⎢ ⎣ ∂x2 (z) ∂x2 (z) ⎦
⎥
∂x1 (0) ∂x2 (0)

These equations have to be integrated from z = 0 to z = 1 to evaluate
∂x2 (1)
(1.51) [df /ds]s=s(k) = Φ21 (1) =
∂x1 (0)
Example 57. Converting a PDE to an ODE-BVP [6]: Consider 2-D
steady state heat transfer problem
∂2T ∂2T
(1.52) + =0
∂x2 ∂y 2
(1.53) x = 0 : T = T ∗; x = 1 : T = T ∗
dT
(1.54) y = 0 : T = T ∗; y = 1 : k = h(T∞ − T )
dx
We construct ny + 1 grid lines parallel to the x-axis. The temperature T along
the j th grid line is denoted as
(1.55) Tj (x) = T (x, yj )
Now, we equate residuals to zero at each internal grid line as
d2 Tj 1
(1.56) 2
= − [Tj+1 (x) − 2Tj (x) + Tj−1 (x)]
dx (∆y)2
j = 2, ....ny − 1
The boundary conditions at y = 0 can be used to eliminate variables on the
corresponding edge. At the boundary y = 1, using the B.C.s we get
dTny
(1.57) k = h(T∞ − Tny )
dx
2. ANALYTICAL SOLUTIONS OF MULTIVARIABLE LINEAR ODE-IVP 135
j = 1, 2, ....ny − 1
The above set of ODE’s can be integrated from x = 0 to x = 1 with initial
condition
(1.58) Tj (0) = T ∗ , (j = 1, 2, ....ny .)
and boundary conditions
(1.59) Tj (1) = T ∗ , (j = 1, 2, ....ny .)
The resulting ODE-BVP can be solved using shooting method.
2. Analytical Solutions of Multivariable Linear ODE-IVP

Consider the problem of solving simultaneous linear ODE-IVP
dx
(2.1) = Ax; x = x(0) at t = 0
dt
x ∈ Rm , A is a (m × m) matrix
To begin with, we develop solution for the scalar case and generalize it to the
multivariable case.
2.1. Scalar Case. Consider the scalar equation

dx
(2.2) = ax; x = x(0) at t = 0
dt
Let the guess solution to this IVP be
(2.3) x(t) = eλt v ; v ∈ R
Now,
(2.4) x = x(0) at t = 0 ⇒ v = x(0)

(2.5) or x(t) = eλt x(0)
This solution also satisfies the ODE, i.e.

dx £ ¤
(2.6) = λ eλt x(0) = λx(t) = ax(t)
dt
(2.7) ⇒ λ = a and x(t) = eat x(0)
Asymptotic behavior of solution can be predicted using the value of parameter

a as follows
• Unstable behavior: a > 0 ⇒ x(t) = eat x(0) → ∞ as t → ∞
• Stable behavior:a < 0 ⇒ x(t) = eat x(0) → 0 as t → ∞
2.2. Vector case. Now consider system of equations given by equation

(2.1). Taking clues from the scalar case, let us investigate a candidate solution
of the form
(2.8) x(t) = eλt v; v ∈ Rm
where v is a constant vector. The above candidate solution must satisfy the
ODE, i.e.,
d λt
(e v) = A(eλt v)
(2.9) dt
⇒ λveλt = Aveλt
Cancelling eλt from both the sides, as it is a non-zero scalar, we get an equation
that vector v must satisfy,
(2.10) λv = Av
This fundamental equation has two unknowns λ and v and the resulting problem
is the well known eigenvalue problem in linear algebra. The number λ is called
the eigenvalue of the matrix A and v is called the eigenvector. Now, λv = Av
is a non-linear equation as λ multiplies v. if we discover λ then the equation
for v would be linear. This fundamental equation can be rewritten as
(2.11) (A − λI)v = 0
This implies that vector v should be ⊥ to the row space of (A − λI). This is
possible only when rows of (A − λI) are linearly dependent. In other words,
λ should be selected in such a way that rows of (A − λI) become linearly
dependent, i.e., (A − λI) is singular. This implies that λ is an eigenvalue of A
if and only if
(2.12) det(A − λI) = 0
This is the characteristic equation of A and it has m possible solutions λ1 , ....., λm .

Thus, corresponding to each eigenvalue λi , there is a vector v(i) that satisfies
(A − λi I)v(i) = 0. This implies that each vector eλi t v(i) is a candidate solution
to equation (2.1). Now, suppose we construct a vector as lineal combination of
these fundamental solutions, i.e.
(2.13) x(t) = c1 eλ1 t v(1) + c2 eλ2 t v(2) + ....... + cm eλm t v(m)

Then, it can be shown that x(t) also satisfies equation (2.1). Thus, a general
solution to the linear ODE-IVP can be constructed as a linear combination of
the fundamental solutions eλi t v(i) .
The next task is to see to it that the above equation reduces to the initial
conditions at t = 0. Defining vectors C and matrix Ψ as
h iT h i
(2.14) C= c1 c2 ... cm ; Ψ= v(1) v(2) ..... v(m)
we can write
(2.15) x(0) = ΨC
If the eigenvectors are linearly independent,
(2.16) C = Ψ−1 x(0)
Thus the solution can be written as
x(t) = [eλ1 t v(1) eλ2 t v(2) .......eλm t v(m) ]Ψ−1 x(0)

⎡ λt ⎤
e 1 0 ... 0
(2.17) ⎢ 0 eλ2 t ... 0 ⎥
⎢ ⎥ −1
⇒ x(t) = [v(1) v(2) .....v(m) ] ⎢ ⎥ Ψ x(0)
⎣ ... ... ... ... ⎦
0 0 0 eλm t
Now let us define the matrix exp(At) as follows
1
(2.18) eAt = I + At + (At)2 + .......
2!
Using the fact that matrix A can be diagonalized as
(2.19) A = ΨΛΨ−1
where matrix Λ is
h i
Λ = diag λ1 λ2 .... λm
we can write
eAt = ΨΨ−1 + ΨΛΨ−1 t + 2!1 ΨΛ2 Ψ−1 t2 + ...

(2.20) = ΨΨ−1 + ΨΛΨ−1 t + 2!1 ΨΛ2 Ψ−1 t2 + ...
= ΨeΛt Ψ−1
Here, the matrix eΛt is limit of infinite sum

eΛt = I + tΛ + 2!1 t2 Λ2 + ...
⎡ λt ⎤
e 1 0 ... 0
(2.21) ⎢ 0 eλ2 t
... 0 ⎥
⎢ ⎥
=⎢ ⎥
⎣ ... ... ... ... ⎦
0 0 0 eλm t
Thus, equation (2.17) reduces to
(2.22) x(t) = ΨeΛt Ψ−1 x(0)
With this definition, the solution to the ODE-IVP can be written as
(2.23) x(t) = ΨeΛt Ψ−1 x(0) = eAt x(0)
2.3. Asymptotic behavior of solutions. In the case of linear multivari-

able ODE-IVP problems, it is possible to analyze asymptotic behavior of the
solution by observing eigenvalues of matrix A.
x(t) = c1 eλ1 t v(1) + c2 eλ2 t v(2) + ....... + cm eλm t v(m)

(2.24)
C = Ψ−1 x(0)
Let λj = αj + iβ j represent j’th eigenvalue of matrix A. Then, we can write
(2.25) eλj t = eαj t .eiβ j t = eαj t [cos β j t + i sin β j t]

As
¯ ¯
(2.26) ¯[cos β j t + i sin β j t]¯ ≤ 1 for all t and all j
the asymptotic behavior of the solution x(t) as t → ∞ is governed by the terms

eαj t . We have following possibilities here
• If αj < 0 then eαj t → 0 as t → ∞
• If αj > 0 then eαj t → ∞ as t → ∞
• If αj = 0 then eαj t → 1 as t → ∞
Thus, we can deduce following three cases
• Case A: ||x(t)|| → 0 as t → ∞ if and only if Re(λi ) < 0 for i =
1, 2, .......m (Asymptotically stable solution)
• Case B: ||x(t)|| ≤ M < ∞ as t → ∞ if and only if Re(λi ) ≤ 0 for
i = 1, 2, .......m (Stable solution)
• Case C: ||x(t)|| → ∞ at t → ∞ if for any λi ,Re(λi ) > 0 for i = 1,2,.......n
(Unstable solution)
Note that asymptotic dynamics of linear ODE-IVP is governed by only eigen-

values of matrix A and is independent of the initial state x(t). Thus, based on
the sign of real part of eignvalues of matrix A, the ODE-IVP is classified as
asymptotically stable, stable or unstable.
Remark 4. The above approach can be extended to obtain local or pertur-

bation solutions of nonlinear ODE-IVP systems
dx
(2.27) = F [x(t), u(t)] ; x = x(0) at t = 0
dt
in the neighborhood of a steady state point x such that
(2.28) F (x) = 0
Using Taylor expansion in the neighborhood of x and neglecting terms higher
that first order, equation (2.27) can be approximated as
∙ ¸
d(x − x) ∂F
= (x − x)
dt ∂x x=x
dδx
(2.29) = A δx ; δx(0) = x(0) − x
dt ∙ ¸
∂F
A=
∂x x=x
Note that the resulting equation is a linear multivariable system of type (2.1)
and the perturbation solution can be computed as
δx(t) = eAt δx(0)
x(t) = x + δx(t)
Example 58. Stirred Tank Reactor
The system under consideration is a Continuous Stirred Tank Reactor (CSTR)

in which a non-isothermal, irreversible first order reaction
A −→ B
is taking place. The dynamic model for a non-isothermal CSTR is given as
follows :
dCA F E
(2.30) = (CA0 − CA ) − k0 exp(− )CA
dt V RT
dT F (−∆Hr ) k0 E Q
(2.31) = (T0 − T ) + exp(− )CA −
dt V ρCp RT V ρCp
aFcb+1
(2.32) Q = µ ¶ (T − Tcin )
aFcb
Fc +
2ρc Cpc
Table 1. Parameters and Steady State Operating Conditions of CSTR
Parameter (↓) Operating Point (→) Stable Unstable

−1
Reaction rate constant (k0 ) min 1010 1010
Inlet concentration of A (CA0 ) kmol/m3 2.0 2.0
Steady state flow rate of A (F ) m3 /min 1.0 1.0
Density of the reagent A (ρ) g/m3 106 106
Specific heat capacity of A(Cp ) cal/g 0 C 1.0 1.0
Heat of reaction (∆Hr ) cal/kmol −130 ∗ 106 −130 ∗ 106
Density of the coolant (ρc ) g/m3 106 106
Specific heat capacity of coolant (Cpc ) cal/g 0 C 1.0 1.0
Volume of the CSTR (V ) m3 1.0 1.0
Coolant flow rate (Fc ) m3 /min 15 15
0
Inlet temperature of the coolant (Tcin ) K 365 365
0
Inlet temperature of A (T0 ) K 323 343
Reactor temperature (T ) K 393.954 349.88
3
Reactor concentration of A (CA ) kmol/m 0.265 1.372
a 1.678X106 0.516X106
Reaction Rate Parameter (E/R ) (0 K)−1 8330 8330
b 0.5 0.5
This system exhibits entirely different dynamic characteristics for different set
of parameter values (Marlin, 1995). The nominal parameter values and nomi-
nal steady state operating conditions of the CSTR for the stable and unstable
operating points are given in Table 1.
• Perturbation model at stable operating point
" # " #" #
d δCA −7.559 −0.09315 δCA
(2.33) =
dt δT 852.7 5.767 δT
∙ ¸
∂F
Eigenvalues of are
∂x x=x
(2.34) λ1 = −0.8960 + 5.9184i ; λ2 = −0.8960 − 5.9184i
and all the trajectories for the unforced system (i.e. when all the inputs
are held constant at their nominal values) starting in the neighborhood
of the steady state operating point converge to the steady state.
3. NUMERICAL METHODS FOR THE SOLUTION OF ODE-IVP 141
• Perturbation model at unstable operating point

" # " #" #
d δCA −1.889 −0.06053 δCA
(2.35) =
dt δT 115.6 2.583 δT
∙ ¸
∂F
Eigenvalues of are
∂x x=x
(2.36) λ1 = 0.3468 + 1.4131i ; λ2 = 0.3468 − 1.4131i
and all the trajectories for the unforced system starting in any small
neighborhood of the steady state operating point diverge from the
steady state.
3. Numerical Methods for the solution of ODE-IVP

There are two basic approaches to integrating ODE-IVP numerically.
• Methods based on Taylor series expansion
- Runge - Kutta methods
• Methods based on polynomial approximation
- Predictor corrector methods
- Orthogonal collocation based methods
In this section, we describe these methods in detail.
3.1. Why develop methods only for the set of first order ODE’s?
In the above illustrations, the system of equations under consideration is the
set of simultaneous first order ODEs represented as
dx
(3.1) = F (x, t) ; x(0) = x0 ; x ∈ Rn
dt
In practice, not all models appear as first order ODEs. In general, one can get
an m’th order ODE of the type:
dm y dy d2 y dm−1 y
(3.2) = f [y, , , ....., , t]
dtm dt dt2 dtm−1
dm−1 y
(3.3) Given y(0), ...... m−1 (0)
dt
Now, do we develop separate methods for each order? It turns out that such a
exercise in unnecessary as a m’th order ODE can be converted to m first order
ODEs. Thus, we can define auxiliary variables

x1 (t) = y(t)
dy
x2 (t) =
dt
(3.4) .......
.......
dm−1 y
xm (t) = m−1
dt
Using these variables, the original nth order ODE can be converted to n first
order ODE’s as,
dx1
= x2
dt
dx2
= x3
dt
(3.5) .......
dxm−1
= xm
dt
dxm
= f [x1 , x2 , x3 , ......., xm , t]
dt
Defining function vector
⎡ ⎤
x2
⎢ ..... ⎥
⎢ ⎥
(3.6) F (x) = ⎢ ⎥
⎣ xm ⎦
f [x1 , x2 , x3 , ......., xm , t]
we can write the above set of
dx
(3.7) = F (x, t)
dt
∙ ¸T
dy dm−1y
(3.8) x(0) = y(0) (0)....... m−1 (0)
dt dt
Thus, it is sufficient to study only the solution methods for solving n first
order ODE’s. Any set of higher order ODEs can be reduced to a set of first
order ODEs. Also, as forced systems (non-homogeneous systems) can be looked
upon as unforced systems (homogenous systems) with variable parameters, it is
sufficient to study the solution methods for homogenous set of equations of the
type (3.1).
3.2. Basic concepts. Consideration is the set of equations defined by

equation (3.1). Let {x∗ (t) : 0 ≤ t ≤ tf } denote the true / actual solution
of the above ODE-IVP. In general, for a nonlinear ODE, it is seldom possible
to obtain the true solution analytically. The aim of numerical methods is to

find an approximate solution numerically. Let t1 , t2 , ......., tn be a sequence of
numbers such that
(3.9) 0 < t1 < t2 < ....... < tn < .... < tf
Instead of attempting to approximate the function x∗ (t), which is defined for

all values of t such that 0 ≤ t ≤ tf , we attempt to approximate the sequence of
vectors {x∗ (tn ) : n = 1, .......f }. Thus, in order to integrate over a large interval
0 ≤ t ≤ tf ,we solve a sequence of ODE-IVPs subproblems
dx
(3.10) = F (x, t) ; x(tn ) = x(n) ;
dt
tn ≤ t ≤ tn+1 ; (n = 1, 2, ......f )
each defined over a smaller interval [tn , tn+1 ] . This generates a sequence of
approximate solution vectors {x(tn ) : n = 1, .......f }. The difference hn = tn −
tn−1 is referred to as the integration step size or the integration interval. Two
possibilities can be considered regarding the choice of the sequence {tn }
• Fixed integration interval: The numbers tn are equispaced, i.e., tn = nh
for some h > 0
• Variable size integration intervals
For the sake of convenience, we introduce the notation
(3.11) F (n) ≡ F [x(tn ), tn ]

(3.12) x(n) ≡ x(tn )
µ ¶ µ ¶
∂F ∂F
(3.13) =
∂x n ∂x (x(tn ),tn )
and use it throughout in the rest of the text.

3.2.1. Two Basic Approaches : Implicit and Explicit. There are two basic
approaches to numerical integrations. To understand these approaches, consider
the integration of the equation (3.1) over the interval [tn , tn+1 ] using Euler’s
method. Let us also assume that the numbers tn are equi-spaced and h is the
integration stepsize.
• Explicit Euler method:If the integration interval is small,
dx ∼ x(n + 1) − x(n)
= = F [x(n), tn ]
(3.14) dt h
x(n + 1) = x(n) + hF (n), (n = 0, 1, ......., n − 1)
The new value x(n + 1) is a function of only the past value of x i.e.,
x(n). This is a non-iterative scheme.
• Implicit Euler method:
dx ∼ x(n + 1) − x(n)
= = F [x(n + 1), tn+1 ]
(3.15) dt h
x(n + 1) = x(n) + hF (n + 1), (n = 0, 1, ......., n − 1)
Each of the above equation has to be solved by iterative method.
For example if we use successive substitution method for solving the
resulting nonlinear equation(s), the algorithm can be stated as follows:
Initialize: x(0), tf , h, ∈, N = tf /h
FOR n = 1 TO n = N
x(0) (n + 1) = x(n) + hF [x(n), tn ]
WHILE ( δ >∈)
x(k+1) (n + 1) = x(n) + hF [x(k) (n + 1), tn+1 ]
||x(k+1) (n + 1) − x(k) (n + 1)||
δ=
||x(k) (n + 1)||
END WHILE
x(n + 1) = x(k) (n + 1)
END FOR
3.2.2. Variable stepsize implementation with accuracy monitoring. One prac-
tical difficulty involved in the integration with fixed stepsize is the choice of
stepsize such that the approximation errors are kept small. Alternatively, a
variable stepsize algorithm is implemented with error monitoring as follows.
Given: tn , x(n) = x(tn ), ε
(1)
• Step 1: Choose stepsize h1 and let tn+1 = tn + h1
• Step 2: Compute x(1) (n + 1) using an integration method (say explicit
Euler).
(2)
• Step 3: Define h2 = h1 /2; tn+1 = tn + h2
(2) (1)
tn+2 = tn + 2h2 (= tn+1 )
(2)
Compute x(2) and xn+2 by the same integration method.
° (1) °
• Step 4: IF (°x (n + 1) − x(2) (n + 2)° < ε),
(Accept x(1) (n + 1) as the new value)
Set x(n + 1) = x(1) (n + 1), and n = n + 1 and proceed to
Step 1.
ELSE
set h1 = h2 and proceed to the step 2.
END IF
3.2.3. Stiffness of ODEs. The problem of integrating multi-variable ODE-

IVP with some variables changing very fast in time while others changing slowly,
is difficult to solve. This is because, the stepsize has to be selected according to
the fastest changing variable / mode. For example, consider the equation
" # " #" #
d y1 −100 0 y1
(3.16) =
dt y2 2 −1 y2
" # " #
−100 0 2
(3.17) A= ; y(0) =
2 −1 1
It can be shown that the solution for the above system of equations is
" # " #
y1 (t) 2e−100t
(3.18) = 103 −t 4 −100t
y2 (t) 99
e − 99
e
It can be observed that the terms with e−100t lead to a sharp decrease in y1 (t)
and to a small maximum in y2 (t) at t = 0.0137. The term y2 (t) is dominated
by e−t which decreases slowly. Thus,
(3.19) y1 (t) < 0.01y1 (0) f or t > 0.03
(3.20) y2 (t) < 0.01y1 (t) for t > 4.65

Now, stepsize should be selected such that the faster dynamics can be captured.
The stiffness of a given ODE-IVP is determined by finding the stiffness ratio
defined as
|Reλi (A)|max
(3.21) S.R. =
|Reλi (A)|min
where matrix A is defined above. Systems with ’large’ stiffness ratio are called
as stiff.
Remark 5. This analysis can be extended to a general nonlinear systems

only locally. Using Taylor’s theorem, we can write
dx
(3.22) = F (x) = F [x(n) + x(t) − x(n)]
dt ∙ ¸
∼ ∂F
(3.23) = F (xn ) + [x(t) − x(n)]
∂x x=x(n)
Local stiffness ratio can be calculated using eigenvalues of the Jacobian and the
ODE-IVP is locally stiff if the local S.R. is high, i.e., the system has at least
one eigenvalue which does not contribute significantly over most of the domain
of interest. In general, eigenvalues of the Jacobian are time dependent and S.R.
is a function of time. Thus, for stiff systems it is better to use variable step size
methods or special algorithms for stiff systems.
3.3. Taylor’s series based methods. Consider a simple scalar case

dx
(3.24) = f (x, t) ; x ∈ R
dt
Suppose we know the exact solution x∗ (t) at time tn , i.e. x∗ (n), then we can
compute x∗ (n + 1) using Taylor series as follows:
dx∗ (tn ) 1 d2 x∗ (tn )
(3.25) x∗ (n + 1) = x∗ + h + h2 + .......
dt 2! dt2
The various derivatives in the above series can be calculated using the differential
equation, as follows:
dx∗ (tn )
(3.26) = f [tn , x∗ (n)]
dt ∙ ¸
d2 x∗ (tn ) ∂f ∗ ∂f [x∗ (n), tn ]
(3.27) = f [x (n), tn ] +
dt2 ∂x (x∗ (n),tn ) ∂t
and so on. Let us now suppose that instead of actual solution x∗ (n), we have
available an approximation to x∗ (n), denoted as x(n). With this information,
we can construct x(n + 1), as
∙µ ¶ µ ¶ ¸
h2 ∂f ∂f
(3.28) x(n + 1) = x(n) + hf (n) + f (n) + + .......
2 ∂x n ∂t n
We can now make a further approximation by truncating the infinite series.
If the Taylor series is truncated after the term involving hk , then the Taylor’s
series method is said to be of order k.
• Order 1(Euler explicit formula)
(3.29) x(n + 1) = x(n) + hf (n)
• Order 2
∙µ ¶ µ ¶ ¸
h2 ∂f ∂f
(3.30) x(n + 1) = x(n) + hf (n) + f (n) +
2 ∂x n ∂t n
Taylor’s series methods are useful starting points for understanding

more sophisticated methods, but are not of much computational use.
First order method is too inaccurate and the higher order methods
require calculation of a lot of partial derivatives.
3.4. Runge-Kutta (R-K) Methods.

3.4.1. Univariate Case. Runge-Kutta methods duplicate the accuracy of the

Taylor series methods, but do not require the calculation of higher partial deriv-
atives. For example, consider the second order method that uses the formula:
(3.31) x(n + 1) = x(n) + ak1 + bk2
(3.32) k1 = hf (tn , x(n)) = hf (n)

(3.33) k2 = hf (tn + αh, x(n) + βk1 )
The real numbers a, b,α, β, are chosen such that the RHS of (3.31) approximates
the RHS of Taylor series method of order 2 (ref. 3.30). To achieve this, consider
Taylor series expansion of the function k2 , about (tn , x(n)).
k2
(3.34) = f (tn + αh, x(n) + βhf (n))
h µ ¶ µ ¶
∂f ∂f
(3.35) = f (tn , x(n)) + αh + βh f (n) + O(h3 )
∂t n ∂x n
Substituting this in equation (3.31), we have
x(n + 1) = x(n) + ahf (n)

∙ µ ¶ µ ¶ ¸
∂f ∂f
(3.36) +bh f (tn , x(n)) + αh + βh f (n) + O(h3 )
∂t n ∂x n
µ ¶ µ ¶
2 ∂f 2 ∂f
(3.37) = x(n) + (a + b)hf (n) + αbh + βbh f (n) + O(h3 )
∂t n ∂x n
Comparing 3.30 and 3.37, we get

a+b=1
(3.38)
αb = βb = 12
There are 4 unknowns and 3 equations and so we can assign one arbitrarily
giving:
a = 1 − b;
(3.39) 1 1
α= 2b
; β = 2b ; b 6= 0
Thus, the general 2nd order algorithm can be stated as
∙ µ ¶¸
h h
(3.40) x(n + 1) = x(n) + h (1 − b)f (n) + bf tn + , x(n) + f (n)
2b 2b
• Heun’s modified algorithm:Set b = 1/2.
∙ ¸
1 1
(3.41) x(n + 1) = x(n) + h ( f (n) + f (tn + h, x(n) + hf (n))
2 2
• Modified Euler-Cauchy Method: Set b = 1.

∙ ¸
h h
(3.42) x(n + 1) = x(n) + hf tn + , x(n) + f (n)
2 2
It must be emphasized that 3.40. and 3.30 do not give identical results.
However, if we start from the same x(n), then x(n + 1) given by 3.30
and 3.40 would differ only by O(h3 ).
3.4.2. Multivariate Case. Even though the above derivation has been worked
for one dependent variable case, the method can be easily extended to multi-
variable case. For example, the most commonly used fourth order R-K method
for one variable can be stated as
h
(3.43) x(n + 1) = x(n) + (k1 + 2k2 + 2k3 + +k4 )
6
(3.44) k1 = f (tn , x(n)) = f (n)

µ ¶
h h
(3.45) k2 = f tn + , x(n) + k1
2 2
µ ¶
h h
(3.46) k3 = f tn + , x(n) + k2
2 2
(3.47) k4 = f (tn + h, x(n) + hk3 )
Now, suppose we want to use this method for solving m simultaneous ODE-IVPs
dx
(3.48) = F(x, t)
dt
(3.49) x(0) = x0
where x ∈ Rm and F (x, t) is a m×1 function vector. Then, the above algorithm
can be modified as follows
h
(3.50) x(n + 1) = x(n) + (k1 + 2k2 + 2k3 + +k4 )
6
(3.51) k1 = F (tn , x(n)) = F(n)

µ ¶
h h
(3.52) k2 = F tn + , x(n) + k1
2 2
µ ¶
h h
(3.53) k3 = F tn + , x(n) + k2
2 2
(3.54) k4 = F (tn + h, x(n) + hk3 )
Note that k1 , k2 , k3 and k4 are n × 1 function vectors.

Note that Runge-Kutta methods can be implemented using variable step
size with accuracy monitoring. Thus, these methods (with variable step size
implementation) are preferred when x(t) is expected to change rapidly in some

regions and slowly in others.
3.5. Multistep Methods.

3.5.1. Univariate Case. The multi-step methods are based on the Weier-
strass theorem, which states that any continuous function over a finite interval
can be uniformly approximated to any desired degree of accuracy by a poly-
nomial of appropriate degree. Thus, we approximate the solution of a given
differential equation by a polynomial in the independent variable t. In order to
understand how this is achieved, consider the scalar differential equation
dx
(3.55) = f (x, t); x(tn ) = x(n); x ∈ R
dt
with uniformly spaced integration (time) intervals. At time t = tn , we have state
and derivative information in the ’past’ i.e.
{x(n), x(n − 1), x(n − 2).........x(0)}
and
{f (n), f (n − 1), f (n − 2).........f (0)}
which can be used to construct the polynomial approximation. We approxi-
mate x(t) in the neighborhood of t = tn by constructing a local polynomial
approximation of type
(3.56) x(n) (t) = a0,n + a1,n t + a2,n t2 + ..... + am,n tm
and use it to estimate or extrapolate x(n + 1).(Note that subscript ’n’ used
for coefficients indicate that the coefficients are corresponding to polynomial
approximation at time t = tn ). Here, the coefficients of the polynomial are
estimated using the state and derivative information from the past and possibly
f (n + 1). In order to see how this can be achieved, consider a simple case where
we want construct a second order approximation
(3.57) x(n) (t) = a0,n + a1,n t + a2,n t2

at instant t = tn . This implies the derivative f (x, t) at time t can be computed
as
dx(n) (t)
(3.58) f (x, t) = = a1,n + 2a2,n t
dt
For the sake of computational convenience, we choose a shifted time scale as
follows:
(3.59) tn = 0; tn+1 = h; tn−1 = −h,
Now, there are several ways we could go about estimating the unknown para-
meters of the polynomial.
• Explicit algorithm: Let us use only the current and the past infor-
mation of state and derivatives, which will lead to an explicit algorithm.
f (n − 1) = a1,n − 2a2,n h
(3.60) f (n) = a1,n
(3.61) x(n) = a0,n

Solving above equations simultaneously, we get coefficients
f (n) − f (n − 1)
(3.62) a0,n = x(n) ; a1,n = f (n) ; a2,n =
2h
which can be used to extrapolate the value of x(t) at t = tn+1 as
(3.63) x(n + 1) = a0,n + a1,n h + a2,n h2
∙ ¸
f (n) − f (n − 1) 2
= x(n) + f (n) h + h
2h
∙ ¸
3 1
= x(n) + h f (n) − f (n − 1)
2 2
• Implicit algorithm: Alternatively, we can choose to estimate x(n+1)
based on derivative at tn+1 ,i.e.
(3.64) f (n + 1) = a1,n + 2a2,n h
(3.65) f (n) = a1,n
(3.66) x(n) = a0,n

These equations yield following set of coefficients
f (n + 1) − f (n)
(3.67) a0,n = x(n) ; a1,n = f (n) ; a2,n =
2h
and x(n + 1) can be estimated as
(3.68) x(n + 1) = a0,n + a1,n h + a2,n h2
∙ ¸
f (n + 1) − f (n) 2
(3.69) = x(n) + f (n) h + h
2h
The above expression can be rearranged as
h
(3.70) x(n + 1) = x(n) + [f (n) + f (n + 1)]
2
which is popularly known as trapezoidal rule or Crank-Nicholson algo-
rithm.
Thus, a more general expression for computational form of x(n + 1) can be

stated as
x(n + 1) = α0 x(n) + α1 x(n − 1) + ....... + αp x(n − p)
£ ¤
(3.71) +h β −1 f (n + 1) + β 0 f (n) + β 1 f (n − 1) + .... + β p f (n − p)
or
p p
X X
(3.72) x(n + 1) = αi x(n − i) + h β i f (n − i)
i=0 i=−1
where p is an integer and αi , β i are real numbers to be selected. Note that if

β −1 6= 0, we get an implicit formula for the unknown quantity x(n + 1), else
we get an explicit formula. An algorithm of the type 3.71 is called (p + 1) step
algorithm because x(n+1) is given in terms of the values of x at previous (p+1)
steps [x(n), ......., x(n − p)] . Formula 3.71 can also be thought of arising from
the discrete approximation of the expression
Z t
∗
(3.73) x(t) = x0 + f [x(τ ), τ ]dτ
0
Order of the algorithm is the degree of the highest-degree polynomial for which
3.71 gives an exact expression of x(n + 1). To see how this definition of order is
used, consider the development of the m’th order algorithm in scalar case. i.e.,
x ∈ R. (similar arguments can be used in vector case). Suppose polynomial
solution of initial value problem is given by
X
m
(n) 2 m
(3.74) x (t) = a0,n + a1,n t + a2,n t + ..... + am,n t = aj,n (t)j
j=0
dx(n) X m
(3.75) f (x, t) = = a1,n + 2a2,n t + ........ + m am,n tm−1 = jaj,n (t)j−1
dt j=1
For the sake of convenience, we choose a shifted time scale as follows:

(3.76) tn = 0; tn+1 = h; tn−1 = −h, .......tn−i = −ih
Thus we have,
(3.77) x(n + 1) = a0,n + a1,n h + a2,n h2 + ..... + am,n hm
(3.78) x(n − i) = a0,n + a1,n (−ih) + a2,n (−ih)2 + ....... + am,n (−ih)m
(3.79) i = 0, 1, ...., p
(3.80) f (n − i) = a1,n + 2a2 , n(−ih) + ........ + m am,n (−ih)m−1

(3.81) i = −1, 0, ....p
Substitution of equations (3.77),(3.78) and (3.80) into (3.71) gives what is known
as the exactness constraints for the algorithm as
p
"m # p
" m #
Xm X X X X
aj,n (h)j = αi aj,n (−ih)j + h βi jaj,n (−ih)j−1
j=0 i=0 j=0 i=−1 j=1
Ã p ! Ã p p
!
X X X
= αi a0,n + (−i)αi + (−i)0 β i a1,n h + ...
i=0 i=0 i=−1
Ã p p
!
X X
(3.82) ... + (−i)m αi + m (−i)m−1 β i am,n hm
i=0 i=−1
Because we would like (3.82) to hold independently of any stepsize, we obtain

the following equations (constraints) by equating like powers of h.
p
X
(3.83) αi = 1 ; (j = 0)
i=0
p p
X X
j
(3.84) (−i) αi + j (−i)j−1 β i = 1 ; (j = 1, 2, ......., m)
i=0 i=−1
Note : (i)j = 1 when i = j = 0

Thus, equations (3.88-3.90) gives m + 1 constraints and the number of variables
are 2p + 3, namely α0 , .......αp , β −1 , β 0 , .......β p . Any choice of these constants
makes the corresponding algorithm 3.71 exact for m’th order polynomial. The
set of equations (3.83) provide (m+1) constraints on (2p+3) variables. Thus, in
order for the algorithm (3.71) to be exact in case of the mth degree polynomial
we must have
(3.85) (m + 1) ≤ 2p + 3
If equality holds, i.e. when
(3.86) m = 2(p + 1)
then we can solve for {αi } and {β i } exactly.

Now, let us re-derive the 2’nd order implicit algorithm again using the above
approach. Constraints for this case can be generated by equating coefficients of
p
X
2
a0,n + a1,n h + a2,n h = αi [a0,n + a1,n (−ih) + a2,n (−ih)2 ]
i=0
p
X
(3.87) +h β i [a1,n + 2a2,n (−ih)]
i=−1
The resulting constraints are

p
X
(3.88) αi = 1
i=0
p p
X X
(3.89) (−iαi ) + βi = 1
i=0 i=−1
p p
X X
(3.90) i2 αi + (−2iβ i ) = 1
i=0 i=−1
Clearly for (3.88-3.90) to hold, we must have 2p + 3 ≥ 3. The second order

algorithm with the smallest number of constants αi , β i is obtained by setting
2p + 3 = 3, i.e., p = 0. In this case,
α0 = 1
(3.91) β −1 + β 0 = 1
2β −1 = 1
which gives
(3.92) α0 = 1; β −1 = 1/2; β 0 = 1/2
and the second order algorithm becomes
h
(3.93) x(n + 1) = x(n) + [f (n) + f (n + 1)]
2
3.5.2. Examples of Multi-step methods. A number of multi-step algorithms
can be obtained by making suitable choices of the parameters {αi } and {β i }.
Some of the popular algorithms are discussed in this sub-section.
Adams-Bashworth (explicit method): Choose
(3.94) α1 = α2 = ....... = αp = 0
(3.95) β −1 = 0
(3.96) p = m−1
These are additional (p + 1) equations.
Total number of constraints = (m + 1) + (p + 1). = 2m + 1
Total number of variables = (2p + 3) = 2m + 1
Out of these, (p +1 = m) variables are selected to be zero and (m +1) constants
namely, α0 , β 0 , .......β p are to be detected. Using constraints for j = 0,
p
X
(3.97) αi = 1; ⇒ α0 = 1
i=0
Using the other constraints,

⎡ ⎤⎡ ⎤ ⎡ ⎤
1 1 ... 1 β0 1/j
⎢ 0 (−1) ... (−p) ⎥⎢ β1 ⎥ ⎢ 1/j ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
(3.98) ⎢ ⎥⎢ ⎥=⎢ ⎥
⎣ ... ... ... ... ⎦⎣ ... ⎦ ⎣ ... ⎦
0 (−1)m−1 ... (−p)m−1 βp 1/j
Solving for the β 0 s, we can write the algorithm as

£ ¤
(3.99) x(n + 1) = x(n) + h β 0 f (n) + β 1 f (n − 1) + ....... + β p f (n − p)
Adam-Moulton Implicit Algorithms: Choose
(3.100) p=m−2
(3.101) α1 = α2 = ....... = αp = 0
For j = 0, we have
p
X
(3.102) αi = 1; ⇒ α0 = 1
i=0
Remaining m variables β −1 , ......., β m−2 can be determined by solving

⎡ ⎤⎡ ⎤ ⎡ ⎤
1 1 ... 1 β −1 1
⎢ 0 (−1) ... (−p) ⎥ ⎢ ⎥ ⎢ ⎥
⎢ ⎥ ⎢ β 0 ⎥ ⎢ 1/2 ⎥
(3.103) ⎢ ⎥⎢ ⎥=⎢ ⎥
⎣ ... ... ... ... ⎦ ⎣ ... ⎦ ⎣ ... ⎦
0 (−1)m−1 ... (−p)m−1 βp 1/m
The algorithm can be written as
" #
β 0 f (n) + β 1 f (n − 1)+
(3.104) x(n + 1) = x(n) + h
... + β p f (n − p)] + β −1 f (n + 1)
(3.105) = yn + hβ −1 f [x(n + 1), tn+1 ]
where y(n) represents sum of all terms which are known from the past data.
The above implicit equation has to be solved iteratively to obtain x(n + 1).
3.5.3. Predictor-Corrector Algorithms. We saw that a m step Adams-Bashworth
algorithm is exact for polynomials of order m, while a m-step Adams-Moulton
algorithm is exact for the polynomials of order (m + 1). However, the Adams-
Moulton algorithm is implicit, i.e.,
(3.106) x(n + 1) = y(n) + hβ −1 f [x(n + 1), tn+1 ]

where the quantity y(n) depends on x(n), ......., x(n − p) and is known. The
above implicit equation can be solved iteratively as
£ ¤
(3.107) x(k+1) (n + 1) = y(n) + hβ −1 f x(k) (n + 1), tn+1
where iterations are terminated when
(3.108) |x(k+1) (n + 1) − x(k) (n + 1)| <∈
If we choose the initial guess x(0) (n + 1) reasonably close to the solution, the
convergence of the iterations is accelerated. To achieve this, we choose x(0) (n +
1) as the value generated by an explicit m−step algorithm and then apply
the iterative formula. This is known as the predictor-corrector method. For
example, a two-step predictor-corrector algorithm can be given as
∙ ¸
(0) 3 1
(3.109) x (n + 1) = x(n) + h f (n) − f (n − 1) (Predictor)
2 2
(3.110) ∙ ¸
(k+1) 1 (k) 1
x (n + 1) = x(n) + h f (x (n + 1), tn+1 ) + f (n) (Corrector)
2 2
If the stepsize is selected properly, relatively few applications of the correction
formula are enough to determine x(n + 1), with a high degree of accuracy.
Gear’s Predictor-Corrector Algorithms: A popular algorithm used
for numerical integration is Gear’s predictor corrector. The equations for this
algorithm are as follows:
• Gear’s m-th order predictor algorithm is an explicit algorithm, with
(3.111) p = m−1
(3.112) β −1 = β 1 = ....... = β p = 0; β 0 6= 0
(3.113) x(n + 1) = α0 x(n) + α1 x(n − 1) + ... + αp x(n − p) + hβ 0 f (n)
• Gear’s m-th order corrector
(3.114) p = m−1
(3.115) β 0 = β 1 = ....... = β p = 0; β −1 6= 0
(3.116)
x(n + 1) = α0 x(n) + α1 x(n − 1) + ....... + αp x(n − p) + hβ −1 f (n + 1)
Coefficients of the above algorithm can be computed by setting up appro-

priate constraint equations as shown above.
3.5.4. Multivariate Case. Even though the above derivations have been worked
for one dependent variable case, these methods can be easily extended to multi-
variable case
dx
(3.117) = F (x, t) ; x ∈ Rn
dt
where F (x, t) is a n × 1 function vector. In the multivariable extension, the
scalar function f (x, t) is replaced by the function vector F (x, t), i.e.
(3.118)
x(n + 1) = α0 x(n) + α1 x(n − 1) + ....... + αp x(n − p)
£ ¤
+h β −1 F (n + 1) + β 0 F (n) + β 1 F (n − 1) + .... + β p F (n − p)
where
(3.119) F (n − i) ≡ F [x(tn − ih), (tn − ih)]
i = −1, 0, 1, ...p
© ª
and the scalar coefficients α0 ....αp , β −1 , β 0 , β 1 , ......β p are identical with the
coefficients derived for the scalar case as described in the above section.
The main advantages and limitations of multi-step methods can be summa-
rized as follows
• Advantages:
There are no extraneous ’inter-interval’ calculations as in the case
of Runge-Kutta methods.
Can be used for stiff equations if integration interval is chosen care-
fully.
• Limitations:
Time instances should be uniformly spaced and selection of integra-
tion interval is a critical issue.
4. Stability Analysis and Selection of Integration Interval

Selection of integration interval is a crucial parameter while solving ODE-
IVPs numerically. In order to see how choice of integration interval can affect
solution behavior consider a scalar linear equation
dx
(4.1) = ax; x(0) = x0
dt
Analytical (true) solution of the above equation is given as
(4.2) x∗ (t) = eat x(0)
Defining x∗ (tn ) = x∗ (n), we can write true solution as a difference equation
(4.3) x∗ (n) = eanh x(0) ⇒ x∗ (n + 1) = eah x∗ (n)
4. STABILITY ANALYSIS AND SELECTION OF INTEGRATION INTERVAL 157
Now consider the approximate solution of the above ODE-IVP by explicit Euler
methods
x(n + 1) = x(n) + hf (n)
(4.4) = (1 + ah)x(n)
⇒ x(n) = (1 + ah)n x(0)
Defining approximation error introduced due to numerical integration,
(4.5) e(n) = x∗ (n) − x(n)
we can write
£ ¤
(4.6) e(n + 1) = (1 + ah)e(n) + eah − (1 + ah) x∗ (n)
Thus, the combined equation becomes

" # " £ ¤ #" #
e(n + 1) (1 + ah) eah − (1 + ah) e(n)
(4.7) =
x∗ (n + 1) 0 eah x∗ (n)
Now, let us consider the situation where a < 0, i.e. the ODE is asymptotically
¯ ¯
stable and x∗ (n) → 0 and n → ∞ as ¯eah ¯ < 1. Thus, we can expect that the
approximate solution {x(n)} should exhibit similar behavior qualitatively and
e(n) → 0 as n → ∞. This requires that the difference equation given by (4.7)
should be asymptotically stable, i.e., all eigen values of matrix
" £ ¤ #
(1 + ah) eah − (1 + ah)
0 eah
should have magnitude strictly less than one. Thus, the approximation error
e(n) → 0 as n → ∞ provided the following condition holds
(4.8) |1 + ah| < 1 ⇒ −2 < ah < 0
This inequality gives constraint on the choice of integration interval h, which

will ensure that approximation error will vanish asymptotically.
Following similar line of arguments, we can derive conditions for choosing
integration interval for different methods. For example,
• Implicit Euler
" # ⎡ ∙ ¸ ⎤" #
1 ah 1
e(n + 1) e − e(n)
(4.9) = ⎣ (1 − ah) (1 − ah) ⎦
x∗ (n + 1) 0 eah x∗ (n)
¯ ¯
¯ 1 ¯
(4.10) ¯ ¯
¯ 1 − ah ¯ < 1 ⇒ ah < 0
• Trapeziodal Rule (Simpson’s method)

" # ⎡ ∙ ¸ ⎤" #
(ah)2 ah 1 + (ah/2)
e(n + 1) 1 + ah + e − e(n)
(4.11) =⎣ 2 1 − (ah/2) ⎦
x∗ (n + 1) x∗ (n)
0 eah
¯ ¯
¯ 1 + (ah/2) ¯
(4.12) ¯ ¯
¯ 1 − (ah/2) ¯ < 1 ⇒ ah < 0
• 2’nd Order Runge Kutta Method

" #
e(n + 1)
(4.13)
x∗ (n + 1)
⎡ µ ¶ ∙ µ ¶¸ ⎤ " #
(ah)2 ah (ah)2
1 + ah + e − 1 + ah + e(n)
(4.14) = ⎣ 2 2 ⎦
x∗ (n)
0 eah
¯ ¯ 2
¯ 2¯
(4.15) ¯1 + ah + (ah) ¯ < 1 ⇒ −2 < ah + (ah) < 0
¯ 2 ¯ 2
Thus, choice of integration interval depends on the parameters of the equa-

tion to be solved and the method used for solving ODE IVP. These simple
example also demonstrates that the approximation error analysis gives consid-
erable insight into relative merits of different methods. For example, in the case
of implicit Euler or Simpson’s rule, the approximation error asymptotically re-
duces to zero for any choice of h > 0. (Of course, larger the value of h, less
accurate is the numerical solution.) Same is not true for explicit Euler method.
This clearly shows that implicit Euler method and Simpson’s rule are superior
to explicit Euler method.
It may be noted that we start solving ODE-IVP from a point x(0) = x∗ (0)
i.e. e(0) = 0.
The above analysis can be easily extended to a coupled system of linear
ODE-IVP of the form
dx
= Ax
dt
(4.16) x = x(0) at t = 0
x ∈ Rn A ≡ (n × n) matrix
Following similar arguments as in the scalar case, it can be shown that condition
for choosing integration interval are as follows
4. STABILITY ANALYSIS AND SELECTION OF INTEGRATION INTERVAL 159
• Explicit Euler
(4.17) x(n + 1) = (I + hA)x(n)
(4.18) ρ [I + hA] < 1
where ρ(.) represents spectral radius of the matrix [I + hA] . When
matrix A is diagonalizable, i.e. A = ΨΛΨ−1 , we can write
(4.19) I + hA = Ψ [I + hΛ] Ψ−1
and eigen values of matrix I + hA are {(1 + hλi ) : i = 1, 2, ..., n} where
{λi : i = 1, 2, ..., n} represent eigenvalues of matrix A. Thus, the stabil-
ity requirement reduces to
(4.20) |1 + hλi | < 1 for i = 1, 2, ..., n
• Implicit Euler
(4.21) x(n + 1) = (I − hA)−1 x(n)
£ ¤
(4.22) ρ (I − hA)−1 < 1
• Trapeziodal Rule:
µ ¶−1 µ ¶
h h
(4.23) x(n + 1) = I − A I + A x(n)
2 2
"µ ¶−1 µ ¶#
h h
(4.24) ρ I− A I+ A <1
2 2
Similar error analysis (or stability analysis) can be performed for other inte-
gration methods. For example, when the 3-step algorithm is used for obtaining
the numerical solution of
x(n + 1) = β −1 hf (n + 1) + α0 x(n) + α1 x(n − 1) + α2 x(n − 2)

= aβ −1 hx(n + 1) + α0 x(n) + α1 x(n − 1) + α2 x(n − 2)
1
= + [α0 x(n) + α1 x(n − 1) + α2 x(n − 2)]
1 − aβ −1 h
(4.25) = η 0 x(n) + η 1 x(n − 1) + η 2 x(n − 2)
The above difference equation can be rearranged in the following form.
⎡ ⎤ ⎡ ⎤⎡ ⎤
x(n − 1) 0 1 0 x(n − 2)
⎢ ⎥ ⎢ ⎥⎢ ⎥
(4.26) ⎣ x(n) ⎦ = ⎣ 0 0 1 ⎦ ⎣ x(n − 1) ⎦
x(n + 1) η2 η1 η0 x(n)
Defining
⎡ ⎤ ⎡ ⎤
x(n − 2) x(n − 1) 0 1 0
⎢ ⎥ ⎢ ⎥
(4.27) z(n) = x(n − 1) ; z(n + 1) = ⎣ x(n) ⎦ ; B = ⎣ 0 0 1 ⎦
x(n) x(n + 1) η2 η1 η0
we have
(4.28) z(n + 1) = Bz(n)

x(n + 1) = z3 (n + 1)
h i
(4.29) = 0 0 1 z(n) = Cz(n)
Similarly, the true solution can be expressed as
(4.30) z∗ (n + 1) = B ∗ z∗ (n)
(4.31) x(n + 1) = Cz∗ (n)
where
⎡ ⎤
eah 0 0
⎢ ⎥
(4.32) B ∗ = ⎣ 0 eah 0 ⎦
0 0 eah
The evolution of the approximation error is given as
(4.33) e(n + 1) = Be(n) + [B ∗ − B] z∗ (n)

(4.34) e(n) = z∗ (n) − z(n)
If the stability criterion that can be used to choose integration interval h can
be derived as
(4.35) ρ(B) < 1
Note that characteristic equation for matrix B is given as
(4.36) λ3 − η 0 λ2 − η 1 λ − η 2 = 0
Thus, eigenvales of matrix B can be directly computed using the coefficients

η 0 , η 1 and η 2, which are functions of integration interval h.
Equations such as (4.18), (4.22) and (4.36) can be used to generate stability
envelopes for each method in the complex plane (eigenvalues of a matrix can be
complex). Stability envelopes for most of the methods are available in literature.
The following general conclusions can be reached by studying these plots [6].
5. SUMMARY 161
• Even though the first and second order Adams-Moulton methods ( im-
plicit Euler and Crank-Nicholson) are A-stable, the higher order tech-
niques have restricted regions of stability. These regions are larger than
the Adams-Bashworth family of the same order.
• All forms of the R-K algorithms with order ≤ 4 have identical stability
envelopes.
• Explicit R-K techniques have better stability characteristics than ex-
plicit Euler.
• For predictor-corrector schemes, accuracy of scheme improves with or-
der. However, stability region shrinks with order.
Remark 6. The conclusions reached from studying linear systems can be
extended to general nonlinear systems locally using Taylor expansion.
dx
(4.37) = F (x)
dt
can be approximated as
∙ ¸
dx ∼ ∂F
(4.38) = F (x(n)) + (x − x(n))
dt ∂x x=x(n)
∙ ¸ " ∙ ¸ #
∼ ∂F ∂F
(4.39) = x + F [x(n)] − x(n)
∂x x=x(n) ∂x x=x(n)
(4.40) ∼
= (A)n x + (d)n
Applying some numerical technique to solve this problem will lead to
(4.41) x(n + 1) = (B)n x(n + 1) + (c)n
and stability will depend on the choice of h such that ρ[(B)n ] < 1 for all n. Note
that, it is difficult to perform global analysis for general nonlinear systems.
5. Summary
In these lecture notes, we undertake the study of solutions of multivariable
and coupled ODE-IVPs. To begin with, we show that variety of problems, such
as solving ODE-BVP, hyperbolic / parabolic PDEs or set of nonlinear algebraic
equations, can be reduced to solving ODE-IVP. A special class of problems,
i.e. solutions of coupled linear ODE-IVPs can be solved analytically. Thus,
before we start the development of the numerical methods, we will develop
analytical solutions for unforced (homogeneous) linear ODE-IVP problem and
investigate their asymptotic behavior using eigenvalue analysis. We later discuss
development of numerical algorithms based on
• Taylor series approximations (Runge-Kutta methods)
• Polynomial approximation based algorithms (Predictor-corrector type

methods).
In the end, we provide a brief introduction to the stability analysis of the
numerical algorithms for solving ODE-IVPs.
6. Exercise
(1) Express the following set of equations in the standard form
dx/dt = Ax; x(0) = x(0)
and solve the resulting initial value problem analytically

(a) Set 1
d2 y/dt2 + 4dy/dt + 3y = 0; y(0) = 1; dy/dt = 0 at t = 0
(b) Set 2
d3 y/dt3 + 6d2 y/dt2 + 11dy/dt + 6y = 0
y(0) = 1; dy/dt = d2 y/dt2 = 0 at t = 0;

(c) Set 3
dy/dt + 3y + z = 0; y(0) = 1
d2 z/dt2 + 3dz/dt + 2z = 0
z(0) = 1; dz/dt = 0
Compare the coefficients of the characteristic equation, i.e. det(
λI − A) = 0, and those of the ODE(s) for the first two sets. Also,
comment upon the asymptotic behavior of the solution in each case
based on eigenvalues of matrix A.
(2) Consider the dynamic model of three isothermal CSTRs in series (Ex-
ample 1). The model parameters are Residences time values: τ 1 = 1
min τ 1 = 2 min τ 3 = 3 min and the reaction rate constant k = 0.5
(min−1 )
(a) Assuming that the CSTR is at a steady state initially ( i.e., dc/dt =
0) with CA0 = 1.8, find the corresponding steady state concentra-
tion by solving the resulting linear algebraic equations.
(b) Suppose, we decide to shutdown the reactor system and reduce
CA0 = 0 at t = 0. Integrate the resulting set of homogeneous ODEs
analytically to obtain the concentration profile C(t), starting with
the steady state obtain above.
6. EXERCISE 163
(c) Use explicit Euler method to integrate the above set of equations
from t =0 to t = 2 with integration interval of 0.1, 0.25, 0.5, 1.0
and compare the approximate solutions with the corresponding
analytical solution in each case.
(d) Repeat (c) for the case h=0.25 using implicit Euler method.
(3) Consider the PDE given below
∂C/∂t = ∂ 2 C/∂z 2
C(0, t) = C(1, t) = 0 for all 0 ≤ t ≤ ∞
C(z, 0) = 1 for 0 ≤ z ≤ 1
(a) Use the finite difference technique on the dimensionless diffusion
equation obtain a set of ODE-IVPs assuming N internal grid points.
Particularly for the case N = 3, obtain the analytical solution to
the resulting ODE-IVP.
(b) Repeat the above exercise using orthogonal collocation to dis-
cretize in space with two internal collocation points.
(4) Consider Van der Pol equation given below
d2 y/dt2 − (1 − y 2 )dy/dt + 3y = 0
y(0) = 2; dy/dt = 0 at t = 0
(a) Express the above ODE-IVP in standard form
dx/dt = F (x); x = x(0) at t = 0
(b) Linearize the resulting equation in the neighborhood of x = [ 0 0 ]
and obtain the perturbation solution analytically. Comment upon
the asymptotic behavior of the solution.
(5) Consider the quadruple tank setup shown in Figure 1.
Figure 1. Quadruple tank setup: Schematic diagram

CHAPTER 5
Optimization and Related Numerical Schemes
1. Introduction
These lecture notes deal with multivariable unconstrained optimization
techniques and their application to computing numerical solutions of various
types of problems. One of the major application of unconstrained optimization
is model parameter estimation (function approximation or multivariate regres-
sion). Thus, we begin by providing a detailed description of the model para-
meter estimation problem. We then derive necessary and sufficient conditions
for optimality for a general multivariable unconstrained optimization problem.
If the model has a nice structure, such as it is linear in parameters or can
be transformed to a linear in parameter form, then the associated parameter
estimation problem can be solved analytically. The parameter estimation of
linear in parameter models (multivariate linear regression) problem is treated
next. Geometric and statistical properties of the linear least square problem are
discussed in detail to provide further insights into this formulation. Numerical
methods for estimating parameters of the nonlinear-in-parameter models are
presented in subsequent section. The applications of optimization formulations
for solving problems such as solving linear/ nonlinear equations and solution of
PDEs using finite element method are discussed in the last section.
2. Principles of Optimization
2.1. Necessary Conditions for Optimality. Given a real valued
scalar function F (z) : Rn → R defined for any z ∈ Rn .
Definition 29. (Global Minimum): If there exists a point z∗ ∈ Rn such

that F (z∗ ) < F (z) for any z ∈ RN , then z∗ is called as global minimum of F (z).
Definition 30. ε-neighborhood of a point a be defined as the set Ne (a) =

{z : kz − ak ≤ ε}
Definition 31. (Local Minimum) : If there exists an ε−neighborhood

NC (z∗ ) round z∗ such that F (z∗ ) < F (z) for each z ∈ Ne (z), then z∗ is called
local minimum.
165
166 5. OPTIMIZATION AND RELATED NUMERICAL SCHEMES
Theorem 11. If F (z) is continuous and differentiable and has an extreme

(or stationary) point (i.e. maximum or minimum ) point at z = z∗ , then
∙ ¸T
∗ ∂F ∂F ∂F
(2.1) ∇F (z ) = .............. =0
∂z1 ∂z2 ∂zN z=z∗
.
Proof: Suppose z = z∗ is a minimum point and one of the partial deriva-

tives, say the kth one, does not vanish at z = z∗ , then by Taylor’s theorem
X
N
∂F
∗ ∗
(2.2) F (z + ∆z) = F (z ) + (z∗ )∆zi + R2 (z∗ , ∆z)
i=1
∂zi
∂F ∗
(2.3) i.e. F (z∗ + ∆z) − F (z∗ ) = ∆zk (z ) + R2 (z∗ , ∆z)
∂zi
Since R2 (z∗ , ∆z) is of order (∆zi )2 , the terms of order ∆zi will dominate over
the higher order terms for sufficiently small ∆z. Thus, sign of F (z∗ +∆z)−F (z∗ )
is decided by sign of
∂F ∗
∆zk (z )
∂zk
Suppose,
∂F ∗
(2.4) (z ) > 0
∂zk
then, choosing ∆zk < 0 implies
(2.5) F (z∗ + ∆z) − F (z∗ ) < 0 ⇒ F (z∗ + ∆z) < F (z∗ )
and F (z) can be further reduced by reducing ∆zk . This contradicts the assump-
tion that z = z∗ is a minimum point. Similarly, if
∂F ∗
(2.6) (z ) < 0
∂zk
then, choosing ∆zk > 0 implies
(2.7) F (z∗ + ∆z) − F (z∗ ) < 0 ⇒ F (z∗ + ∆z) < F (z∗ )
and F (z) can be further reduced by increasing ∆zk . This contradicts the as-
sumption that z = z∗ is a minimum point. Thus, z = z∗ will be a minimum of
F (z) only if
∂F ∗
(2.8) (z ) = 0 F or k = 1, 2, ...N
∂zk
Similar arguments can be made if z = z∗ is a maximum of F (z).
2. PRINCIPLES OF OPTIMIZATION 167
2.2. Sufficient Conditions for Optimality. Before we prove sufficient

condition for optimality, we revise some relevant definitions from linear algebra.
Definition 32. (Positive Definite Matrix) A n × n matrix A is called

positive definite if for every z ∈Rn
(2.9) zT Az >0
whenever z 6=0.
Definition 33. (Positive Semi-definite Matrix) A n × n matrix A is

called positive semi-definite if for every z ∈Rn we have
(2.10) zT Az ≥0
Definition 34. .(Negative Definite Matrix) A n × n matrix A is called

negative definite if for every z ∈Rn
(2.11) zT Az <0
whenever z 6=0.
Definition 35. (Negative Semi-definite Matrix) A n × n matrix A is

called negative semi-definite if for every z ∈Rn we have
(2.12) zT Az ≤0
The sufficient condition for optimality, which can be used to establish whether
a stationary point is a maximum or a minimum, is given by the following theo-
rem.
∗
Theorem 12. A sufficient ¸ for a stationary point z = z to be an
∙ condition
2
∂ F
extreme point is that matrix (Hessian of F )evaluated at z = z∗ is
∂zi ∂zj
(1) positive definite when z = z∗ is minimum
(2) negative definite when z = z∗ is maximum
Proof: Using Taylor series expansion, we have
X
N
∂F 1 XX ∂ 2 F (z∗ + λ∆z)
N N
F (z∗ + ∆z) = F (z∗ ) + (z∗ )∆z + ∆zi ∆zj
i=1
∂zi 2! i=1 j=1 ∂zi ∂zj
(2.13) (0 < λ < 1)
.Since z = z∗ is a stationary point we have
(2.14) ∇F (z∗ ) = 0
Thus, above equation reduces to
1 XX ∂ 2 F (z∗ + λ∆z)
N N
∗ ∗
(2.15) F (z + ∆z) − F (z ) = ∆zi ∆zj
2! i=1 j=1 ∂zi ∂zj
(0 < λ < 1)
This implies that sign of F (a + ∆z) − F (a)at ∙extreme¸ point z∗ is same as
∂2F
sign of R.H.S. Since the 2’nd partial derivative is continuous in the
∂zi ∂zj
neighborhood of z = z∗ , its value at z = z∗ + λ∆z will have same sign as its
value at z = z∗ for all sufficiently small ∆z. If the quantity
X
N X
N
∂ 2 F (z∗ + λ∆z)
(2.16) ∆zi ∆zj ' (∆z)T [∇2 F (z∗ )]∆z ≥0
i=1 j=1
∂zi ∂zj
for all ∆z, then z = z∗ will be a local minimum. In other words, if Hessian ma-
trix [∇2 F (z∗ )] positive semi-definite, then z = z∗ will be a local minimum.
If the quantity
X
N X
N
∂ 2 F (z∗ + λ∆z)
(2.17) ∆zi ∆zj ' (∆z)T [∇2 F (z∗ )]∆z ≤0
i=1 j=1
∂zi ∂zj
for all ∆z, then z = z∗ will be a local maximum. In other words, if Hessian
matrix [∇2 F (z∗ )] negative semi-definite, then z = z∗ will be a local maxi-
mum.
Remark 7. It should be noted that the need to define a positive definite

or negative definite matrix naturally arises from the geometric considerations
while qualifying a stationary point in multi-dimensional optimization. When
the Hessian is positive definite, a better insight into the local geometry can be
obtained by plotting function
(2.18) q(∆z) = (∆z)T [∇2 F (z∗ )]∆z = 1
To begin with, consider a special case when z ∈ R2 and [∇2 F (z∗ )] = I. In this
case
(2.19) (∆z)T [∇2 F (z∗ )]∆z = (∆z1 )2 + (∆z2 )2 = 1
which represents a unit circle in the neighborhood of z = z∗ . In R3 , (∆z)T ∆z = 1
represents a sphere in the neighborhood of z = z∗ . Now, suppose we consider
h i
[∇2 F (z∗ )] = diag 4 1 1/9
2. PRINCIPLES OF OPTIMIZATION 169
then
(2.20) (∆z)T [∇2 F (z∗ )]∆z = 4 (∆z1 )2 + (∆z2 )2 + (1/9) (∆z3 )2 = 1
As the coefficients of quadratic terms are unequal and positive, we get an ellip-
soid instead of a sphere in the neighborhood of z = z∗ .
Now, suppose we consider a dense and positive definite [∇2 F (z∗ )] such
as
" #
5 4
(2.21) [∇2 F (z∗ )] =
4 5
then, we have
(2.22) (∆z)T [∇2 F (z∗ )]∆z = 5 (∆z1 )2 + 8 (∆z1 ∆z2 ) + 4 (∆z2 )2 = 1
This is still an ellipse in the neighborhood of z = z∗ , however, its axis are not
aligned parallel to the coordinate axis. Matrix [∇2 F (z∗ )] can be diagonalized as
" # " √ √ #" #" √ √ #T
5 4 1/ 2 1/ 2 1 0 1/ 2 1/ 2
(2.23) = √ √ √ √
4 5 −1/ 2 1/ 2 0 9 −1/ 2 1/ 2
Defining a rotated coordinates ∆y as
" #
∆y1
(2.24) ∆y = = ΨT ∆z
∆y2
" √ √ #T " #
1/ 2 1/ 2 ∆z1
(2.25) = √ √
−1/ 2 1/ 2 ∆z2
we have
" #
1 0
(∆z)T [∇2 F (z∗ )]∆z = ∆yT ∆y
0 9
(2.26) = (∆y1 )2 + 9 (∆y2 )2 = 1
h iT
∗
Figure 1 shows ellipsoid when z = 0 0 . Note that the coordinate trans-
formation ∆y = ΨT ∆z has rotated the axis of the space to match the axes of the
ellipsoid. Moreover, the major axis is aligned along the eigenvector correspond-
ing to the largest magnitude eigenvalue and the minor axis is aligned along the
smallest magnitude eigenvalue.
In more general settings, when z ∈ Rn , let 0 < λ1 ≤ λ2 ≤ .... ≤ λn
represent eigenvalues of the Hessian matrix. Using the fact that Hessian is
positive definite, we can write
(2.27) [∇2 F (z∗ )] = ΨΛΨT
Figure 1
where
h i
Ψ= (1) (2) (n)
v v .... v
represents unitary matrix with eigenvectors of the Hessian as its columns and Λ
is a diagonal matrix with eigenvalues on the diagonal. Using this transforma-
tion, we have
¡ ¢
(2.28) q(∆z) = (ΨT ∆z)T Λ ΨT ∆z = (∆y)T Λ (∆y)
(2.29) = λ1 (∆y1 )2 + λ2 (∆y2 )2 + ....λn (∆yn )2 = 1
From the above expression, it is easy to see that q(∆y) is an ellipsoid in n

dimensions with its major axis aligned along the eigenvector v(n) and the minor
axis along the eigenvector v(1) . Thus, function F (z) looks like an ellipsoid in
the neighborhood of z = z∗ when z∗ is a minimum.
Remark 8. Whether a matrix is positive (semi)definite, negative (semi)

definite or indefinite can be established using eigen values of the matrix. If
eigenvalues of a matrix are all real positive (i.e. λi ≥ 0 for all i) then, the
matrix is positive semi-definite. If eigenvalues of a matrix are all real negative
3. MODEL PARAMETER ESTIMATION 171
(i.e. λi ≤ 0 for all i) then, the matrix is negative semi-definite. When eigen
values have mixed signs, the matrix is indefinite.
3. Model Parameter Estimation

3.1. Mathematical Models in Engineering. Mathematical model-
ing is an elegant tool for describing various processes and phenomena occurring
in a processing plant or a manufacturing system. Mathematical models play
important role in design and scaling of a new process or understanding sta-
tic/dynamic behavior of an existing plant. Typically, such models are gray box
models and are developed by judiciously combining first principles (i.e. energy,
momentum and material balances) with semi-empirical correlations developed
from experimental data. As a consequence, such models involve a number of
parameters, such as heat and mass transfer coefficients, parameters, reaction
kinetics, correlation coefficients etc., which have to be estimated from the ex-
perimental data. In general, these models can be expressed in abstract form
as
y = f (x, θ)
where x ∈ Rm represents vector of independent variables (e.g.. temperature,
pressure, concentration, current, voltage etc.) and let y ∈ R denotes dependent
variable, f (.) represents proposed functional relationship that relates y with x
and θ ∈ Rl represent vector of model parameters.
Example 59. Correlations

(1) Specific heat capacity at constant pressure (Cp, ), as a function of tem-
perature
(3.1) CP = a + bT + cT 2
h iT
y ≡ Cp ; x ≡ T ; θ ≡ a b c
(2) Dimensionless analysis is mass transfer / heat transfer
(3.2) Sh = α0 Reα1 Scα2

h iT
y = Sh ; x = [Re Sc]T ; θ ≡ α0 α1 α2
¡ ¢α
(3.3) Nu = α0 Reα1 Prα2 µa /µp 3
h iT
y = Nu ; x = [Re Pr]T ; θ ≡ α0 α1 α2 α3
(3) Friction factor as a function of Reynold’s number for turbulent flow

p p
(3.4) 1/ f = α log(Re f ) − β
h iT
y = f ; x = Re ; θ ≡ α β
(4) Equation(s) of state: e.g. Redlish Kwong equation
RT a
(3.5) P = − 1/2
V − b T (V + b)V
h iT h iT
y=P ; x= T V ; θ≡ a b
or Van der Waals equation

a
(3.6) (P + )(V − b) = RT
V2
h iT h iT
y=P ; x= T V ; θ≡ a b
(5) Antonie equation for estimating vapor pressure of a pure component
B
(3.7a) log(Pv ) = A −
T +C
Example 60. Reaction rate models:
µ ¶
dCA
(3.8) −rA = − = ko exp(−E/RT ) (CA )n
dt
h iT
y ≡ −rA ; x ≡ [CA T ] T ; θ ≡ n E ko
Example 61. Step response of a second order process (dynamic modeling

for process control )
(3.9) δT (t) = K [1 + α1 exp(−t/τ 1 ) + α2 exp(−t/τ 2 )] ∆Fc
h i
y = δT ; x ≡ t ; θ = K α1 α2 τ 1 τ 2
Here δT represents deviation temperature obtained in response to step change

in cooling water flow rate of magnitude ∆Fc . This is a nonlinear in parameter
model.
3.2. Classification of Models. Based on the manner in which the para-

meters appear in model equations, we can categorize the model as follows:
• Linear in parameter models: The most common type of approxi-
mation considered is from the class of functions
(3.10) yb = θ1 f1 (x) + θ2 f2 (x) + ........ + θm fm (x)
As the parameters θ1 , ...θm appear linearly in the model, the model
is called as linear in parameter model. Note that fi (x) can be nonlinear
functions of x. More commonly used linear forms are
— Simple polynomials
yb = θ1 + θ2 x + θ3 x2 + ......... + θm xm−1
— Legendre polynomials
(3.11) yb = θ1 L0 (x) + θ2 L1 (x) + ........ + θm Lm−1 (x)
— Chebysheve polynomials
(3.12) yb = θ1 T0 (x) + θ2 T1 (x) + ....... + θm Tm−1 (x)
— Fourier series
(3.13) yb = θ1 sin(ωx) + θ2 sin(2ωx) + ........ + θm sin(mωx)
— Exponential form with α1 ....αm specified
(3.14) yb = θ1 eα1 x + θ2 eα2 x + .......... + θm eαm x

• Nonlinear in parameter models: In many problems the parameters
appear nonlinearly
in the model, i.e.
(3.15) yb = f (x ; θ1 , .., θ m ) ; (i = 1, ....N)
where f is a nonlinear function of parameters θ1 ..., θ m .
Example 62. • Linear and Nonlinear in Parameter Models

• Cp as a function of temperature (equation 3.1) is a linear in parameter
model.
• Reaction rate model (equation 3.8), model for friction factor (equation
3.4), Antonie equation (equation 3.7a), second order dynamics (equa-
tion 3.9), heat and mass transfer correlations (equations 3.3 and 3.2)
are examples of nonlinear in parameter models. However, some of these
models can be transformed to linear in parameter models. For example,

the transformed reaction rate model
µ ¶
E 1
log(−rA ) = log(ko ) + n log CA −
R T
(1)
Example 63. • is a linear in parameter model.

3.3. Formulation of Parameter Estimation Problem. Estimation of
model parameter from experimental data is not an easy task as the data ob-
tained from experiments is always influenced by uncertain variation of uncon-
trollable and unknown variables, which occur while conducting experiments and
collecting data. In modeling parlance, the data is corrupted with unmeasured
disturbances and measurement errors. For example,
• If we measure flow, pressure, temperature etc., through electronic trans-
mitters, there are errors or noise in the measurements due to local
electrical disturbances.
• While conducting experiments involving heating with a steam coil, un-
measured fluctuations in steam header pressure may introduce varia-
tions in the rate of heating
In any experimental evaluation, we can list many such unknown factors
which influence the data. Apart from these influences, the proposed mathe-
matical models are often approximate descriptions of underlying phenomenon
and additional errors are introduced due to limitations imposed by modeling
assumptions. Thus, when we develop a model from experimental data, we can
identify three possible sources of error
• Measurement errors : Errors in measurements of various recorded
variables
• Unmeasured disturbances: Unrecorded influences
• Modeling Errors : Errors arising due to fact that the model equa-
tion(s) represents only an approximate description of the reality.
When we develop mathematical models using data corrupted with measure-
ment errors and unmeasured disturbances, it becomes necessary to characterize
the unknown component in the data for estimating the model parameters ac-
curately. Let xt ∈ Rn denote a vector of true values of independent variables
(e.g.. temperature, pressure, concentration, current, voltage etc.) and let yt ∈ R
denote the true value of dependent variable. This can be expressed as
yt = FT (xt , Θ)
where FT (.) represents true functional relationship that relates yt with xt and
Θ represent the parameter vector. When we collect data from an experiment,
we get a set of measurements x(k) and yk such that
(k)
x(k) = xt +ε(k)
yk = yt,k + vk
k = 1, 2, ......N
where ε(k) and v (k) represent errors in measurements of independent and depen-
dent variables, respectively, and N represents the size of data sample. Given
these measurements, the model relating these measured quantities can be stated
as ³ ´
(k)
yt,k = F xt , θ +ek
where f (.) represents the proposed approximate functional relationship, θ rep-
resent the parameter vector and ek represents the equation error for k’th sample
point. Thus, the most general problem of estimating model parameters from
experimental data can be stated as follows:
Estimate of θ such that
min ¡ ¢
f e1 , e2 , .....eN , ε(1) , ....ε(N) , v1 , ...vN
θ
subject to
h i
(i)
ei = yi − F xt , θ
(i)
ε(i) = x(i) − xt
vi = yi − yt,i
for i = 1, 2, ....N
where, f (.) represents some scalar objective function.
Given a data set, formulation and solution of the above general modeling
problem is not an easy task. In these lecture notes, we restrict ourselves to a
special class of models, which assume that
(1) Measurement errors in all independent variables are negligible i.e. x =
xT
(2) Effect of all unknown disturbances and modeling errors can be ade-
quately captured using equation error model, i.e.
y = F (x, θ) +e
The term e on R.H.S. can be interpreted either as modeling error or as
measurement error while recording y.
These assumption considerably simplifies problem formulation. Under these

assumptions, the model parameter estimation problem can be stated as estima-
tion of θ such that
min
f (e1 , e2 , .....eN )
θ
£ ¤
ei = yi − yb x(i) , θ ; i = 1, 2, ....N
3.4. Interpolation and Approximation. Problem definition : Let

(i)
a set {x : i = 1, ...N } of variable x corresponds to a set {yi : i = 1, .......N} of
variable y and let
(3.16) yb = f (x, θ1 , .....θ m )
represent the proposed model where f is a continuous function linear in para-

meters. Substituting xi we get
¡ ¢
(3.17) ybi = f x(i) , θ1 , .....θ m ( i = 1, 2......N)
a set of N linear equations in m unknowns.

Case (N = m) : If N equations are independent then a unique solution
exists and we can obtain the function yb = f (x, θ1 , ...θm ) passing through each
of these N points i.e. we have done an interpolation.
Case (N > m) : When we have an overdetermined set of equations , in
general there is neither a solution nor a function f (x, θ1 , ...θm ) passing through
all points. Define the errors ei as
(3.18) ei = yi − ybi for i = 1, 2, ....N

¡ (i) ¢
(3.19) = yi − f x , θ1 , .....θm
This is a system of N linear equations in (m + N) unknowns, namely m parameters

θj and the N error terms ei . Such a system has infinite solutions. Among these,
we choose the one that minimizes
X
N
(3.20) || e ||2 = e2i wi
i=1
Mathematically, we have two different problems for a given set of N pairs

(xi , yi ) with m parameters.
• Interpolation (m = N) : Interpolation is used in the case of ‘precisely’
known values (xi , yi ) and when it is required to predict y at points x
other than support points xi . e.g. when using logarithmic tables.
Figure 2. Interpolation
Figure 3. Approximation
• Approximation (N > m) : In case of experimental values, imprecise

values are obtained due to measurement noise and other uncontrolled
perturbations. The aim of approximation ( which is also called smooth-
ing ) is to try to eliminate the effects of noise as much as possible from
experimental data.
Example 64. Consider energy consumption of a boat as a function of dis-

tance (see Figures 2 and 3). It is clear that total energy consumed is directly
proportional to distance but the atmospheric factors namely wind, water current
etc. create perturbations. Figures 2 and 3 also shows results of interpolation
and smoothing, respectively. Clearly, in this case the interpolation is unable to
bring out the trend in energy consumption while smoothing produces acceptable
results.
In the case of imprecise data, the result of interpolation is absurd when
the original problem is sensitive to errors. Consequently, these results cannot be
used for any future predictions. On the other hand, smoothing allows us to bring
out a tendency to reduce the probable energy consumption y for any arbitrary
distance x.
3.5. Quality of Approximation. When we approximate a real con-

tinuous function{y(x) : x ∈ [a, b]} or a set of numerical data {yi : i = 1, 2......N } by
an analytic function yb(x), it is desirable to choose yb(x) such that error between
the approximate function and real function / measurements is small in some
sense. The ’distance’ between the real function/ measurements and its model
predictions can be measured by the norm function. For the set of numerical
¡ ¢ ¡ ¢
data x(i) , yi where yi = y x(i) , let us define vectors Y of measurements
h i
y = y1 y2 .... yN
N×1
b of model predictions as
and y
h i
b=
y yb1 yb2 .... ybN
N×1
Then, we choose parameters of (3.10) such that
bkp
f (e1 , e2 , .....eN ) = ky − y
is minimized. Commonly used norms are

• Laplace - norm (1 - norm )
X
N
(3.21) b ||1 =
|| y − y | yi − ybi |
i=1
• Euclidean - norm ( 2 - norm )
X
N
(3.22) b ||2 =
|| y − y ( yi − ybi )2
i=1
4. MULTIVARIATE LINEAR REGRESSION 179
• Laplace - Chebyshev norm ( ∝ - norm )

(3.23) b ||∝ = max | yi − ybi |
|| y − y
i=1....N
By definition, the best approximation yb of y in the least squares will imply

that || y − yb ||2 is minimum. The 2 - norm defined above is often inadequate in
practice as some values can be more precise than the others and approximation
yb should be more influenced by precise values than the others. For the above
reasons, a weighted least square norm is introduced as
X
N
(3.24) b ||2,w =
|| y − y ( yi − ybi )2 wi
i=1
where wi ≥ 0 ( for i = 1, 2............N )are weights associated with

experimental values.
In the case of approximation of a continuous function defined on an interval
[a, b] we define the following norms.
Zb
(3.25) || y(z) − yb(z) ||1 = | y(z) − yb(z) | dz
a
Zb
(3.26) || y(z) − yb(z) ||2 = [ y(z) − yb(z) ]2 dz
a
Zb
(3.27) || y(z) − yb(z) ||2,w = [ y(z) − yb(z) ]2 w(z) dz
a
(3.28) || y(z) − yb(z) ||∝ = max | y(z) − yb(z) |

z∈[a,b]
The procedures for the determination of the best approximation are both for
continuous and discrete cases.
4. Multivariate Linear Regression

4.1. Least Square Formulation for Linear In Parameter Models.
Suppose the following model is proposed for a phenomenon
X
m
(4.1) yb = θj fj (x)
j=1
where x ∈ Rr and we have N experimental data sets { (x(i) , yi ) : i = 1, ......N }.

Defining the ith approximation error as
X
m
(4.2) ei = yi − ybi = yi − θj fj (x(i) )
j=1
it is desired to choose a solution that minimizes the scalar quantity

" #
min XN
(4.3) f= e2i wi
θ1 ....θm i=1
where wi ≥ 0 are the weights associated with the individual measurements.

These weights can be chosen to reflect reliability of each experimental data. A
relatively large weight wi can be selected for the experimental data set (x(i) , yi )
that is more reliable and vice-versa.
The above optimization problem can be expressed in compact vector-matrix
notation as follows
¡ ¢
(4.4) yb1 = θ1 f1 (x(1) ) + θ2 f2 x(1) + .......... + θm fm (x(1) )
¡ ¢
(4.5) yb2 = θ1 f1 (x(2) ) + θ2 f2 x(2) + .......... + θm fm (x(2) )
..... = .................................................
¡ ¢
(4.6) ybN = θ1 f1 (x(N ) ) + θ2 f2 x(N) + .......... + θm fm (x(N) )
Defining
h iT
(4.7) θ = θ1 θ2 .... θm ∈ Rm
h iT
(4.8) ϕ(i) = f1 (x(i) ) f2 (x(i) ) .... fm (x(i) ) ∈ Rm
h i
(4.9) b =
y yb1 yb2 .... ybN ∈ RN
and
⎡ ⎤ ⎡ ¡ ¢T ⎤
f1 (x(1) ) ........ fm (x(1) ) ϕ(1)
⎢ ⎥ ⎢ ⎥
(4.10) Φ = ⎣ ...... ......... ...... ⎦ = ⎣ ...... ⎦
¡ (N ) ¢T
f1 (x(N) ) ........ fm (x(N) ) N x m ϕ
we get an over - determined set of equations
(4.11) b = Φθ
y
Let the vector of measurements be defined as

h iT
(4.12) y = y1 y2 .... yN ∈ RN
4. MULTIVARIATE LINEAR REGRESSION 181
Now, defining approximation error vector e ∈ RN as

(4.13) e b = y − Φθ
= y−y
(4.14)
it is desired to choose θ such that the quantity Φ = eT We is minimized, i.e.
(4.15) b = min eT We
θ
θ
where
h i
(4.16) W = diag w1 w2 .... wN
is a diagonal weighting matrix.
4.2. Solution of Linear Least Square Problem. Consider the min-

imization problem
b = min T
(4.17) θ e We
θ
e b = y − Aθ
= y−y
Using the necessary condition for optimality, we have
£ ¤
∂ eT We
=0
∂θ
Rules of differentiation of a scalar function f = uT Bv with respect to vectors
u and v can be stated as follows
∂ ∂ T
(4.18) (uT Bv) = Bv ; [u Bv] = B T u
∂u ∂v
∂
(4.19) [uT Bu] = 2 Bu when B is symmetric
∂u
Now, applying the above rules
(4.20) eT We = [y − Φθ]T W [y − Φθ]
= yT Wy − (Φθ)T Wy − yT WΦθ + θT (ΦT WΦ)θ
£ T ¤
∂ e We b =0
(4.21) = −ΦT Wy − ΦT Wy + 2(ΦT WΦ)θ
∂θ
(4.22) bLS = ΦT Wy
⇒ (ΦT WΦ) θ
In the above derivation, we have used the fact that matrix ΦT WΦ is symmetric.
b can
If matrix (ΦT WΦ) is invertible, the least square estimate of parameters θ
computed as
£ ¤ ¡ ¢
(4.23) bLS = ΦT WΦ −1 ΦT W y
θ
Using sufficient condition for optimality, Hessian matrix should be positive

definite or positive semi-definite for the stationary point to be a minimum.
Now,
" £ ¤#
∂ 2 eT We
(4.24) = 2(ΦT WΦ)
∂θ2
It can be easily shown that

¡ ¢
(4.25) vT ΦT WΦ v ≥ 0 for any v ∈Rm
and the sufficiency condition is satisfied and the stationary point is a minimum.
ˆ
As Φ is a convex function, it can be shown that the solution θ is the global
minimum of Φ.
Thus, linear least square estimation problem is finally reduced to solving
equation of the form Ax = b where
(4.26) A = ΦT WΦ and b =ΦT Wy
Note that ΦT WΦ is symmetric and positive semi-definite and Cholensky

decomposition method can be used to solve the resulting linear system.
5. Projections: Geometric Interpretation of Linear Regression

5.1. Distance of a Point from a Line. Suppose we are given a
3
point y ∈R in space and we want to find its distance from the line in the
direction of vector a ∈R3 . In other words, we are looking for a point p along
the line that is closest to y (see Figure 4),i.e p =θa such that
(5.1) Φ = kp − yk2 = kθa − yk2
is minimum. This problem can be solved by minimizing Φ with respect to

θ,which is equivalent to
min 2 min
(5.2) Φ = hθa − y,θa − yi
θ θ
min £ 2 ¤
(5.3) = θ ha, ai − 2θ ha, yi + hy, yi
θ
5. PROJECTIONS: GEOMETRIC INTERPRETATION OF LINEAR REGRESSION 183
Figure 4
Using necessary condition for optimality,

∂Φ2
(5.4) = θ ha, ai − ha, yi = 0
∂θ
ha, yi
(5.5) ⇒ θ=
ha, ai
ha, yi
(5.6) p = θa = a
ha, ai
Now, equation (5.4) can be rearranged as
(5.7) ha,θai − ha, yi = ha,θa − yi = ha, p − yi = 0
From school geometry we know that if p is such a point, then the vector
(y − p) is perpendicular to direction a. We have derived this geometric result
using optimization. Equation (5.6) can be rearranged as
* +
a a
(5.8) p= p ,y p a, yi b
= hb a
ha, ai ha, ai
a
where ba= p is unit vector along direction of a.and point p is the projec-
ha, ai
tion of vector y along direction ba. Note that the above derivation holds in any
general n dimensional space a, y ∈Rn or even any infinite dimensional vector
space.
Figure 5
The equation can be rearranged as

µ ¶ ∙ ¸
aT y 1 £ ¤
(5.9) p= a = aaT y = Pr .y
aT a aT a
T
where Pr = aT1 a aa is a n × n matrix is called as projection matrix, which
projects vector y into its column space.
5.2. Distance of a point from Subspace. The situation is exactly

same when we are given a point y ∈R and plane S in R3 passing through
3
origin,we want to find distance of y from S,i.e. a point p ∈S such that kp − yk2
is minimum (see Figure 5). Again, from school geometry, we know that such
point can be obtained by drawing a perpendicular from y to S ; p is the point
where this perpendicular meets S (see Figure 5). We would like to formally
derive this result using optimization.
More generally, given a point y ∈Rm and subspace S of Rm , the problem
is to find a point p in subspace S such that it is closest to vector y. Let
© ª
S = span a(1) , a(2) , ...., a(m) and as p ∈ S we have
X
m
(1) (2) (m)
(5.10) p = θ1 a + θ2 a + .... + θm a = θi a(i)
i=1
We want to find a point find p such that

°Ã m ! °
° X °
° °
(5.11) Φ = kp − yk2 = ° θi a(i) −y°
° °
i=1 2
is minimum. This problem is equivalent to

*Ã m ! Ãm !+
min 2 min X X
(i) (i)
(5.12) Φ = θi a −y , θi a −y
θ θ i=1 i=1
Using necessary condition for optimality,

* Ãm !+
∂Φ2 X ¡ ¢ ®
(j) (i)
(5.13) = a , θi a −y = a(i) , (p − y) = 0
∂θj i=1
j = 1, 2, ...m
Equation (5.13) has a straight forward geometric interpretation. Vector p − y

is orthogonal to each vector a(i) , which forms the basis of S. The point p is a
projection of y onto subspace S. This is exactly what we learn in the school
geometry (see Figure 5).
Now, let us calculate optimal parameters θi using equation (5.13). Equation
(5.13) can be rearranged as
* +
Xm Xm
® ®
(j) (i)
(5.14) a , θi a = θi a(j) , a(i) = a(j) , y
i=1 i=1
(5.15) j = 1, 2, ...m
Collecting the above set of equations and using vector-matrix notation, we have
(5.16)
⎡ (1) (1) ® (1) (2) ® (1) (m) ® ⎤ ⎡ ⎤ ⎡ (1) ® ⎤
a ,a a ,a .... a ,a θ1 a ,y
⎢ a(2) , a(1) ® a(2) , a(2) ® .... a(2) , a(m) ® ⎥ ⎢ θ ⎥ ⎢ a(2) , y® ⎥
⎢ ⎥⎢ 2 ⎥ ⎢ ⎥
⎢ ⎥⎢ ⎥=⎢ ⎥
⎣ ..... ..... ..... ..... ⎦ ⎣ .... ⎦ ⎣ .... ⎦
(m) (1) ® (m) (2) ® (m) (m) ® (m) ®
a ,a a ,a ..... a , a θm a ,y
This is nothing but the classic normal equation derived in the above subsec-
tion.
Let us now interpret the least square parameter estimation problem stated in
the last section using the above geometric arguments. The least square problem
was posed as choosing the parameter vector θ b such that
q
(5.17) || e ||W,2 = || y − Φθ ||W,2 = [y − Φθ]T W [y − Φθ]
is minimized. The subscript (W,2 ) indicates that k.kW,2 is a 2-norm defined

using matrix W. This is exactly the geometrical problem of finding distance of
vector Y from a subspace S . The sub-space involved is nothing but the column
space of Φ. Let Φ be represented as
h i
(5.18) Φ = a(1) a(2) .... a(m)
where a(i) ∈ RN are columns of matrix Φ. Let us define the inner product for
any u, v ∈ RN as
(5.19) hu, vi = uT Wv
Let the vector p such that
(5.20) θ1 a(1) + b
p=b θ2 a(2) + .... + b b
θm a(m) = Φθ
Then, using geometric arguments, we have

D E £ ¤ £ ¤ h i
(5.21) b = a(1) T Wy − a(1) T W Φθ
a(1) , y − Φθ b =0
D E £ (2) ¤T £ (2) ¤T h i
(5.22) (2) b
a , y − Φθ = a Wy − a b =0
W Φθ
.................... = ........................................... = 0
D E £ ¤ £ ¤ h i
(5.23) a(m) , y − Φθ b = a(m) T Wy − a(m) T W Φθ b =0
Rearranging above equations, we have

⎡ £ ¤T ⎤ ⎡ ⎤ ⎡ £ ¤T ⎤ ⎡ £ ¤T ⎤
a (1) b
θ1 a(1) Wy a(1)
⎢ £ (2) ¤T ⎥ ⎢b ⎥ ⎢ £ ¤ ⎥ ⎢ £ (2) ¤T ⎥
⎢ a ⎥ ⎢ θ2 ⎥ ⎢ a(2) T Wy ⎥ ⎢ a ⎥
⎢ ⎥ ⎢ ⎥=⎢ ⎥ Wy
(5.24)⎢ ⎥ WΦ ⎢
⎣ .... ⎦
⎥ = ⎢ ⎥ ⎢ .... ⎥
⎣ ....
£ (m) ¤T ⎦ ⎣ ....
£ (m) ¤T ⎦ ⎣ £ ¤T ⎦
a b
θm a Wy a(m)
£ T ¤ £ T ¤
(5.25) Φ WΦ θ b = Φ W y
It can be easily shown that

⎡ (1) (1) ® (1) (2) ® (1) (m) ® ⎤
a ,a a ,a .... a ,a
£ T ¤ ⎢ a(2) , a(1) ® (2) (2) ®
a ,a ....
(2) (m) ®
a ,a ⎥
⎢ ⎥
(5.26) Φ WΦ = ⎢ ⎥
⎣ ..... ..... ..... ..... ⎦
(m) (1) ® (m) (2) ® ®
a ,a a ,a ..... a(m) , a(m)
⎡ (1) ® ⎤
a ,y
£ T ¤ ⎢ a(2) , y® ⎥
⎢ ⎥
(5.27) Φ W y = ⎢ ⎥
⎣ .... ⎦
(m) ®
a ,y
5.3. Additional Geometric Insights.

• Let us consider the special case where W = I,i.e. identity matrix. If
columns of Φ are linearly independent then ΦT Φ is invertible and, the
point p, which is projection of y onto column space of Φ (i.e. R(Φ))
is given as
£ ¤ £ ¤
(5.28) p = Φθ b = Φ ΦT Φ −1 ΦT y = [Pr ] y
£ ¤−1 £ T ¤
(5.29) Pr = Φ ΦT Φ Φ
Here matrix Pr is the projection matrix, which projects matrix y onto
R(Φ), i.e. the column space of Φ. Note that [Pr ] y is the component
of y in R(Φ)
(5.30) y − (Pr )y = [I − Pr ] y
is component of y ⊥ to R(Φ). Thus we have a matrix formula of
splitting a vector into two orthogonal components.
• Projection matrix has two fundamental properties.
(1) [Pr ]2 = Pr
(2) [Pr ]T = Pr
Conversely, any symmetric matrix with Φ2 = Φ represents a projection
matrix.
• Suppose then y ∈ R(Φ), then y can be expressed as linear combination
of columns of Φ i.e.,the projection of y is still y itself.
(5.31) b=y
p = Φθ
This implies
¡ ¢
(5.32) p = Φ(ΦT Φ)−1 ΦT y = Φ(ΦT Φ)−1 ΦT Φ θb = Φθ
b=y
The closest point of p to y is y itself

• At the other extreme, suppose y ⊥ R(Φ). Then
(5.33) p = Φ(ΦT Φ)−1 ΦT y = Φ(ΦT Φ)−1 0̄ = 0̄
• When Φ is square and invertible, every vector projects onto itself, i.e.
(5.34) p = Φ(ΦT Φ)−1 ΦT y = (ΦΦ−1 )(ΦT )−1 ΦT y = y
Suppose Φ is only a column vector. Then
aT y
(5.35) θ=
aT a
£ ¤−1 £ T ¤
Remark 9. Matrix ΦT Φ Φ is called as pseudo-inverse of matrix Φ.
5.4. Projection Theorem in a general Hilbert Space. Equations

we have derived in the above sub-sections are special cases of a very general
result called projection theorem, which holds in any Hilbert space. Although
we state this result here without giving a formal proof, the discussion in the
above subsections provided sufficient basis for understanding the theorem.
Theorem 13. Classical Projection Theorem : Let X be a Hilbert space
and S be a finite dimensional subspace of X. Corresponding to any vector y ∈X,
there is unique vector p ∈S such that ky − pk2 ≤ ky − sk2 for any vector s ∈
S. Furthermore, a necessary and sufficient condition for p ∈S be the unique
minimizing vector is that vector (y − p) is orthogonal to S.
Thus, given any finite dimensional sub-space S spanned by linearly inde-
© ª
pendent vectors a(1) , a(2) , ......., a(m) and an arbitrary vector y ∈X we seek a
vector p ∈S
p = θ1 a(1) + θ2 a(2) + .... + θm a(m)
such that
° ¡ ¢°
(5.36) °y− θ1 a(1) + θ2 a(2) + .... + θm a(m) °
2
is minimized with respect to scalars bθ1, ....b

θm . Now, according to the projection
theorem, the unique minimizing vector p is the orthogonal projection of y on
S. This translates to the following set of equations
® ¡ ¢ ®
(5.37) y − p, a(i) = y− θ1 a(1) + θ2 a(2) + .... + θm a(m) , a(i) = 0
for i = 1, 2, ...m. This set of m equations can be written as Gθ = b. i.e.
(5.38)
⎡ (1) (1) ® (1) (2) ® (1) (m) ® ⎤⎡ ⎤ ⎡ (1) ® ⎤
a ,a a ,a .... a ,a θ1 a ,y
⎢ a(2) , a(1) ® a(2) , a(2) ® ....
(2) (m) ®
a ,a ⎥⎢ θ2 ⎥ ⎢ a(2) , y® ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
⎢ ⎥⎢ ⎥=⎢ ⎥
⎣ ..... ..... ..... ..... ⎦⎣ .... ⎦ ⎣ .... ⎦
(m) (1) ® (m) (2) ® ® (m) ®
a ,a a ,a ..... a(m) , a(m) θm a ,y
This is the general form of normal equation resulting from the minimization
problem. The m × m matrix G on L.H.S. is called as Gram matrix. If vectors
© (1) (2) ª
a , a , ......., a(m) are linearly independent, then Gram matrix is nonsingu-
© ª
lar. Moreover, if the set a(1) , a(2) , ......., a(m) is chosen to be an orthonormal
© ª
set, say e(1) , e(2) , ......., e(m) , then Gram matrix reduces to identity matrix i.e.
G = I and we have
(5.39) p = θ1 e(1) + θ2 e(2) + .... + θm e(m)
where
®
θi = e(i) , y
6. STATISTICAL INTERPRETATIONS OF LINEAR REGRESSION 189
®
as e(i) , e(j) = 0 when i 6= j. It is important to note that, if we choose orthonor-
© ª
mal set e(1) , e(2) , ......., e(m) and we want to include an additional orthonormal
vector, say e(m+1) , to this set, then we can compute θm+1 as
®
θm+1 = e(m+1) , y
without requiring to recompute θ1 , ....θ m .
Remark 10. Given any Hilbert space X and a orthonormal basis for the
© ª
Hilbert space e(1) , e(2) , .., e(m) , ... we can express any vector u ∈X as
(5.40) u = α1 e(1) + α2 e(2) + .... + αm e(m) + ......

®
(5.41) αi = e(i) , u
The series
(1) ® (1) (2) ® (2) ®
(5.42) u = e , u e + e , u e + ........... + e(i) , u e(i) + ....
X∞
(i) ® (i)
(5.43) = e ,u e
i=1
which converges to element u ∈X is called as generalized Fourier series

®
expansion of element u and coefficients αi = e(i) , u are the corresponding
Fourier coefficients. The well known Fourier expansion or a continuous func-
tion over interval [−π, π] using sin(it) and cos(it) is a special case of this more
general result.
6. Statistical Interpretations of Linear Regression

6.1. Review of Fundamentals of Statistics. Consider a vector of ran-
dom variables (RVs), say ϕ ∈Rm , where each element ϕi of ϕ is a random
variable. Let ϕ represents population mean (average of all possible outcomes of
ϕ), i.e.
h iT
(6.1) ϕ = E (ϕ) = E (ϕ1 ) ... E (ϕm )
where operator E (.) represents expected value of RVs ϕ defined as

Z ∞ Z ∞
(6.2) E (ϕi ) = .... ϕi p(ϕ1 , ..., ϕm )dϕ1 ...dϕm
−∞ −∞
where p(ϕ1 , ..., ϕm ) = p(ϕ) represent multivariate probability density function

of ϕ.Also, let Σ represents population covariance, which is defined as
h i
(6.3)Σ = E (ϕ−ϕ) (ϕ−ϕ)T
⎡ h i h i ⎤
E (ϕ1 −ϕ1 ) (ϕ1 −ϕ1 )T .... E (ϕ1 −ϕ1 ) (ϕm −ϕm )T
⎢ ⎥
= ⎢ ⎣ h .... i .... h .... ⎥
i ⎦
E (ϕm −ϕm ) (ϕ1 −ϕ1 )T .... E (ϕm −ϕm ) (ϕm −ϕm )T
In fact, operator E (.) can be used for computing ensemble (population) expec-
tation of any arbitrary function of ϕ, say g(ϕ),i.e.,
Z ∞ Z ∞
(6.4) E [g (ϕ)] = .... g(ϕ)p(ϕ)dϕ1 ...dϕm
−∞ −∞
Mean and covariance are specific forms function g (ϕ) .

Given a vector of random variables (RVs) ϕ, in general, we may not have
exact knowledge of p(ϕ1 , ..., ϕm ), ϕ and Σ. However, we can generate estimates
of ϕ and Σ if we can conduct experiments and collect data for ϕ in each
experiments. Let us assume that we have carried out experiments and collected
© ª
N data vectors ϕ(1) , ...., ϕ(N ) , which can be can be arranged into a matrix
as follows
⎡ ¡ ¢T ⎤
ϕ(1)
⎢ ⎥
(6.5) Φ = ⎣ .... ⎦
¡ (N ) ¢T
ϕ
¡ (i) ¢T h i
(6.6) ϕ = φi,1 φi,2 ..... φi,m
which contains various values random variable ϕi takes during the experiments.
The data matrix Φ can also be expressed as
h i
(6.7) Φ = η (1) η (2) ..... η (m)
⎡ ⎤
φ1,j
⎢ ⎥
(6.8) η (j) = ⎣ .... ⎦
φN,j
where η (j) represents j th column vector of matrix Φ consists of vector.
The arithmetic average or sample mean of ϕi can be computed as
XN
(6.9) bi = 1
ϕ φ ; i = 1, 2, ...m
N j=1 j,i
b i denotes esti-
Here, ϕi represents true mean or (entire) population mean and ϕ
mate of population mean generated using the given data set. The sample mean
can also be viewed as coefficient of projection of vector η (i) on unit vector 1

defined as
⎡ ⎤
1
⎢ ⎥
(6.10) 1 = ⎣ .... ⎦
1
(j) ®
η ,1
(6.11) bj =
ϕ ; j = 1, 2, ...m
h1, 1i
Alternatively, sample mean of random variable vector ϕ can be estimated as

follows
XN
(6.12) b = 1
ϕ ϕ(j)
N j=1
It may be noted that the vector 1 makes makes equal angles with all coordinate
axis in RN and the vector ϕ b j 1 represents the projection of η (j) along vector 1.
© ª
A measure of spread of data elements φj,i : j = 1, 2...N around the esti-
mated sample mean ϕ b i is given by sample variance defined as
1 X£
N
¤2
(6.13) s2i = φj,i − ϕi
N − 1 j=1
The sample variance can also be estimated as follows
(6.14) b i1
e(i) = η (i) − ϕ
1 £ (i) ¤T (i)
(6.15) s2i = e e
N −1
³ ´ ³ ´
b i 1 is orthogonal to vector ϕ
Note that the vector η (i) − ϕ b i 1 , which is best
approximation of η (i) along vector 1. Square root of sample variance is called
sample standard deviation.
Now consider data obtained for two different random variables, say ϕi and
ϕk . A measure of linear association between these two variables can be estimated
as
1 Xh ih i
N
£ (i) ¤T (k)
(6.16) si,k = φj,i − ϕ bk = 1
b i φj,k − ϕ e e
N − 1 j=1 N −1
i = 1, 2, ...m ; i = 1, 2, ...m
Here si,k are (i, k)th elements of sample covariance matrix Sϕ of the random
variable vector ϕ. Alternatively, this matrix can also be estimated as
1 X h (j) b i h (j) b iT
N
(6.17) Sϕ = Cov(ϕ) = ϕ −ϕ ϕ −ϕ
N − 1 j=1
∙ ³ ´T ¸T ∙ ³ ´T ¸
1 b b
(6.18) = Φ−1 ϕ Φ−1 ϕ
N −1
It may be noted that sample covariance matrix Sϕ is an estimated of population
covariance matrix Σ.
It may be noted that Finally, sample correlation coefficients (normalized or
standardized covariances) are defined as
PN h ih i
φj,i − b
ϕ i φ j,k − b
ϕ k
j=1
(6.19) ri,k = r h i r h i2
PN 2 PN
b
j=1 φj,i − ϕi j=1 φj,k − ϕk
b
si,k si,k
= √ √ =√ √
si,i sk,k si sk
£ (i) ¤T (k)
e e
= q q = cos(θι,k )
T T
[e(i) ] e(i) [e(k) ] e(k)
i = 1, 2, ...m ; i = 1, 2, ...m
where si and sk represents sample standard deviations of ϕi and ϕk , respectively,

and ri,k represents (i, k)th element of sample correlation matrix Rϕ . Here θι,k
is angle between vectors e(i) and e(k) . From the above equations it follows that
(6.20) ri,k = cos(θι,k ) ⇒ −1 ≤ ri,k ≤ 1
If two deviation vectors e(i) and e(k) have nearly same orientation, the sample
correlation coefficient will be close to 1. If two deviation vectors e(i) and e(k)
are nearly perpendicular, the sample correlation coefficient will be close to 0.
if two deviation vectors e(i) and e(k) have nearly opposite orientation, the sam-
ple correlation coefficient will be close to -1. Thus, ri,k is a measure of linear
association between two random variables.
ri,k = 0 ⇒ Indicates Lack of linear association between ϕi and ϕk
tendency for one value in pair to be larger than
ri,k < 0 ⇒
its average when the other is smaller than its average
tendency for one value in pair to be large when the
ri,k > 0 ⇒
other is large and for both values to be small together
It may be noted that two random variables having nonlinear association may
have ri,k = 0 i.e. lack of linear association. Thus, ri,k = 0 implies only lack of
linear association and not lack of association between two random variables.
The sample correlation matrix Rϕ can also be estimated as
(6.21) Rϕ = D−1/2 Sϕ D−1/2
where
h √ √ √ i
(6.22) D1/2 = diag s1 s2 ... sm
Suppose we define a variable transformation where φi,j are replaced by nor-
malized or standardized variables defined as
³ ´
b
φi,j − ϕj
(6.23) υ i,j = √
sj
Then the vector υ of these scaled random variables will have zero mean and its
sample covariance will be identical to its sample correlation matrix.
6.2. Commonly used probability distributions.

6.2.1. Multivariable Gaussian / Normal Distribution. Consider a vector or
random variables ϕ with multivariate normal (Gaussian) distribution, which is
generalization of univariate normal density to dimensions higher that one. Let ϕ
and Σ denote population mean and population covariance matrix, respectively.
Then, the probability density function for this distribution has form
h i
(6.24) f (ϕ) =C exp k(ϕ−ϕ)k2,Σ−1
where C is a constant and k(ϕ−ϕ)k2,Σ−1 is a normalized distance measure of

vector ϕ from its mean defined as
(6.25) k(ϕ−ϕ)k2,Σ−1 = (ϕ−ϕ)T Σ−1 (ϕ−ϕ)
Note that Σ is a positive definite and symmetric matrix. From the requirement
that
Z ∞ Z ∞
(6.26) .... f (ϕ1 , ..., ϕm )dϕ1 ...dϕm = 1
−∞ −∞
it follows that
1
(6.27) C=
(2π)m/2 [det(Σ)]1/2
Thus, multivariate Gaussian distribution has form
1 h i
T −1
(6.28) f (ϕ) = exp (ϕ−ϕ) Σ (ϕ−ϕ)
(2π)m/2 [det(Σ)]1/2
In the univariate case, this reduces to familiar form

∙ ¸
1 (ϕ−ϕ)
(6.29) f (ϕ) = p exp
σ (2π) σ2
6.2.2. Chi-square (χ2 ) distribution. This distribution is used in connection
with
• Testing goodness of fit of experimental observations to hypothesized
probability distributions
• Obtaining confidence limits for the variance and the standard devia-
tions
• Testing independence of variables.
Let (ϕ1 , ..., ϕm ) represent a set of m independent normally distributed ran-
dom variables with parameters (µ1 , σ 21 ), ...., (µm , σ 2m ). If we calculate the squares
of the standard normal variables
µ ¶2
2 ϕi − ϕi
(6.30) ui =
σi
and sum the u2i s, then we have a new random variable χ2 as follows
X
m
(6.31) χ2 = u2i
i=1
Here, m is called degrees of freedom for χ2 . The distribution of χ2 depends only

on m because ui are standardized. The probability density function for χ2 can
be shown to be
µ 2¶
¡ 2¢ 1 ¡ 2 ¢ m2 −1 χ
(6.32) p χ = m/2
χ exp −
(2) Γ(m/2) 2
0 < χ2 < ∞
p
For m > 30, 2χ2 is approximately distributed as a normal variable with
√
µ = 2m − 1 and σ2 = 1.
6.2.3. Student t distribution. Given a random variable ϕ, the random
qvari-
χ2
able t represents the ratio of two independent ransom variables, u, and m
u u b−ϕ
ϕ
(6.33) t= q = =
χ2 sϕ /σ ϕ sϕ
m
where ϕb is the sample mean and sϕ is the sample standard deviation. The
probability density function for t is
¡ ¢∙ ¸− m+1
1 Γ m+1 2 t2 2
(6.34) p(t) = √ ¡m¢ 1 +
πm Γ 2 m
where m is the degrees of freedom associated with s2ϕ .

6.2.4. Fisher distribution. If two samples are taken, one consisting of n1
independent measurements of a normal random variable ϕ1 , which has mean µ1
and variance σ 21 , and other sample consisting of n2 independent measurements
of a normal random variable ϕ2 , which has mean µ2 and variance σ 22 , then the
random variable F is defined as
s21 /σ 21
(6.35) F (n1 − 1, n2 − 1) =
s22 /σ 22
with degrees of freedom (n1 − 1) and (n2 − 1). If σ 21 = σ 22 = σ 2 , then F can be
related to χ2 as
s21 χ21 /(n1 − 1)
(6.36) F (n1 − 1, n2 − 1) = =
s22 χ22 /(n2 − 1)
The probability density function of F is given by
(6.37)
¡ ¢
Γ n1 +n2 2 −2 n1 −1 n2 −1 F ( n12+1 )
p(F ) = ¡ n1 −1 ¢ ¡ n2 −1 ¢ [n1 − 1] 2 [n2 − 1] 2 n1 +n2 −2
Γ 2 Γ 2 [(n2 − 1) + (n1 − 1)F ]( 2 )
Ensemble mean and variances of F are
n2 − 1
(6.38) E (F ) = for n2 > 2
n2 − 3
2(n2 − 1)2 (n1 + n2 − 4)
(6.39) V ar(F ) =
(n1 − 1)(n2 − 3)2 (n2 − 5)
6.3. Statistical Interpretation of Linear Least Squares. In order
to interpret the linear least square approximation from statistical viewpoint, we
make following assumptions :
• The weighting matrix is an identity matrix, i.e. W = I. In other
words, all the measurements are assumed to be equally likely.
• Only equation / modelling errors are present. Alternatively, measure-
ment errors are present only in dependent variable and there are no
measurement errors in independent variables.
We further hypothesize that
y has a Gaussian / normal distribution N(0, σ 2 I).i.e.,
• Error vector e = y−b
(6.40) mean(e) =E(e) = 0 and cov(e) =σ 2 I
Note that the above covariance expression implies that each element
ei of vector e, are independent and normally distributed variables such
that
£ (i) ¤T
(6.41) yi = ϕ θ + ei for i = 1, 2...N
(6.42) cov(ei , ej ) = 0 when i 6= j ; i = 1, 2...N, j = 1, 2...N
(6.43) var(ei ) = σ 2 for i = 1, 2...N
• The parameter vector θ and the error vector e are not correlated.
Thus, statistical model for experimental data given by equation (6.41) can
be expressed as
(6.44) y = ΦθT + e
⎡ £ ¤T ⎤
ϕ(1)
⎢ ⎥
(6.45) Φ = ⎣ ........ ⎦
£ (N) ¤T
ϕ
where θ T represent the true parameter values and vectors y and e have been
defined by equations (4.12) and (4.13), respectively. Taking expectation (mean)
on both sides of the above equation., we have
(6.46) E(y) = E(ΦθT + e) = ΦθT
From least square analysis, we have least square solution , given as
(6.47) b = (ΦT Φ)−1 ΦT y
θ
(6.48) b = (ΦT Φ)−1 ΦT E(y) = (ΦT Φ)−1 ΦT E(Φθ T + e)
E(θ)
(6.49) = (ΦT Φ)−1 ΦT ΦθT = θT
The above result guarantees that, if we collect sufficiently large number of sam-
ples, the least square estimate will approach the true values of the model para-
b is an unbiased estimate of θT . To calculate
meters. In statistical terms, θ
b we proceed as follows
covariance of θ,
(6.50) b θ)
V = cov (θ, b = E [(θ b − θT )T ]
b − θT )(θ
Now
(6.51) b − θT
θ = (ΦT Φ)−1 ΦT (ΦθT + e) − θ T
(6.52) = θT + (ΦT Φ)−1 ΦT e − θ T
(6.53) = (ΦT Φ)−1 ΦT e
This implies
(6.54) b θ)
V = cov(θ, b = E [ (ΦT Φ)−1 ΦT (eeT ) Φ (ΦT Φ)−1 ]
(6.55) = (ΦT Φ)−1 ΦT E(eeT ) Φ (ΦT Φ)−1
Using the hypothesis
(6.56) E(eeT ) = σ 2 I
we have
(6.57) b θ)
V =cov(θ, b = σ 2 (ΦT Φ)−1
Thus, covariance matrix of regression coefficients θ b is proportional to (ΦT Φ)−1 . If

N is large, there is negligible bias in the estimation of σ 2 . The knowledge of
b is important because it constitutes an estimation on precision with which
cov(θ)
each θi are determined. Note that diagonal elements of cov(θ) b are variances of
individual parameters θi , i.e.
£ ¤
(6.58) var [θi ] = cov(θi , θi ) = Vii = σ 2 (ΦT Φ)−1 ii
Now, ³since e ís zero mean and normally distributed vector, it can be shown
that θ b − θT is also a zero mean and normally distributed vector with the
covariance matrix given by equation (6.57). Thus, given matrix Φ and σ2 ,we
can determine confidence interval of each parameter using the parameter
variances..
In general, σ 2 may not be known apriori for a given set of data. However,
an estimate of σ2 (denoted as σb2 ) can be obtained from the estimated error
vector as
1 XN
eT b
b e
(6.59) b2
σ = eb2i =
N − m i=1 N−m
(6.60) b b
e = y − Φθ
where θb is the least square estimate. Estimate of V (denoted as V)

b can be
computed as
µ T ¶
b b
e b
e
(6.61) V= (ΦT Φ)−1
N−m
To appreciate the validity of uniformity of a model after smoothing, normally
we calculate the correlation coefficient R.
yT y
b b
yT (Φθ)
2
(6.62) R = T =
y y yT y
This quantity indicates the ratio of response y known from the model. For
example, if R2 = 0.95 then we can say that 95% of variation of yi is known
from ybi . Thus, this correlation coefficient can be used to compare different mod-
els developed using same data. Further it can be shown that the quantity
µ ¶
N −m R2
(6.63) f=
m + 1 1 − R2
has Fisher distribution with (m + 1) and (N − m) degrees of freedom. When
regression is significant, we can test the hypothesis H0 by rejecting H0 with risk
of α%. The test statistic is with risk of α% can be computed as
(6.64) ε = Fα (m + 1, N − m)
where F denotes Fisher distribution. If f > ε,then we conclude that f is
probably not a the Fisher distribution and the model is not suitable. If f < ε
,then smoothing is significant and the model is satisfactory.
In many situations, the model parameters have physical meaning. In such
cases, it is important to determine confidence interval for parameter θi . Defining
variable
b
θi − θi
(6.65) ti = q
Vbii
where θi is the true value of parameter and bθi is the estimated value of parame-
ter, it can be shown that ti is a t (Student) distribution with N − m degrees of
freedom. Thus, confidence interval with the risk of α% for θi is
q q
(6.66) b
θi − Vbii t(α/2, N − m) < θi < b θi + Vbii t(α/2, N − m)
In many situations, when we are fitting a functional form to explain variation
in experimental data, we do not know apriori the exact function form (e.g.
polynomial order) that fits data best. In such a case, we need to assess whether
a particular term fj (x) contributes significantly to yb or otherwise. In order to
measure the importance of the contribution of θi in
X
m
(6.67) yb = θj fj (x)
j=1
one can test hypothesis H0 namely: θi = 0 (i.e. fi (x) does not influence yb).
Thus, we have
b
θi − 0
(6.68) τi = q
b Vbii
σ
If
(6.69) −t(α/2, N − m) < τ i < t(α/2, N − m)
7. SIMPLE POLYNOMIAL MODELS AND HILBERT MATRICES 199
then fi (x) has significant influence on yb. If not, then we can remove term fi (x)
from yb.
7. Simple Polynomial Models and Hilbert Matrices

7.1. Approximation of Continuous Function by a Polynomial. Con-
sider problem of approximating a continuous function, say f (z), over interval
[0, 1] by a simple polynomial model of the form
(7.1) yb(z) = θ1 p(1) (z) + θ2 p(2) (z) + θ3 p(3) (z) + ............ + θm p(m) (z)
= θ1 + θ2 z + θ3 z 2 + ................... + θm z m−1
Let the inner product on C2 [0, 1] is defined as
Z1
(7.2) hh(z), g(z)i = h(z)g(z)dz
0
We want to find a polynomial of the form (7.1) which approximates f (z) in the
least square sense. Geometrically, we want to project f (z) in the m dimensional
subspace of C2 [0, 1] spanned by vectors
(7.3) p(1) (z) = 1; p(2) (z) = z ; p(3) (z) = z 2 , ........, p(m) (z) = z m−1
Using projection theorem, we get the normal equation

⎡ ⎤⎡ ⎤ ⎡ ⎤
h1,1i h1,zi .... h1,z m−1 i θ1 h1,f (z)i
⎢ hz,1i hz,zi .... hz,z m−1
i ⎥ ⎢ ⎥ ⎢ hz,f (z)i ⎥
⎢ ⎥ ⎢ θ2 ⎥ ⎢ ⎥
(7.4) ⎢ ⎥⎢ ⎥=⎢ ⎥
⎣ ..... ..... ..... ..... ⎦ ⎣ .... ⎦ ⎣ .... ⎦
m−1 m−1 m−1 m−1
hz ,1i hz ,zi ..... hz ,z i θm hz m−1 ,f (z)i
Element hij of the matrix on L.H.S. can be computed as
Z1
® 1
(7.5) hij = x(i) (z), x(j) (z) = z j+i−2 dz =
i+j−1
0
and this reduces the above equation to

⎡ ⎤ ⎡ ⎤
θ1 h1,f (z)i
⎢ θ ⎥ ⎢ hz,f (z)i ⎥
⎢ 2 ⎥ ⎢ ⎥
(7.6) H⎢ ⎥=⎢ ⎥
⎣ .... ⎦ ⎣ .... ⎦
θm hz m−1 , f (z)i
where
⎡ ⎤
1 1/2 1/3 ... ... 1/m
⎢ 1/2 1/3 1/4 ... ... 1/(m + 1) ⎥
⎢ ⎥
(7.7) H = ⎢ ⎥
⎣ ... ... ... ... ... ... ⎦
1/m ... ... ... ... 1/(2m − 1)
The matrix H is known as Hilbert matrix and this matrix is highly ill-conditioned
for m > 3. The following table shows condition numbers for a few values of m.
m 3 4 5 6 7 8
(7.8)
c(H) 524 1.55e4 4.67e5 1.5e7 4.75e8 1.53e10
Thus, for polynomial models of small order, say m = 3 we obtain good sit-
uation, but beyond this order, what ever be the method of solution, we get
approximations of less and less accuracy. This implies that approximating a
continuous function by polynomial of type (7.1) with the choice of basis vectors
as (7.3) is extremely ill-conditioned problem from the viewpoint of numerical
computations. Also, note that if we want to increase the degree of polynomial
to say (m + 1)from m, then we have to recompute θ1 , ...., θm along with θm+1 .
On the other hand, consider the model
(7.9) yb(z) = α1 p1 (z) + α2 p2 (z) + α3 p3 (z) + ............. + αm pm (z)
where pi (z) represents the i’th order orthonormal basis function on C2 [0, 1].This
corresponds to choice of basis functions as
(7.10) x(1) (z) = p1 (z); x(2) (z) = p2 (z) ; ........, x(m) (z) = pm (z)
and since
( )
1 if i = j
(7.11) hpi (z), pj (z)i =
0 if i 6= j
the normal equation reduces to
⎡ ⎤⎡ ⎤ ⎡ ⎤
1 0 .... 0 α1 hp1 (z),f (z)i
⎢ 0 1 .... 0 ⎥⎢ α2 ⎥ ⎢ hp2 (z),f (z)i ⎥
⎢ ⎥⎢ ⎥ ⎢ ⎥
(7.12) ⎢ ⎥⎢ ⎥=⎢ ⎥
⎣ ..... ..... ..... ..... ⎦⎣ .... ⎦ ⎣ .... ⎦
0 0 ..... 1 αm hpm (z), f (z)i
or simply
(7.13) αi = hpi (z),f (z)i ; i = 1, 2, ....m
Obviously, the approximation problem is extremely well conditioned in this case.
In fact, if we want to increase the degree of polynomial to say (m + 1)from m,
then we do not have to recompute α1 , ...., αm as in the case basis (7.3) where
7. SIMPLE POLYNOMIAL MODELS AND HILBERT MATRICES 201
vectors are linearly independent but not orthogonal. We simply have to compute
the αm+1 as
(7.14) αm+1 = hpm+1 (z),f (z)i
The above illustration of approximation of a function by orthogonal polynomials
is a special case of what is known as generalized Fourier series expansion.
7.2. Approximation of Numerical Data by a Polynomial. Sup-

pose we only know numerical {y1 , y2 , ......yN } at points {z1 , z2 , ......zN } ∈ [0, 1]
and we want to develop a simple polynomial model of the form given by equa-
tion (7.1). Substituting the data into the polynomial model leads to an over-
dertermined
(7.15) yi = θ1 + θ2 zi + θ3 zi2 + .......................... + θm zim−1 + ei
(7.16) i = 1, 2, .....N
The least square estimates of the model parameters ( for W = I) can be obtained
by solving normal equation
ˆ
(7.17) (ΦT Φ)θ = ΦT y
where
⎡ ⎤
1 z1 z12 ... z1m−1
⎢ ⎥
(7.18) Φ = ⎣ ... ... ... ... ....... ⎦
2 m−1
1 zN zN ... zN
⎡ P P 2 P m−1 ⎤
N zi zi .... z
⎢ P z P z2 Pi m ⎥
⎢ i i ..... .... zi ⎥
(7.19) ΦT Φ = ⎢ ⎥
⎣ ..... ..... ..... .... ....... ⎦
..... ..... ..... .... .......
i.e.,
X
N
(7.20) (ΦT Φ)ik = zij+k−2
i=1
Let us assume that zi is uniformly distributed in interval [0, 1]. For large N,
approximating dz ' 1/N, we can write
X
N Z1
∼ N
(7.21) [Φ]jk = zij+k−2 = N z j+k−2 dz =
i=1
j+k−1
0
(7.22) ( j, k = 1, 2, ............m )
Thus, we can approximate (ΦT Φ) matrix by the Hilbert matrix

⎡ ⎤
1 1/2 1/3 ... ... 1/m
⎢ 1/2 1/3 1/4 ... ... 1/(m + 1) ⎥
⎢ ⎥
(7.23) (ΦT Φ) = N(H) = N ⎢ ⎥
⎣ ... ... ... ... ... ... ⎦
1/m ... ... ... ... 1/(2m − 1)
which is highly ill- conditioned for large m. Thus, whether we have a continuous
function or numerical data over interval [0, 1], the numerical difficulties persists
as the Hilbert matrix appears in both the cases.
Similar to the previous case, modelling in terms of orthogonal polynomials
can considerably improve numerical accuracy. The orthogonal set under con-
sideration is now different. Let pm (zi ) denote a orthogonal polynomial of order
m. The inner product is now defined as
X
N
(7.24) hpj (z) , pk (z)i = wi pj (zi ) pk (zi )
i=1
By definition, a set of polynomials {pj (zi )}are orthogonal over a set of points
{zi } with weights wi , if
X
N
wi pj (zi ) pk (zi ) = 0 ; j, k = 1...............m and ( j 6= k)
i=1
Let a linear polynomial model be defined as

Xm
(7.25) yi = αj pj (zi ) + ei ; i = 1, ...................N
j=1
Then the normal equation becomes

(7.26)
⎡ ⎤ ⎡ ⎤
P
N
⎡ ⎤ P
N
⎢ wi p21 (zi ) 0 0 ⎥ ⎢ wi yi p1 (zi ) ⎥
⎢ i=1 ⎥ α1 ⎢ i=1 ⎥
⎢ ... ⎥ ⎢ .. ⎥ ⎢ .. ⎥
⎢ 0 0 ⎥ ⎣ . ⎦ = ⎢ . ⎥
⎢ ⎥ ⎢ N ⎥
⎣ P
N ⎦ αm ⎣ P ⎦
0 0 wi p2m (zi ) wi yi pm (zi )
i=1 i=1
Thus, each term αj pj (zi ) is (statistically) independent of any other term and
contains information that other terms do not have and the resulting parameter
estimation problem is highly well conditioned. A set of orthogonal polynomials
can be obtained by different approaches. One simple algorithm is
INITIALIZE
p−1 (zi ) = 0
p0 (zi ) = 1 (f or i = 1, .............N)
8. NONLINEAR IN PARAMETER MODELS 203
Ψ0 = 0
FOR (k = 0, 1, .............m − 1)
FOR (i = 0, 1, .............N )
PN
wi [ pk (zi ) ]2
Ψk = Ni=1
P
wi [ pk−1 (zi ) ]2
i=1
P
N
wi zi [ pk (zi ) ]2
i=1
Γk+1 =
PN
wi [ pk (zi ) ]2
i=1
pk+1 (zi ) = [zi − Γk+1 ] pk (zi ) − [Ψk ] ρk−1 (zi )

END FOR
END FOR
8. Nonlinear in Parameter Models

In many problems the parameters appear nonlinearly in the model
¡ ¢
(8.1) ybi = f x(i) ; θ1 ........, θ m ; (i = 1, .........N)
or in the vector notation
(8.2) b = F [X, θ]
y
where
h iT
(8.3) b =
y yb1 yb2 .... ybN
h ¡ ¢ ¡ ¢ ¡ ¢ iT
(8.4) F = f x(1) , θ f x(2) , θ .... f x(N) , θ
© ª
and X = x(1) , · · · , x(N ) represents data set. The problem is to determine
b such that
vector θ
(8.5) Ψ = eT We
(8.6) b − F (X, θ)
e = y
is minimized. Note that. in general, the above problem cannot be solved ana-
lytically and we have to resort to iterative procedures. There are three solution
approaches:
• Approximate solution using weighted least square when the model is
analytically linearizable
• Successive linear least square approach ( Gauss-Newton method )
• Use of direct optimization (nonlinear programming)
The first two approaches use the linear least square formulation as basis
while the nonlinear programming approaches is a separate class of algorithms.
8.1. Weighted Least Square For Analytically Linearizable Models.

Many nonlinear-in-parameter forms can be converted into linear-in-parameter
forms by means of some linearizing transformations. In such cases, it appears
that we can apply linear least square to obtain parameter estimates. However,
this may not minimize eT We as explained below
Consider linearizable form of model
(8.7) yb = θ0 [ f1 (x)]θ1 [ f2 (x)]θ2 · · · · · · [fm (x)]θm
which can be transformed as
(8.8) ln yb = ln θ1 + θ2 ln [ f1 (x)] + θ3 ln [ f2 (x)] · · · θm ln [ fm (x)]
Now, parameter set that minimizes
X
N X
N
(8.9) e=
Ψ 2
( ln yi − ln ybi ) = ei )2
(e
i=1 i=1
may not minimize

X
N
2
X
N
(8.10) Ψ= ( yi − ybi ) = (ei )2
i=1 i=1
A rigorous approach to avoid this problem is to use nonlinear programming of

Gauss-Newton approach. However, it is often possible to use weighted coeffi-
cients wi to account for the linearizing transformation.
Let ei denote the error associated with nonlinear model and let eei denote
the error associated with transformed model .We want to approximate e2i in
terms of ee2i , i.e.
(8.11) e2i ' ee2i wi
Note that e
ei is a complex function of ei . Let us denote this function
(8.12) e
ei = g (ei ) (i = 1, · · · N)
∂g
(8.13) ⇒ de
ei = dei
∂ei
One possibility is to use Taylor series expansion of function g (.) at point (ei = 0, eei = 0)
∙ ¸
∂g
(8.14) eei ' g(0) + (ei − 0)
∂ei (ei =0)
∙ ¸
∂g
(8.15) ' ei
∂ei (ei =0)
8. NONLINEAR IN PARAMETER MODELS 205
Then, we can use

⎡Ã !2 ⎤−1
∙ ¸
⎣ ∂ēi ⎦
(8.16) wi =
∂ei ei =0
ēi =0
(8.17) (i = 1, .........N)
For example, in the above case
(8.18) eei = ln yi − ln ye = ln( yei + ei ) − ln yei
∂ēi 1 1
(8.19) = =
∂ei yei + ei yi
(8.20) ⇒ wi = (yi )2
and optimization problem can be formulated as minimization of

X
N X
N
(8.21) e=
Ψ 2
(yi ) ( ln yi − ln ybi ) = 2
(yi eei )2
i=1 i=1
The resulting weighted least square problem can be solved analytically.
8.2. Method of Successive Linearization (Gauss-Newton method).

This approach is iterative. Start with an initial guess vector θ (0) . By some
process, generate improved guess θ(k) from θ (k−1) . At kth iteration let θ(k−1)
be the guess solution. By expanding the model as Taylor series in the neigh-
borhood of θ = θ(k−1) and neglecting higher order terms we have
³ ´ ∙ ∂F ¸ ³ ´
(k) (k−1) (k)
(8.22) e
y ' F X, θ + 4θ
∂θ θ=θ(k−1)
where
∙ ¸
(k−1) ∂F
(8.23) J =
∂θ
is a (N × m) matrix with elements
∙ ¸ " ¡ (i) ¢ #
∂F ∂F x ,θ
(8.24) = i = 1, . . . N and j = 1, . . . m
∂θ ij ∂θi
θ=θ (k−1)
Let us denote
∙ ¸
(k−1) ∂F
(8.25) J =
∂θ θ=θ(k−1)
and
³ ´
(8.26) F (k−1) = F X, θ (k−1)
Then approximate error vector at kth iteration can be defined as

£ ¤
(8.27) e(k) = y − y
e e(k) = y − F (k−1) − J(k−1) 4θ(k)
and kth linear sub-problem is defined as
£ (k) ¤T
(8.28) min e e e(k)
We
4θ (j)
The least square solution to above sub problem can be obtained by solving the
normal equation
¡ (k−1) ¢T ¡ ¢T £ ¤
(8.29) J W J(k−1) 4θ (k) = J(k−1) W y − F (k−1)
h¡ ¢T i−1 ¡ ¢T £ ¤
(8.30) 4θ(k) = J(k−1) W J(k−1) J(k−1) W y − F (k−1)
and an improved guess can be obtained as
(8.31) θ(k) = θ(k−1) + 4θ(k)
Termination criterion : Defining e(k) = y − F (k) and
£ ¤T
(8.32) Φ(k) = e(k) W e(k)
terminate iterations when Φ(k) changes only by a small amount, i.e.
| Φ(k) − Φ(k−1) |
(8.33) < ε
| Φ(k) |
Gauss Newton Algorithm:
INITIALIZE: θ(0) , ε1 , ε2 , α, k, δ 1 , δ 2 , kmax
e(0) = y − F [ X, θ(0) ]
δ 1 = e(0)T W e(0)
WHILE [(δ 1 > ε1 ) AND (δ 2 > ε2 ) AND (k < kmax )]
e(k) = y − F [ X, θ(k) ]
£ (k)T ¤
J W J (k) 4 θ (k) = J (k)T W e(k) )
λ(0) = 1, j = 0
θ(k+1) = θ(k) + λ(0) 4 θ(k)
δ 0 = e(k) W e(k)
δ 1 = e(k+1) W e(k+1)
WHILE[δ1 > δ 0 ]
λ(j) = αλ(j−1)
θ(k+1) = θ(k) + λ(j) 4 θ(k)
δ 1 = e(k+1) W e(k+1)
END WHILE
δ 2= ||θ(k+1) − θ(k) || / ||θ(k+1) ||
k =k+1
9. UNCONSTRAINED NONLINEAR PROGRAMMING 207
END WHILE
9. Unconstrained Nonlinear Programming

Any iterative method for unconstrained optimization consists of two steps.
• Step 1: given a guess solution vector x(k) ,find a search direction s(k)
to move in the direction where f(x) decreases (or increases).
• Step 2: After the direction S(k) is determined, estimate the step length
λ to calculate
(9.1) x(k+1) = x(k) + λs(k)
i.e. new guess. Typically, λ is selected by minimizing f (x(k) +λs(k) ) with respect
to, i.e. by carrying out one dimensional optimization with respect to λ.
The iterations are terminated when either one of the following termination
criteria are satisfied:
|f (x(k+1) ) − f (x(k) )|
(9.2) < 1
|f (x(k+1) )|
||(x(k+1) ) − (x(k) )||

(9.3) < 2
||(x(k+1) )||
The iterative nonlinear unconstrained optimization algorithms can be cate-
gorized as follows
• Analytical Methods : Solve ∇Φ(z) = 0 analytically
• Direct Methods: need only computation of Φ(z) at each step
• Gradient based methods: Require computation of ∇Φ(z) and Φ(z)at
each step
• Hessian based methods: require computation of ∇2 Φ(z), ∇Φ(z)and
Φ(z) at each step
In this section, we a brief introduction to some of the direct, gradient and
Hessian based approaches.
9.1. Simplex Method (Direct Method). The geometric figure formed

by (n + 1) points in n dimensional are called as simplex. Simplex in two di-
mensional space is a triangle and three dimensional space is tetrahedron. Basic
idea in simplex method is to evaluate f (x) at (n + 1) vertices of a simplex and

use this information to move towards the optimum. Let x(m) be such that
max
(9.4) x(m) = f (x(i) )
i ∈ (1, ....n + 1)
and x(c) represent centroid of the remaining points defined as
1 X (i)
n+1
(c)
(9.5) x = x
n i=1,i6=m
Then, we find a new point x(new) new such that by moving in direction of
x(c) − x(m) as follows
¡ ¢
(9.6) x(new) = x(m) + λ x(c) − x(m)
The choice of λ = 1 ⇒ x(new) = x(c) and λ = 0 ⇒ x(new) = x(m) . In order

to maintain the regularity of simplex, the reflection should be symmetric and
λ = 2 is selected for achieving symmetry. The property of simplex used by
algorithm is that a new simplex- can be generated on any face of the old one
by projecting any chosen vertex a suitable distance through the centroid of the
remaining vertices of old simplex. The new Simplex is then formed by replacing
old vertices with newly generated projected point.
Initialization step involves generation of a regular simplex given a starting
point x(0) and a scale factor α. Remaining n vertices of a regular simplex (with
equidistant neighboring points) are given by
( )
(0)
(i) xj + δ 1 if j 6= i
(9.7) xj = (0)
xj + δ 2 if j = i
∙ ¸
(N + 1)1/2 + N − 1
(9.8) δ1 = √ α
N 2
∙ ¸
(N + 1)1/2 − 1
(9.9) δ2 = √ α
N 2
Scale factor α is selected by user. Choice α = 1 generates a simplex with

unit length. If cycling occurs, i.e., a given vertex remains unchanged for more
than M iterations , then the simplex size is reduced by some factor. A new
simplex is setup with the currently lowest points as base points. This rule
requires specification of some reduction factor. The search is terminated when
Figure 6. f (x) curves for n = 2
the simplex gets small enough or else if

X
n+1
[f (xi ) − f (xc )]2
(9.10) ≤
i=1
n+1
There are other modifications of this method where by λ is selected smaller or

larger than 2, i.e. contraction or expansion of simplex. Main advantages of
simplex method are (a) simple calculations and uncomplicated logic (b) few ad-
justable parameters(α,β,λ). It may be noted that the variables should be scaled
to avoid difficulties due to different ranges of variables. The main disadvantage
of this method is that the movement towards the optimum can be very slow.
9.2. Gradient / Steepest Descent / Cauchy’s method.
Definition 36. Set of vectors z ∈ RN such that Φ(z) = C where C is a

constant, is called the level surface of Φ(z) for value C.
By tracing out one by one level surfaces we obtain contour plot (see Figure
6 and Figure 7). Suppose z =z is a point lying on one of the level surfaces. If
Φ(z) is continuous and differentiable then, using Taylor series expansion in a
neighborhood of z we can write
1 £ ¤
(9.11) Φ(z) = Φ(z) + [∇Φ(z)]T (z−z) + (z−z)T ∇2 Φ(z) (z−z) + ....
2
If we neglect the second and higher order terms, we obtained
Figure 7. Level surfaces for n = 2
(9.12) Φ(z) = Φ(z + ∆z) ' Φ(z) + [∇Φ(z)]T ∆z = C

(9.13) ∆z = (z−z)
This is equation of the plane tangent to surface Φ(z) at point z. The equation
of level surface through z is
(9.14) C = Φ(z) = Φ(z)
Combining above two equations are obtain the equation of tangent surface at
z =z as
(9.15) (z−z)T ∇Φ(z) = 0
Thus, gradient at z = z is perpendicular to the level surface passing through
Φ(z) (See Figure 8).
We will now show that it points in the direction in which the function in-
creases most rapidly, and in fact, ∇Φ(z) is the direction of maximum slope.
If
[∇Φ(z)]T ∆z < 0
then
(9.16) Φ(z + ∆z) < Φ(z)
and ∆z is called as descent direction. Suppose we fix our selves to unit sphere
in the neighborhood of z =z i.e. set of all z such that k∆zk ≤ 1 and want to
Figure 8
find direction ∆z such that ∆Φ(z)T ∆z algebraically minimum. Using Cauchy-

Schwartz inequality together with k∆zk ≤ 1, we have
¯ ¯
¯ T ¯
(9.17) ¯[∇Φ(z)] ∆z¯ ≤ k∇Φ(z)k k∆zk ≤ k∇Φ(z)k
This implies
(9.18) − k∇Φ(z)k ≤ [∇Φ(z)]T ∆z ≤ k∇Φ(z)k

and minimum value [∇Φ(z)]T ∆z can attain when ∆z is restricted within the
unit ball equals − k∇Φ(z)k . In fact, the equality will hold if and only if ∆z is
colinear with ∇Φ(z), i.e.
∇Φ(z)
(9.19) z=−
∆b
k∇Φ(z)k
Thus, unit direction ∆b z given by the above equation is the direction of steepest
or maximum descent in which Φ(z + ∆z) − Φ(z) reduces at the maximum rate.
Algorithm
Given a point z(z) ,the steepest descent method performs a line search in the
∇Φ(z)
direction of − , i.e. the direction of steepest descent.
k∇Φ(z)k
INITIALIZE: z(0) , ε, kmax , λ(0)
k=0
δ = 100 ∗ ε
(k) ∇Φ(z(k) )
s =
k∇Φ(z(k) )k
min ¡ (k) ¢
λ∗k = Φ z − λs(k)
λ
z(k+1)
= z(k) − λ∗k s(k)
° °
δ = °∇Φ(z k+1 )° 2
END WHILE
A numerical approach to the above one dimensional minimization problem
is given in the Appendix. Alternate criteria which can be used for termination
of iterations are as follows
¯ ¯
¯Φ(z(k+1) ) − Φ(z(k) )¯
≤ ε
|Φ(z (k+1) )|
¯ ¯
¯ ∂Φ(z(k+1) ) ¯
Maz ¯¯ ¯ ≤ ε
¯
i ∂zi
° (k+1) °
°z − z(k) ° ≤ ε
The Method of steepest descent may appear to be the best unconstrained min-
imization method. However due to the fact that steepest descent is a local
property, this method is not effective in many problems. If the objective func-
tion are distorted, then the method can be hopelessly slow.
9.3. Method of Conjugate Gradients. Convergence characteristics
of steepest descent can be greatly improved by modifying it into a conjugate
gradient method. Method of conjugate gradients display positive characteris-
tics of Cauchy’s and second order (i.e. Newton’s) method with only first order
information. This procedure sets up each new search direction as a linear com-
bination of all previous search directions and newly determined gradient . The
set of directions p(1) , p(2) ........ are called as A-conjugate if they satisfy the
condition.
(9.20) [p(k) ]T Ap(k−1) = 0 for all k

where A is a symmetric positive definite matrix. In conjugate directions
method the successive directions are selected such that successive directions are
conjugate with respect to the local Hessian .
Theorem 14. If a quadratic function

(9.21) Φ(z) = 0.5zT Ax + bT z + c
is minimized sequentially, once along each direction of a set of n linearly inde-
pendent A-conjugate directions,the global minimum of Φ(z) will be reached at or
before nth step regardless of the starting point (initial guess) z(0) .
Proof. Let z =z minimize Φ(z), then applying necessary condition for op-
timality, we have
(9.22) ∇Φ(z) = b + Az = 0
Now , given a point z(0) and n linearly independent directions s(0) ,.....,s(n−1) ,
constants β i can be found such that
X
n−1
(9.23) z = z(0) + β i s(i)
i=0
If {s(i) }are A-conjugate and none of them is zero, then

X
n
(9.24) ∇Φ(z) = b + Az(0) + A β i s(i) = 0
i=1
h i £ ¤T
(9.25) [s(j) ]T ∇Φ(z) = [s(j) ]T b + Ax(1) + β j s(j) As(j) = 0
[b + Ax(0) ]T s(j)
(9.26) βj = −
[s(j) ]T As(j)
Now, consider an iterative procedure for minimization that starts at z(0) and
successively minimizes Φ(z) in directions s(0) .......s(n−1) where the directions are
A-conjugate .The successive points are determined by relation
(k)
(9.27) z(k+1) = z(k) + λmin s(k)
(k)
Whereλmin is found by minimizing f (z(k) + λs(k) ) with respect to λ. At the
optimum λ, we have
∙ ¸T
∂Φ ∂Φ ∂z(k+1)
(9.28) = =0
∂λ ∂z z=z(k+1) ∂λ
(9.29) ⇒ [∇Φ(z(k+1) )]T [s(k) ] = 0
(9.30) ∇Φ(z(k+1) ) = b + A{z(k) + λs(k) }

h iT
(k)
b + Az s(k)
(9.31) ⇒ λkmin = −
[s(k) ]T As(k)
Now
z(i+1) = z(i) + λimin s(i) for i = 1...n
X
k−1
(9.32) ⇒z (k)
=z(0)
+ λ(j) s(j)
j=1
X
i−1
T
(9.33) [z(k) ]T As(k) = [z(0) ]T As(k) + λkmin s(j) As(k)
j=1
(0) T (k)
(9.34) = [z ] As
h iT
b + Az(0) s(k)
(9.35) ⇒ λkmin = −
[s(k) ]T As(k)
which is identical with β j . Thus z can be expressed as
X
n−1 X
n−1
(i)
(0) (j) (0)
(9.36) z=z + βj s =z + λmin s(i)
i=1 i=1
This implies that z can be reached in n steps or less. Since above equation holds
good for any z(0) , the process converges independent of any choice of starting
point z(0) and any arbitrary set of conjugate directions. ¤
Now for development of algorithm we assume that the objective function to

be quadratic.
(9.37) Φ(z) = 0.5z(T ) Ax + bT z + c
Defining
(9.38) g(z) = ∇Φ(z) = Az + b
we have
(9.39) g(k) = g(z(k) ) = Az(k) + b

(9.40) g(k−1) = g(z(k−1) ) = Az(k−1) + b
⇒ 4g(z) = g(k) − g(k−1) = 4g(k) = A 4 z(k)

Now, let us form an iterative scheme
(9.41) z(k+1) = z(k) + λ(k) s(k)
Where we would like s(0) .......s(k) to be A-conjugate. To generate such directions,

consider
X
k−1
(k) (k)
(9.42) s = − 5 Φ(z ) + αi s(i)
i=0
X
k−1
(9.43) = −g(k) + αi s(i)
i=0
which is linear combination of all previous directions with s(0) = −g(0) . Note
that at each iteration, a line search is performed to determine optimum step
length λk such that
δΦ(z(k+1) )
(9.44) = 5Φ[z(k+1) ]T s(k)
δλ
£ ¤T
(9.45) = g(k+1) s(k) = 0
Now, we want to choose αi such that (i = 1, .......k − 1) such that s(k) is a

conjugate to all previous directions. Consider first direction
(9.46) s(1) = −g(1) + α0 s(0) = −g(1) ± α0 g(0)
We force it to be A-conjugate with respect to s(0) , i.e.,

(9.47) [s(1) ]T As(0) = 0
(9.48) [g(1) + α0 g(0) ]T As(0) = 0

At iteration k = 1, we have
∆x(1) = λ0 s(0)
∆x(1)
(9.49) s(0) =
λ0
∆z(1)
(9.50) [g(1) + α0 g(0) ]T A =0
λ0
But, we have
(9.51) A∆z(1) = ∆I (1)
(9.52) [g(1) + α0 g(0) ]T ∆I (1) = 0
−(∆I (1) )T g(1)

(9.53) α0 =
(∆I (1) )T g(0)
−(∆I (1) − g(0) )T g(1)

(9.54) ⇒ α0 =
(∆I (1) − g(0) )T g(0)
(9.55) ⇒ [g(1) ]T g(1) + α0 (g(0) )T g(1) − g(1) g(0) − α0 (g(0) )T g(0) = 0

Now, we have
(9.56) [g(k+1) ]T s(k) = 0
[g(1) ]T g(0) = 0
||g(1) ||2 ||∇f (x1 )||2

(9.57) α0 = (0) 2 =
||g || ||∇f (x0 )||2
Continuing the process, we form next direction as
(9.58) s(2) = −g(2) + α0 s(0) + α1 s(1)
we want to choose α0 and α1 such that
[s(2) ]T As(0) = 0 and [s(2) ]T As(1) = 0
(9.59) [−g(2) + α0 s(0) + α1 s(1) ]T As(0) = 0
(9.60) −g(2) As(0) + α1 [s(2) ]T As(0) = 0
(g(2) )T As(0) (g(2) )T A∆z(1)

(9.61) α1 = =
[s(0) ]T As(0) [s(1) ]T As(1)
(g(2) )T [g(1) − g(0) ] (g(2) )T ∆I (1)

(9.62) =
g(0) Ag(0) [s(1) ]T As(1)
Continuing this process leads to the following general iteration scheme
∙ ¸
(k) (k) ||g(k) ||2
(9.63) s = −g + (k−1) 2
s(k+1)
||g ||
These are called Fletcher and Reev’s iterations. For quadratic Φ(z), the min-
imum reached in maximum n steps. For non-quadratic Φ(z), additional line
searches and steps may be required.
Algorithm
k=0
δ = 100 ∗ ε
° ¡ ¢°
°∇Φ z(k) °2
(k)
α = 2
k∇Φ (z(k−1) )k
∇Φ(z(k) )
s(k) =
k∇Φ(z(k) )k
p = s − α(k) p(k−1)
(k) (k)
min ¡ (k) ¢
λ∗k = Φ z − λp(k)
λ
z(k+1)
= z(k) − λ∗k p(k)
° °
δ = °∇Φ(z(k+1) )°2
END WHILE
Polak and Ribiere have developed conjugate gradient method by considering

more general Φ(x) than quadratic. They have suggested use of αk as
[∆g(k) ]T g(k)
(9.64) αk =
||g(k−1) ||22
9.4. Newton’s Method. The necessary condition for optimization of
a scalar function Φ(z) is
(9.65) ∇Φ(z) = 0
if z = z is the optimum. Note that equation (9.65) defines a system of m
equations in m unknowns. If ∇Φ(z) is continuously differentiable in the neigh-
borhood of z = z, then, using Taylor series expansion, we can express the
optimality condition (9.65) as
£ ¤ £ ¤
(9.66) ∇Φ(z) = ∇Φ z(k) + (z − zk ) ' ∇Φ[z(k) ] + ∇2 Φ(z(k) ) ∆z(k) = 0
Defining Hessian matrix H (k) as
£ ¤
H (k) = ∇2 Φ(z(k) )
an iteration scheme an iteration scheme can be developed by solving equation
(9.66)
z(k+1) = zk + λ∆zk ;
£ ¤−1
∆z(k) = − H (k) ∇Φ[z(k) ]
In order that ∇z(k) is a descent direction it should satisfy the condition
£ ¤T
(9.67) ∇Φ[z(k) ] ∆z(k) < 0
or
£ ¤T £ (k) ¤−1
(9.68) ∇Φ[z(k) ] H ∇Φ[z(k) ] > 0
i.e. in order that ∆z(k) is a descent direction, Hessian H (k) should be a positive
definite matrix. This method has good convergence but demands large amount
of computations i.e. solving a system of linear equations and evaluation of
Hessian at each step
Algorithm
k=0
δ = 100 ∗ ε
Solve H (k) s(k) = −∇Φ(k)
min ¡ (k) ¢
λ∗k = Φ z − λs(k)
λ
z(k+1)
= z(k) − λ∗k s(k)
° £ ¤°
δ = °∇Φ z(k+1) ° 2
END WHILE
9.4.1. Quasi- Newton Method. Major disadvantage of Newtons method
or its variants is the need to compute the Hessian of each iteration . The
quasi-Newton methods over come this difficulty by constructing an approxi-
mate Hessian from the gradient information available at the successive itera-
tion. Quasi-Newton methods thus mimic positive characteristics of Newton’s
method using only gradient information. All methods of this class generate
search direction as
(9.69) s(k) = −D(k) ∇f (k)
which is similar to the Newton’s update, i.e.
s(k) = −[H (k) ]−1 ∇f (k)
Here, D(k) is a n × n matrix (called metric), which changes with iterations
(variable metric methods). A variable metric method is quasi-Newton method
if it is designed so that the iterates satisfy the following quadratic function
property
(9.70) ∆z = A−1 ∆g
Let us assume a recursion for the estimate to inverse of Hessian
(9.71) D(k+1) = D(k) + Dc(k)
(k)
Basic idea is to form Dc such that D(0) , D(1) ..........D (k+1) → [H(z)]−1 =
[∇2 f (z)]−1 . We know that for a quadratic f (z) of the form
(9.72) f (z) = 0.5zT Az + bz + c
we can show
(9.73) ∆z = A−1 ∆g
Let us assume that our approximation for A−1 is of the form
(9.74) A−1 = βD(k)
where β is a scalar. We would like D(k) to satisfy
(9.75) ∆z(k) = z(k+1) − z(k) = βD(k+1) ∆g(k)
= βD(k+1) [g(k+1) − g(k) ]
(k+1) (k) ∆z(k)

(9.76) =D ∆g =
β
Now
(9.77) D(k+1) ∆g(k) = D(k) ∆g(k) + Dc(k) ∆g(k)
∆z(k)
(9.78) = [D(k) + Dc(k) ]∆g(k)
β
∆z(k)
(9.79) Ac(k) ∆g(k) = − D(k) ∆g(k)
β
One can verify by direct substitution that
∙ ¸
(k) 1 ∆z(k) y T A(k) ∆I (k) η T
(9.80) Ac = −
β y T ∆I (k) η T ∆I (k)
is a solution. As y and η are arbitrary vectors, this is really a family of solutions
.If we let
(9.81) y = ∆z(k) and η = A(k) ∆I (k)
we get the Davidon-Fletcher-Powel update
∆z(k−1) [∆z(k−1) ]T A(k−1) ∆I (k−1) [∆I (k−1) A(k−1) ]T
(9.82) A(k) = A(k−1) + −
[∆z(k−1) ]T ∆I (k−1) [∆I (k−1) ]T A(k−1) ∆I (k−1)
Thus, matrix A(k) is iteratively computed
(9.83) A(k+1) = A(k) + M (k) − N (k)

(9.84) q(k) = ∇Φk+1 − ∇Φ(k)
Ã !
λ ∗ h£ ¤ £ (k) ¤T i
k
(9.85) M (k) = T
∆z (k)
∆z
[∆z(k) ] q(k)
Ã !
1 £ (k) (k) ¤ £ (k) (k) ¤T
(9.86) Nk = T
L q L q
[q(k) ] L(k) q(k)
starting from some initial guess, usually a positive definite matrix. Typical
choice is
(9.87) L(0) = I
Properties
• It can be shown that if A(0) is symmetric and positive definite def-
inite, then A(1) , A(2) ...............A(k) will all be symmetric and positive
definite.(Typically A0 = I).
• If f(x)is quadratic, it can be shown that D(k) converges to A−1 at opti-

mum, i.e., D(n+1) = A−1
• The directions generated are A-conjugate
There are several variations of this method depending upon choice of β, y, and z.
Broyder-Fletcher-Shanno update(BSF).
∙ ¸ ∙ ¸
(k) ∆x(k−1) [∆I (k−1) ]T (k−1) ∆x(k−1) [∆I (k−1) ]T
A = I− A I−
[∆x(k−1) ]T ∆I (k−1) [∆x(k−1) ]T ∆I (k−1)
∆x(k−1) [∆x(k−1) ]T
(9.88) +
[∆x(k−1) ]T ∆x(k)
9.5. Leverberg-Marquardt Method. It is known from the expe-
rience that the steepest descent method produces large reduction in objec-
tive function when z(0) is far away from z,The optimal solution. However,
steepest descent method becomes notoriously slow near optimum, on the other
hand Newtons method generates ideal search directions near the solution. The
Leverberg- Marquardt approach combines as follows.
£ (k) ¤
H + λk I ∆z(k) = −∇Φ[z(k) ]
z(k+1) = z(k) + ∆z(k)
Here λk is used to set the search direction and step length. To begin the search,
=104 )is selected so that
a large value of λo (e
h i
H (0) + λ(0) I =e [λo I]
Thus, for sufficiently large λ(0) , ∆z(0) is in the negative of the gradient direction
i.e. −∇Φ(k) .As λk → 0, ∆z(k) goes from steepest descent to Newtons method.
• Advantages:Simplicity, excellent convergence near z
• Disadvantages: Need to compute Hessian matrix, H (k) and set of
linear equations has to be solved at each iteration
Algorithm
k=0
δ = 100 ∗ ε
STEP 1 : Compute H (k) and ∇Φ[z(k) ]
£ ¤
STEP 2 : Solve H (k) + λk I s(k) = −∇Φ[z(k) ]
¡ ¢
IF Φ[z(k+1) ] < Φ[z(k) ]
λ(k+1) = 12 λ(k)
° °
δ = °∇Φ[z(k+1) ]°
k =k+1
ELSE
λ(k) = 2λ(k)
GO TO STEP 2
END WHILE
9.6. Line Search: One Dimensional Optimization. In any multi-dimensional

optimization problem, we have to solve a one dimensional minimization problem
min min ¡ (k) ¢
(9.89) Φ(λ) = Φ z − λs(k)
λ λ
where s(k) is the search direction. There are several numerical approaches avail-
able for solving this one dimensional optimization problem. In this sub-section,
we discuss the cubic interpolation method for performing the line search.
The first step in the line search is to find bounds on the optimal step size λ∗ .
These are established by finding two points, say α and β, such that the slope
dΦ/dλ
∙ ¸T
∂Φ(z(k) − λs(k) ) ∂(z(k) − λs(k) )
(9.90) dΦ/dλ =
∂(z(k) − λs(k) ) ∂λ
¡ ¢T
(9.91) = − ∇Φ(z(k) − λs(k) ) s(k)
has opposite signs at these points. We know that at λ = 0,
¡ ¢T
(9.92) dΦ(0)/dλ = − ∇Φ(z(k) ) s(k) < 0
as s(k) is assumed to be a descent direction. Thus, we take α corresponding to
λ = 0 and try to find a point λ = β such that dΦ/dλ > 0. The point β can be
taken as the first value out of λ = h, 2h, 4h, 8h, .... for which dΦ/dλ > 0,where
h is some pre-assigned initial step size. As dΦ/dλ changes sign in the interval
[0, β], the optimum λ∗ is bounded in the interval [0, β].
The next step is to approximate Φ(λ) over interval [0, β] by a cubic interpo-
lating polynomial for the form
(9.93) Φ(λ) = a + bλ + cλ2 + dλ3
The parameters a and b be computed as
Φ(0) = a = Φ(z(k) )
¡ ¢T
dΦ(0)/dλ = b = − ∇Φ(z(k) ) s(k)
The parameters c and d can be computed by solving
Φ(β) = a + bβ + cβ 2 + dβ 3
dΦ(β)/dλ = b + 2cβ + 3dβ 2
i.e. by solving
" #" # " ¡ ¢ #
β2 β3 c Φ z(k) − βs(k) − a − βb
= ¡ ¢T ¡ ¢
2β 3β 2 d − s(k) ∇Φ z(k) − βs(k) − b
The application of necessary condition for optimality yields
(9.94) dΦ/dλ = b + 2cλ + 3dλ2 = 0
i.e.
p
∗ −c ± (c2 − 3bd)
(9.95) λ =
3d
One of the two values correspond to the minimum. The sufficiency condition
for minimum requires
(9.96) d2 Φ/dλ2 = 2c + 6dλ∗ > 0
The fact that dΦ/dλ has opposite sign at λ = 0 and λ = β ensures that the
equation 9.94 does not have imaginary roots.
Algorithm
INITIALIZE: z(k) , s(k) , h
Step 1: Find β
β=h
WHILE [dΦ(β)/dλ < 0]
β = 2β
END WHILE
Step 2: Solve for a, b, c and d using z(k) , s(k) and β
Step 3: Find λ∗ using sufficient condition for optimality
10. Numerical Methods Based on Optimization Formulation

10.1. Simultaneous Solutions of Linear Algebraic Equations. Con-
sider system of linear algebraic equations
(10.1) Ax = b ; x, b ∈ Rn
where A is a non-singular matrix. Defining objective function

1
(10.2) Φ(x) = (Ax − b)T (Ax − b)
2
10. NUMERICAL METHODS BASED ON OPTIMIZATION FORMULATION 223
the necessary condition for optimality requires that
(10.3) AT (Ax − b) = 0
Since A is assumed to be nonsingular, the stationarity condition is satisfied only

at the solution of Ax = b. The stationary point is also a minimum as AT A is
a positive definite matrix. Thus, we can compute the solution of Ax = b by
minimizing
1 ¡ ¢ ¡ ¢T
(10.4) Φ(x) = xT AT A x− AT b x
2
If conjugate gradient method is used for solving the optimization problem, it
can be theoretically shown that the minimum can be reached in n steps. In
practice, however, we require more than n steps to achieve Φ(x) <ε due to
the rounding off errors in computation of conjugate directions. Nevertheless,
when n is large, this approach can generate a reasonably accurate solution with
considerably less computations.
10.2. Simultaneous Solutions of Nonlinear Algebraic Equations.

Consider problem of solving n nonlinear algebraic equations in n variables, which
have to be solved simultaneously. These can be expressed in the following
abstract form
(10.5) f1 (x1 , x2 , x3, ......xn ) = 0

(10.6) f2 (x1 , x2 , x3, ......xn ) = 0
............................ = 0
(10.7) fn (x1 , x2 , x3, ......xn ) = 0
Or
(10.8) F (x) = 0 ; x ∈ Rn
h iT
x = x1 x2 ... xn
where 0 represents n × 1 zero vector. Here F (x) ∈ Rn represents n dimensional

function vector defined as
h iT
(10.9) F (x) = f1 (x) f2 (x) ... fn (x)
Defining a scalar function
(10.10) Φ(x) = [F (x)]T F (x) = [f1 (x)]2 + [f2 (x)]2 + .... + [fn (x)]2
finding solution to above set of equations can be formulated as minimization of

min
(10.11) Φ (x)
x
The necessary condition for unconstrained optimality is
∙ ¸T
∂Φ (x) ∂F (x)
(10.12) = F (x) = 0
∂x ∂x
∙ ¸
∂F (x)
If we ignore the degenerate case where matrix is singular and vector
∂x
F (x) belongs to null space of this matrix, the necessary condition for optimality
is satisfied when
(10.13) F (x) = 0
which also corresponds to the global minimum of Φ (x) . The Leverberg-Marquardt

method is known to work well for solving optimization formulation of this type.
10.3. Finite Element Method for Solving ODE-BVP and PDEs [15,
16]. The finite element method is a powerful tool for solving PDEs particularly
when the system under consideration has complex geometry. This method is
based on the optimization formulation. In this section, we provide a very brief
introduction to the method of finite element.
10.3.1. Raleigh-Ritz method. Consider linear system of equations
(10.14) Ax = b
where A is a n × n positive definite and symmetric matrix and it is desired

to solve for vector x. We can pose this as a minimization problem by defining
objective function
(10.15) Φ(x) = (1/2)xT Ax − xT b

(10.16) = (1/2) hx, Axi − hx, bi
If Φ(x) minimum at x = x∗ , the necessary condition for optimality requires
(10.17) ∂Φ/∂x = Ax∗ − b = 0
which is precisely the equation we want to solve. Since the Hessian matrix
∂ 2 Φ/∂x2 = A
is positive definite, the solution of x = x∗ of Ax = b is the global minimum of

objective function Φ(x).
In the above demonstration, we were working in space Rn . Now, let us see if

a similar formulation can be worked out in another space, namely C (2) [0, 1], i.e.
the set of twice differentiable continuous functions on [0, 1]. Consider ODE-BVP
(10.18) Lu = −d2 u/dz 2 = f (z);

(10.19) B1 : u(0) = 0
(10.20) B2 : u(1) = 0
Similar to the linear operator (matrix) A,which operates on vector x ∈ Rn to

produce another vector b ∈Rn , the linear operator L = [−d2 /dz 2 ]operates on
vector u(z) ∈ C (2) [0, 1] to produce f (z) ∈ C (2) [0, 1]. Note that the matrix A in
our example is symmetric and positive definite, i.e.
hx, Axi > 0 for all x 6= 0

and AT = A
In order to see how the concept of symmetric matrix can be generalized to

operators on infinite dimensional spaces, let us first define adjoint of a matrix.
Definition 37. (Adjoint of Matrix): A matrix A∗ is said to be adjoint

of matrix A if it satisfies hx, Ayi = hA∗ x, yi . Further, the matrix A is called
self adjoint if A∗ = A.
When matrix A has all real elements, we have
xT (Ay) = (AT x)T y
and it is easy to see that A∗ = AT , i.e.

®
(10.21) hx, Ayi = AT x, y
The matrix A is called self-adjoint if AT = A. Does operator L defined by equa-

tions (10.18-10.20) have some similar properties of symmetry and positiveness?
Analogous to the concept of adjoint of a matrix, we first introduce the concept
of adjoint of an operator L on any inner product space.
Definition 38. (Adjoint of Operator) An operator L∗ is said to be ad-

joint of operator L if it satisfies
hv, Lui = hL∗ v, ui
Further, the operator L is said to be self-adjoint, if L∗ = L, B1∗ = B1 and

B2∗ = B2.
To begin with, let us check whether the operator L defined by equations

(10.18-10.20) is self-adjoint.
Z1
(10.22) hv, Lui = v(z)(−d2 u/dz 2 )dz
0
∙ ¸1 Z1
du dv du
= −v(z) + dz
dz 0 dz dz
0
∙ ¸1 ∙ ¸1 Z1 µ 2 ¶
du dv dv
= −v(z) + u(z) + − 2 u(z)dz
dz 0 dz 0 dz
0
Using the boundary conditions u(0) = u(1) = 0, we have

∙ ¸1
dv dv dv
(10.23) u(z) = u(1) − u(0) = 0
dz 0 dz dz
If we set
(10.24) B1∗ : v(0) = 0
(10.25) B2∗ : v(1) = 0
then
∙ ¸1
du
(10.26) v(z) = 0
dz 0
and we have
Z1 µ 2 ¶
dv
(10.27) hv, Lui = − 2 u(z)dz = hL∗ v, ui
dz
0
In fact, it is easy to see that the operator L is self adjoint as L∗ = L, B1∗ = B1

and B2∗ = B2. In addition to the self-adjointness of L, we have
∙ ¸1 Z1 µ ¶2
du du
(10.28) hu, Lui = −u(z) + dz
dz 0 dz
0
Z1 µ ¶2
du
(10.29) = dz > 0 for all u(z)
dz
0
when u(z) is a non-zero vector in C (2) [0, 1]. In other words, solving the ODE-
BVP is analogous to solving Ax = b by optimization formulation where A is
symmetric and positive definite matrix, i.e.
£ ¤
A ↔ −d2 /dz 2 ; x ↔ u(z); b ↔ f (z)
Let u(z) = u∗ (z) represent the true solution of the ODE-BVP. Now, taking mo-
tivation from the optimization formulation for solving Ax = b, we can formulate
a minimization problem to compute the solution
®
(10.30) Φ [u(z)] = (1/2) u(z), −d2 u/dz 2 − hu(z), f (z)i
Z1 Z1
= 1/2 u(z)(−d2 u/dz 2 )dz − u(z)f (z)dz
0 0
Min
u∗ (z) = Φ[u(z)]
u(z)
Min
(10.31) = (1/2) hu(z), Lui − hu(z), f (z)i
u(z)
(10.32) u(z) ∈ C (2) [0, 1]
subject to u(0) = u(1) = 0

Thus, solving the ODE − BV P has been converted solving a minimization
problem. Integrating the first term in equation (??) by parts, we have
Z1 Z1 1
2 2
(10.33) u(z)(−d u/dz )dz = (du/dz)2 dz − [u(du/dz) ]
0
0 0
Now, using boundary conditions, we have

1
(10.34) [u(du/dz) ] = [u(0)(du/dz)z=0 − u(1)(du/dz)z=1 ] = 0
0
This reduces Φ(u) to
⎡ ⎤
⎡ 1 ⎤
Z1 Z
(10.35) Φ(u) = ⎣1/2 (du/dz)2 dz ⎦ − ⎣ uf (z)dz ⎦
0 0
The above equation is similar to an energy function, where the first term is
analogous to kinetic energy and the second term is analogous to potential energy.
As
Z1
(du/dz)2 dz
0
is positive and symmetric, we are guaranteed to find the minimum. The main
difficulty in performing the search is that, unlike the previous case where we were
working in Rn , the search space is infinite dimensional as u(z) ∈ C (2) [0, 1]. One
remedy to alleviate this difficulty is to reduce the infinite dimensional search
problem to a finite dimensional search space by constructing an approximate
solution using n trial functions. Let v (1) (z), ....., v (n) (z) represent trial function.
The approximate solution is constructed as
(10.36) b(z) = c0 v(0) (z) + ..... + cn v(n) (z)

u
where v(i) (z) represents trial functions. Using this approximation, we convert
the infinite dimensional optimization problem to a finite dimensional optimiza-
tion problem as follows
⎡ ⎤ ⎡ 1 ⎤
Z1 Z
b
(10.37)MinΦ(c) = ⎣1/2 u/dz)2 dz ⎦ − ⎣ u
(db bf (z)dz ⎦
C
0 0
Z1
£ ¡ (0) ¢ ¡ ¢¤2
(10.38) = 1/2 c0 dv (z)/dz + ..... + cn dv (n) (z)/dz dz
0
Z1
(10.39) − f (z)[c0 v (0) (z) + ..... + cn v (n) (z)]dz
0
The trial functions v(i) (z) are chosen in advance and coefficients c1 , ....cm are
treated as unknown. Then, the above optimization problem can be recast as
Min b Min £ ¤
(10.40) Φ(c) = 1/2 cT Ac − cT b
c c
h iT
(10.41) c = c0 c2 ... cn
⎡ ¿ À ¿ (0) À ⎤
dv (0) dv (0) dv dv (n)
⎢ , ........ , ⎥
⎢ dz dz dz dz ⎥
(10.42) A=⎢
⎢ .................. ........ ............. ⎥
⎥
⎣ ¿ (n) À ¿ (n) À ⎦
dv dv (0) dv dv (n)
, ........ ,
dz dz dz dz
⎡ ® ⎤
v(1) (z), f (z)
⎢ ⎥
(10.43) b = ⎣ .................... ⎦
(n) ®
v (z), f (z)
It is easy to see that matrix A is positive definite and symmetric and the global
minimum of the above optimization problem can be found by using necessary
condition for optimality as follows
(10.44) b
∂ Φ/∂c = Ac∗ − b = 0
or
(10.45) c∗ = A−1 b
Note the similarity of the above equation with the normal equation arising
from the projection theorem. Thus, steps in the Raleigh-Ritz method can be
summarized as follows
(1) Choose an approximate solution.
(2) Compute matrix A and vector b
(3) Solve for Ac = b
Similar to finite difference method, we begin by choosing (n − 1) equidistant
internal node (grid) points as follows
zi = i∆z (i = 0, 1, 2, ....n)
and defining (n − 1) finite elements
(10.46) zi−1 ≤ z ≤ zi for i = 1, 2, ...n − 1
Then, by invoking Weierstarss theorem, we formulate the approximate solution

using piecewise constant polynomials on each finite element.. The simplest
possible choice is a line
(10.47) bi (z) = ai + bi z
u
(10.48) zi−1 ≤ z ≤ zi for i = 0, 2, ...n − 1
Figure 9
At he boundary points of the element, we have
(10.49) u bi−1 = ai + bi zi−1

bi (zi−1 ) = u
(10.50) bi (zi ) = u
u bi = ai + bi zi
This implies
u bi zi−1
bi−1 zi − u bi − u
u bi−1
(10.51) ai = ; bi =
∆z ∆z
Thus, the polynomial on the i’th segment can be written as
µ ¶
u bi zi−1
bi−1 zi − u bi − u
u bi−1
(10.52) ubi (z) = + z
∆z ∆z
zi−1 ≤ z ≤ zi for i = 1, 2, ...n − 1
and now we can work in terms of unknown values {b b1 , ....b

u0, u un } instead of
parameters ai and bi. A more elegant and useful form of equation (10.52) can be
found by defining shape functions (see Figure 9)
z − zi z − zi−1
(10.53) Mi (z) = ; Ni (z) =
∆z ∆z
The graphs of these shape functions are straight lines and they have fundamental
properties
( )
1 ; z = zi−1
(10.54) Mi (z) =
0 ; z = zi
( )
0 ; z = zi−1
(10.55) Ni (z) =
1 ; z = zi
bi (z) as
This allows us to express u
bi (z) = u
u bi−1 Mi (z) + u
bi Ni (z)
i = 1, 2, ...n − 1
Figure 10
bi appears in polynomials u
Note that the coefficient u bi (z) and u
bi+1 (z), i.e.
u bi−1 Mi (z) + u
bi (z) = u bi Ni (z)
bi+1 (z) = u
u bi Mi+1 (z) + u
bi+1 Ni+1 (z)
Thus, we can define a continuous trial function by combining Ni (z) and Mi+1 (z)
as follows
⎧ ⎫
⎪ z − zi−1
⎪
⎨ Ni (z) = ∆z
⎪ ; zi−1 ≤ z ≤ zi ⎪ ⎪
⎪
⎬
(i)
(10.56) v (z) = z − zi
⎪ −Mi+1 (z) = 1 − ; zi ≤ z ≤ zi+1 ⎪
⎪
⎪ ∆z ⎪
⎪
⎩ 0 Elsewhere ⎭
i = 1, 2, ....n − 1
The simplest and most widely used are these piecewise linear functions (or hat
function) as shown in Figure 10. This is a continuous linear function of z, but,
it is not differentiable at zi−1, zi, and zi+1 . Also, note that at z = zi , we have
( )
1 if i = j
(10.57) v(i) (zi ) =
0 otherwise
i = 1, ....n − 1
Thus, plot of this function looks like a symmetric triangle. The two functions
at the boundary points are defined as ramps
( z )
−M 1 (z) = 1 − ; 0 ≤ z ≤ z1
(10.58) v (0) (z) = ∆z
0 Elsewhere
⎧ ⎫
⎨ N (z) = z − zn−1 ; z ⎬
n n−1 ≤ z ≤ zn
(10.59) v(n) (z) = ∆z
⎩ 0 Elsewhere ⎭
Introduction of these trial functions allows us to express the approximate solu-

tion as
(10.60) b0 v(0) (z) + ...... + u

b(z) = u
u bn v(n) (z)
h iT
b
and now we can work with u = u b0 u b2 ... u bn as unknowns. The optimum
b can be computed by solving equation
parameters u
(10.61) u−b=0
Ab
where
¿ À
dv (i) dv (j)
(10.62) (A)ij = ,
dz dz
and
( )
dv (i) 1/∆z on interval left of zi
=
dz −1/∆z on interval right of zi
If intervals do not overlap, then
¿ À
dv (i) dv (j)
(10.63) , =0
dz dz
The intervals overlap when
¿ À Zzi Zzi
dv (i) dv (i) 2
(10.64) i=j : , = (1/∆z) dz + (−1/∆z)2 dz = 2/∆z
dz dz
zi−1 zi−1
or
¿ À Zzi
dv (i) dv (i−1)
(10.65)i = j + 1 : , = (1/∆z).(−1/∆z)dz = −1/∆z
dz dz
zi−1
¿ À Zzi
dv (i) dv (i+1)
(10.66)i = j − 1 : , = (1/∆z).(−1/∆z)dz = −1/∆z
dz dz
zi−1
Thus, the matrix A is a tridiagonal matrix

⎡ ⎤
2 −1 .... .... 0
⎢ −1 2 −1 .... ... ⎥
⎢ ⎥
(10.67) A = 1/∆z ⎢ ⎥
⎣ .... .... .... .... ... ⎦
0 .... .... −1 2
which is similar to the matrix obtained using finite difference method. The
components of vector b on the R.H.S. is computed as
(i) ®
(10.68) bi = v , f (z)
Zzi Zzi µ ¶
z − zi−1 z − zi
(10.69) = f (z) dz + f (z) 1 − dz
∆z ∆z
zi−1 zi−1
which is a weighted average of f (z) over the interval zi−1 ≤ z ≤ zi+1 . Note that
the R.H.S. is different from finite difference method.
The Raleigh-Ritz method can be easily applied to problems in higher dimen-
sions when the operators are self-adjoin. Consider Laplace / Poisson’s equation
(10.70) Lu = −∂ 2 u/∂x2 − ∂ 2 u/∂y 2 = f (x, y)

in open set S and u(x, y) = 0 on the boundary. Let the inner product on the
space C (2) [0, 1] × C (2) [0, 1] be defined as
Z1 Z1
(10.71) hf (x, y), g(x, y)i = f (x, y) g(x, y) dxdy
0 0
We formulate and optimization problem

®
(10.72) Φ(u) = 1/2 u(x, y), −∂ 2 u/∂x2 − ∂ 2 u/∂y 2 − hu(x, y), f (x, y)i
Integrating by parts, we can show that
Z Z
(10.73)
Φ(u) = [1/2(∂u/∂x)2 + 1/2(∂u/∂y)2 − f u]dxdy
(10.74) = (1/2) h∂u/∂x, ∂u/∂xi + 1/2 h∂u/∂y, ∂u/∂yi − hf (x, y), u(x, y)i
We begin by choosing (n − 1) × (n − 1) equidistant (with ∆x = ∆y = h)
internal node (grid) points at (xi , yj ) where
xi = ih (i = 1, 2, ....n − 1)
yj = ih (j = 1, 2, ....n − 1)
In two dimension, the simplest element divides region into triangles on which
simple polynomials are fitted. For example, u(x, y) can be approximated as
(10.75) b(x, y) = a + bx + cy
u
where vertices a, b, c can be expressed in terms of values of u b(x, y) at the tri-

angle vertices. For example, consider triangle defined by (xi , yj ), (xi+1 , yj ) and
(xi , yj+1 ). The value of the approximate solution at the corner points is denoted
by
u b(xi , yj ) ; u
bi,j = u bi+1,j = u
b(xi+1 , yj ) ; u
bi,j+1 = u
b(xi , yj+1 )
b(x, y) can be written in terms of shape functions as follows

Then, u
bi+1,j − u
u bi,j bi,j+1 − u
u bi,j
b(x, y) = u
(10.76) u bi,j + (x − xi,j ) + (y − yi,j )
∙ h ¸ h
(x − xi,j ) (y − yi,j )
bi,j 1 −
= u −
h h
∙ ¸ ∙ ¸
(x − xi,j ) (y − yi,j )
(10.77) +bui+1,j bi,j+1
+u
h h
Now, coefficient u bi,j appears in the shape functions of four triangular element
around (xi , yj ). Collecting these shape functions, we can define a two dimen-
sional trial function as follows
(10.78) ⎧ ⎫
⎪ (x − xi,j ) (y − yi,j ) ⎪
⎪
⎪ 1 − − ; x i ≤ x ≤ xi+1 ; y j ≤ y ≤ yj+1 ⎪ ⎪
⎪
⎪ h h ⎪
⎪
⎪
⎪ (x − x ) (y − y ) ⎪
⎪
⎪
⎪ 1 +
i,j
−
i,j
; x ≤ x ≤ x ; y ≤ y ≤ y ⎪
⎪
⎪
⎨ h h
i−1 i j j+1 ⎪
⎬
(i,j)
v (z) = (x − x i,j ) (y − y i,j )
⎪
⎪ 1− + ; xi ≤ x ≤ xi+1 ; yj−1 ≤ y ≤ yj ⎪ ⎪
⎪
⎪ h h ⎪
⎪
⎪
⎪ (x − xi,j ) (y − yi,j ) ⎪
⎪
⎪
⎪ 1 + + ; xi−1 ≤ x ≤ xi ; y j−1 ≤ y ≤ y j ⎪
⎪
⎪
⎪ h h ⎪
⎪
⎩ ⎭
0 Elsewhere
The shape of this trial function is like a pyramid (see Figure 11). We can define
trial functions at the boundary points in a similar manner. Thus, expressing the
approximate solution using trial functions and using the fact that u b(x, y) = 0
at the boundary points, we get
b1,1 v (1,1) (x, y) + .... + u

b(x, y) = u
u bn−1,n−1 v (n−1,n−1) (x, y)
where v(i,j) (z) represents the (i,j)’th trial function. For the sake of convenience,
let us re-number these trial functions and coefficients using a new index l =
Figure 11
0, 1, ....., N such that
l = i + (n − 1)j
i = 1, ...n − 1 and j = 0, 1, ...n − 1
N = (n − 1) × (n − 1)
The approximate solution can now be expressed as
(10.79) b(x, y) = u
u bN v N (x, y)
b0 v0 (x, y) + .... + u
The minimization problem an be reformulated as

(10.80)
∙ ¿ À ¿ À ¸
Min Min 1 ∂b u ∂bu 1 ∂b u ∂bu
Φ(b
u) = , + , b(x, y)i
− hf (x, y), u
b
u b
u 2 ∂x ∂x 2 ∂y ∂y
where
h iT
(10.81) b=
u u b2 ... u
b0 u bN
Thus, the above objective function can be reformulated as
Min Min ¡ ¢
Φ(b
u) = uT Ab
1/2b bT b
u−u
b
u b
u
where
® ®
(10.82) (A)ij = (1/2) ∂v (i) /∂x, ∂v (j) /∂x + (1/2) ∂v(i) /∂y, ∂v (j) /∂y
®
(10.83) bi = f (x, y), v (j) (x, y)
Again, the matrix A is symmetric and positive definite matrix and this guar-
antees that stationary point of Φ(u) is the minimum. At the minimum, we
have
(10.84) ∂Φ/∂b u−b=0

u = Ab
The matrix A will also be a sparse matrix. The main limitation of Relay-Ritz
method is that it works only when the operator L is symmetric or self adjoint.
10.3.2. Gelarkin’s method. The Gelarkin’s method can be applied for any
problem where differential operator is not self adjoint or symmetric. Instead of
minimizing Φ(b u), we solve for
(i) ® ®
(10.85) u(z) = v (i) (z), f (z)
v (z), Lb
i = 0, 2, ....n
b(z) is chosen as finite dimensional approximation to u(z)
where u
(10.86) b0 v(0) (z) + ...... + u
b(z) = u
u bn v(n) (z)
Rearranging above equations as
(i) ®
(10.87) v (z), (Lbu(z) − f (z)) = 0 ; (i = 0, 1, 2, ....n)
we can observe that parameters u0 , .....cn are computed such that the error or
residual vector
(10.88) u(z) − f (z))
e(z) = (Lb
is perpendicular to the subspace (n + 1) dimensional subspace spanned by set
S defined as
© ª
(10.89) S = v(i) (z) : i = 0, 1, 2, ....n
This results in a linear algebraic equation of the form
(10.90) Ab
u=b
where
⎡ ® ® ⎤
v(0) , L(v(0) ) ........ v(1) , L(v(n) )
⎢ ⎥
(10.91) A = ⎣ .................. ........ ............. ⎦
(n) (0)
® (n) (n)
®
v , L(v ) ........ v , L(v )
⎡ ® ⎤
v (0) (z), f (z)
⎢ ⎥
b = ⎣ .................... ⎦
(n) ®
v (z), f (z)
b gives approximate solution given by equation (10.86).When the

Solving for u
operator is L self adjoint, the Gelarkin’s method reduces to the Raleigh-Ritz
method.
Example 65. Consider ODE-BVP
(10.92) Lu = ∂ 2 u/∂z 2 − ∂u/∂z = f (z)

(10.93) in ( 0 < x < 1)
(10.94) subject to u(0) = 0; u(1) = 0
It can be shown that
L∗ (= ∂ 2 /∂z 2 + ∂/∂z) 6= (∂ 2 /∂z 2 − ∂/∂z) = L
Thus, Raleigh-Ritz method cannot be applied to generate approximate solution

to this problem, however, Gelarkin’s method can be applied.
Example 66. [6]Consider the ODE-BVP describing steady state conditions

in a tubular reactor with axial mixing (TRAM) in which an irreversible 2nd
order reaction is carried out.
1 d2 C dC
(10.95) LC = 2
− = DaC 2 (0 ≤ z ≤ 1)
P e dz dz
dC
(10.96) = P e(C − 1) at z = 0;
dz
dC
(10.97) = 0 at z = 1;
dz
The approximate solution is chosen as
X
n
(10.98) b
C(z) =C bn v(n) (z) =
b0 v () (z) + ...... + C bi v (i) (z)
C
i=0
and elements of matrix A are computed as

®
(10.99) Aij = v(i) , L(v(j) )
* Ã n !2 +
X
(10.100) bi = (i)
v , Da bi v (z)
C (i)
i=0
and using the boundary conditions. This yields nonlinear algebraic equations,
which have to be solved simultaneously to compute the unknown coefficients
b0 , ...C
C bn .
11. Summary
In these lecture notes, we have presented various numerical schemes based on
multivariate unconstrained optimization formulation. One of the major appli-
cation of unconstrained optimization is function approximation or multivariate
regression. Thus, we begin by providing a detailed description of the model pa-
rameter estimation problem. We then derive necessary and sufficient conditions
for optimality for a general multivariable unconstrained optimization problem.
If the model has some nice structure, such as it is linear in parameters, then
the parameter estimation problem can be solved analytically. Thus, the linear
model parameter estimation (linear regression) problem is treated elaborately.
Geometric and statistical properties of the linear least square problem are dis-
cussed in detail to provide further insights into this formulation. Numerical
methods for estimating parameters of the nonlinear-in-parameter models are
presented next. Other applications of optimization formulations, such as solv-
ing nonlinear algebraic equations and finite element method for solving PDEs
and ODE-BVPs, have been discussed at the end.
12. Exercise
(1) Square the matrix P = aaT /aT a, which projects onto a line and show
that p2 = p. Is projection matrix invertible?
(2) Compute a matrix that projects every point in the plane onto line
x + 2y = 0.
(3) Solve Ax = b by least square and find p = Ax if
⎡ ⎤ ⎡ ⎤
1 0 1
⎢ ⎥ ⎢ ⎥
A = ⎣ 0 1 ⎦; b = ⎣ 1 ⎦
1 1 0
verify that b − p is perpendicular to the columns of A.

(4) Given the measurements b1 , .., bn at distant points t1 , ..., tn . Show that
the straight line
b = C + Dt + e
P
which minimizes e2 comes from the least squares:
" P #" # " P #
n ti C b
P P = P
ti ( ti )2 D bi ti
12. EXERCISE 239
(5) The following system has no solution

⎡ ⎤ ⎡ ⎤
1 1 " # 4
⎢ ⎥ C ⎢ ⎥
⎣ 1 0 ⎦ =⎣ 5 ⎦
D
1 1 9
sketch and solve a straight line fit that leads to minimization of the
quadratic
(C − D − 4)2 + (C − 5)2 + (C + D − 9)2
(6) Find the best straight line fit to the following measurements, and sketch
your solution
y = 2 at t = −1 ; y = 0 at t = 0
y = −3 at t = −1 ; y = −5 at t = 2
(7) Suppose that instead of a straight line, we fit in the previous exercise
by a parabola:
y = C + Dt + Et2
In the inconsistent set of systems Ax = b that comes from the four
measurements, what are the coefficient matrix A, the unknown vector
x and the data vector b? Compute the least square solution.
(8) (a) If P is the projection matrix onto k-dimensional subspace S of the
whole space Rn , then what is the column space of P and its rank?
(b) If P = A(AT A)−1 AT b is the projection onto the column space
of A, what is the projection of vector a onto row space of A?
(9) It is desired to fit the experimental data in the table below with a
second -degree polynomial of the form
y = a0 + a1 x + a2 x2
using least squares. It is known from external information that the
fitted equation should have a maximum value at x = 0.6 and it should
pass through the point x = 1, y = 0. Determine the fitted equation
that satisfies these requirements.
x : 0.2 0.6 0.8 1.0
y : 0.7 2.0 1.0 0.5
(10) It is desired to fit the heat capacity data for methylecyclohexane (C7 H14 )
to a polynomial function in temperature
model 1: cp = a + bT
model 2: cp = a + bT + cT 2
where cp is the heat capacity, T is the absolute temperature.
• Determine the model parameters in each case using least square

estimation method using the following data:
• Formulate the problem as minimization of ||e||2 = ||Ac − Y ||2
and find the projection matrix Pr that projects vector Y onto the
column space of A.
• Find the model prediction vector Y = Ac and the error vector
e = Y − Y using the projection matrix.
• Assuming that the errors are normally distributed, estimate the
covariance matrix of the estimated parameters and correlation co-
efficient in each case and compare the two models.
cp (KJ/kgK) T (K) cp (KJ/kgK) T (K)

1.426 150 1.627 230
1.469 170 1.661 240
1.516 190 1.696 250
1.567 210 1.732 260
(11) Very common model for a dimensionless first-order chemical reaction

is
dC/dt = −kC
with C(0) = 1. The integrated form of this model is C = exp(−kt),

which is non linear in the parameter k. Solve this problem by first
transforming the model to linear form to first estimate of k to be used
in calculating the nonlinear value. How much different are the linear
and nonlinear values of k
t 0.2 0.5 1 1.5 2

C 0.75 0.55 0.21 0.13 0.04
Reconsider the above problem. Since we do not have a measurement of

the initial concentration at t = 0, it is possible that there is a systematic
error (bias) in the concentrations, so that C(0) is not equal to unity.
To test for this possibility, extend the previous model to form C =
A exp(−kt), where both A and k are to be determined. A simple test
for bias involves re-computing to see if the two-parameter model causes
the value of ”Error” to be significantly reduced. First transform the
original model to a linear form and estimate the model parameters.
Using these estimates of the parameters, solve the nonlinear model. Is
there evidence of a significant bias?
12. EXERCISE 241
(12) Use all of the data given in the following table to fit the following
two-dimensional models for diffusion coefficient D as a function of tem-
perature (T) and weight fraction (X).
T(0 C) 20 20 20 25 25 25 30 30 30
X 0.3 0.4 0.5 0.3 0.4 0.5 0.3 0.4 0.5
D ×105 cm2 /s 0.823 0.639 0.43 0.973 0.751 0.506 1.032 0.824 0.561
Model 1 : D = c1 + c2 T + c3 X
Model 2 : D = c1 + c2 T + c3 X + c2 T 2 + c5 T X + c6 X 2
In each case, calculate D at T = 22, X = 0.36 and compare the two
results.
(13) Weighed least square method is to be used in the following linearizable
model
y1 = exp(−θ1 x1,i − θ2 /x2,i ); i = 1, 2, , .........n
Estimate the weights that should be used to formulate the weighted

least square problem.
(14) Consider the problem of minimizing the following functions
• Function 1
f (x) = 4[(x1 )2 + (x2 )2 ] − 2x1 x2 − 6(x1 + x2 )
Initial guess x(0) = [3 2]T

• Rosenbrok banana function
f (x) = [(x1 )2 − (x2 )2 + (1 − x1 )2 ]
Initial guess x(0) = [−0.5 0.5 2]T

Determine the minimum by the following approaches (a) Analytical
approach (b) Steepest descent method (with 1-D minimization for step
length) (c) Conjugate gradient method (with 1-D minimization for step
length) (d) Newton’s optimization method
(15) Apply Newton’s optimization method and perform 3 iterations each
(without 1-D minimization and t(k) = 1 for all k) to minimize
• f (x) = x1 − x2 + 2(x1 )2 + 2x1 x2 + (x2 )2
x(0) = [0 0]T
• f(x)= -1/[(x1 )2 + (x2 )2 + 2]
x(0) = [4 0]T
Comment upon the convergence of iterations in case (ii). What
measure will ensure convergence?
(16) Solve the following equations using Steepest descent method and New-
ton’s optimization method
2x1 + x2 = 4; x1 + 2x2 + x3 = 8; x2 + 3x3 = 11
by taking the starting point as x = [0 0 0]T .
(17) In optimization algorithms, instead of performing 1-D minimization
to find step length λ(k) at each iteration, an alternate approach is to
choose λ(k) such that
f [x(k+1) ] = f [x(k) + λ(k) s(k) ] < f [x(k) ]
Develop an algorithm to choose λ(k) at each iteration step by the
later approach.
Bibliography
[1] Atkinson, K. E.; An Introduction to Numerical Analysis, John Wiley, 2001.

[2] Axler, S., Linear Algebra Done Right, Springer, San Francisco, 2000.
[3] Bazara, M.S., Sherali, H. D., Shetty, C. M., Nonlinear Programming, John Wiley, 1979.
[4] Demidovich, B. P. and I. A. Maron; Computational Mathematics. Mir Publishers,
Moskow, 1976.
[5] Gourdin, A. and M Boumhrat; Applied Numerical Methods. Prentice Hall India, New
Delhi.
[6] Gupta, S. K.; Numerical Methods for Engineers. Wiley Eastern, New Delhi, 1995.
[7] Kreyzig, E.; Introduction to Functional Analysis with Applications,John Wiley, New
York, 1978.
[8] Linfield, G. and J. Penny; Numerical Methods Using MATLAB, Prentice Hall, 1999.
[9] Linz, P.; Theoretical Numerical Analysis, Dover, New York, 1979.
[10] Luenberger, D. G.; Optimization by Vector Space Approach , John Wiley, New York,
1969.
[11] Luenberger, D. G.; Optimization by Vector Space Approach , Joshn Wiley, New York,
1969.
[12] Moursund, D. G., Duris, C. S., Elementary Theory and Application of Numerical Analy-
sis, Dover, NY, 1988.
[13] Rao, S. S., Optimization: Theory and Applications, Wiley Eastern, New Delhi, 1978.
[14] Rall, L. B.; Computational Solutions of Nonlinear Operator Equations. John Wiley, New
York, 1969.
[15] Strang, G.; Linear Algebra and Its Applications. Harcourt Brace Jevanovich College
Publisher, New York, 1988.
[16] Strang, G.; Introduction to Applied Mathematics. Wellesley Cambridge, Massachusetts,
1986.
[17] Vidyasagar, M.; Nonlinear Systems Analysis. Prentice Hall, 1978.
243

Lecture Notes On Modelling PDF

Uploaded by

Document Informationclick to expand document information

Document Informationclick to expand document information

Copyright:

Available Formats

Lecture Notes On Modelling PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture Notes On Modelling PDF

Uploaded by

Copyright:

Available Formats

Lecture Notes for

Computational Methods in Chemical

Chapter 1. Mathematical Modeling and Simulation 1

Chapter 2. Fundamentals of Functional Analysis 23

Chapter 3. Linear Algebraic Equations and Related Numerical Schemes 55

Chapter 4. ODE-IVPs and Related Numerical Schemes 127

Chapter 5. Optimization and Related Numerical Schemes 165

Mathematical Modeling and Simulation

— Is it possible to operate the plant at a diﬀerent production rate?

In order to solve process design or retrofitting problems, mathematical mod-

• Mass balances: overall, component.

• Distributed parameter model: These models capture the relationship

Figure 1. Typical processing plant: Schematic daigram

2. Mechanistic Models and Abstract Equation Forms

Example 1. Recovery of acetone from air -acetone mixture is achieved us-

(2.2) Air : 0.92Ai = 0.97Ao

Figure 3. Flash Drum: Schematic Diagram

2.2. Nonlinear Algebraic Equations. Consider a stream of two

Example 2. Consider flash vaporization unit shown in Figure 4. A hydro-

Note that this results in a set of simultaneous 5 nonlinear

which represent coupled nonlinear algebraic equations. These equations

The above 5 equations can also be further simplified as follows

(2.20) f1 (x1 , x2 , x3, ......xn ) = 0

Using vector notation, we can write

where 0 represents n × 1 zero vector. Here F (x) ∈ Rn represents n dimensional

2.3. Optimization Based Formulations. Variety of modeling and de-

Example 3. Consider a simple reaction

modelled using the following reaction rate equation

Reaction Rate Concentration Temperature

Suppose we use −raie to denote the estimated reaction rate

Example 4. Cooling water is to be allocated to three distillation columns.

Note that this is an example of a typical multi-dimensional optimization

2.4. ODE - Initial Value Problem (ODE-IVP). For most of the

Figure 5. General lumped parameter system

+ [Amount of S generated within the system]

Mass Balance of the component A

ρi : density of the material in the i’th inlet stream

Example 5. Stirred Tank Heater (STH) System (Figure 6): Total

Assuming that the density is independent of the temperature,

Hot Water Flow Cold Water Flow

Figure 6. Stitted Tank Heater (STH) System

Now, flow out due to the gravity is also a function of height

However, since tank does not move

(2.36) H = ρV Cp (T − Tref ) = ρAhCp (T − Tref )

Figure 7. System response to step change in Ti

Figure 8. System reponse for step change in Fi

solution of the equation (2.39) using the initial as condition T (0) = T ,

• Fi is decreased by 10% from its steady state value F i : Since Fi appears

(2.47) x(t) = [x1 (t)......xn (t)]T ∈ Rn

and u(t) is a forcing function vector defined over t ≥ 0.

obtained by setting dx/dt = 0 where 0 represents n × 1 zero vector.

(2.51) u(t) = [u1 (t) u2 (t)......um (t)]T

(2.53) x(t) = [x1 (t) x2 (t)..........xn (t)]T

Figure 9. Shell and tube heat exchanger

Since u(t) is a known function of time, we re-state the above problem

2.5. PDEs and ODE-Boundary value Problems. Most of the systems

Steady State Simulation: Find T (z) given T (z = 0) = T1 when ∂T /∂t = 0,