Introduction to OR and Linear Algebra

Part-I

Dr. Prasun Das

Indian Statistical Institute
SQC & OR Unit
Kolkata, India
[email protected]
 OR-The Historical Development

• The beginning of operations research has generally been attributed

to the military services during World War II.

• There was an urgent need to allocate scarce resources to the various

military operations in an effective manner.

• The military commands of U.K. and U.S.A engaged several inter-

disciplinary teams of scientists to undertake scientific research into
strategic and tactical military operations related to air and land
defense of the country (the efficient ocean transport, effective
bombing etc.).

• Hence OR can be associated with “an art of winning the war

without actually fighting it”.
 OR-The Historical Development

• As the name implies, ‘Operations Research’ was apparently invented

because the team was dealing with research on (military) operations.

• Following the end of war, the success of military teams attracted the
attention of industry people. In this way, operations research began to
creep into industry, business and civil government.

• American Mathematician, George B. Dantzig, developed the first

mathematical technique in this field in 1947. By 1951, it had already
taken hold in Great Britain and was in the process of doing so in the
United States for the invention of new flight patterns, planning sea
mining and effective utilization of electronic equipment.
 OR-The Historical Development

• Today, the impact of OR can be felt in many areas like transportation

system, libraries, hospitals and city planning, financial institutions
apart from military and business applications. In fact, linear
programming can be viewed as part of a great revolutionary
development to lay out a path of detailed decisions to take in order to
achieve its goals when faced with practical situations of great
complexity.

• Some of the well-known names for the contribution during the

beginning of operations research are von Neumann, Kantorovich,
Leontief, Koopmans and Dantzig.
 What is OR ?

Techniques for Optimizing the Performance of a System.

System: a Process, Product, Institute, Organization etc.

Performance Measure: Any criterion/property measuring the way system is

working. Some important performance measures are profit, cost, profitability,

market share etc. Performance measures are of two types: PROFIT measure

(higher is desirable) and COST measure (lower is desirable).

Optimizing a Performance: either maximizing or minimizing a

criterion/property.
 OR Techniques

Linear Programming: To optimize a single linear objective function subject to

linear constraints.
Nonlinear Programming: Techniques for solving mathematical models
involving nonlinear functions.
Integer and Combinatorial Optimization: Techniques for solving mathematical
models involving discrete decision variables.
Dynamic Programming: Technique for solving multi-period or multi-stage
decision problems.
 OR Techniques
Multi-criteria Decision-making: To choose the best among a small number of
alternatives, when there are many important criteria to consider.
Multi-objective Programming: Techniques for obtaining good (compromise)
solutions involving two or more objective functions that need to be optimized
simultaneously. Goal Programming is one technique for solving multi-objective
programs in classical domain.
Project Scheduling (PERT, CPM): Techniques for controlling, scheduling
various jobs in a project, which have to obey a precedence relationship.
Inventory Management: Techniques for maintaining inventories to meet
demands for goods, while at same time minimizing inventory costs.
Queuing Problems: To optimize performance of systems involving waiting lines.
 Basic Concepts of Linear Algebra

• Vectors, Matrices, Linear Independence/Dependence of vectors, Rank, Basis,

System of linear eqns, Hyperplane, Convex set, Convex polyhedron, Extreme
points, Basic feasible solution etc.
• Graphical solution method (for 2-variable problems)
 Definitions – Euclidean Space
n-dimensional real Euclidean space (n): Cartesian product of n sets, each
equal to 1:

n = 1 x 1 … 1 x 1 n times
So n consists of n-tuples of real numbers as 1xn = {(x1, x2,…,xn): xi 1} or
nx1 = {(x1, x2,…,xn)t: xi 1}

Vector addition and Scalar multiplication: The set of all n-component vectors
with addition [ x + y = (x1+y1, x2 +y2,…, xn +yn)t ] and multiplication by scalar
[ .x = (x1, x2,…, xn)t ] and the laws of matrix algebra being used to carry on
n 2

the algebra in the set, together with distance | x  y |  x

i 1
i  yi  gives the n-
dimensional Euclidean Space denoted by n or En.
 Definitions – Vector Space
A non-empty set S  n is called a Vector Space in n (or, a subspace in n),

if both of the following conditions hold:

a) S is closed under vector addition (xS, yS  x+yS)

b) S is closed under scalar multiplication (xS  xS for all  1)

In other words, a non-empty set S  n is a subspace in n if x+yS for every

x, yS and every  1.

Trivial subspace: The set {0}  n is itself a subspace of n

 Definitions – L.C. and Span
Linear Combination: Let A = {a1, a2,…,an} be a set of n vectors with aim,
i=1,…,n and x1, x2,…,xn are real numbers.
n
Then, the vector b = xi ai= x1a1+ x2a2+…+xnan is called a linear combination of
i 1
a1, a2,…,an.
Span of A: The set of all possible linear combinations of the ai’s is called the
span of A, written as Sp(A) = {1a1+ 2a2+…+nan: i1, i = 1,2,…,n}

Result: Let A be a non-empty set of vectors in m. The span of A is a subspace

of m.

Result: Let A = {a1, a2,…,an} be a set of vectors from some subspace S  m.
Let A = [a1, a2,…,an] be the m x n matrix with ai as the i-th column. Then A
spans S iff Ax = b is a consistent system for every bS.
 Definitions – Linear Independence (l.i.)

Let A = {a1, a2,…,an} be a finite set of vectors with each aim.

Linear independence (l.i.): A is said to be l.i. if whenever there exist

real numbers x1, x2,…,xn, s.t.
x1a1+ x2a2 +…+ xnan = 0,  x1= x2=…= xn = 0.

On the other hand,

Linear dependence (l.d.): A is said to be l.d. if whenever there exist
real numbers x1, x2,…,xn, not all zero, such that x1a1+ x2a2 +…+ xnan = 0.
 Definitions – Basis

Let, Bmxm be a submatrix of Amxn comprising of m(<n) l.i columns of A.

Then, Bmxm = {a1,a2,…,am} is called a basis set for Em if,
 B is a l.i. set of vectors;
 B spans Em i.e., any member of Em can be expressed as a linear
combination of others.

Example: {e1,e2,…,en} is called a basis for En since

 iei = 0  i = 0 for all i = 1,2,…,n
 Let, xEn, i.e. x = (x1, x2,…, xn). Then, x = xiei
 Definitions – System of linear equations

Ax = b, where A = (aij)m×n; x = (x1, x2,…, xn)tnx1; b = (b1, b2,…, bm)tmx1

If r(A) = r(A b) = k (k m, kn)  system is consistent (at least one soln.)
If r(A)  r(A b)  system is inconsistent (no soln. exists)

Case-1. k < m  system is redundant (at least one eqn. may be dropped)
Case-2. k = m  system is not redundant
Case-3. k = n  system is having unique solution.
Case-4. k < n  system is having infinitely many solutions.

Rank of A: (i) maximum number of l.i. rows/columns of A; or

(ii) order of the largest non-vanishing minor of A.
 Definitions – Basic Variables & Basis Matrix
If rank (A) = m (no-redundancy) with m  n (unique/infinite solutions) for a
consistent system Ax = b, then the matrix A has at least one set of m l.i. columns.

Variables, xj, j = 1,…,m, associated with the jth column of the matrix A are called
basic variables. The remaining (n-m) variables are called non-basic variables.

The original matrix A can be partitioned into two matrices B and N respectively
such that A = [B N] where, B is called the basis matrix associated with m basic
variables and N is called non-basis matrix associated with (n-m) non-basic
variables.
 Definitions – Basic Solution & B.F.S

• Linear System: Ax = b. Assume r(A) = m  n.

• Partition columns as A = [B:N], where B is m x m basis matrix and N is
m x (n-m) non-basis matrix.
• The linear system becomes: BxB + NxN = b, where xB is vector of basic
variables and xN is vector of non-basic variables.
• Since B is non-singular, it is invertible. Hence, xB = B-1b – B-1NxN.
• Basic Solution for a linear system: Set xN = 0, then, xB = B-1b.
• If xB  0 (i.e. B–1b  0), then the solution x = (B-1b, 0)t is called a basic
feasible solution (b.f.s).
 Definitions - Hyperplane and Half Space

• Hyperplane H: {x: ctx = }, where cRn and R1

(Ex: Line c1x1 + c2x2 =  in R2, Plane c1x1+c2x2+c3x3 =  in R3)
• Closed halfspaces H+: {x: ctx  } and H-: {x: ctx  }
• Open halfspaces H+: {x: ctx > } and H-: {x: ctx < }
• A polyhedral set is the intersection of halfspaces.
 Definitions - Convexity and Convex polyhedron
Point Set: Any subset XEn will be called a point set of En.
Line Segment in En: line joining distinct points x1, x2 in En is defined as S = {x:
x = (1-)x1 + x2, 01}
Convex Set: Let XEn. Then X is called a convex set if for x1, x2En, (1-)x1 +
x2 X where, 01.
 Definitions - Convexity and Convex polyhedron

Convex Polyhedran:
Let x1, x2,…, xp  En. If x = ixi, i  0, i = 1, then x is called a convex
combination of x1, x2,…, xp. The set of all convex combinations of x1, x2,…, xp
is called Convex Polyhedran generated (or, spanned) by the points x1, x2,…, xp.

• A Convex Polyhedron is an intersection of half spaces.

• It is Not necessarily bounded.
• A bounded convex polyhedron is a polytope.
• A polytope in R2 is a convex polygon.
Geometry: Convex Polyhedron
 Definition: Extreme Points

A corner
corner--point (extreme point) feasible solution is a feasible solution
that cannot be written as a convex combination of other two feasible
solutions. (does not lie on any line segment connecting 2 other
feasible solutions.
 Definition: Extreme Points

Extreme Points of Convex Set:

Let X be a convex set in En and  X. Then, is called an extreme point of X if
there do not exist x1, x2En, x1 x2 s.t. = (1-)x1 + x2 X where, 0<<1. ie.,
should not occur strictly in between any line segment in X.
Remark: A convex set with no extreme point is a hyperplane
Question: If X is defined as the interior of the circle as well as points on the
circumference, then what about existence of extreme points of X?
Definition: Extreme Points in case of a LPP
 Definition: Extreme Points in case of a LPP

Feasible region is a convex

polyhedron; Optimal solution is
a corner.
 Some Results of Linear Algebra

Result-1: A set of vectors in En is linearly dependent (l.d.) iff at least one of this
can be expressed as a linear combination (l.c.) of the others.
Proof: use the definition of l.d. set of vectors. [Hint]
Result-2: A linearly independent set of vectors cannot contain zero vectors.
Proof: Try yourself
Result-3: Subset of a linearly independent set is linearly independent (l.i.).
Proof: Let, S = {a1,a2,…,ap} be a set of l.i. vectors and T = {a1,a2,…,an} (n<p) be
any subset of S.
Hint: Try by contradiction theory on the set T and show that S is l.d.
Result-4: Any superset of a linearly dependent set is linearly dependent.
Proof: Try yourself

03-12-2020 26
 Some Results of Linear Algebra

Result-5: Representation of any vector in terms of a given set of basis vectors is

unique.

Proof. Let B = {a1,a2,…,ap} be a set of basis vectors and let xEn.

Then, x = iai =iai i = 1,2,…,p
Hint: Now, use the definition of Basis (property of l.i.)

03-12-2020 27
 Some Results of Linear Algebra

Result-6: Replacement of a basis vector forms another basis.

Proof: Let, B = {a1,a2,…,an} be a set of basis vectors in En and 0bEn.
To show that, B1 = {a1,a2,…,an-1, b} is also a basis.
Condition-1: B1 is a l.i. set of vectors.
If possible, suppose B1 is l.d., then iai + .b = 0 for i = 1,2,…,n-1 and with at
least one i  0. Now   0, otherwise, a1,a2,…,an-1 are l.d. which is impossible
being subset of a l.i. set (Result 3). Thus,
1a1 + 2a2 + … + n-1an-1 + .iai = 0 (since, b = iai , for i = 1,2,…,n with
n  0, say)
or, (i +.i)ai + (.n) an = 0 where .n  0 (since, 0, n 
0), i = 1,2,…,n-1
This shows, a1,a2,…,an are l.d. which is contradictory !!!
Hence, B1 is a l.i. set.

03-12-2020 28
 Some Results of Linear Algebra

Result-6 [contd…]: Replacement of a basis vector forms another basis.

Condition-2: B1 spans En
Let, xEn. Then, x = iai (Since, B is a basis of En), i = 1,2,…,n
So, x =  iai + nan, i = 1,2,…,n-1 ……………..(1)
Now, b =  iai + nan, i = 1,2,…,n-1
or, an = (1/n) [b -  iai ], i = 1,2,…,n-1 and n  0
Replacing an in (1), we get
x =  iai + n.(1/n) [b - iai ], i = 1,2,…,n-1
=  [i – (n./n).i]ai + (n./n).b, i = 1,2,…,n-1
This shows, x is a l.c. of elements of B1. Thus B1 spans En.
Hence the proof.

03-12-2020 29
 Some Results of Linear Algebra

Home Work:
Result-7: Hyperplane is a Convex Set.
Result-8: Half Spaces are Convex Sets.
Result-9: Intersection of two convex sets is Convex.
[Is it true for Union of two convex sets? Comment]

03-12-2020 30
 Some Results of Linear Algebra

Result-10: Convex Polyhedran is a Convex Set.

Proof: Let S = {x: x = ixi, i  0, i = 1} and u1, u2 S.
Then, u1 = ixi, i  0, i = 1 and u2 = ixi, i  0, i = 1
If u = (1-)u1 + u2, 01, then
Show that,
(i) u is expressed as convex combination of x1, x2,…, xp, say, u = ixi,
(ii) i  0, i = 1.
Then, u S and hence, S is convex.

03-12-2020 31
 Some Results of Linear Algebra

Result-11: In the general LPP, the set of all feasible solutions is a Convex Set.
Proof: Let K denote the set of all f.s. and x1, x2K.
Then, x1 0, x2 0, Ax1= b, Ax2= b
Let, x = (1-)x1 + x2, 01
Then, prove…. x  0 and Ax = b so that, xK.
Therefore, K is Convex.

03-12-2020 32