Lecture 2: Fundamentals of Optimization Models

Dabeen Lee

Industrial and Systems Engineering, KAIST

IE 331 Operations Research (OR): Optimization

February 28, 2024

Outline

• Quick review of linear algebra: vectors and matrices.

• Components of optimization models.
• Optimization terminologies.
• Introduction to linear programming.
Quick review of linear algebra

We write a vector x 2 Rd in the d-dimensional space as

2 3
x1
6 x2 7
6 7
x = 6 . 7 or x = (x1 , . . . , xd ).
4 .. 5
xd
Quick review of linear algebra

We write a vector x 2 Rd in the d-dimensional space as

2 3
x1
6 x2 7
6 7
x = 6 . 7 or x = (x1 , . . . , xd ).
4 .. 5
xd

The first represents x as a column vector, while the second is the row vector
representation.
Quick review of linear algebra

Rigorously, they are the transpose of one another, i.e.

2 3
x1
6 x2 7
6 7 >
6 . 7 = (x1 , . . . , xd ) .
4 .. 5
xd
Quick review of linear algebra

Rigorously, they are the transpose of one another, i.e.

2 3
x1
6 x2 7
6 7 >
6 . 7 = (x1 , . . . , xd ) .
4 .. 5
xd

We interchangeably use them depending on the context.

Quick review of linear algebra

Rigorously, they are the transpose of one another, i.e.

2 3
x1
6 x2 7
6 7 >
6 . 7 = (x1 , . . . , xd ) .
4 .. 5
xd

We interchangeably use them depending on the context.

x1 , . . . , xd are called the components or coordinates of vector x.

Quick review of linear algebra

An m ⇥ d matrix A 2 Rm⇥d is written as

2 3 2 >3
a11 a12 ··· a1d a1
6 a21 a22 ··· a2d 7 6a2> 7
6 7 6 7
A=6 . .. .. .. 7 = 6 .. 7 .
4 .. . . . 5 4 . 5
>
am1 am2 ··· amd am
Quick review of linear algebra

An m ⇥ d matrix A 2 Rm⇥d is written as

2 3 2 >3
a11 a12 ··· a1d a1
6 a21 a22 ··· a2d 7 6a2> 7
6 7 6 7
A=6 . .. .. .. 7 = 6 .. 7 .
4 .. . . . 5 4 . 5
>
am1 am2 ··· amd am

Here, aij for i = 1, . . . , m and j = 1, . . . , d are the entries of A.

Quick review of linear algebra

An m ⇥ d matrix A 2 Rm⇥d is written as

2 3 2 >3
a11 a12 ··· a1d a1
6 a21 a22 ··· a2d 7 6a2> 7
6 7 6 7
A=6 . .. .. .. 7 = 6 .. 7 .
4 .. . . . 5 4 . 5
>
am1 am2 ··· amd am

Here, aij for i = 1, . . . , m and j = 1, . . . , d are the entries of A.

Moreover, ai 2 Rd for i = 1, . . . , m are called the rows of A and given by

2 3
ai1
6 7
⇥ ⇤ 6ai2 7
ai> = ai1 a12 · · · aid and ai = 6 . 7 .
4 .. 5
aid
Quick review of linear algebra

An m ⇥ d matrix A 2 Rm⇥d is written as

2 3 2 >3
a11 a12 ··· a1d a1
6 a21 a22 ··· a2d 7 6a2> 7
6 7 6 7
A=6 . .. .. .. 7 = 6 .. 7 .
4 .. . . . 5 4 . 5
>
am1 am2 ··· amd am

Here, aij for i = 1, . . . , m and j = 1, . . . , d are the entries of A.

Moreover, ai 2 Rd for i = 1, . . . , m are called the rows of A and given by

2 3
ai1
6 7
⇥ ⇤ 6ai2 7
ai> = ai1 a12 · · · aid and ai = 6 . 7 .
4 .. 5
aid

Again, ai> is a row vector while ai is a column vector.

Components of optimization models

A decision-making problem has the following three components.

• Decisions
Decisions : parameters and variables that we need to determine.
For example, the number of units to produce, a schedule of sports games,
and a regression model.

Constraints
• Constraints : restrictions and requirements that we need to satisfy.
For example, the limits on how much material we use for production and
balancing the number of away games and that of home games for a team.

objective: what we are trying to achieve.

• Objective

Typically either maximizing profits or minimizing costs.

Components of optimization models

Optimization models formulate the three components of a decision-making

problem mathematically and quantitatively.
Components of optimization models

Optimization models formulate the three components of a decision-making

problem mathematically and quantitatively.

• Decisions are encoded by variables x = (x1 , . . . , xd ) where each xj has to

assign a numerical value.
Components of optimization models

Optimization models formulate the three components of a decision-making

problem mathematically and quantitatively.

• Decisions are encoded by variables x = (x1 , . . . , xd ) where each xj has to

assign a numerical value.

• Constraints are modeled by some functions of variables.

Components of optimization models

Optimization models formulate the three components of a decision-making

problem mathematically and quantitatively.

• Decisions are encoded by variables x = (x1 , . . . , xd ) where each xj has to

assign a numerical value.
right-hand side

O
• Constraints are modeled by some functions of variables.

A constraint is of the form g (x)  b for some function g : Rd ! R and

some bound b 2 R. I
construint functi
Components of optimization models

Optimization models formulate the three components of a decision-making

problem mathematically and quantitatively.

• Decisions are encoded by variables x = (x1 , . . . , xd ) where each xj has to

assign a numerical value.

• Constraints are modeled by some functions of variables.

A constraint is of the form g (x)  b for some function g : Rd ! R and

some bound b 2 R.
Here, without loss of generality, we may assume that b = 0 for a
constraint because g (x)  b is equivalent to g (x) b  0.
Components of optimization models

Optimization models formulate the three components of a decision-making

problem mathematically and quantitatively.

• Decisions are encoded by variables x = (x1 , . . . , xd ) where each xj has to

assign a numerical value.

• Constraints are modeled by some functions of variables.

A constraint is of the form g (x)  b for some function g : Rd ! R and

some bound b 2 R.
Here, without loss of generality, we may assume that b = 0 for a
constraint because g (x)  b is equivalent to g (x) b  0.

• The objective is a function f : Rd ! R that want to maximize or minimize.

Components of optimization models

Optimization models formulate the three components of a decision-making

problem mathematically and quantitatively.

• Decisions are encoded by variables x = (x1 , . . . , xd ) where each xj has to

assign a numerical value.

• Constraints are modeled by some functions of variables.

A constraint is of the form g (x)  b for some function g : Rd ! R and

some bound b 2 R.
Here, without loss of generality, we may assume that b = 0 for a
constraint because g (x)  b is equivalent to g (x) b  0.
> objectiv function
-
• The objective is a function f : Rd ! R that want to maximize or minimize.

We may have multiple decisions (multi-objective optimization) and

constraints but usually there is a single objective.
Optimization problem

Given the formalism, we may state an optimization problem as follows.

Choose
decision variables x
to make
the objective f (x)
as large (or small) as possible under the restrictions that x satisfy

constraints g1 (x)  b1 , . . . , gm (x)  bm .

Optimization problem

Standard ways to represent the optimization problem is

min / max f (x)

x x

s.t. gi (x)  bi , i 2 [m]

Optimization problem

Standard ways to represent the optimization problem is

min / max f (x)

x x

s.t. gi (x)  bi , i 2 [m]

and
min / max {f (x) : gi (x)  bi , i 2 [m]}
x x
Optimization problem

Standard ways to represent the optimization problem is

min / max f (x)

x x

O
s.t. g (x)  b ,
i i i 2 [m]

and
min / max {f (x) : gi (x)  bi , i 2 [m]}
x x

where
• s.t. stands for subject to,
Optimization problem

Standard ways to represent the optimization problem is

min / max f (x)

x x

s.t. gi (x)  bi , i 2 [m]

and
min / max {f (x) : gi (x)  bi , i 2 [m]}
x x

where
• s.t. stands for subject to,
• [m] denotes the set {1, . . . , m},
Optimization problem

Standard ways to represent the optimization problem is

min / max f (x)

x x

s.t. gi (x)  bi , i 2 [m]

and
min / max {f (x) : gi (x)  bi , i 2 [m]}
x x

where
• s.t. stands for subject to,
• [m] denotes the set {1, . . . , m},
• subscript x under min / max means that the optimization problem is over
decision variables x.
Optimization problem
< die) i t (d)
0 .

that x 2 Rd .
-
O
Unless there is some constraint on x, e.g. x 2 Zd and x 2 [0, 1]d , we assume
Optimization problem

Unless there is some constraint on x, e.g. x 2 Zd and x 2 [0, 1]d , we assume

that x 2 Rd .

To be more specific, we may write the optimization problem as

min / max f (x) for all

s.t. gi (x)  bi , #
i 2 [m],
x 2 Rd .
Optimization problem

Unless there is some constraint on x, e.g. x 2 Zd and x 2 [0, 1]d , we assume

that x 2 Rd .

To be more specific, we may write the optimization problem as

min / max f (x)

s.t. gi (x)  bi , i 2 [m],
d
x 2R .

It is common to omit the constraint x 2 Rd .

Example

Let us get back to the production planning question.

A B
X 2 2
Y 3 1
Recall that we have 150 units of material A and 70 units of material B.
S
• Decisions: the number x of product X and the number y of product Y to
produce.
• Constraints: the amount of material A required for production is at most
150, given by
2x2x++ 3y  150,
by
2150

and the amount of material B required for production is at most 70, given
by
2x + y Eno
2x + y  70.
Lastly, the quantities x and y cannot be negative.
• Objective: the objective is to maximize the total number of products
produced. Here, the number of produced products would be x + y
x+
y
Example

Hence, the problem can be formulated as the following optimization problem.

maxx +
y
diy
=150
s t
.

.
2x + By
Eno
City
d .
y
10
Optimization terminologies

Let us take an optimization problem

min f (x)
s.t. gi (x)  bi , i 2 [m],
d
x 2R .

• f (x) is the objective funcin

• g1 (x), . . . , gm (x) are constraint functions

• b1 , . . . , bm are called righthand sides or righthand side values

• Any vector x 2 Rd is a solution

• A solution x satisfying all the constraints, gi (x)  bi for i 2 [m] is called
feasible feasible
feasible or a feasible solutionsolution

• If a solution x violates some constraint, i.e. gi (x) > bi for some i 2 [m],
we call x infeasible infeasible
infeasibleor an infeasible solutionsolution
Optimization terminologies

• The value p ⇤ = minx {f (x) : gi (x)  bi , i 2 [m]}, as long as it is finite, is

value
optimal
the

• We also call it the

optimum
objective value of x.
• Given a solution x, we call f (x) the objective value

• A feasible solution x ⇤ whose objective value f (x ⇤ ) is equal to the optimal

-

optimal
value p ⇤ is called optimal or an optimal solution
optimal solution
• In fact, there can be more than one optimal solutions.
Optimization terminologies

• We say that the optimization problem

min f (x)
s.t. gi (x)  bi , i 2 [m],
d
x 2R .

feasible if it admits a feasible solution.

is feasible

infeasible if there is no feaasible solution, in

• We say that the problem is infeasible
which case, we set p ⇤ = 1 by convention.

• We say that the problem is unbounded if

fr there is a tensible solutin >
in which case, we set p ⇤ = 1. with f() <U .
Optimization terminologies

The definitions and terminologies apply to maximization problems, as a

maximization problem can be equivalently transformed into a minimization
problem.

Note that

max {f (x) : gi (x)  bi , i 2 [m]}

x

= -

x
min)-ful)
min { f (x) : gi (x)  bi ,
Sik) [bi
i 2: [m]} . ,
it
my
Here, the maximization problem being infeasible means that p ⇤ = 1 while
being unbounded means p ⇤ = 1.
 El
Example
o
O

• In the production planning example, any vector (x, y ) 2 R2 is a solution

• Note that (0, 0), (10, 10), (20, 20) all satisfy the constraints, so they are
feasible solutions
• However, (30, 30) violates the second constraint as 2 ⇥ 30 + 30 = 90 > 70,
- - - -

so (30, 30) is an
infeasible solutio
• As the problem has a feasible solution, the problem is feasible
• We have seen that (15, 40) is an
optimal solution
• 15 + 40 = 55 is the
optimum ,
the optimal value

• Therefore, the problem is not unbounded

Linear functions

lineup if
We say that a function f : Rd ! R is linear

f (↵x + y ) = ↵f (x) + f (y )
d
for any x, y 2 R and ↵, 2 R.

In fact, a function f is linear if and only if there exists some vector c 2 Rd such
that
d
X
is) 1 x1 + c2 x2 + · · · + Hold
f (x) = c > x = cClitlicht ·

cd xd = cj xj Gilli
,
.
j=1

Example 1: x1 + x2 + x3 , x1 + 2x3 , x2 , and x1 x2 + 3x3 are all linear functions.

Example 2: x1 x3 and x12 + 2x1 x2 + x22 are nonlinear.

"biline
L gundratic
quadratic
Linear programs

A linear program (in short, LP) is an optimization of the following form.

min c(Tx
>
x

;
s.t. aai> x D
=bib
, ; i 2 [m],
x 2 Rd

where c 2 Rd , a1 , . . . , am 2 Rd , and b1 , . . . , bm 2 R.

Here, the objective function c > x is equal to

< p1 + Che
c1 x1 + c2 x2 + · · · + cd xd ,
. + Gild
and the constraint ai> x  bi is equivalent to

ai1 x1 + ai2 x2 + · · · + aid xd  bi . b;

ailthith-o-42ds <
Example

For the production planning problem, the objective function is x + y , which is

linear.
Example

For the production planning problem, the objective function is x + y , which is

linear.

Moreover, 2x + 3y , 2x + y , x, and y are all linear.

Example

For the production planning problem, the objective function is x + y , which is

linear.

Moreover, 2x + 3y , 2x + y , x, and y are all linear.

Therefore, all constraints are given by linear functions.

Example

For the production planning problem, the objective function is x + y , which is

linear.

Moreover, 2x + 3y , 2x + y , x, and y are all linear.

Therefore, all constraints are given by linear functions.

Hence, the production planning problem is a linear program.

Production planning (general version)

Imagine a company producing a variety of products.

Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
We enumerate them by material 1, material 2, and so forth.
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
We enumerate them by material 1, material 2, and so forth.

*
• One unit of product j sells for price pj for i 2 [d].
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
We enumerate them by material 1, material 2, and so forth.
• One unit of product j sells for price pj for i 2 [d].
• The current stock of material i is bi for i 2 [m].
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
We enumerate them by material 1, material 2, and so forth.
• One unit of product j sells for price pj for i 2 [d].
• The current stock of material i is bi for i 2 [m].
O
• Producing one unit of project j requires aij amount of material i for every
pair (i, j) 2 [m] ⇥ [d] where [m] ⇥ [d] = {(i, j) : i 2 [m], j 2 [d]}.
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
We enumerate them by material 1, material 2, and so forth.
• One unit of product j sells for price pj for i 2 [d].
• The current stock of material i is bi for i 2 [m].
• Producing one unit of project j requires aij amount of material i for every
pair (i, j) 2 [m] ⇥ [d] where [m] ⇥ [d] = {(i, j) : i 2 [m], j 2 [d]}.
• We assume that all data are nonnegative, i.e. pj 0, bi 0, and aij 0
for all i 2 [m] and j 2 [d].
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
We enumerate them by material 1, material 2, and so forth.
• One unit of product j sells for price pj for i 2 [d].
• The current stock of material i is bi for i 2 [m].
• Producing one unit of project j requires aij amount of material i for every
pair (i, j) 2 [m] ⇥ [d] where [m] ⇥ [d] = {(i, j) : i 2 [m], j 2 [d]}.
• We assume that all data are nonnegative, i.e. pj 0, bi 0, and aij 0
for all i 2 [m] and j 2 [d].
• We assume that each product is divisible, which means that the amount of
each product can be any real number.
Production planning (general version)

Imagine a company producing a variety of products.

• There are d di↵erent products in the product portfolio of the company.
We enumerate them by product 1, product 2, and so forth.
• There are m di↵erent kinds of materials necessary for producing the
products.
We enumerate them by material 1, material 2, and so forth.
• One unit of product j sells for price pj for i 2 [d].
• The current stock of material i is bi for i 2 [m].
• Producing one unit of project j requires aij amount of material i for every
pair (i, j) 2 [m] ⇥ [d] where [m] ⇥ [d] = {(i, j) : i 2 [m], j 2 [d]}.
• We assume that all data are nonnegative, i.e. pj 0, bi 0, and aij 0
for all i 2 [m] and j 2 [d].
• We assume that each product is divisible, which means that the amount of
each product can be any real number.
For example, we may produce 1.5 units of product 1.
Production planning (general version)

Goal: decide the production quantity of each project under the current stock
level of materials while maximizing the total revenue.

Here, the revenue refers to the gross income by selling the products.

Write a linear program to solve the problem.

Production planning (general version)

• Decisions: let xj denote the amount of product j that we produce for

j 2 [d].
• Objective: the total revenue is given by

Pjcts X
pj xj .
j2[d]

• Constraints: the total consumption of material i should be at most the

current stock. The total consumption is

hij
X
Dj
aij xj .
j2[d]

As the current stock of material i is given by bi , we can write the

hijdly
constraint as X
aij xj  b<
i.
b ; Vielny
j2[d]

Moreover, we know that the production quantity of each product cannot

be less than 0. Hence,
x 0. xj
10
Ojccd)
Production planning (general version)

In summary, the production planning problem can be formulated as the

Pilg
following LP.

max
sit
ai;, bi Fiel
dj zo Fjeld) .
Production planning - profit maximization

Next let us consider a di↵erent objective.

• Suppose that one unit of material i incurs a cost of si for i 2 [m].

The profit is defined as the net income after deducting costs from earnings.

The company may attempt to maximize the profit instead of the revenue. How
can we model the new objective?
Production planning - profit maximization

Note that producing one unit of product j incurs a cost of

si
X

hij
si aij .
i2[m]

Hence, the profit from selling one unit of product j is

X
pj si aij .
Pj
-

Silig
i2[m]

Then the new objective is to maximize

I Pj
0 1
X X
9ij) diy
@p j si aij ASi
xj .
j2[d] i2[m]
-
Production planning - profit maximization

Then the linear program for profit maximization is given by

max P; -Si hij) By

d
x <bi Vitcng
S
9i;
T
. .
;

xj 20 Fjela