ISyE/Math/CS/Stat 525

Linear Optimization

Fall 2021 Course Introduction

Prof. Alberto Del Pia

University of Wisconsin-Madison

Based on the book Introduction to Linear Optimization by D. Bertsimas and J.N. Tsitsiklis
Linear Optimization
Optimization
Optimization

This course is about taking decisions.

Optimization

This course is about taking OPTIMAL decisions.

Common sense vs Optimization

Common sense is often sufficient in everyday life.

Common sense vs Optimization

Common sense is not sufficient to make decisions in complex systems.

Common sense vs Optimization
A simple example

▶ Find the best assignment of 70 people to 70 jobs.

bc bc bc bc
bc bc bc bc
bc bc bc bc
bc bc bc bc
.. .. .. ..
. . . .

bc bc bc bc
Can growing computing power help?
Can growing computing power help?

▶ Simple: check each possible assignment and select the best.

Can growing computing power help?

▶ Simple: check each possible assignment and select the best.

▶ But how long will it take? Let’s use an IBM BlueGene/L:
Can growing computing power help?

▶ Simple: check each possible assignment and select the best.

▶ But how long will it take? Let’s use an IBM BlueGene/L:
#people Solutions to check Time
3 6 0 sec
10 3.6 × 106 0 sec
15 1.3 × 1012 1 sec
30
70
Can growing computing power help?

▶ Simple: check each possible assignment and select the best.

▶ But how long will it take? Let’s use an IBM BlueGene/L:
#people Solutions to check Time
3 6 0 sec
10 3.6 × 106 0 sec
15 1.3 × 1012 1 sec
30 2.7 × 1032
70
Can growing computing power help?

▶ Simple: check each possible assignment and select the best.

▶ But how long will it take? Let’s use an IBM BlueGene/L:
#people Solutions to check Time
3 6 0 sec
10 3.6 × 106 0 sec
15 1.3 × 1012 1 sec
30 2.7 × 1032 14 billion years(a)
70
(a) That’s four times the age of the universe!
Can growing computing power help?

▶ Simple: check each possible assignment and select the best.

▶ But how long will it take? Let’s use an IBM BlueGene/L:
#people Solutions to check Time
3 6 0 sec
10 3.6 × 106 0 sec
15 1.3 × 1012 1 sec
30 2.7 × 1032 14 billion years(a)
70 1.2 × 10100 (b)
(a)That’s four times the age of the universe!
(b)That’s more than the number of particles in the observable
universe!
Modelling Approach

PROBLEM ANALYSIS

MODELLING

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Modelling Approach

PROBLEM ANALYSIS

real-world relationships ⇔ math relationships

MODELLING

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Modelling Approach

PROBLEM ANALYSIS

real-world relationships ⇔ math relationships

MODELLING min f(x)
s.t. x ∈ S

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Modelling Approach

PROBLEM ANALYSIS

real-world relationships ⇔ math relationships

MODELLING min f(x) objective function
s.t. x ∈ S feasible set

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Modelling Approach

PROBLEM ANALYSIS

real-world relationships ⇔ math relationships

MODELLING min f(x) objective function
s.t. x ∈ S feasible set

study the mathematical properties

MODEL ANALYSIS
of the model

SOLUTION METHOD

VALIDATION
Modelling Approach

PROBLEM ANALYSIS

real-world relationships ⇔ math relationships

MODELLING min f(x) objective function
s.t. x ∈ S feasible set

study the mathematical properties

MODEL ANALYSIS
of the model

apply an algorithm to compute an

SOLUTION METHOD
optimal solution x∗

VALIDATION
Modelling Approach

PROBLEM ANALYSIS

real-world relationships ⇔ math relationships

MODELLING min f(x) objective function
s.t. x ∈ S feasible set

study the mathematical properties

MODEL ANALYSIS
of the model

apply an algorithm to compute an

SOLUTION METHOD
optimal solution x∗

VALIDATION
verification and simulation
Optimization and Programming

For us Optimization ≡ Programming.

▶ “Programming” was used in the 1930s, before computers.

▶ It means “Planning”, based on the military definition of “Program”.
▶ Lately “Programming” is getting replaced by “Optimization”.
Linear Optimization
Linear
Linear functions
Linear functions

A function f : Rn → R is linear if:

1. f(x + y) = f(x) + f(y) ∀x, y ∈ Rn
2. f(λx) = λf(x) ∀x ∈ Rn , ∀λ ∈ R

Equivalently, f is linear if it can be written as:

∑
n
c′ x = ci xi = c1 x1 + c2 x2 + · · · + cn xn ,
i=1

where c1 , c2 , . . . , cn are real numbers.

Linear Optimization

PROBLEM ANALYSIS

MODELLING

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Linear Optimization

real-world relationships ⇔ math relationships

PROBLEM ANALYSIS

MODELLING

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Linear Optimization

real-world relationships ⇔ math relationships

PROBLEM ANALYSIS

min f(x)
s.t. x ∈ S
MODELLING

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Linear Optimization

real-world relationships ⇔ math relationships

PROBLEM ANALYSIS

min f(x) LINEAR objective function

s.t. x ∈ S LINEAR inequalities
MODELLING

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Linear Optimization

real-world relationships ⇔ math relationships

PROBLEM ANALYSIS

min c′ x LINEAR objective function

′
s.t. ai x ≥ bi i ∈ M1
MODELLING
a′i x ≤ bi i ∈ M2 LINEAR inequalities
a′i x = bi i ∈ M3

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
A Linear Programming (LP) problem

minimize 2x1 − x2 + 4x3

subject to x1 + x2 + x4 ≤ 2
3x2 − x3 = 5
x3 + x4 ≥ 3
x1 ≥ 0
x3 ≤ 0.

▶ Strict inequalities like x3 + x4 > 3 are not allowed!

Algorithms for Linear Programming
Algorithms for LP

real-world relationships ⇔ math relationships

PROBLEM ANALYSIS

min c′ x LINEAR objective function

′
s.t. ai x ≥ bi i ∈ M1
MODELLING
a′i x ≤ bi i ∈ M2 LINEAR inequalities
a′i x = bi i ∈ M3

MODEL ANALYSIS

SOLUTION METHOD

VALIDATION
Algorithms for LP

real-world relationships ⇔ math relationships

PROBLEM ANALYSIS

min c′ x LINEAR objective function

′
s.t. ai x ≥ bi i ∈ M1
MODELLING
a′i x ≤ bi i ∈ M2 LINEAR inequalities
a′i x = bi i ∈ M3

MODEL ANALYSIS
▶ optimality conditions
▶ duality

SOLUTION METHOD

VALIDATION
Algorithms for LP

real-world relationships ⇔ math relationships

PROBLEM ANALYSIS

min c′ x LINEAR objective function

′
s.t. ai x ≥ bi i ∈ M1
MODELLING
a′i x ≤ bi i ∈ M2 LINEAR inequalities
a′i x = bi i ∈ M3

MODEL ANALYSIS
▶ optimality conditions
▶ duality

SOLUTION METHOD
SIMPLEX ALGORITHM

VALIDATION
Can growing computing power help?
bc bc bc bc
bc bc bc bc
bc bc bc bc
bc bc bc bc
.. .. .. ..
. . . .

bc bc bc bc

#people Solutions to check Time

3 6 0
10 3.6 × 106 0
15 1.3 × 1012 1 sec
30 2.7 × 1032 14 billion years(a)
70 1.2 × 10100 (b)
(a)That’s four times the age of the universe!
(b)That’s more than the number of particles in the observable
universe!
Can growing computing power help?

▶ It takes only a moment to find the optimum solution by

posing the problem as a LP problem and applying the simplex
algorithm.
▶ The theory behind LP drastically reduces the number of
possible solutions that must be checked.
▶ LP is now a technology: It can be solved efficiently in theory
and in practice.
LP is everywhere!

▶ Many practical problems in Operations Research can be

expressed as LP problems.
▶ Application areas include engineering, computer science,
physics, biology, finance, and economics.
▶ Some examples: Network problems, scheduling,
microeconomics, company management, planning, production,
transportation, technology, military operations.
▶ A number of algorithms for other types of optimization
problems work by solving LP problems as sub-problems.
Course details
Purpose of this course

▶ Solid understanding of Linear Optimization.

▶ Special attention to theory.
▶ We will study the available solution methods:
▶ HOW they work.
▶ WHY they work.
▶ Our focus is on the interplay between algebra and geometry.
▶ Ideas require a geometric view.
▶ Problems are solved by algorithms, which can only be
described algebraically.
What this course is NOT about:
▶ We won’t spend time on software or on implementation.
▶ We won’t spend much time on how to model problems as LP
problems. (See ISyE 323: Operations Research - Deterministic
Modeling.)
Recommended textbook

Introduction to Linear Optimization,

by Dimitris Bertsimas and John N.
Tsitsiklis (1997).
▶ Developed for a course at M.I.T.
▶ Rent ∼ ?
▶ Buy used ∼ $58.
▶ Buy new ∼ $85.
What we will cover

Ch. 1: Introduction §1.1–1.4

▶ The LP problem and examples.
Ch. 2: The geometry of linear programming §2.1–2.8
▶ Polyhedra and their geometric and algebraic properties.
Ch. 3: The simplex method §3.1–3.7 and 1.6
▶ The main method used to solve LP problems.
Ch. 4: Duality theory §4.1–4.6
▶ A new LP problem and its relation with the original one.
Ch. 7: Network flow problems §7.1–7.3
▶ Solving network flow problems with the simplex algorithm.
Ch. 8: Complexity of linear programming and the ellipsoid method
§8.1–8.5
▶ The first polynomial time algorithm for LP problems.
Warning on course difficulty

This is a challenging course ⇒ Much of the material in this course

is deep.
▶ Requires serious math.
▶ Proofs will be done by me and by you.
Warning on course difficulty

▶ “Hard but rewarding”

Prerequisites
Essential background that will be assumed:
▶ Working knowledge of linear algebra.
See Section 1.5 Linear algebra background and notation of
the textbook (6 pages).
▶ E.g., set theoretic notation, vectors and matrices, matrix
inversion, subspaces and bases, affine subspaces, linear
independence, and the rank of a matrix.
▶ Basic proof techniques.
See Introduction to mathematical arguments by Michael
Hutchings in Canvas.
▶ Mathematical notation.
See Math symbols cheat sheet in Canvas.
Prerequisites
Essential background that will be assumed:
▶ Working knowledge of linear algebra.
See Section 1.5 Linear algebra background and notation of
the textbook (6 pages).
▶ E.g., set theoretic notation, vectors and matrices, matrix
inversion, subspaces and bases, affine subspaces, linear
independence, and the rank of a matrix.
▶ Basic proof techniques.
See Introduction to mathematical arguments by Michael
Hutchings in Canvas.
▶ Mathematical notation.
See Math symbols cheat sheet in Canvas.

Now in Canvas:
▶ Assignment 0, to test your Math skills and Linear Algebra.
▶ You do not need to submit it since it will not be graded.
▶ Solutions are already in Canvas.
Other Optimization courses

▶ ISyE/Math/CS 425: Introduction to Combinatorial

Optimization.
▶ CS/ECE/ISyE 524: Introduction to Optimization.
▶ CS/ISyE 526: Advanced Linear Programming.
▶ ISyE/Math/CS/Stat 726: Nonlinear Optimization I.
▶ ISyE/Math/CS 728: Integer Optimization.

If interested in research in Optimization, consider applying to our

Graduate programs.
Technical details
Video lectures

▶ Backup option for students that cannot attend live lectures.

▶ Available in YouTube in the following playlist.
▶ Videos will be topic based, therefore their length will vary
widely.
▶ Videos will be more concise than the live lectures.
▶ Useful to subscribe and turn on notifications.
Office hours

My office hours
▶ For discussions on the material presented in the lectures and
related theoretical questions.
▶ Tuesday and Thursday 10:45–11:45am.
▶ Office hours take place in WID 1157/1158 and in Zoom.
▶ Meeting ID: 934 3380 4819. Passcode: 863306.
▶ You can just use this link.
Office hours

TA office hours
▶ The TA is Dekun Zhou <[email protected]>.
▶ For questions regarding exercises and assignments.
▶ Monday 11am–12pm and Friday 4–5pm.
▶ Office hours take place in ME 3146 and in Zoom.
▶ Same Meeting ID, passcode, and link.
Homework

▶ Homework assignments will be given roughly every two weeks.

Assignment # Topic Available on Graded Due on

0 Course intro Aug. 27 no N/A
1 Ch. 1 Sep. 16 yes Oct. 4
2 Ch. 2 Oct. 5 yes Oct. 18
3 Ch. 3 part 1 Oct. 19 yes Nov. 10
4 Ch. 3 part 2 Nov. 11 yes Nov. 24
5 Ch. 4 Nov. 25 yes Dec. 8
6 Ch. 7 Dec. 9 no N/A
Homework

▶ Homework assignments are posted in Canvas.

▶ 5 compulsory exercises per assignment.
▶ Some exercises in the assignments are specific to
undergraduate and graduate students.
▶ Optional exercises for keen students.

▶ Strongly encouraged to work in groups of 2 or 3 (same type).

▶ Submit pdf in Canvas (one submission per group).

▶ Assignments will not be completely graded. 3/5 randomly

selected.
▶ Complete solutions will be published in Canvas.
Grading

Components of grade:
▶ 40% Homework.
▶ 30% Midterm exam.
▶ 30% Final exam.

▶ The midterm exam will take place on October 28,

9:30am–10:45am (class time) in 311 and 332 Wendt
Commons.
▶ The final exam will take place in the course slot in final week,
which is December 22, 10:05am–12:05pm. Classroom
TBA.

∗ Notify me this week if you have a conflict with the exams times.
Canvas web page

https://canvas.wisc.edu/courses/258459

▶ Syllabus, lecture slides, introductory material, homework

assignments and solutions, grades.
▶ Use “Discussions” section to post questions about the course,
and to find a teammate for the assignments.
▶ I cannot give you access to the Canvas webpage. You are
given access when enrolled.

▶ See syllabus in Canvas for more info: /files/525syllabus

About me…
▶ BS and MS, University of Padova, Italy, 2005.
▶ PhD, University of Padova, Italy, 2009.
About me…
▶ BS and MS, University of Padova, Italy, 2005.
▶ PhD, University of Padova, Italy, 2009.

▶ 2009 – 2010: Otto-von-Guericke University, Magdeburg,

Germany.
▶ 2010 – 2013: ETH Zürich, Zürich, Switzerland.
▶ 2013 – 2014: IBM, Watson Research Center, New York.
About me…
▶ BS and MS, University of Padova, Italy, 2005.
▶ PhD, University of Padova, Italy, 2009.

▶ 2009 – 2010: Otto-von-Guericke University, Magdeburg,

Germany.
▶ 2010 – 2013: ETH Zürich, Zürich, Switzerland.
▶ 2013 – 2014: IBM, Watson Research Center, New York.
▶ Since 2014: University of Wisconsin-Madison.
▶ Wisconsin Institute for Discovery.
▶ Dept. Industrial and Systems Engineering.
▶ Dept. Mathematics.
▶ Dept. Computer Sciences.
About me…
▶ BS and MS, University of Padova, Italy, 2005.
▶ PhD, University of Padova, Italy, 2009.

▶ 2009 – 2010: Otto-von-Guericke University, Magdeburg,

Germany.
▶ 2010 – 2013: ETH Zürich, Zürich, Switzerland.
▶ 2013 – 2014: IBM, Watson Research Center, New York.
▶ Since 2014: University of Wisconsin-Madison.
▶ Wisconsin Institute for Discovery.
▶ Dept. Industrial and Systems Engineering.
▶ Dept. Mathematics.
▶ Dept. Computer Sciences.

▶ Research areas: Integer optimization, Combinatorial

optimization, Machine learning.
About you...

Please complete the Student Survey quiz in Canvas!

Questions about the course?
Questions about the course?

▶ If you are not yet enrolled in this course and want to, hang on
for student to drop this course!