AT&T

Linear Optimization
and Extensions:
Theory and Algorithms
Shu-Cherng Fang
North Carolina State University

Sarai Puthenpura
AT&T Bell Labs

Prentice Hall, Englewood Cliffs, New Jersey 07632

Contents

PREFACE

1 INTRODUCTION

1.1 History of Linear Programming 1

1.2 The Linear Programming Problem 2
1.2.1 Standard-Form Linear Program, 3
1.2.2 Embedded Assumptions, 3
1.2.3 Converting to Standard Form, 4

1.3 Examples of Linear Programming Problems 5

1.4 Mastering Linear Programming 9
References for Further Reading 10
Exercises 11
2 GEOMETRY OF LINEAR PROGRAMMING

2.1 Basic Terminologies of Linear Programming 14

2.2 Hyperplanes, Halfspaces, and Polyhedral Sets 15
2.3 Affine Sets, Convex Sets, and Cones 17
VIII Contents

2.4 Extreme Points and Basic Feasible Solutions 19

2.5 Nondegeneracy and Adjacency 21
2.6 Resolution Theorem for Convex Polyhedrons 23
2.7 Fundamental Theorem of Linear Programming 24
2.8 Concluding Remarks: Motivations of Different
Approaches 25
References for Further Reading 26

Exercises 26

3 THE REVISED SIMPLEX METHOD 29

3.1 Elements of an Iterative Scheme 29

3.2 Basics of the Simplex Method 30
3.3 Algebra of the Simplex Method 31
3.3.1 Stopping the Simplex Method—Checking
for Optimality, 33
3.3.2 Iterations ofthe Simplex Method—Moving
for Improvement, 33
3.4 Starting the Simplex Method 39
3.4.1 Two-Phase Method, 39
3.4.2 Big-M Method, 41
3.5 Degeneracy and Cycling 42
3.6 Preventing Cycling 44
3.6.1 Lexicographic Rule, 44
3.6.2 Bland 's Rule, 44
3.7 The Revised Simplex Method 45
3.8 Concluding Remarks 50
References for Further Reading 50
Exercises 51
4 DUALITY THEORY AND SENSITIVITY ANALYSIS 55

4.1 Dual Linear Program 56

4.2 Duality Theory 57
Contents

4.3 Complementary Slackness and Optimality Conditions 61

4.4 An Economic Interpretation of the Dual Problem 63
4.4.1 Dual Variables and Shadow Prices, 63
4.4.2 Interpretation of the Dual Problem, 64
4.5 The Dual Simplex Method 65
4.5.1 Basic Idea ofthe Dual Simplex Method, 65
4.5.2 Sherman-Morrison-Woodbury Formula, 66
4.5.3 Computer Implementation of the Dual Simplex
Method, 70
4.5.4 Find an Initial Dual Basic Feasible Solution, 72
4.6 The Primal Dual Method 73
4.6.1 Step-by-Step Procedure for the Primal-Dual
Simplex Method, 75
A.l Sensitivity Analysis 78
4.7.1 Change in the Cost Vector, 78
4.7.2 Change in the Right-Hand-Side Vector, 80
4.7.3 Change in the Constraint Matrix, 82
4.8 Concluding Remarks 86
References for Further Reading 87
Exercises 87
5 COMPLEXITY ANALYSIS AND THE ELLIPSOID METHOD

5.1 Concepts of Computational Complexity 93

5.2 Complexity of the Simplex Method 94
5.3 Basic Ideas of the Ellipsoid Method 96
5.4 Ellipsoid Method for Linear Programming 100
5.5 Performance of the Ellipsoid Method for LP 103
5.6 Modifications of the Basic Algorithm 104
5.6.1 Deep Cuts, 104
5.6.2 Surrogate Cuts, 106
5.6.3 Parallel Cuts, 106
5.6.4 Replacing Ellipsoid by Simplex, 107
5.7 Concluding Remarks 108
References for Further Reading 108
Exercises 109
X Contents

6 KARMARKAR'S PROJECTIVE SCALING ALGORITHM 112

6.1 Basic Ideas of Karmarkar's Algorithm 112

6.2 Karmarkar's Standard Form 114
6.2.1 The Simplex Structure, 115
6.2.2 Projective Transformation on the Simplex, 116
6.3 Karmarkar's Projective Scaling Algorithm 117
6.4 Polynomial-Time Solvability 120
6.5 Converting to Karmarkar's Standard Form 126
6.6 Handling Problems with Unknown Optimal Objective
Values 128
6.7 Unconstrained Convex Dual Approach 135
6.7.1 e-Optimal Solution, 136
6.7.2 Extension, 139
6.8 Concluding Remarks 141
References for Further Reading 141
Exercises 142

7 AFFINE SCALING ALGORITHMS 144

7.1 Primal Affine Scaling Algorithm 145

7.1.1 Basic Ideas of Primal Affine Scaling, 145
7.1.2 Implementing the Primal Affine Scaling
Algorithm, 155
7.1.3 Computational Complexity, 160
7.2 Dual Affine Scaling Algorithm 165
7.2.7 Basic Ideas of Dual Affine Scaling, 165
7.2.3 Dual Affine Scaling Algorithm, 167
7.2.3 Implementing the Dual Affine Scaling
Algorithm, 169
7.2.4 Improving Computational Complexity, 172
7.3 The Primal-Dual Algorithm 177
7.3.1 Basic Ideas of the Primal-Dual Algorithm, 178
7.3.2 Direction and Step-Length of Movement, 180
7.3.3 Primal-Dual Algorithm, 184
7.3.4 Polynomial-Time Termination, 184
7.3.5 Starting the Primal-Dual Algorithm, 188
7.3.6 Practical Implementation, 189
Contents xi

7.3.7 Accelerating via Power-Series Method, 193

7.4 Concluding Remarks 194
References for Further Reading 195
Exercises 197

8 INSIGHTS INTO THE INTERIOR-POINT METHODS 201

8.1 Moving Along Different Algebraic Paths 201

8.1.1 Primat Affine Scaling with Logarithmic Barrier
Function, 203
8.1.2 Dual Affine Scaling with Logarithmic Barrier
Function, 204
8.1.3 The Primal-Dual Algorithm, 205
8.2 Missing Information 207
8.2.1 Dual Information in the Primal Approach, 207
8.2.2 Primal Information in the Dual Approach, 207
8.3 Extensions of Algebraic Paths 208
8.4 Geometrie Interpretation of the Moving Directions 209
8.4.1 Primal Affine Scaling with Logarithmic Barrier
Function, 211
8.4.2 Dual Affine Scaling with Logarithmic Barrier
Function, 212
8.4.3 The Primal-Dual Algorithm, 213
8.5 General Theory 217
8.5.1 General Primal Affine Scaling, 217
8.5.2 General Dual Affine Scaling, 219
8.6 Concluding Remarks 220
References for Further Reading 221
Exercises 221

9 AFFINE SCALING FOR CONVEX QUADRATIC PROGRAMMING 224

9.1 Convex Quadratic Program with Linear Constraints 225

9.1.1 Primal Quadratic Program, 225
9.1.2 Dual Quadratic Program, 225
9.2 Affine Scaling for Quadratic Programs 227
9.2.1 Primal Affine Scaling for Quadratic
Programming, 227
xii Contents

9.2.2 Improving Primal Affine Scalingfor Quadratic

Programming, 237
9.3 Primal-Dual Algorithm for Quadratic Programming 241
9.3.1 Basic Concepts, 241
9.3.2 A Step-By-Step Implementation Procedure, 243
9.3.3 Convergence Properties ofthe Primal-Dual
Algorithm, 245
9.4 Convex Programming with Linear Constraints 246
9.4.1 Basic Concepts, 246
9.4.2 A Step-by-Step Implementation Procedure, 248

9.5 Concluding Remarks 249

References for Further Reading 249

Exercises 250

10 IMPLEMENTATION OF INTERIOR-POINT ALGORITHMS 253

10.1 The Computational Bottleneck 253

10.2 The Cholesky Factorization Method 254

10.2.1 Computing Cholesky Factor, 255
10.2.2 Block Cholesky Factorization, 257
10.2.3 Sparse Cholesky Factorization, 259
10.2.4 Symbolic Cholesky Factorization, 263
10.2.5 Solving Triangulär Systems, 263

10.3 The Conjugate Gradient Method 265

10.4 The LQ Factorization Method 268

10.5 Concluding Remarks 275

References for Further Reading 276

Exercises 277

BIBLIOGRAPHY 280

INDEX 295