ISyE6669 Homework 5 Solution

Fall 2021

1. Compute the gradient rf (x) and Hessian r2 f (x) of the Rosenbrock function

f (x1 , x2 ) = 100(x2 − x21 )2 + (1 − x1 )2 .

Solution:

200 (x2 − x21 ) (−2x1 ) + 2 (1 − x1 ) (−1)

 
rf (x) =
200(x2 − x21 )
400 (x31 − x1 x2 ) + 2 (x1 − 1)
 
= (1)
200(x2 − x21 )
1200x21 − 400x2 + 2, −400 x1
 
2
r f (x) = (2)
−400 x1 , 200

2. Implement the Newton’s Method with line search given in Algorithm 1. Use the Newton’s

Algorithm 1 Newton’s Method with Line Search

Start with x0 . Set k = 0,  = 10−4 .
Set d0 ← −(r2 f (x0 ))−1 rf (x0 )
while krf (xk )k >  do
Choose ᾱ > 0, ρ ∈ (0, 1), c ∈ (0, 1); Set αk ← ᾱ
while f (xk + αk dk ) > f (xk ) + cαk rf (xk )> dk do
αk ← ραk
end while
xk+1 ← xk + αk dk
dk+1 ← −(r2 f (xk+1 ))−1 rf (xk+1 )
k ←k+1
end while

Method to minimize the Rosenbrock function in Problem 1. Set the initial stepsize ᾱ = 1.
Select your own choice of ρ ∈ (0, 1), c ∈ (0, 1). First run the algorithm from the initial point
x0 = (1.2, 1.2)> , and then try the more difficult starting point x0 = (−1.2, 1)> . For each
starting point, print out the step length αk used by the algorithm as well as the point xk for
every step k. You should observe that Newton’s Method converges very fast.

1
Note that answers may vary depending on what ρ and c values were used. We use ρ = 0.9
and c = 10−5 . We do not break the inner backtracking while loop in our code, but you can if
α falls below 0.2 or 0.3. Such answers will be accepted. Using initial point x0 = (1.2, 1.2)> ,
we get the following output from our code:

iter =

alpha =

x =

1.195918367346939
1.430204081632654

iter =

alpha =

0.656100000000000

x =

1.067803199227371
1.123784446698436

iter =

alpha =

2
 0.900000000000000

x =

1.053558314632527
1.108140283036032

iter =

alpha =

0.900000000000000

x =

1.018347492953793
1.035607330512453

iter =

alpha =

0.900000000000000

x =

1.005495680368007
1.010713965562699

iter =

3
alpha =

0.900000000000000

x =

1.000836216112118
1.001620661106868

iter =

alpha =

0.900000000000000

x =

1.000091437375132
1.000177081378216

iter =

alpha =

0.900000000000000

x =

1.000009239115984
1.000017891387492

iter =

4
 9

alpha =

0.900000000000000

x =

1.000000924887601
1.000001791013948

Note that Newton’s method converges very fast. Our code converges in 9 iterations to optimal
solution x∗ = (1, 1)> .
Using initial point x0 = (−1.2, 1)> , we get the following output from our code:

iter =

alpha =

x =

-1.175280898876405
1.380674157303371

iter =

alpha =

0.166771816996666

5
x =

-0.852011114246042
0.620910454427278

iter =

alpha =

0.900000000000000

x =

-0.776255621024348
0.586332645981484

iter =

alpha =

0.729000000000000

x =

-0.471434147807713
0.124932946380622

iter =

alpha =

6
 0.900000000000000

x =

-0.406719192152260
0.151500754845643

iter =

alpha =

0.729000000000000

x =

-0.135706497703422
-0.058803878313132

iter =

alpha =

0.900000000000000

x =

-0.073548007735245
-0.006176381616507

iter =

7
alpha =

0.810000000000000

x =

0.188597803134905
-0.035352576110539

iter =

alpha =

0.900000000000000

x =

0.236690898089599
0.046617464710355

iter =

10

alpha =

0.810000000000000

x =

0.451295330945479
0.155825440991456

iter =

8
 11

alpha =

0.900000000000000

x =

0.498022733151194
0.241058989144317

iter =

12

alpha =

0.900000000000000

x =

0.686772990971616
0.435333715941717

iter =

13

alpha =

0.900000000000000

x =

0.720882495066941
0.514875770905670

9
iter =

14

alpha =

0.900000000000000

x =

0.849103637916556
0.704056746370642

iter =

15

alpha =

0.900000000000000

x =

0.880081108112290
0.771891129041179

iter =

16

alpha =

0.900000000000000

x =

10
 0.950606625890149
0.898413945746921

iter =

17

alpha =

0.900000000000000

x =

0.972314794184649
0.944400913275701

iter =

18

alpha =

0.900000000000000

x =

0.993095511134515
0.985707341467110

iter =

19

alpha =

0.900000000000000

11
x =

0.998712618051308
0.997342206287941

iter =

20

alpha =

0.900000000000000

x =

0.999851964205533
0.999694183499308

iter =

21

alpha =

0.900000000000000

x =

0.999984936676745
0.999968879216080

iter =

22

12
 alpha =

0.900000000000000

x =

0.999998490972099
0.999996882326324

Using this more difficult initial point, we converge in 22 iterations to optimal solution x∗ =
(1, 1)> .
Here is the Matlab code (you can use Python alternatively) that generates the above output.
1 clc
2 clear a l l
3 close a l l
4 x0 = [ − 1 . 2 ; 1 ] ;
5 e p s i l o n =10ˆ−4;
6 a l p h a b a r =1;
7 c =1∗10ˆ −5;
8 rho = 0 . 9 ;
9
10 f=@( x1 , x2 ) 1 0 0∗ ( x2−x1 ˆ 2 ) ˆ2+(1−x1 ) ˆ 2 ;
11 g r a d f=@( x1 , x2 ) [ 4 0 0 ∗ ( x1ˆ3−x1 ∗ x2 ) +2∗(x1 −1) ; 2 0 0 ∗ ( x2−x1 ˆ 2 ) ] ;
12 h e s s f=@( x1 , x2 ) [ 1 2 0 0 ∗ x1 ˆ2−400∗ x2+2 −400∗x1 ; −400∗x1 2 0 0 ] ;
13
14 d=−inv ( h e s s f ( x0 ( 1 ) , x0 ( 2 ) ) ) ∗ g r a d f ( x0 ( 1 ) , x0 ( 2 ) ) ;
15
16 x=x0 ;
17 z=x+a l p h a b a r ∗d ;
18 i t e r =0;
19 while norm ( g r a d f ( x ( 1 ) , x ( 2 ) ) )>e p s i l o n
20 a l p h a=a l p h a b a r ;
21 while f ( z ( 1 ) , z ( 2 ) )>f ( x ( 1 ) , x ( 2 ) ) +(c ∗ a l p h a ) ∗ t r a n s p o s e ( g r a d f ( x ( 1 ) ,
x ( 2 ) ) ) ∗d
22 a l p h a=rho ∗ a l p h a ;
23 z=x+a l p h a ∗d ;
24 end
25 i t e r=i t e r +1
26 alpha
27 x=x+a l p h a ∗d
28
29
30 d=−inv ( h e s s f ( x ( 1 ) , x ( 2 ) ) ) ∗ g r a d f ( x ( 1 ) , x ( 2 ) ) ;

13
31 end

Observe that Newton’s method converges in less iterations for initial point x0 = (1.2, 1.2)>
than initial point x0 = (−1.2, 1)> because x0 = (1.2, 1.2)> is closer in distance to the optimal
solution x∗ = (1, 1)> . Note we also take c to be very small (i.e. 10−5 ) and ρ to be large (i.e.
0.9) so the condition in the backtracking line search is satisfied faster and so we do not reduce
the stepsize as much in this loop. This way, since the stepsize is not too small, we make
significant progress towards the optimal solution at each iteration and thus converge in a very
small number of iterations.

3. Figure
7. The1figure
belowbelow illustrates thenetwork
illustrates the water water networkTheoflinesNewvillage.
of Newvillage. are water piplelinesThe lines are water piplelines
numbered
numbered from
from 1 through 1 through
13. The arrows on the13. The
lines are arrows
possible on ofthe
direction(s) flow lines
of water are possible
in these pipelines. direction(s) of flow of
The circles are water sources numbered A, B, C. The rectangles are houses D, E, F, G, H.
water in these pipelines. The circles are water sources numbered A, B, C. The rectangles are
The maximum possible capacity of the water sources are (the sources can operate at less than the
houses D, E,
maximum F, G, H. The maximum possible capacity of the water sources are (the sources
capacity):
can A:operate
100 Units, at lessUnits,
B: 100 than
C: 80the
Units maximum capacity): A: 100 Units, B: 100 Units, C: 80 Units

Demands
Demandsof water
of water inhouses
in the the houses
are: are: D: 50 Units, E: 60 Units, F: 40 Units, G: 30 Units, H: 70
D: 50 Units, E: 60 Units, F: 40 Units, G: 30 Units, H: 70 Units
Units Since the houses are at different elevation and the pipes are of different diameter, the
Since
costwater the houses are at different
of transporting waterelevation and the pipes
is different are ofdifferent
in the different diameter,
pipes.the cost
Theseof transporting
costs per unit of water are:
is different in the different pipes. These costs per unit of water are:
PipePipe
1: 1:$2, Pipe
$2, Pipe 2:Pipe
2: $3, $3,3: Pipe
$4, Pipe3: $4,Pipe
4: $2, Pipe
5: $3,4:
Pipe$2, Pipe
6: $2, 5:$4,$3,
Pipe 7: PipePipe
8: $1, 6: $2, Pipe 7: $4, Pipe 8: $1,
Pipe 9: $2, Pipe 10: $4, Pipe 11: $5, Pipe 12: $1, Pipe 13: $2.
Pipe 9: $2, Pipe 10: $4, Pipe 11: $5, Pipe 12: $1, Pipe 13: $2. Formulate an LP to minimize
the Formulate
total cost
house.
an LPofto transporting
minimize the total cost of transporting
water so as water so as tothe
to meet meetwater
the waterdemands
demands of each
of each house.

A B
1 2
D
6
3 5
4 G
7
E 8
10 F 11 12
9

C
H 13

Figure 1: Water Network for Question 2

Solution:
We introduce the following decision variables. xA , xB , xC , respectively, is the amount of
water pumped through water source A, B, C, and fij is the flow of water from location i to
location j, for arcs (i, j) in the set of arc A := {(A, D), (A, E), (B, D), (B, F ), (B, G), (C, F ),
(C, G), (C, H), (D, F ), (E, F ), (E, H), (F, D), (F, G), (H, E), (H, F )}.

14
min 2fAD + 4fAE + 3fBD + 3fBF + 2fBG + 5fCF + fCG + 2fCH + 2fDF + 4fEF + 2fEH + 2fF D
+ fF G + 2fHE + 4fHF (3)
s.t. xA − fAD − fAE = 0 (4)
xB − fBD − fBF − fBG = 0 (5)
xC − fCF − fCG − fCH = 0 (6)
fAD + fBD − fDF + fF D = 50 (7)
fAE + fHE − fEH − fEF = 60 (8)
fBF + fCF + fDF + fEF + fHF − fF D − fF G = 40 (9)
fBG + fF G + fCG = 30 (10)
fCH + fEH − fHE − fHF = 70 (11)
xA ≤ 100, (12)
xB ≤ 100, (13)
xC ≤ 80, (14)
xi ≥ 0, ∀i ∈ {A, B, C}, (15)
fij ≥ 0, ∀(i, j) ∈ A. (16)

Note: Edges DF and EH are bidirectional, meaning that there is flow in both directions.
Thus we have two variables for each edge: fDF and fF D for DF, and fEH and fEH for HE.
Some of the students assumed that arrows pointing to both directions, that may be flow in
one direction or the other, but not both at the same time. We will accept that formulation
too. Following is the solution in that case.
We introduce the following decision variables. xA , xB , xC , respectively, is the amount of
water pumped through water source A, B, C, and fij is the flow of water from location i to
location j, for arcs (i, j) in the set of arc A := {(A, D), (A, E), (B, D), (B, F ), (B, G), (C, F ),
(C, G), (C, H), (D, F ), (E, F ), (E, H), (F, G), (H, F )}. And let D = {(D, F ), (E, H)} ⊂ A
denotes set of edges for which flow can go in either direction.

15
min 2fAD + 4fAE + 3fBD + 3fBF + 2fBG + 5fCF + fCG + 2fCH + 2|fDF | + 4fEF + 2|fEH |
+ fF G + 4fHF
s.t. xA − fAD − fAE = 0
xB − fBD − fBF − fBG = 0
xC − fCF − fCG − fCH = 0
fAD + fBD − fDF = 50
fAE + fHE − fEH − fEF = 60
fBF + fCF + fDF + fEF + fHF − fF G = 40
fBG + fF G + fCG = 30
fCH + fEH − fHF = 70
xA ≤ 100,
xB ≤ 100,
xC ≤ 80,
xi ≥ 0, ∀i ∈ {A, B, C},
fij ≥ 0, ∀(i, j) ∈ A \ D.

This is not a linear problem due to two absolute values in the objective function. Hence it
will need to be converted into:

min 2fAD + 4fAE + 3fBD + 3fBF + 2fBG + 5fCF + fCG + 2fCH + 2z + 4fEF + 2w
+ fF G + 4fHF
s.t. xA − fAD − fAE = 0
xB − fBD − fBF − fBG = 0
xC − fCF − fCG − fCH = 0
fAD + fBD − fDF = 50
fAE + fHE − fEH − fEF = 60
fBF + fCF + fDF + fEF + fHF − fF G = 40
fBG + fF G + fCG = 30
fCH + fEH − fHF = 70
xA ≤ 100,
xB ≤ 100,
xC ≤ 80,
xi ≥ 0, ∀i ∈ {A, B, C},
fij ≥ 0, ∀(i, j) ∈ A \ D
− z ≤ fDF ≤ z
− w ≤ fEH ≤ w.

16
4. Consider the following electric power network shown in Figure 2. This network is taken from
a real-world electric power system. Electricity generators are located at nodes 1, 3, and 5 and
producing p1 , p2 , p3 amounts of electricity, respectively. Electricity loads are located at nodes
2, 4, and 6 and are consuming d1 , d2 , d3 amounts of electricity, respectively.
The demand is fixed and given as d1 = 110, d2 = 65, d3 = 95.
Each generator i’s production must be within an upper and a lower bound as pmin
i ≤ pi ≤ pmax
i .
The bounds are given as p1 = 20, p1 = 200, p2 = 20, p2 = 150, p3 = 10, pmax
min max min max min
3 =
150.
max = 100, f max = 110, f max = 50, f max = 80, f max =
The flow limits over lines are given as f12 23 34 45 56
max
60, f61 = 40.
The line parameters are given as B12 = 11.6, B23 = 5.9, B34 = 13.7, B45 = 9.8, B56 =
5.6, B61 = 10.5. The unit generation costs are given as c1 = 16, c2 = 20, c3 = 8.

(a) Formulate the power system scheduling problem using the model discussed in Lecture 2.
(b) Implement and solve the model using CVXPY. Write down the optimal solution.
(c) Find the electricity prices for demand at nodes 2, 4, 6. To do this, use the command
constraints[0].dual value to find the dual variable of constraints[0]. Hint: Recall the elec-
tricity price at node i is the dual variable for the flow conservation constraint at node
i.

Figure 2: Electric network for Question 3.

17
(a) The LP can be formulated as follows:
X
min ci pi
i=1..3
s.t.
Flow Conservation:
f12 − f61 = p1 ,
f34 − f23 = p2 ,
f56 − f45 = p3 ,
f23 − f12 = −d1 ,
f45 − f34 = −d2 ,
f61 − f56 = −d3 ,
Constraints linking branch flow and nodal potential:
f12 = B12 (θ1 − θ2 ),
f23 = B23 (θ2 − θ3 ),
f34 = B34 (θ3 − θ4 ),
f45 = B45 (θ4 − θ5 ),
f56 = B56 (θ5 − θ6 ),
f61 = B61 (θ6 − θ1 ),
Flow limit constraints:
max max
− f12 ≤ f12 ≤ f12 ,
max max
− f23 ≤ f23 ≤ f23 ,
max max
− f34 ≤ f34 ≤ f34 ,
max max
− f45 ≤ f45 ≤ f45 ,
max max
− f56 ≤ f56 ≤ f56 ,
max max
− f61 ≤ f61 ≤ f61 ,
Generation constraints:
pmin
1 ≤ p1 ≤ pmax
1 ,
pmin
2 ≤ p2 ≤ pmax
2 ,
pmin
3 ≤ p3 ≤ pmax
3 .
(b) The optimal objective value is . The quantum of electricity produced are p1 = 113.12, p2 =
20.0, p3 = 136.88.
Following is the output from python model
-The optimal value is 3304.96
-The quantum of electricity produced at node 1: p1 = 113.12 , at node 3: p2 = 20.0 , at
node 5: p3 = 136.88
-The flows in different lines are f12 =78.12 , f23 = -31.88 , f34 = -11.88, f45 = -76.88, f56 =
60.0 and f61 = -35.0
(c) The electricity price at node 2 is 14.4 , at node 4 is 9.9 and at node 6 is 17.77 .

18