Operations Research Models and Methods

Paul A. Jensen and Jonathan F. Bard

Nonlinear Programming Methods.S1

Separable Programming
Separable programming is important because it allows a convex nonlinear program to be
approximated with arbitrary accuracy with a linear programming model. The idea is to
replace each nonlinear function with a piecewise linear approximation. Global solutions
can then be obtained with any number efficient LP codes. For nonconvex problems, the
approach is still valid but more work needs to be done. Either a mixed-integer linear
programming problem must be solved as discussed in Section 8.8, or a modified version
of the simplex algorithm with a limited basis entry rule can be applied to the model
directly. The candidates for the entering variable must be restricted to maintain the
validity of the LP approximation. In this case a local optimum is obtained but it is
possible to find the global optimum with the help of branch and bound.

Problem Statement

Consider again the general nonlinear programming problem

Minimize{f(x) : gi(x) ≤ bi , i = 1, . . . ,m}

with two additional provisions: (i) the objective function and all
constraints are separable, and (ii) each decision variable xj is bounded
below by 0 and above by a known constant uj, j = 1, . . . ,n. Recall that a
function, f(x), is separable if it can be expressed as the sum of functions of
the individual decision variables.
n
f(x) = ∑ fj(xj)
j=1

The separable nonlinear programming problem has the following

structure.
n
Minimize ∑ fj(xj)
j=1
n
subject to ∑ gij(xj) ≤ bi, i = 1, . . . ,m
j=1
0 ≤ xj ≤ uj, j =1, . . . ,n

The key advantage of this formulation is that the nonlinearities are

mathematically independent. This property in conjunction with the finite
 Separable Programming 2

bounds on the decision variables permits the development of a piecewise

linear approximation for each function in the problem.

Linearization

Consider the general nonlinear function f(x) depicted in Fig. 5. To form a

piecewise linear approximation using, say, r line segments, we must first
select r+1 values of the scalar x within its range 0 ≤ x ≤ u (call them
x- , x- , . . . ,x- ) and let f = f(x- ) for k = 0, 1 , . . . ,r. At the boundaries we have
0 1 r k k
x- 0 = 0 and x- r = u. Notice that the values of x- k do not have to be evenly
spaced.

f(x)

Piecewise linear approximation

x-0 = 0 x-1 x-2 x-3 x- 4 x-5 = u x

Figure 5. Piecewise linear approximation of a nonlinear function

Recall that any value of x lying between the two endpoints of the
kth line segment may be expressed as

x = αx- k+1 + (1 – α)x- k or x – x- k = α(x- k+1 – x- k) for 0 ≤ α ≤ 1.

where x- k (k = 0 , 1 , . . . ,r) are data and α is the decision variable. This

relationship leads directly to an expression for the kth line segment.

fk+1 – fk
(x – x- k) = αfk+1 – (1 – α)fk for 0 ≤ α ≤ 1
^
f(x) = fk +
x- k+1 – x- k
 Separable Programming 3

^
The approximation f(x) becomes increasingly more accurate as r gets
larger. Unfortunately, there is a corresponding growth in the size of the
resultant problem.

For the kth segment, let α = αk+1 and (1 – α) = αk. As such, for x- k ≤
x ≤ x- k+1, the above expresses is

x = αk+1x- k+1 + αk x- k and f(x) = αk+1fk+1 + αk fk

^

where αk + αk+1 = 1 and αk ≥ 0, αk+1 ≥ 0. Generalizing this procedure to

cover the entire range over which x is defined gives
r

∑ αkx- k,
r r
∑ αkfk, ∑ αk
^
x= f(x) = = 1, αk ≥ 0, k = 0, . . . ,r
k=0 k=0 k=0

such that at least one and no more than two αk can be greater than zero.
Furthermore, we require that if two αk are greater than zero, their indices
must differ by exactly 1. In other words, if αs is greater than zero then
only one of either αs+1 or αs-1 can be greater than zero. If the last
condition, known as the adjacency criterion, is not satisfied, the
^
approximation to f(x) will not lie on f(x).
To apply the above transformations, a grid of rj + 1 points must be
defined for each variable xj over its range. This requires the use of an
additional index for each variable and function. For the jth variable, for
example, f + 1 data points result: x- , x- , . . . , x- . With this notation in
j 1j 2 r jj
mind, the separable programming problem in x becomes the following
"almost" linear program in .

n rj

Minimize f( ) = ∑ ∑ αkjfkj(x- kj)

j=1 k=0

n rj

subject to gi( ) = ∑ ∑ αkjgkij(x- kj) ≤ bi , i = 1, . . . ,m

j=1 k=0
rj

∑ αkj = 1, j =1, . . . ,n
k=0
 Separable Programming 4

αkj ≥ 0, k = 0, . . . ,rj , j =1, . . . ,n.

The reason that this is an "almost" linear programming problem is that the
adjacency criterion must be imposed on the new decision variables αkj
when any of the functions are nonconvex. This can be accomplished with
a restricted basis entry rule. When all the functions are convex, the
adjacency criterion will automatically be satisfied so no modifications of
the simplex algorithm are necessary. Note that the approximate problem
has m + n constraints and Σj rj + n variables.

From a practical point of view, one might start off with a rather
large grid and find the optimum to the corresponding approximate
problem. This should be easy to do but the results may not be very
accurate. To improve on the solution, we could then introduce a smaller
grid in the neighborhood of the optimum and solve the new problem. This
idea is further discussed by Bard et al. (2000).

Example 13

Consider the problem below whose feasible region is shown graphically in

Fig. 6. All the functions are convex but the second constraint is g2(x) ≥
10. This implies that the feasible region is not convex so the solution to
the approximate problem may not be a global optimum.
2
Minimize f(x) = 2x1 – 3x1 + 2x2
2 2
subject to g1(x) = 3x1 + 4x2 ≤ 8

g2(x) = 3(x1 – 2)2 + 5(x2 – 2)2 ≥ 10

g3(x) = 3(x1 – 2)2 + 5(x2 – 2)2 ≤ 21

0 ≤ x1 ≤ 1.75, 0 ≤ x2 ≤ 1.5
The upper bounds on the variables have been selected to be
redundant. The objective function and constraints are separable with the
individual terms being identified in Table 3.
 Separable Programming 5

Figure 6. Feasible region for example

Table 3. Separable functions for example

j 1 2
fj(xj) 2x21 – 3x1 2x2

g1j(xj) 3x21 4x22

g2j(xj) 3(x1 – 2)2 5(x2 – 2)2

g3j(xj) 3(x1 – 2)2 5(x2 – 2)2

To develop the piecewise linear approximations we select six grid points

for each variable and evaluate the functions at each point. The results are
given in Table 4. For this example n = 2, m = 3, r1 = 5, and r2 = 5. As an
illustration, the piecewise linear approximations of f(x1) and g12(x2) along
with the original graphs are depicted in Fig. 7. The full model has 5
constraints and 12 variables. The coefficient matrix is given in Table 5
where the last two rows correspond to the summation constraints on the
two sets of variables.
 Separable Programming 6

Table 4. Grid points and corresponding function values

k 0 1 2 3 4 5
xk1 0 0.4 0.75 1.0 1.25 1.75
xk2 0 0.3 0.6 0.9 1.2 1.5
fk1 0 – 0.88 – 1.125 –1 – 0.625 0.875
gk11 0 0.48 1.6875 3 4.6875 9.1875
gk21 12 7.68 4.6875 3 1.6875 0.1875
gk31 12 7.68 4.6875 3 1.6875 0.1875
fk2 0 0.6 1.2 1.8 2.4 3
gk12 0 0.36 1.44 3.24 5.76 9
gk22 20 14.45 9.8 6.05 3.2 1.25
gk32 20 14.45 9.8 6.05 3.2 1.25

f(x1) g12(x2)

0 6
1 x1

-1
0.5 1 1.5 x2

Figure 7. Piecewise linear approximations for f(x1) and g12(x2)

Table 5. Coefficients of the linear programming model

 Separable Programming 7

α01 α11 α21 α31 α41 α51 α02 α12 α22 α32 α42 α52

f 0 0.88 1.125 1 0.625 - 0.875 0 - 0.6 - 1.2 - 1.8 - 2.4 -3

g1 0 0.48 1.6875 3 4.6875 9.1875 0 0.36 1.44 3.24 5.76 9

g2 12 7.68 4.6875 3 1.6875 0.1875 20 14.45 9.8 6.05 3.2 1.25

g3 12 7.68 4.6875 3 1.6875 0.1875 20 14.45 9.8 6.05 3.2 1.25

x1 1 1 1 1 1 1 0 0 0 0 0 0

x2 0 0 0 0 0 0 1 1 1 1 1 1

The problem will be solved with a linear programming code modi-

fied to enforce the adjacency criterion. In particular, for the jth variable
we do not allow an αkj variable to enter the basis unless αk–1,j or αk+1,j is
already in the basis, or no αkj (k = 0, 1, . . . , 5) is currently basic. The
following slack and artificial variables will be used to put the problem into
standard simplex form.
s1 = slack for constraint 1, g1
s2 = surplus for constraint 2, g2
a2 = artificial for constraint 2, g2
s3 = slack for constraint 3, g3

a4 = artificial for constraint 4, Σk αk1

a5 = artificial for constraint 5, Σk αk2

The initial basic solution is

xB = (s1, a2, s3, a4, a5) = (8, 10, 21, 1, 1)
which is seen to contain three artificial variables. The phase 1 procedure
required five iterations to drive the objective function to zero. Phase 2
also required five iterations. The corresponding data are presented Table
6. The most negative rule was used to select the entering variable but if
that variable led to a violation of the adjacency criterion the next one on
the list was selected. For example, at iteration 3.1, α21 is selected to enter
the basis but since it is not adjacent to α51 it is not permitted to do so. The
nonbasic variable with the next smallest reduced cost is α11 but it too does
not satisfy the adjacency criterion. Finally, at iteration 3.4 the variable
with the smallest reduced cost not yet examined is α41 which meets the
 Separable Programming 8

criterion and so is allowed to enter the basis. A pivot is executed with a4

the leaving variable.

Table 6. Iterations of restricted entry simplex algorithm

Iter. Entering Leaving Basic variables

0 s1, a2, s3, a4, a5
1 α02 a2 s1, α02, s3, a4, a5
2 α51 s1 α51, α02, s3, a4, a5
3.1 α21 –– not adjacent to α51
3.2 α11 –– not adjacent to α51
3.3 α31 –– not adjacent to α51
3.4 α41 a4 α51, α02, s3, α41, a5
4.1 α52 –– not adjacent to α02
4.2 α42 –– not adjacent to α02
4.3 α32 –– not adjacent to α02
4.4 α22 α02 α51, α22, s3, α41, a5
5.1 α52 –– not adjacent to α22
5.2 α42 –– not adjacent to α22
5.3 α32 a5 α51, α22, s3, α41, α32
Begin phase 2
5.3 α41, α32, s3, α51, α22
5.4 α02 –– not adjacent to α22
5.5 α42 –– not adjacent to α22
5.6 s1 α51 α41, α32, s3, s1, α22
6 α31 α41 α31, α32, s3, s1, α22
7 s2 α32 α31, s2, s3, s1, α22
8.1 α02 –– not adjacent to α22
8.2 α12 α22 α31, s2, s3, s1, α12
9 α02 s3 α31, s2, α02, s1, α12

At the end of iteration 9, the algorithm terminates with the optimal

solution
^
s1 = 4.87, s2 = 11, α31 = 1, α02 = 0.6396, α12 = 0.3604 and f = – 0.7838.
In terms of the original problem variables, this corresponds to
 Separable Programming 9

x1 = α31x- 31 = 1 and x2 = α02x- 02 + α12x- 12 = 0.1081.

When comparing these values with the true optimum x1* = 0.9227, x2* =
0.1282, and f(x*) = – 0.8089, we can observe the effect of the linearization
process on the accuracy of the solution. With respect to the objective
function, our six-point approximation is off by 3.2%. For this particular
problem, however, we can get as close as desired by making the grid size
arbitrarily small in the neighborhood of the true optimum.
It is interesting to note that while the basic solutions given in Table
6 lie at vertices in the space, they do not necessarily have any relation to
the vertices in the original x space. This is illustrated in Fig. 8 which plots
the feasible region and several isovalue contours of f(x). Also shown are
the 6 basic solutions that comprise phase 2. The first basic solution
corresponds to iteration 5.3 in Table 6. As can be seen, none of the 6
iterates in the figure lies at a vertex of the feasible region. At the optimum
only g3 is binding.

Figure 8. Sequence of six basic solutions during phase 2

If the objective in the example is changed from minimization to

maximization and the same approximation is used, the algorithm
converges to x = (1.458, 0.6). This point is a local maximum
approximating the rightmost intersection of the boundaries of constraints
g1 and g2 located at x = (1.449, 0.652). The global maximum resides at x
= (0, 1.414). The fact that it was not found can be attributed to the
nonconvexity of the constraint region. We are assured that a local
maximum is a global maximum if and only if a concave function is being
maximized over a convex constraint region. Although the algorithm
found the global minimum to the example, this was only because we were
minimizing a convex objective function and the global optimum happened
to reside on the boundary of a constraint, g3, whose feasible region is
convex.
 Separable Programming 10

Convex Programming Problems

These observations stem directly from the fact that the separable
programming method guarantees an approximate global optimum to the
original problem only when one is minimizing a convex function
(maximizing a concave function) over a convex set. When these
conditions hold, the accuracy of the approach is limited only by the
coarseness of the piecewise linear approximations that are used.
Furthermore, when solving a convex programming problem, we may solve
the approximate problem using ordinary linear programming methods
without enforcing the adjacency restrictions.

Nonconvex Programming

If the conditions that define a convex program are not present, several
outcomes may occur.
1. An approximate global optimum is found (as in the minimization
example above).
2. An approximate local optimum is found that is not the global
optimum.
3. The solution to the approximate problem may be infeasible with
respect to the original problem or be nowhere near a corresponding
local or global optimum. These outcomes are due to the fact that
an insufficient number of lines segments were chosen for the
approximation. In many cases, however, infeasible solutions will
be only slightly infeasible, and thus present no practical difficulty.

Notwithstanding the possibility of obtaining erroneous results,

separable programming methods have proven to be very useful in a variety
of practical applications. In addition, it is possible to modify the basic
transformations by introducing integer variables and related constraints so
that approximate global optima are always obtained, regardless of the
convexity of the original problem. An experimental code called MOGG
was developed by Falk and Soland (1969) along these lines.
Unfortunately, the modified formulation often yields problems that are
extremely difficult to solve within acceptable time limits.