Computers & Geosciences 37 (2011) 1894–1899

Contents lists available at ScienceDirect

Computers & Geosciences

journal homepage: www.elsevier.com/locate/cageo

Application of MATLAB and Python optimizers to two case studies involving

groundwater flow and contaminant transport modeling$
L. Shawn Matott n, Kenny Leung, Junyoung Sim
Department of Civil and Environmental Engineering, University of Waterloo, 200 University Avenue West, Waterloo, Ontario, Canada N2L 3G1

a r t i c l e i n f o a b s t r a c t

Article history: One approach for utilizing geoscience models for management or policy analysis is via a simulation-
Received 7 October 2010 based optimization framework—where an underlying model is linked with an optimization search
Received in revised form algorithm. In this regard, MATLAB and Python are high-level programming languages that implement
24 February 2011
numerous optimization routines, including gradient-based, heuristic, and direct-search optimizers. The
Accepted 31 March 2011
Available online 12 April 2011
ever-expanding number of available algorithms makes it challenging for practitioners to identify
optimizers that deliver good performance when applied to problems of interest. Thus, the primary
Keywords: contribution of this paper is to present a series of numerical experiments that investigated the
Reactive transport performance of various MATLAB and Python optimizers. The experiments considered two simulation-
Pump and treat
based optimization case studies involving groundwater flow and contaminant transport. One case
Simulation-based optimization
study examined the design of a pump-and-treat system for groundwater remediation, while the other
Model calibration
MATLAB considered least-squares calibration of a model of strontium (Sr) transport. Using these case studies, the
Python performance of 12 different MATLAB and Python optimizers was compared. Overall, the Hooke–Jeeves
direct search algorithm yielded the best performance in terms of identifying least-cost and best-fit
solutions to the design and calibration problems, respectively. The IFFCO (implicit filtering for
constrained optimization) direct search algorithm and the dynamically dimensioned search (DDS)
heuristic algorithm also consistently yielded good performance and were up to 80% more efficient than
Hooke–Jeeves when applied to the pump-and-treat problem. These results provide empirical evidence
that, relative to gradient- and population-based alternatives, direct search algorithms and heuristic
variants, such as DDS, are good choices for application to simulation-based optimization problems
involving groundwater management.
& 2011 Elsevier Ltd. All rights reserved.

1. Introduction Numerous optimization search algorithms have been devel-

oped and may be classified according to a variety of alternative
A common approach for utilizing geoscience models in a taxonomies (e.g., Dréo et al., 2006; Talbi, 2002). For this paper, it
management or policy-analysis context is to incorporate them is convenient to informally divide the spectrum of optimizers into
into a simulation-based optimization framework—where an four categories: (1) gradient-based algorithms that use derivative
underlying process-based geoscience model is linked with an information to guide the search; (2) heuristic algorithms that do
optimization search algorithm (Banta et al., 2008; Freeze and not utilize derivative information but instead follow empirical
Gorelick, 1999). During optimization, a given search algorithm guidelines and incorporate elements of structured randomness;
iteratively adjusts various model inputs (i.e., parameters or design (3) direct search algorithms that are derivative free but utilize
variables) in order to minimize an application-specific cost func- deterministic stencil-based procedures; and (4) hybrid methods
tion computed on the basis of model outputs (i.e., response that combine aspects of the previous three categories.
variables). The fundamental output of an optimization algorithm Example applications of simulation-based optimization in a
is an ‘‘optimal’’ configuration of design variables along with the geoscience context include the design of pump-and-treat systems
corresponding value of the cost function. (McKinney and Lin, 1996), groundwater supply systems (Chu and
Chang, 2010), landfill liners (Bartelt-Hunt et al., 2006), agricultural
operations (Gitau et al., 2006), waste load allocation strategies
$
(Burn and Yulianti, 2001), and geothermal reservoirs (Akin et al.,
Code available from server at http://www.civil.uwaterloo.ca/lsmatott/Pigeon/
Pigeon.html
2010). In addition, calibration of geoscience models is typically
n
Corresponding author. Tel.: þ1 519 888 4567x38807; fax: þ 1 519 888 4349. formulated as a simulation-based optimization problem—where
E-mail address: [email protected] (L.S. Matott). uncertain model parameters are adjusted to minimize differences

0098-3004/$ - see front matter & 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.cageo.2011.03.017
 L.S. Matott et al. / Computers & Geosciences 37 (2011) 1894–1899 1895

between simulated outputs and corresponding observational data 200

Na
(Massoudieh et al., 2008). 180 K
For a given application, researchers have generally used optimi- Mg

Na, K, Mg, Ca (mg/L) ; Sr (uq/L)

160
zation routines that are embedded within the code of the underlying Ca
geoscience model (e.g., Akin et al., 2010; Lim et al., 2010). However, 140
Sr
various efforts by commercial entities, research labs, academics, and 120
independent enthusiasts have led to the development of numerous 100
general-purpose optimization libraries using the MATLAB and
80
Python programming languages (Dahl and Vandenberghe, 2008;
Hart, 2009; Kelley, 1999; Venkataraman, 2009). These libraries can 60
potentially eliminate the need to implement embedded optimiza- 40
tion codes if the associated external optimizers are sufficiently
20
robust and deliver good performance when applied to problems of
interest. 0
0 5 10 15 20 25 30 35 40 45 50 55 60 65
To assist practitioners with identifying ‘‘best-in-class’’ MATLAB Time from start of experiment (days)
and Python optimizers for a given simulation-based optimization
problem, a series of numerical experiments were performed Fig. 1. Measured column effluent for the MOUSER calibration problem (adapted
from Rabideau et al. (2005)).
involving two case studies relevant to the geosciences community
and a suite of 12 MATLAB and Python optimizers. The case studies
consider optimization from two distinctly different contexts—a
pump-and-treat case study emphasizes least-cost remedial sys- Northern boundary (no flow)

tem design, while a subsurface strontium transport application Candidate Remediation Wells
Gravel
considers the best-fit calibration of model parameters. Pond
The remainder of the paper is organized as follows: Section 2
describes the selected case-study simulation-based optimization
problems; Section 3 describes the optimizers that were selected

Eastern boundary (specified head)

for consideration in this study along with the associated numer-
ical experiments; Section 4 presents the results of the numerical K=15 m/d

experiments and compares the performance of the selected

optimizers; and Section 5 provides some concluding remarks. K=3 m/d

2. Case-study simulation-based optimization problems

To evaluate the usefulness of MATLAB and Python optimizers K=4.6 m/d

in a geosciences context, selected algorithms were applied to two 300 meters
case-study simulation-based optimization problems involving
Bedrock Outcrop (no flow)
groundwater flow and contaminant transport. The first case study
(i.e., ‘‘the calibration problem’’) considers the calibration of a Fig. 2. Pump-and-treat problem setup (K values are hydraulic conductivities of
subsurface reactive transport model where the primary solute of various zones of heterogeneity).
interest is strontium. The second case study (i.e., ‘‘the pump-and-
treat problem’’) involves the design of a pump-and-treat contain- outputs. In the WSSR formulation, data uncertainty may be
ment system for a site containing contaminated groundwater. accommodated via an appropriate assignment of residual weights
Additional case study details are given below. (Poeter and Hill, 1999). As shown in Fig. 1, sampling of the column
effluent yielded multiple observations of Na (the reference ion-
2.1. The calibration problem exchange species), K, Mg, Ca, and Sr.

The calibration benchmark problem is motivated by the use of 2.2. The pump-and-treat problem
a natural zeolite permeable reactive barrier (PRB) to remediate
groundwater contaminated with strontium 90 at the West Valley The pump-and-treat benchmark problem involves a hypothe-
Demonstration Project site in western New York State (Rabideau tical groundwater contamination scenario based on the Lockwood
et al., 2005). To assess the effectiveness of the PRB, a one- Solvent Groundwater Plume Site (LSGPS) located near Billings,
dimensional reactive transport model was developed using the Montana (Tetra Tech Inc., 2003). At the site, industrial practices
MOUSER (MOderately USEr-friendly Reactive transport model) have resulted in the development of two separate plumes con-
software (Rabideau, 2003). MOUSER is a multipurpose finite- taining chlorinated solvents. As illustrated in Fig. 2, the plumes
element reactive transport model which uses an operator split- (i.e., Plume A in the south and Plume B in the north) are migrating
ting algorithm in which the nodal reaction equations are solved toward Yellowstone River and six pump-and-treat wells (2 for
sequentially over each time step. In addition to simulating Plume A and 4 for Plume B) have been proposed to prevent
advective–dispersive transport, MOUSER incorporates competitive further expansion. Therefore, optimization objectives are to
cation-exchange reaction processes, wherein the Gaines–Thomas successfully contain both plumes using a minimum amount of
convention is used for sorbed phase activity (Gaines and Thomas, pumping.
1953). Data from laboratory column experiments served as obser- Alternative configurations of the proposed pump-and-treat sys-
vations for a calibration exercise in which the values of four species tem were modeled using the Bluebird (Craig and Matott, 2005)
selectivity coefficients (CK, CMg, CCa, and CSr) were estimated by engine, which uses a high-order implementation (Jankovic and
minimizing the weighted sum of squared residuals (WSSR) Barnes, 1999) of the analytic element method (AEM) (Hunt, 2006)
between observed data and corresponding MOUSER-generated and assumes two-dimensional steady-state flow in a single-layer
1896 L.S. Matott et al. / Computers & Geosciences 37 (2011) 1894–1899

aquifer. Unlike finite difference and finite element approaches, AEM routine is an implementation of implicit filtering for constrained
groundwater models incorporate hydrologic features as distinct optimization (IFFCO), a robust stencil-based approach that is
model elements rather than distributed over a spatial grid or mesh. designed to tolerate noise (e.g., from numerical errors (Kavetski,
Plume containment goals were evaluated using Bluebird’s zone 2006)) in the optimization cost function. The fminsearch routine
budget feature, which measures the simulated amount of ground- is provided with the MATLAB optimization toolbox and is an
water flux exiting each plume boundary. Following Matott et al. implementation of the classic Nelder–Mead downhill simplex
(2006), the exit flux was required to be identically zero or else a algorithm (Nelder and Mead, 1965).
penalty was assessed and incorporated into the cost function using Two direct search routines implemented in Python were eval-
the additive penalty method (APM) (Hilton and Culver, 2000). uated, namely fmin_powell and HookeJeeves. The fmin_powell routine
implements Powell’s classic method of conjugate directions (Powell,
1977) and is provided with the scipy.optimize library. The Hooke-
3. Numerical experiments Jeeves routine is a Python implementation of the Hooke–Jeeves
pattern search algorithm (Hooke and Jeeves, 1961)—the Python
A software code known as PIGEON (Program for Interfacing code was directly translated from a freely available C-language
Geoscience models with External Optimization routiNes) was implementation (Amit Saha, personal communication).
developed to link the selected Python and MATLAB optimizers
with the selected case-study models (i.e., MOUSER and Bluebird, 3.3. Heuristic global optimizers (HGO)
for the calibration pump-and-treat problems, respectively).
PIGEON provides users with a human-readable text file interface Two MATLAB heuristic optimization routines were included in
for configuring the necessary communication between a given the comparison experiments, namely DDSmincon and TurboGA.
combination of geoscience model and external optimizer. The The DDSmincon routine implements dynamically dimensioned
PIGEON software and user manual is freely available for download search (DDS) (Tolson and Shoemaker, 2007), an algorithm that
from the following web site: www.civil.uwaterloo.ca/lsmatott/ focuses on finding ‘‘good-enough’’ global solutions within a
Pigeon/Pigeon.html. Incorporation of PIGEON into other geoscience specified maximum computational budget. Designed to mimic
toolkits, such as HYDROMAD (Hydrological Model Assessment and the trial-and-error approach commonly employed by practi-
Development) (Andrews, 2010) and OpenDA (Open-source Data tioners, DDS may be viewed as a kind of stochastic direct search
Assimilation) (Verlaan et al., 2010), would increase the choice of procedure (Tolson, personal communication). The algorithm
optimizers available to users of these packages. requires no tuning and the search dimension is dynamically
The numerical experiments considered a total of 12 optimiza- refined as optimization proceeds. The TurboGA routine is an
tion routines, corresponding to 6 MATLAB and 6 Python optimi- implementation of the recently developed ‘‘turbo’’ genetic algo-
zers. The chosen algorithms are briefly described in the following rithm (TGA) (Burjorjee, 2010). As a variant of the classic genetic
subsections. Consisting of four gradient-based optimizers, four algorithm, TGA emulates the evolutionary processes of gene
direct search optimizers, and four heuristic global optimizers, the encoding, natural selection, mutation, and crossover while incor-
selected algorithms represent a manageable cross section of porating new procedures that allow for speedier convergence. The
different approaches. DDS and TGA MATLAB codes utilized in this work are freely
available from the original developers of these algorithms (i.e.,
3.1. Gradient-based optimizers (GBO) Bryan Tolson for DDS and Keki Burjorjee for TGA).
Two heuristic optimization routines implemented in Python were
Two gradient-based optimizers from the MATLAB optimization evaluated, namely anneal and DiffEvolution. The anneal routine is part
toolbox were evaluated: fminunc and fminimax. Both of these routines of the scipy.optimize library and is an implementation of simulated
are highly flexible and will select from a variety of optimization annealing (SA) (Kirkpatrick et al., 1983)—an algorithm that emulates
procedures depending on problem characteristics and the settings the process of physical annealing, where a solid is heated to a high
of various algorithm options. In this regard, the fminunc routine energy state and slowly cooled to drive the material to its lowest
was configured to use the Broyden–Fletcher–Goldfarb–Shanno energy state. The DiffEvolution routine is an enthusiasts’ (i.e., Amit
(BFGS) (Broyden, 1970; Fletcher, 1970; Goldfarb, 1970; Shanno, Saha) freely available implementation of differential evolution (DE)
1970) algorithm, while the fminimax procedure defaulted to using (Storn and Price, 1997)—a population-based algorithm that perturbs
Newton’s method (Polyak, 2007) for the selected problems. Both parameter values using a combination of vector differences and
algorithms were instructed to approximate gradients using finite evolutionary operators (i.e., mutation, selection, and crossover).
differences.
Two gradient-based optimizers from the Python scipy.optimize 3.4. Numerical experiments
library were also evaluated: namely fmin_cg and fmin_tnc. The
fmin_cg routine is an implementation of the Polak–Ribiere con- All 12 of the selected optimizers were applied to each of the
jugate gradient (PRCG) algorithm (Grippo and Lucidi, 1997), while benchmark problems. Because heuristic optimizers are stochastic,
the fmin_tnc routine implements a truncated Netwon code (TNC) multiple (i.e., 9) trials of these algorithms were evaluated in order
(Dixon and Price, 1988)—an approach that is similar to Newton’s to determine the central tendency of each approach, expressed in
method but uses coarser estimates of the optimal search direction terms of the median trial. For each problem, and as summarized
to improve efficiency. As with the MATLAB gradient-based opti- in Table 1, the various optimizers were provided with the same
mizers, both fmin_cg and fmin_tnc were instructed to approximate initial iterate and, where required, the same lower and upper
gradients using finite differences. bounds for design variables.
Most algorithm parameters were allowed to default to the
3.2. Direct search optimizers (DSO) internal settings of the associated Python or MATLAB implementa-
tion. Notable exceptions are as follows: (1) DDS, Powell, and SA were
Two MATLAB routines implementing alternative derivative- configured to use budgets of 1000 and 500 evaluations for the
free direct search procedures were evaluated: namely, imfil pump-and-treat and MOUSER calibration problems, respectively;
(Kelley, 2009) and fminsearch. Along with many other optimizers, (2) PRCG was configured to use a maximum of 100 iterations; (3) DE
the imfil routine is provided as a companion to Kelley (1999)—the was configured to use population sizes of 20 and 15 and maximum
 L.S. Matott et al. / Computers & Geosciences 37 (2011) 1894–1899 1897

Table 1
Parameter bounds, initial values, and objective functions for the benchmark problems (sim and obs subscripts denote simulated and observed concentrations, respectively;
Qout,A and Qout,B are the simulated fluxes exiting the boundaries of Plume A and B, respectively).

(a) MOUSER calibration problem

Parameter Initial value Lower bound Upper bound Units

CK 80 1 100 n/a
CMg 950 100 1000
CCa 880 100 1000
CSr 1290 1000 10000
!
10 h
P i P16
WSSR ¼ ðKi,obs Ki,sim Þ2 þ ðMgi,obs Mgi,sim Þ2 þ ðCai,obs Cai,sim Þ2 þ 1000  ðSri,obs Sri,sim Þ2
i¼1 j¼1

(b) Pump-and-treat problem

Design variable Initial value Lower bound Upper bound Units

Q1 283 0.00 566 m3/day

Q2 283 0.00 566
Q3 283 0.00 566
Q4 283 0.00 566
Q5 283 0.00 566
Q6 283 0.00 566
!
6
P  
Cost ¼ $35  Qi þ $100000  ðQout,A þ Qout,B Þ
i¼1 |fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}
penalty function

Table 2
Optimization results for the MOUSER calibration problem.

Algorithma Optimal Computational Optimal parameters Engine (routine) Typeb

WSSR costc
CK CMg CCa CSr

Hooke–Jeeves 9364 300 28 299 348 4698 Python (Hooke–Jeeves) DSO

TNC 9366 990 28 302 347 4704 Python (fmin_tnc) GBO
IFFCO 9366 354 28 294 343 4643 MATLAB (imfil) DSO
DE 9445 497 29 259 355 4655 Python (DiffEvolution) HGO
DDS 9468 486 30 329 394 5274 MATLAB (ddsmincon) HGO
TGA 10,120 480 30 488 475 6173 MATLAB (tga) HGO
SA 10,712 481 22 476 348 4717 Python (anneal) HGO
Nelder–Mead 10,758 284 26 898 333 4707 MATLAB (fminsearch) DSO
Powell 10,775 498 23 200 220 3098 Python (fmin_powell) DSO
PRCG 11,113 627 25 860 270 3935 Python (fmin_cg) GBO
Newton 11,517 356 37 551 656 8685 MATLAB (fminimax) GBO
BFGS 61,304 293 5 939 100 1984 MATLAB (fminunc) GBO

a
BFGS, Broyden–Flecther—Goldfarb—Shanno; DDS, dynamically dimensioned search; DE, differential evolution; IFFCO, implicit filtering for constrained optimization;
PRCG, Polak–Robiere conjugate gradient; SA, simulated annealing; TGA, turbo genetic algorithm; TNC, truncated Newton code.
b
DSO, direct search optimizer; GBO, gradient-based optimizer; HGO, heuristic global optimizer.
c
Number of required model evaluations before convergence on the reported optimal parameter set.

generations of 50 and 33 for the pump-and-treat and MOUSER in Table 2, the top 5 optimizers (i.e., Hooke–Jeeves, TNC, IFFCO,
calibration problems, respectively; (4) initial step-size vectors for DE, and DDS) delivered comparable results (i.e., within 1% of each
the Hooke–Jeeves algorithm were [5000,5000,5000,5000,5000, other) in terms of minimizing the weighted sum of squared
5000]T and [50,500,500,5000]T for the pump-and-treat and MOUSER residuals (WSSR) cost function. Of these algorithms, the two
calibration problems, respectively; (5) for the SA algorithm, a value direct search optimizers (i.e., Hooke–Jeeves and IFFCO) were the
of 10 was used for the dwell parameter and initial temperatures of most efficient in terms of computational costs (i.e., lowest
5  104 and 1  106 were used for the pump-and-treat and MOUSER required number of model evaluations). Although it yielded a
calibration problems, respectively; and (6) the TGA was configured low-WSSR solution, the gradient-based truncated Newton code
to use a binary gene encoding of 15 bits per parameter, population was by far the least efficient—requiring between 150 and 350%
sizes of 50 and 24, and maximum generations of 19 and 20 for the more evaluations, relative to the other algorithms. Interestingly,
pump-and-treat and MOUSER calibration problems, respectively. the other 3 gradient-based optimizers (PRCG, Newton, and BFGS)
Native MATLAB and Python command-lined interfaces were utilized performed poorly and yielded the highest optimized WSSR values.
to assign the aforementioned algorithm settings. Fig. 3 plots the convergence history of the various algorithms,
as applied to the MOUSER calibration problem and where WSSR
values have been converted to equivalent root mean squared
4. Results and discussion error (RMSE) values. The convergence behavior of most of the
algorithms is characterized by an initial fast reduction in the cost
The performance of the selected algorithms, as applied to the function over the course of the first 10–20 model evaluations,
MOUSER calibration problem, is summarized in Table 2. As shown followed by much more gradual refinements. For example, within
1898 L.S. Matott et al. / Computers & Geosciences 37 (2011) 1894–1899

the first 10 model evaluations the Hooke–Jeeves algorithm specific wells. For Plume A, the optimal configurations set Q1 at a
identifies a solution that is only 20% greater than the WSSR value low rate and Q2 at a high rate; and for Plume B, Q3 is high rate, Q4
at convergence. is medium rate, and Q5 and Q6 are low rate.
With respect to the pump-and-treat problem, the results gener- Fig. 4 illustrates the convergence behavior of the chosen
ated by the chosen algorithms are given in Table 3, sorted according algorithms, as applied to the pump-and-treat problem. Four of
to optimized cost values. For this problem, the top 3 algorithms the selected algorithms (i.e., DDS, Hooke–Jeeves, IFFCO, and
(i.e., Hooke–Jeeves, SA, and DDS) yielded optimized cost values that Newton) are characterized by an initial rapid reduction in the
are within 7.5% of each other, and the optimized costs of the top cost function, followed by more gradual refinements. Conversely,
6 algorithms (i.e., Hooke–Jeeves, SA, DDS, DE, IFFCO, and Newton) three of the four heuristic algorithms (i.e., differential evolution,
are within 15% of each other. The Hooke–Jeeves direct search simulated annealing, and Turbo GA) yielded reductions in the cost
algorithm yielded the least-cost solution, followed by three of the function that were more evenly distributed over the entire course
four heuristic optimizers (i.e., simulated annealing, dynamically of the optimization. The TNC and Nelder–Mead algorithms appear
dimensioned search, and differential evolution). Of the top 6 algo- to quickly ‘‘stall,’’ possibly becoming entrapped in local minima.
rithms, the IFFCO, Newton, and DDS algorithms are notably more
efficient and required between 200 and 700 fewer model evalua-
tions. Unlike the MOUSER calibration problem, where it yielded the 5. Conclusions
second-lowest optimized WSSR value, the truncated Newton code
yielded the highest optimized cost when applied to the pump-and- Overall, the results from the two benchmark problems indicate
treat problem. that the Hooke–Jeeves algorithm yielded the best performance in
Examination of the optimized pumping rates reported by each terms of identifying least-cost solutions. For the MOUSER calibra-
algorithm (see Table 3) suggests the presence of multiple nearly tion problem the Hooke–Jeeves algorithm was also highly efficient
equally optimal pumping strategies. However, these strategies are and required only 300 model evaluations. For the pump-and-treat
fairly consistent with respect to the required pumping rates for problem, the Hooke–Jeeves algorithm was not as efficient and

36 60,000
Pump-and-Treat Cost Function (ΣQ×$35/m3/d)
55,000
32
50,000
Root Mean Squared Error

28
45,000

24 40,000

35,000
20
30,000
16
25,000

12 20,000
0 100 200 300 400 500 600 700 0 200 400 600 800 1000
Number of Model Evaluations Number of Model Evaluations

Fig. 3. Convergence behavior of the selected algorithms (as applied to the Fig. 4. Convergence behavior of the selected algorithms (as applied to the pump-
MOUSER calibration problem). and-treat problem).

Table 3
Optimization results for the pump-and-treat problem.

Algorithma Optimal cost Computational Optimal pumping rates (m3/day) Engine (routine) Typeb
P6 costc
Qi  $35=ðm3 =dÞ
i¼1
Plume A Plume B

Q1 Q2 Q3 Q4 Q5 Q6

Hooke–Jeeves 23,714 993 10 170 383 70 22 17 Python (Hooke–eeves) DSO

SA 25,018 969 9 196 384 84 17 18 Python (anneal) HGO
DDS 25,441 768 16 169 348 52 23 113 MATLAB (ddsmincon) HGO
DE 26,392 992 7 174 369 132 45 21 Python (DiffEvolution) HGO
IFFCO 27,137 210 0 178 317 61 0 213 MATLAB (imfil) DSO
Newton 27,193 603 6 155 354 52 70 133 MATLAB (fminimax) GBO
TGA 31,705 650 83 176 300 40 100 199 MATLAB (tga) HGO
BFGS 34,943 325 10 143 308 79 106 157 MATLAB (fminunc) GBO
Powell 35,176 1021 3 158 199 46 177 410 Python (fmin_powell) DSO
Nelder–Mead 38,805 858 258 243 338 27 232 0 MATLAB (fminsearch) DSO
PRCG 47,277 1393 223 223 223 223 223 223 Python (fmin_cg) GBO
TNC 47,886 317 272 269 272 272 0 272 Python (fmin_tnc) GBO

a
BFGS, Broyden—Flecther—Goldfarb—Shanno; DDS, dynamically dimensioned search; DE, differential evolution; IFFCO, implicit filtering for constrained optimization;
PRCG, Polak–Robiere conjugate gradient; SA, simulated annealing; TGA, turbo genetic algorithm; TNC, truncated Newton code.
b
DSO, direct search optimizer; GBO, gradient-based optimizer; HGO, heuristic global optimizer.
c
Number of required model evaluations before convergence on the reported optimal parameter set.
 L.S. Matott et al. / Computers & Geosciences 37 (2011) 1894–1899 1899

required 993 model evaluations before converging on a solution. Fletcher, R., 1970. A new approach to variable metric algorithms. The Computer
If trading off optimized cost for computational efficiency was Journal 13, 317–322.
Freeze, R.A., Gorelick, S.M., 1999. Convergence of stochastic optimization and
acceptable, alternative algorithms such as IFFCO (requiring 79% decision analysis in the engineering design of aquifer remediation. Ground
fewer model evaluations, but yielding a solution that is only 14% Water Monitoring & Remediation 37, 934–954.
more expensive than Hooke–Jeeves) or DDS (requiring 23% fewer Gaines, G.L., Thomas, H.C., 1953. Adsorption studies on clay minerals. II. A
formulation of the thermodynamics of exchange adsorption. Journal of
evaluations and yielding only a 7% more expensive solution) might Chemical Physics 21, 714–718.
be considered as practical alternatives. Both of these algorithms Gitau, M.W., Veith, T.L., Gburek, W.J., Jarrett, A.R., 2006. Watershed level best
also delivered good performance when applied to the MOUSER management practice selection and placement in the town brook watershed,
New York. Journal of the American Water Resources Association 42,
calibration problem. 1565–1581.
MATLAB and Python have become popular delivery mechan- Goldfarb, D., 1970. A family of variable-metric methods derived by variational
isms for both classical (e.g., Hooke–Jeeves, BFGS, and simulated means. Mathematics of Computation 24, 23–26.
Grippo, L., Lucidi, S., 1997. A globally convergent version of the Polak–Ribiere
annealing) and up-and-coming (e.g., DDS, IFFCO, and the Turbo GA)
conjugate gradient method. Mathematical Programming 78, 375–391.
optimization routines. This paper represents an initial exploration Hart, W., 2009. Python optimization modeling objects (Pyomo). In: Chinneck, J.W.,
of the suitability of such optimizers for use within the geosciences Kristjansson, B., Saltzman, M. (Eds.), Operations Research and Cyber-Infra-
community. The results lay the groundwork for consideration of structure. Springer, New York, NY, pp. 3–20.
Hilton, A.B.C., Culver, T.B., 2000. Constraint handling for genetic algorithms in
additional case studies extending beyond problems involving optimal remediation design. Journal of Water Resources Planning and Man-
subsurface flow and contaminant transport. agement 126, 128–137.
From a practical perspective, simulation-based optimization is Hooke, R., Jeeves, T.A., 1961. ‘Direct search’ solution of numerical and statistical
problems. Journal of the ACM 8, 212–229.
usually employed within a broader context that seeks to utilize Hunt, R.J., 2006. Ground water modeling applications using the analytic element
geoscience models in support of policy analysis and decision method. Ground Water 44, 5–15.
making. Thus, although this study focused on identifying ‘‘best in Jankovic, I., Barnes, R., 1999. High-order line elements in modeling two-dimen-
sional groundwater flow. Journal of Hydrology 226, 211–223.
class’’ optimization routines, there are a number of additional Kavetski, D., 2006. Calibration of conceptual hydrological models revisited: 1.
considerations that need to be addressed to ensure that a given Overcoming numerical artefacts. Journal of Hydrology 320, 173–186.
simulation-based optimization effort is appropriate for its intended Kelley, C.T., 1999. Iterative Methods for Optimization. Society for Industrial and
Applied Mathematics, Philadelphia, PA.
purpose. For example, sources of uncertainty and variability must Kelley, C.T., 2009. Users’ Guide for imfil Version 0.85. North Carolina State
be carefully considered using robust methods. Further, calibration University, Raleigh, NC /w?w?w?4.ncsu.edu/  ?ctk/matlab_darts.htmlS.
efforts must guard against the possibility of overfitting—where Kirkpatrick, S., Gelatt Jr., C.D., Vecchi, M.P., 1983. Optimization by simulated
annealing. Science 220, 671–680.
adjusted model parameters yield excellent fits to historical data
Lim, K.J., Park, Y.S., Kim, J., Shin, Y.-C., Kim, N.W., Kim, S.J., Jeon, J.-H., Engel, B.A.,
but nonetheless generate poor predictions. 2010. Development of genetic algorithm-based optimization module in WHAT
system for hydrograph analysis and model application. Computers & Geos-
ciences 36, 936–944.
Appendix A. Supporting information Massoudieh, A., Mathew, A., Ginn, T.R., 2008. Column and batch reactive transport
experiment parameter estimation using a genetic algorithm. Computers &
Geosciences 34, 24–34.
Supplementary data associated with this article can be found Matott, L.S., Rabideau, A.J., Craig, J.R., 2006. Pump-and-treat optimization using
in the online version at doi:10.1016/j.cageo.2011.03.017. analytic element method flow models. Advances in Water Resources 29,
760–775.
McKinney, D.C., Lin, M.-D., 1996. Pump-and-treat ground-water remediation
system optimization. Journal of Water Resources Planning and Management
References 122, 128–136.
Nelder, J.A., Mead, R., 1965. A simplex method for function minimization. The
Computer Journal 7, 308–313.
Akin, S., Kok, M.V., Uraz, I., 2010. Optimization of well placement geothermal Poeter, E.P., Hill, M.C., 1999. UCODE, a computer code for universal inverse
reservoirs using artificial intelligence. Computers & Geosciences 36, 776–785. modeling. Computers & Geosciences 25, 457–462.
Andrews, F., 2010. HYDROMAD Tutorial. The Australian National University, Polyak, B.T., 2007. Newton’s method and its use in optimization. European Journal
Canberra, Australia. of Operational Research 181, 1086–1096.
Banta, E.R., Hill, M.C., Poeter, E., Doherty, J.E., Babendreier, J., 2008. Building model Powell, M.J.D., 1977. Restart procedures for the conjugate gradient method.
analysis applications with the joint universal parameter identification and Mathematical Programming 12, 241–254.
evaluation of reliability (JUPITER) API. Computers & Geosciences 34, 310–319. Rabideau, A.J., 2003. MOUSER Version 1 User’s Manual. University at Buffalo,
Bartelt-Hunt, S.L., Culver, T.B., Smith, J.A., Matott, L.S., Rabideau, A.J., 2006. Optimal Department of Civil, Structural, and Environmental Engineering, Buffalo, NY
design of a compacted soil liner containing sorptive amendments. Journal of /www.groundwater.buffalo.eduS.
Environmental Engineering 132, 769–776. Rabideau, A.J., Van Benschoten, J., Patel, A., Bandilla, K., 2005. Performance
Broyden, C.G., 1970. The convergence of a class of double-rank minimization assessment of a zeolite treatment wall for removing Sr-90 from groundwater.
algorithms. 1. General considerations. IMA Journal of Applied Mathematics 6, Journal of Contaminant Hydrology 79, 1–24.
76–90. Shanno, D.F., 1970. Conditioning of quasi-Newton methods for function mini-
Burjorjee, K.M., 2010. Generative fixation: a unified explanation for the adaptive mization. Mathematics of Computation 24, 647–656.
capacity of simple recombinative genetic algorithms. SIGEVOlution 4, 12–13. Storn, R., Price, K., 1997. Differential evolution—a simple and efficient heuristic for
Burn, D.H., Yulianti, J.S., 2001. Waste-load allocation using genetic algorithms. global optimization over continuous spaces. Journal of Global Optimization 11,
Journal of Water Resources Planning and Management 127, 121–129. 341–359.
Chu, H.-J., Chang, L.-C., 2010. Optimizing capacity-expansion planning of ground- Talbi, E.G., 2002. A taxonomy of hybrid metaheuristics. Journal of Heuristics 8,
water supply system between cost and subsidence. Journal of Hydrologic 541–564.
Engineering 15, 632–641. Tetra Tech Inc., 2003. Remedial Investigation Report Lockwood Solvent Groundwater
Craig, J.R.,Matott, L.S. 2005. Visual Bluebird User’s Manual: Version 2.0, University Plume Site. Montana Department of Environmental Quality, Helena, Montana,
at Buffalo, Department of Civil, Structural, and Environmental Engineering, /www.epa.gov/region8/superfund/mt/lockwood_solvents/RIreport06-03.pdfS.
Buffalo, NY. Available from: /www.groundwater.buffalo.eduS. Tolson, B.A., Shoemaker, C.A., 2007. Dynamically dimensioned search algorithm for
Dahl, J., Vandenberghe, L., 2008. CVXOPT: A Python Package for Convex Optimiza- computationally efficient watershed model calibration. Water Resources
tion. University of California at Los Angeles, Los Angeles, CA /http://abel.ee. Research 43, W01413.
ucla.edu/cvxoptS. Venkataraman, P., 2009. Applied Optimization with MATLAB Programming. John
Dixon, L.C.W., Price, R.C., 1988. Numerical experience with the truncated Newton Wiley & Sons, Hoboken, NJ.
method for unconstrained optimization. Journal of Optimization Theory and Verlaan, M., Velzen, N. v., Hummel, S.,Gerritsen, H., 2010. OpenDA, a generic
Applications 56, 245–255. toolbox for data-assimilation in numerical modelling. Paper presented at the
Dréo, J., Pétrowski, A., Siarry, P., Taillard, E. (Eds.), 2006. Metaheuristics for Hard 15th Biennial Conference of the Joint Numerical Sea Modelling Group, Delft,
Optimization. Springer, Heidelberg, Germany. The Netherlands, May 10–12, 2010.