The irace Package

Manuel Lopez-Ibanez, Jeremie Dubois-Lacoste, Thomas Stutzle, and

Mauro Birattari. The irace package, Iterated Race for Automatic
Algorithm Configuration. Technical Report TR/IRIDIA/2011-004,
IRIDIA, Universite Libre de Bruxelles, Belgium, 2011.
http://iridia.ulb.ac.be/irace

The irace Software Package

Implementation of Iterated Racing in R

Manuel Lopez-Ibanez Goal 1: Flexible
[email protected]
Goal 2: Easy to use
COMEX Workshop on Practical Automatic Algorithm Configuration

Manuel Lopez-Ibanez The irace Software Package

The irace Package The irace Package

Manuel Lopez-Ibanez, Jeremie Dubois-Lacoste, Thomas Stutzle, and Manuel Lopez-Ibanez, Jeremie Dubois-Lacoste, Thomas Stutzle, and
Mauro Birattari. The irace package, Iterated Race for Automatic Mauro Birattari. The irace package, Iterated Race for Automatic
Algorithm Configuration. Technical Report TR/IRIDIA/2011-004, Algorithm Configuration. Technical Report TR/IRIDIA/2011-004,
IRIDIA, Universite Libre de Bruxelles, Belgium, 2011. IRIDIA, Universite Libre de Bruxelles, Belgium, 2011.
http://iridia.ulb.ac.be/irace http://iridia.ulb.ac.be/irace

Use it from inside R . . .

R package available at CRAN: R> result <- irace(tunerConfig = list(maxExperiments = 1000),
parameters = parameters)
http://cran.r-project.org/package=irace

R> install.packages("irace") . . . or through command-line: (See irace --help)

irace --max-experiments 1000 --param-file parameters.txt

4 No knowledge of R needed

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package
The irace Package The irace Package: Instances

TSP instances
Parameter Configuration $ dir Instances/
Instances space of irace
3000-01.tsp 3000-02.tsp 3000-03.tsp ...

Continuous functions
$ cat instances.txt
function=1 dimension=100
function=2 dimension=100
...
calls with i, returns c(i,)
irace
irace Parameters for an instance generator
$ cat instances.txt
I1 --size 100 --num-clusters 10 --sym yes --seed 1
I2 --size 100 --num-clusters 5 --sym no --seed 1
...
hookRun Script / R function that generates instances
+ if you need this, tell us!

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package

The irace Package: Parameter space The irace Package: Options

Categorical (c), ordinal (o), integer (i) and real (r) digits: number of decimal places to be considered for the real
parameters (default: 4)
Subordinate parameters (| condition)
maxExperiments: maximum number of runs of the algorithm
$ cat parameters.txt being tuned (tuning budget)
# Name Label/switch Type Domain Condition testType: either F-test or t-test
LS "--localsearch " c {SA, TS, II}
rate "--rate=" o {low, med, high}
firstTest: specifies how many instances are seen before the
population "--pop " i (1, 100)
temp "--temp " r (0.5, 1) | LS == "SA" first test is performed (default: 5)

For real parameters, number of decimal places is controlled by eachTest: specifies how many instances are seen between
option digits (--digits) tests (default: 1)

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package
The irace Package: hook-run Example: ACOTSP

Thomas Stutzle. ACOTSP: A software package of various ant

colony optimization algorithms applied to the symmetric
A script/program that calls the software to be tuned:
traveling salesman problem, 2002.
./hook-run instance candidate-number candidate-parameters ... http://www.aco-metaheuristic.org/aco-code/

An R function: Command-line program:

hook.run <- function(instance, candidate, extra.params = NULL, ./acotsp -i instance -t 5 --mmas --ants 10 --rho 0.95 ...
config = list())
{
...
}
Goal: find best parameter settings of ACOTSP for solving
random Euclidean TSP instances with n [1000, 3000]
Flexibility: If there is something you cannot tune, let us know! within 1 CPU-second

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package

Example: ACOTSP Example: ACOTSP

$ cat parameters-acotsp.txt $ cat hook-run

# name switch type values conditions #!/bin/bash

algorithm "--" c (as,mmas,eas,ras,acs)
INSTANCE=$1
localsearch "--localsearch " c (0, 1, 2, 3)
CANDIDATENUM=$2
alpha "--alpha " r (0.00, 5.00)
beta "--beta " r (0.00, 10.00) CAND PARAMS=$*
rho "--rho " r (0.01, 1.00) STDOUT="c${CANDIDATENUM}.stdout"
ants "--ants " i (5, 100) FIXED PARAMS=" --time 1 --tries 1 --quiet "
nnls "--nnls " i (5, 50) | localsearch %in% c(1,2,3) acotsp $FIXED PARAMS -i $INSTANCE $CAND PARAMS 1> $STDOUT
dlb "--dlb " c (0, 1) | localsearch %in% c(1,2,3) COST=$(grep -oE Best [-+0-9.e]+ $STDOUT |cut -d -f2)
q0 "--q0 " r (0.0, 1.0) | algorithm == "acs"
if ! [[ "${COST}" = [-+0-9.e]+$ ]] ; then
rasrank "--rasranks " i (1, 100) | algorithm == "ras"
elitistants "--elitistants " i (1, 750) | algorithm == "eas" error "${STDOUT}: Output is not a number"
fi
echo "${COST}"
exit 0

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package
Example: ACOTSP Questions

$ cat tune-conf

execDir <- "./acotsp-execdir"

instanceFile <- "./training.txt"
maxExperiments <- 300
digits <- 2

4 Good to go:

$ make -C ACOTSP-1.04-tuning all

$ mkdir acotsp-execdir
$ irace

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package

Exercise #1 ACOTSP-VAR Example: Tuning for anytime

Setup irace for your own problem Like ACOTSP, but with more parameters
OR The output is now a Pareto front of (time, quality) pairs
Use a common reference point (2.1, 2.1, . . . )
Run the ACOTSP example:
Normalize the objectives range to [1, 2] per instance
http://iridia.ulb.ac.be/manuel/comex workshop/ without predefined maximum / minimum
acotsp-example.tar.gz
+ We need all Pareto fronts for computing the normalization!
Add another default configuration and make irace use it
See default.txt 8 We cannot simply use hook-run
Add another forbidden configuration and make irace use it 4 We use hook-evaluate !
See forbidden.txt hook-evaluate hook-run
Use --debug-level 1 to see what irace is executing Executes after all hook-run for a given instance
Returns the cost value instead of hook-run

./hook-evaluate instance candidate-number total-candidates

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package
Exercise #2 Exercise #2

Continue setting up irace for your own problem 1 Compile ACOTSPvar, hv and nondominated
OR
2 Add training instances: use the ones from the ACOTSP
Run the ACOTSP-VAR (anytime) example:
example
http://iridia.ulb.ac.be/manuel/comex workshop/
acotspvar-example.tar.gz 3 Examine all irace files
Note the use of fixed parameters in the parameters file
Note that hook-run does not return a value
Note that tune-conf needs adjusting

4 Run irace and try to understand what it is doing

Manuel Lopez-Ibanez The irace Software Package Manuel Lopez-Ibanez The irace Software Package

Exercise #3

Analyzing the results of irace:

$ R
R> load("acotsp-execdir/irace.Rdata")
R> names(tunerResults)
R> print(tunerResults$experiments)
R> print(tunerResults$allCandidates)

Manuel Lopez-Ibanez The irace Software Package