0% found this document useful (0 votes)
70 views1 page

MLR 3 Tuning

This document provides a summary of the mlr3tuning package which allows users to: 1) Define hyperparameter tuning instances to map configurations to performance values. 2) Run black-box optimizers on tuning instances to find optimal hyperparameters. 3) Combine learners and tuners for nested resampling of hyperparameters.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
70 views1 page

MLR 3 Tuning

This document provides a summary of the mlr3tuning package which allows users to: 1) Define hyperparameter tuning instances to map configurations to performance values. 2) Run black-box optimizers on tuning instances to find optimal hyperparameters. 3) Combine learners and tuners for nested resampling of hyperparameters.
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

Hyperparameter Tuning with mlr3tuning::CHEAT SHEET

Class Overview TuningInstance* - Search Scenario Execute Tuning and Access Results Nested Resampling
The package provides a set of R6 classes which allow to (a) Evaluator and container for resampled performances of HPCs. tuner$optimize(instance)

Just resample AutoTuner; now has inner and outer loop.


define general hyperparameter (HP) tuning instances, i.e., the The (internal) eval_batch(xdt) calls benchmark() to eval a as.data.table(instance$archive)

black-box objective that maps HP configurations (HPCs) to


resampled performance values; (b) run black-box optimzers; (c)
table of HPCs. Stores archive of all evaluated experiments and
final result.
## > cost gamma classif.ce uhash x_domain_cost x_domain_gamma
Example
## > 1: 3.13 5.55 0.56 b8744... 3.13 5.55

combine learners with tuners (for nested resampling). ## > 2: -1.94 1.32 0.10 f5623... -1.94 1.32

inner = rsmp("holdout")

at = auto_tuner(method = "gensa", learner, inner, measure, term_evals = 20)

instance = TuningInstanceSingleCrit$new(task,
instance$result # datatable row with optimal HPC and estimated perf
outer = rsmp("cv", folds = 2)

learner, resampling, measure,terminator, ss)


rr = resample(task, at, outer, store_models = TRUE)

Get evaluated HPcs and performances; and result. x_domain_*

store_benchmark_result = TRUE to store resampled cols contain HP values after trafo (if any). as.data.table(rr)

evals and store_models = TRUE for fitted models. ## > learner resampling iteration

learner$param_set$values =
## > 1: <AutoTuner[37]> <ResamplingCV[19]> 1

Example instance$result_learner_param_vals ## > 2: <AutoTuner[37]> <ResamplingCV[19]> 2

# optimize HPs of RBF SVM on logscale

Set optimal HPC in Learner.


[NB: In many table prints we suppres cols for readability.] learner = lrn("classif.svm", kernel = "radial", type = "C-classification")
extract_inner_tuning_results(rr)

ss = ps(cost = p_dbl(1e-4, 1e4, logscale = TRUE),


# > iteration cost gamma classif.ce learner_param_vals x_domain

ParamSet - Parameters and Ranges


gamma = p_dbl(1e-4, 1e4, logscale = TRUE))
Example # > 1: 1 1.222198 -0.4974749 0.08 <list[4]> <list[2]>

evals = trm("evals", n_evals = 20)


# > 2: 2 2.616557 -3.1440039 0.08 <list[4]> <list[2]>
instance = TuningInstanceSingleCrit$new(task, learner, resampling, measure, evals, learner = lrn("classif.svm", type = "C-classification", kernel = "radial",

ss)

Scalar doubles, integers, factors or logicals are combined to cost = to_tune(1e-4, 1e4, logscale = TRUE),

tuner = tnr("random_search")
Check inner tuning results for stable HPs.
define a multivariate search space (SS). gamma = to_tune(1e-4, 1e4, logscale = TRUE))

tuner$optimize(instance)
instance = tune(method = "grid_search", task = tsk("iris"), learner = learner,

instance$result
resampling = rsmp ("holdout"), measure = msr("classif.ce"), resolution = 5) rr$score()

ss = ps(

# > cost gamma learner_param_vals x_domain classif.ce


# > learner iteration prediction classif.ce

<id> = p_int(lower, upper),

# > 1: 5.852743 -7.281365 <list[4]> <list[2]> 0.04 # > 1: <AutoTuner[40]> 1 <PredictionClassif[19]> 0.05333333

<id> = p_dbl(lower, upper),


Use tune()-shortcut.
# > 2: <AutoTuner[40]> 2 <PredictionClassif[19]> 0.02666667
<id> = p_dct(levels),

<id> = p_lgl()) Use TuningInstanceMultiCrit for multi-criteria tuning.


AutoTuner - Tune before Train Predictive performances estimated on the outer resampling.
id is identifier. lower/upper ranges, levels categories. Tuner - Search Strategy Wraps learner and performs integrated tuning. extract_inner_tuning_archives(rr)

# > iteration cost gamma classif.ce runtime resample_result

learner = lrn("classif.rpart",
Generates HPCs and passes to tuning instance for evaluation at = AutoTuner$new(learner, resampling, measure,
# > 1: 1 -7.4572 4.1506 0.68 0.013 <ResampleResult[20]>

cp = to_tune(0.001, 0.1, logscale = TRUE))


until termination. Creation: tnr(.key, ...) terminator, tuner) # > 21: 2 1.0056 0.4003 0.12 0.014 <ResampleResult[20]>
learner$param_set$search_space() # for inspection
grid_search (resolution, batch_size)
Grid search. Inherits from class Learner. Training starts tuning on the All evaluated HP configurations.
Or, use to_tune() to set SS for each param in Learner. SS is training set. After completion the learner is trained with the
random_search (batch_size)

auto-generated when learner is tuned. Params can be arbitrarily “optimal” configuration on the given task.
transformed by setting a global trafo in SS, or p_* shortforms, Random search. rr$aggregate()

logscale = TRUE is short for most common choice. design_points (design)


#> classif.ce

Search at predefined points. at$train(task)


#> 0.04
random_search (batch_size)
at$predict(task, row_ids)
Terminators - When to stop Random search. Aggregates performances of outer resampling iterations.
nloptr (algorithm)
at$learner
Construction: trm(.key, ...) Non-linear optimization. rr = tune_nested(method = "grid_search", task,

gensa (smooth, temperature)


learner, inner, outer, measure, term_evals = 20)
evals (n_evals)
Returns tuned learner trained on full data set.
Generalized Simulated Annealing.
After iterations. irace

run_time (secs)
at$tuning_result
Use shortcut to execute nested resampling.
Iterated racing.
# > cost gamma learner_param_vals x_domain classif.ce

After training time.


# > 1: 5.270814 -4.414869 <list[4]> <list[2]> 0.08
clock_time (stop_time)
as.data.table(mlr_tuners) # list all
At given timepoint.
perf_reached (level)
Access tuning result.
After performance was reached. Logging and Parallelization
stagnation (iters, threshold)
at = auto_tuner(method = "grid_search", learner,

After performance stagnated. resampling, measure, term_evals = 20)


lgr::get_logger("bbotk")$set_threshold("<level>")
combo (list_of_terms, any=TRUE)

Combine terminators with AND or OR. Use shortcut to create AutoTuner.


Change log-level only for mlr3tuning.
as.data.table(mlr_terminators) # list all
future::plan(strategy)

Sets the parallelization backend. Speeds up tuning by running


iterations in parallel.
mlr-org.com, cheatsheets.mlr-org.com

You might also like