What Is Hyperparameter Optimization?
Two types of parameters govern the operation of machine learning (ML) models, model
parameters and hyperparameters. Model parameters are parameters the model has
learned during training, such as weights and biases in a neural network.
Hyperparameters are settings and regulations that are set by the developer before
training to control how the algorithm applies the model to the data. When a model
is first being created, the best combinations and settings for the hyperparameters
are unknown. This creates a need for hyperparameter optimization.
Hyperparameter optimization is the process of finding the best set of
hyperparameter values for a given data set and problem. It begins by identifying
the search space, which is a theoretical space where each hyperparameter to be
optimized acts as a dimension; each point in this space represents a possible
configuration of the model. There are several methods developers use to search
through these configurations and identify the best candidates. H2O.ai provides
defaults for many data sets to act as a starting point for additional optimization.
H2O.ai assists clients with this process by grouping the hyperparameters by their
importance to the Flow UI. All users should look closely at those labeled as
critical, while the others may only be necessary in special cases.
Hyperparameter Optimization Methods
Manual Search
Manual search is the traditional method of hyperparameter optimization. Users
manually select several settings for each hyperparameter and then compare the
performance of the model with each combination of those selected settings. For
instance, users might select ntrees settings of 50, 100, and 200, and max_depth
settings of 5, 10, 15, and 20. This would give users twelve combinations and twelve
models to compare. However, this method becomes less practical when there are many
hyperparameters and settings to compare.
Grid Search
A grid search is similar to a manual search. It involves determining a set of
values for each hyperparameter and evaluating the model with each possible
combination of these values. It is less tedious than manual search but still
involves the developers guessing at which values may be optimal.
Random Search
The random search method involves the system randomly selecting the values for each
hyperparameter and then running the model to see how it performs. H2O.ai
specializes in random search optimization and has proven to yield models that are
equal or better than those generated by grid and manual optimization in a shorter
timespan. In addition, H2O.ai allows users to compare the various models generated
and then narrow the search parameters to those areas which look promising allowing
developers to refine their search and be more efficient.
Bayesian Optimization
Bayesian optimization involves running sets of hyperparameters then using the
results to improve the hyperparameter sets for the next search. Bayesian
optimization is more efficient than other methods and can reduce optimization time
and improve results.
Why Does Hyperparameter Optimization Matter?
Hyperparameter optimization is essential to machine learning and AI systems because
it allows developers to identify the best configurations of a model for a given
situation. It helps machine learning models generalize, which is the ability to
operate just as effectively whether encountering training data or new data. This
leads to better performance of the model and better results for users.
