HyperAI Hyperparameter Tuning Introduction

Building machine learning models requires many hyperparameters, and the choice of hyperparameters greatly affects the performance of the final model. Automatic hyperparameter tuning refers to methods for systematically obtaining the optimal set of hyperparameters.

Automatic hyperparameter tuning currently supports creation through both the interface and commands. To create through the command line, please first install the bayes command-line tool and understand its basic usage.

To use automatic hyperparameter tuning, you need to do the following:

1. Complete the hyperparameter tuning configuration, which mainly includes two parts:
   • Determine which hyperparameters need to be automatically tuned, and specify to the system which hyperparameters need to be tuned and their tuning ranges according to the automatic tuning configuration documentation
   • Determine the required key metrics, and submit them to the HyperAI system through openbayestool during model training
2. Modify existing code to support reading the hyperparameters generated by the automatic tuning system

Quick Try

openbayes-hypertuning-example is a sample code repository. The repository currently includes examples for both TensorFlow and PyTorch versions, which can be found in the /tensorflow and /pytorch directories respectively. Each directory contains a complete openbayes.yaml and corresponding supporting code. Following the steps within, you can run a hyperparameter tuning example.

Modify Your Own Code to Support Automatic Hyperparameter Tuning

Obtaining Parameters Generated by the Hyperparameter Tuning Service

Tasks launched by automatic hyperparameter tuning can obtain the parameters it provides in two ways:

1. Read openbayes_params.json: The above parameter_specs will generate four parameters, and an openbayes_params.json file containing these contents will appear in the container's working directory:

   {
     "regularization": 6.933098216541059,
     "latent_factors": 29,
     "unobs_weight": 3.439490399012204,
     "feature_wt_factor": 102.9461653166788
   }

   Simply read the parameters from this file and use them before code execution

2. Read command-line arguments: Parameters for tasks created through automatic hyperparameter tuning will be added to the execution command in the following format:

   python main.py \
       --regularization=6.933 \
       --latent_factors=29 \
       --unobs_weight=3.439 \
       --feature_wt_factor=102.9

   You can use argparse to parse and use these parameters.

Reporting Key Metrics

In the System Metrics and Custom Metrics section, it was introduced that the openbayestool tool has a log_metric method to report custom metrics. Hyperparameter tuning also uses this tool to report metrics. At the end of the program, reporting can be achieved through the following code:

import openbayestool

openbayestool.log_metric("precision", 0.809)

Note that the specific metric name reported here should be consistent with the hyperparameter_metric configured in openbayes.yaml.

Creating Hyperparameter Tuning

Creating via Interface

After logging into the console, you can create from "Hyperparameter Tuning" in the sidebar, or from "Create Hyperparameter Tuning" in the upper right corner of the list.

Currently we support two ways to configure hyperparameter tuning parameters: visual editor and YAML editor. Edits made in either editor will be synchronized and updated. This means when users update parameters in the visual editor and switch to the YAML editor, the configuration will be synchronized, and vice versa.

Creating via Command Line Tool

:::caution Note When creating hyperparameter tuning through the command line tool, you need to be familiar with the bayes command line tool. Please refer to bayes Command Line Tool Getting Started, and you also need to understand the openbayes configuration file. :::

After preparing the hyperparameter tuning configuration file, create a hyperparameter tuning task through the command bayes gear run hypertuning:

Viewing Hyperparameter Tuning

As shown above, the hyperparameter tuning page displays the current execution status through "Charts" and "Parallel Coordinates". Users can select the best execution as the result based on the results, or update the hyperparameter tuning parameter ranges based on current results and continue with new hyperparameter tuning tasks. Among them, "Parallel Coordinates" is a very suitable view for filtering hyperparameter tuning results. By defining parameter ranges on the interface, you can control the displayed parameter combinations.

Hyperparameter Tuning Configuration Description

In the command line, bind the current directory to a container through bayes gear init <container-name> (related documentation). After executing this command, the file openbayes.yaml will appear in the directory. In openbayes.yaml, specify hyperparameters and their ranges according to configuration requirements.

An openbayes.yaml document is divided into two parts:

1. Basic configuration, which is introduced in the Configuration File section
2. Hyperparameter tuning configuration, only configurations related to hyperparameter tuning under the hyper_tuning parameter

The default generated openbayes.yaml contains the following sample hyperparameter tuning configuration section:

hyper_tuning:
  max_job_count: 3
  hyperparameter_metric: precision
  side_metrics: []
  goal: MINIMIZE
  algorithm: Bayesian
  parameter_specs:
    - name: regularization
      type: DOUBLE
      min_value: 0.001
      max_value: 10.0
      scale_type: UNIT_LOG_SCALE
    - name: latent_factors
      type: INTEGER
      min_value: 5
      max_value: 50
      scale_type: UNIT_LINEAR_SCALE
    - name: unobs_weight
      type: DOUBLE
      min_value: 0.001
      max_value: 5.0
      scale_type: UNIT_LOG_SCALE
    - name: feature_wt_factor
      type: DOUBLE
      min_value: 1
      max_value: 200
      scale_type: UNIT_LOG_SCALE
    - name: level
      type: DISCRETE
      discrete_values: [1, 2, 3, 4]
    - name: category
      type: CATEGORICAL
      categorical_values: ['A', 'B', 'C']

The parameters under hyper_tuning are the configurations needed for automatic hyperparameter tuning:

1. max_job_count

The number of attempts for one automatic tuning session, supporting up to 100 attempts maximum.

2. parallel_count

The number of tasks executed in parallel. This value is limited by the user's maximum parallel count for a single resource type (see Parallel Quota for Computing Resources). The higher the parallel count, the faster one automatic tuning task will execute.

3. hyperparameter_metric

The key metric that needs to be obtained, which will be used in the Report Key Metrics section below.

4. goal

The direction of the optimal solution for the key metric to be obtained (MAXIMIZE or MINIMIZE).

5. parameter_specs

The specification of hyperparameters, which defines the types and ranges of hyperparameters that can be set. The hyperparameter types can be one of the following four:

For INTEGER and DOUBLE types, the distribution type for parameter sampling (scale_type) can also be set:

6. algorithm

Algorithms that can be used for automatic hyperparameter tuning, supporting Bayesian, Random, or Grid:

7. parameters

:::caution Note This parameter does not appear under the hyper_tuning configuration, but appears at the root level, as introduced in the config configuration section. Providing this field when creating an "automatic hyperparameter tuning task" will still take effect. However, the priority of the parameter configuration provided here is lower than the configuration generated during the automatic tuning process. This means that if there are parameters in this configuration section with the same name as those under parameter_specs, that parameter will be overridden by the parameter under parameter_specs. :::

For example, we configured an openbayes.yaml as follows:

...other parts omitted...
parameters:
  filter_size: 5
hyper_tuning:
  parameter_specs:
    - name: filter_size
      type: DISCRETE
      discrete_values: [1, 2, 3, 4]
...other parts omitted...

Then each parameter combination generated by automatic tuning will definitely have the filter_size field, and its generated value will override the value of the filter_size field under parameters.

8. side_metrics

As mentioned above, hyperparameter_metric is the key metric used for automatic hyperparameter tuning. In addition, multiple side_metrics can be set as auxiliary metrics for the final metric. For example, relative to accuracy, the f1_score metric or the loss during training can be added as side_metrics for reference. This metric has no impact on the automatic tuning process and is only displayed in the metric visualization as a reference for the quality of automatic tuning results.

As shown in the figure above, the reference metric loss for side_metrics will also be displayed. However, note that during the code execution, you must also use the openbayestool.log_metric method to record the metric, otherwise it cannot be displayed in the visualization. You can refer to the handling of the loss metric in openbayes hypertuning in pytorch.

Status Explanations for Hyperparameter Tuning

During the execution of hyperparameter tuning tasks, the following statuses may appear. Here are explanations for each status:

1. Running: The execution under hyperparameter tuning is running normally
2. Canceled: The hyperparameter tuning task was closed due to active termination by the user
3. No corresponding metrics: Indicates that the code for metric reporting was not provided in the execution under hyperparameter tuning, and the hyperparameter tuning system terminated the tuning process after discovering this issue
4. Execution error: Indicates that the execution under hyperparameter tuning encountered an error or was terminated for some reason (possibly a single execution was closed, such as being shut down by the system due to insufficient funds)
5. System error: Indicates that the hyperparameter tuning system itself encountered some issues that caused the tuning to be terminated. In this case, you can contact the system administrator to report the problem
6. Search space exhausted: This result can occur in two situations: one is when the hyperparameter tuning system determines that the parameter combinations generated from the hyperparameter ranges provided by the user have been exhausted and any subsequently generated parameters would be duplicates; the other is when the system determines that there is currently no way to produce better results and proactively terminates the search