Our right to determine the maximum number of collisions.

[airvzxf]

Is your feature request related to a problem? Please describe.

I am a perfectionist person and every time I use Hyperband the number of possible combinations given the hyperparameters is different from the result I get, except when I use a reduced number of combinations of hyperparameters (less than 10 combinations). For example, for a combination of 5 hyperparameters that end up giving 48 combinations, I only get 41 trials. Another example is given 5 hyperparameters that get 120 combinations, I get 110 trials, but I want to get all 120 combinations or trials.

I understand that Hyperband is probability-based, so avoid running all possible combinations, as described in their official article “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”.

However, I am convinced that we as users have the right to determine the maximum number of collisions to randomly determine how many trials will be executed in the end.

Describe the solution you'd like

Add a parameter in Hyperband, that can modify the maximum number of collisions (max_collisions).

Hyperband(
    MyHyperModel(),
    objective=[Objective('val_accuracy', direction='max'),],
    seed=42,
    overwrite=True,
    directory=tuner_directory,
    project_name='PredictClassification',
    max_collisions=27,
)

Describe alternatives you've considered

I tried to use all the parameters of the Hyperband class, but none worked, since the trials' execution is directly related to the maximum number of collisions.

I tried using the Bayesian Optimization, and Random Search algorithms, but the one that best suits my needs is the Hyperband one, except that if I want to run the maximum number of trials, this doesn't allow it. Which has a solution if the feature I propose is implemented.

Additional context

I made these changes in a fork of the keras_tuner project, this works great. This is found in pull request #902 – Added the parameter for max collisions in the Hyperband tuner.

In my run, adding the seed=42 parameter to generate the same combinations for the random hyperparameter combination values, I tried changing max_collisions and these are the results I got:

5 Hyperparameter that generate 48 combinations.

2 * 2 * 2 * 3 * 2 = 48

It reads: Hyperparameter 1 with 2 combinations * Hyperparameter 2 with 2 combinations * Hyperparameter 3 with 2 combinations * Hyperparameter 4 with 3 combinations * Hyperparameter 5 with 2 combinations = Total of combinations.

Max collisions = 20 | default value
Got: 41 trails

Max collisions = 27
Got: 41 trails

Max collisions = 28
Got: 41 trails

Max collisions = 29
Got: 48 trails

Max collisions = 30
Got: 48 trails