Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

Note that this repository only contains the implementation of the search spaces, evaluation pipelines, and experiment scripts. The main implementation (e.g., how to construct architectures etc.) of our approach is implemented as part of the NePS project.

This repository contains the implementation of the experiments of our NeurIPS 2023 paper "Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars" that takes a functional view on neural architecture search by constructing architectures based on context-free grammars.

If you would like to learn more about our work, please refer to our paper. If you find our approach interesting for your own work, please cite the paper:

@inproceedings{schrodi2023construction,
    title={Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars},
    author={Schrodi, Simon and Stoll, Danny and Ru, Binxin and Sukthanker, Rhea and Brox, Thomas and Hutter, Frank},
    booktitle={Advances in Neural Information Processing Systems},
    year={2023},
}

A short form of this work was also previously presented in the NeurIPS2022 Meta-Learning Workshop with the title "Towards Discovering Neural Architectures from Scratch".

1. Installation

1. Clone this repository.

2. Create a conda environment

conda create -n hnas python=3.7

and activate it

conda activate hnas

3. Install poetry

bash install_dev_utils/poetry.sh

4. Run poetry install (this can take quite a while) and then run pip install opencv-python.

2. Reproducing the paper results

2.1 Search on the cell-based or hierarchical NAS-Bench-201 search space

To reproduce those search experiments, run

python experiments/optimize.py \
--working_directory $working_directory \
--data_path $data_path \
--search_space $search_space \
--objective $objective \
--searcher $searcher \
--surrogate_model $surrogate_model \
--seed $seed \
--pool_strategy evolution \
--pool_size 200 \
--n_init 10 \
--log \
--p_self_crossover 0.5

where $working_directory and $data_path are the directory you want to save to or the path to the dataset, respectively. The other variables can be set as follows:

variable	options
search_space	nb201_variable_multi_multi (hierarchical) or nb201_fixed_1_none (cell-based)
objective	nb201_cifar10, nb201_cifar100, nb201_ImageNet16-120, nb201_cifarTile, or nb201_addNIST
searcher	bayesian_optimization, random search, regularized_evolution, or àssisted_regularized_evolution
surrogate_model	gpwl_hierarchical (hWL), gpwl (WL), or gp_nasbot (NASBOT) (only active if searcher is set to bayesian_optimization)
seed	777, 888, 999

To run DARTS (or improved versions of DARTS) on the cell-based NAS-Bench-201 search space, run

python darts_train_search.py \
    --working_directory $working_directory \
    --data_path $data_path \
    --objective $objective \
    --seed $seed \
    --method $method

where working_directory and data_path are the directory you want to save to or the path to the dataset, respectively. The other variables can be set as follows:

variable	options
objective	nb201_cifar10, nb201_cifar100, nb201_ImageNet16-120, nb201_cifarTile, or nb201_addNIST
seed	777, 888, 999
method	darts, dirichlet
Note that add the --progressive flag to the above command to run DrNAS with progressive learning scheme.

To evaluate the found architectures, run

python $WORKDIR/hierarchical_nas_experiments/darts_evaluate.py \
--working_directory $working_directory \
--data_path $data_path \
--objective $objective

where working_directory and data_path are the directory you saved the data to or the path to the dataset, respectively. The other variable can be set as follows:

variable	options
objective	nb201_cifar10, nb201_cifar100, nb201_ImageNet16-120, nb201_cifarTile, or nb201_addNIST

Note that DARTS (or improved versions of it) cannot be applied with further adoption to our hierarchical NAS-Bench-201 search space due to the exponential number of parameters that the supernet would contain.

2.2 Search on the activation function search space

To reproduce this search experiment, run

python experiments/optimize.py \
--working_directory $working_directory \
--data_path $data_path \
--search_space act_cifar10 \
--objective act_cifar10 \
--searcher $searcher \
--surrogate_model $surrogate_model \
--seed $seed \
--pool_strategy evolution \
--pool_size 200 \
--n_init 50 \
--log \
--p_self_crossover 0.5 \
--max_evaluations_total 1000

where $working_directory and $data_path are the directory you want to save to or the path to the dataset, respectively. The other variables can be set as follows:

variable	options
searcher	bayesian_optimization, random search, or regularized_evolution
surrogate_model	gpwl_hierarchical (hWL), gpwl (WL), or gp_nasbot (NASBOT) (only active if searcher is set to bayesian_optimization)
seed	777, 888, 999 (note that we only ran on the seed 777 in our experiments)

2.3 Surrogate experiments

Search has to be run beforehand or data needs to be provided!

To reproduce our surrogate experiments, run

python experiments/surrogate_regression.py \
--working_directory $working_directory \
--search_space $search_space \
--objective $objective \
--surrogate_model $surrogate_model \
--n_train $n_train \
--log

where $working_directory is the directory where the data from the search runs has been saved to and the surrogate results will be saved to. Other variables can be set as follows:

variable	options
search_space	nb201_variable_multi_multi (hierarchical) or nb201_fixed_1_none (cell-based)
objective	nb201_cifar10, nb201_cifar100, nb201_ImageNet16-120, nb201_cifarTile, or nb201_addNIST
surrogate_model	gpwl_hierarchical (hWL), gpwl (WL), or nasbot (NASBOT) (only active if searcher is set to bayesian_optimization)
n_train	10, 25, 50, 75, 100, 150, 200, 300, or 400

2.4 Zero-cost proxy experiments

Search has to be run beforehand or data needs to be provided!

To reproduce our zero-cost proxy rank correlation experiments, run

python experiments/zero_cost_proxy_rank_correlation.py \
--working_directory $working_directory \
--search_space $search_space \
--objective $objective \
--data_path $data_path \
--log

where $working_directory and $data_path are the directory you want to save to and the data from the search runs has been saved to or the path to the dataset, respectively. Other variables can be set as follows:

variable	options
search_space	nb201_variable_multi_multi (hierarchical) or nb201_fixed_1_none (cell-based)
objective	nb201_cifar10, nb201_cifar100, nb201_ImageNet16-120, nb201_cifarTile, or nb201_addNIST

2.5 NASWOT search experiment

To reproduce the NASWOT search experiment, run

python experiments/optimize_naswot.py \
--working_directory $working_directory \
--search_space $search_space \
--objective $objective \
--data_path $data_path \
--seed $seed \
--naslib

where $working_directory and $data_path are the directory you want to save to or the path to the dataset, respectively. The other variables can be set as follows:

variable	options
search_space	nb201_variable_multi_multi (hierarchical) or nb201_fixed_1_none (cell-based)
objective	nb201_cifar10, nb201_cifar100, nb201_ImageNet16-120, nb201_cifarTile, or nb201_addNIST
seed	777, 888, 999

2.6 DARTS search experiment

Code is coming soon

2.7 Transformer search experiment

Code is coming soon

3. Acknowledgements

We thank the authors of following works for open sourcing their code:

• NAS-BOWL: GPWL surrogate model base implementation, NASBOT's graph encoding scheme
• NASLib: base graph class, zero-cost proxies
• NAS-Bench-201: training protocols of NAS-Bench-201 search space
• CVPR-NAS 2021 Competition Track 3: dataset generation and training protocols for AddNIST and CIFARTile
• NASWOT: implementation of zero-cost proxy search
• DARTS, DrNAS: implementation of DARTS training pipeline and DARTS (+ improved versions) search algorithms