Retiarii Roadmap

[QuanluZhang]

Retiarii is a new NAS (exploratory-training) framework. We have the following goals when implementing it on NNI:

• Our basic design philosophy of model APIs is "writing DNN model as a space". That is why we have provided APIs like LayerChoice, InputChoice, ValueChoice, which can be directly embedded in a traditional PyTorch/TensorFlow model.
• High expressiveness. Any model space can be expressed with our provided APIs (e.g., mutators)
• High effectiveness and reusability of model search algorithms.
• Support more system-level optimizations to speedup the search, deduplicate computation as much as possible.

Guided by the abovementioned goals, we draft the following roadmap:

V2.1

• Improved the robustness of graph generation and code generation for PyTorch models
• Supported the inline mutation API ValueChoice
• Improved the design and implementation of Model Evaluator
• Supported Random/Grid/Evolution exploration strategies (i.e., search algorithms)

V2.2

• Support debug mode
• More high-level APIs for users to easily express model space
• Migrate more one-shot trainer from NAS v1 to Retiarii framework (i.e., NAS v2)

V2.3

• Deprecate NAS v1
• Support high-level APIs ValueChoice, Repeat, Cell
• Support pure-python execution engine
• Support RL exploration strategy

V2.4

• Initial support of TensorFlow
• Support more-friendly debug mode
• Support super-graph weight sharing
• Support a fancy NAS algorithm which could leverage architectural information
• Migrate search space zoo to NAS 2.0
• Migrate Cream to NAS 2.0
• Adapt NAS benchmarks to NAS 2.0
• Support one system-level optimization: CSE optimization to deduplicate computation
• Further improve the logic of graph generation and code generation
  • Support control flow in graph/code generation
  • Further integrated with TorchScript

V2.5

• Support BrainTrust which directly proposes good candidates for user-defined model search space, or as initial trials.
• Support fancy search algorithms which could jointly optimize architectures and hyper-parameters

[ultmaster]

As of 2022, this roadmap is outdated, we have another one here.