Single Train Multi Deploy on Topology Search Spaces using Kshot-Hypernet

TMLR Paper5554 Authors

Abstract:

Neural Architecture Search (NAS) has become a crucial research direction for automating the design of neural networks.
The introduction of weight sharing has significantly reduced the computational and time costs of NAS.
Recent approaches enable the simultaneous training of numerous sub-networks without the need for retraining; however, these methods are primarily limited to the Size Search Space (SSS), which provides limited architecture diversity.
To date, methods based on the more diverse Topology Search Space (TSS) remain unexplored. TSS has greater potential for hardware-aware architecture search.
In this work, we propose a novel NAS method that operates on TSS, while maintainting high efficiency.
To do so, we introduce Kshot-Hypernet, that extends in-place distillation to TSS, significantly improving supernetwork training.
Experiments on NASBench-201 show that, once the supernet is trained, most sub-networks can match or even exceed the performance of those trained from scratch.
Furthermore, our method achieves 80.7% top-1 accuracy on ImageNet with only 8.7M parameters.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

1. Corrected figure and text errors
2. Added description of the work scope in Section 1
3. Updated table 1 and respective main text
4. Updated table 5 and added more text to Section 4.2

Assigned Action Editor: ~Yunhe_Wang1

Submission Number: 5554