Decoupled Differentiable Neural Architecture Search: Memory-Efficient Differentiable NAS via Disentangled Search Space
Keywords: Differentiable Neural Architecture Search, Decoupled Search Space, Memory-Efficient Algorithm
TL;DR: D2NAS proposes a method to reduce GPU memory usage in Differentiable NAS by decoupling weight updates, leading to significant optimization parameter reduction
Abstract: Differentiable Neural Architecture Search (NAS) is a popular paradigm, but scaling this approach to models with larger parameters is severely hampered by the fact that the entire supernet resides in GPU memory. In this paper, we rethink the gradient propagation processs of Differentiable NAS and propose Decoupled Differentiable Neural Architecture Search (D2NAS). In our method, the branch structure is designed to decouple the weight update of the trainable parameters from the backbone network, and the candidate operation selection is redesigned with Gumbel-Softmax to make the overall differentiable process more stable. Experiments show that D2NAS achieves both performance and stability, with 67\% memory cost compared to the best other differentiable methods.
Submission Number: 16
Loading