SEEKING THE SEARCH SPACE FOR SIZE-AWARE VISION TRANSFORMER ARCHITECTURE

Chih-Han Tsou, Yu-Chen Lin, Yi Cheng Huang, WeiHua Li, Jun-Cheng Chen, Chu-Song Chen

Keywords: Neural Architecture Search, Vision Transformer, Size-aware

Abstract: Recently, vision transformer methods have gained significant attention due to their superior performance in various tasks, whereas their architectures still highly rely on manual design. Although neural architecture search (NAS) has been introduced to automate the process, it still requires humans to manually specify a fixed search space. Even allowing search space updates, existing methods tend to lose control of model size and result in large and complex models for satisfactory performance. To address these issues, we introduce a constrained optimization framework to Seeking the Search Space for Size-aware transformer architecture, named S4, which allows the search space to evolve to neighbor search space under user-specified constraints (e.g., model size, FLOPS, etc.). With extensive experiments on various benchmarks, including Cifar10, Cifar100, Tiny ImageNet, and SUN397, the results demonstrate that S4 can consistently find architectures that align with model size expectations while achieving better performance than those searched by the original search space or with larger size from compared NAS methods. Moreover, we demonstrate the plug-and-play characteristic of S4 by finding effective yet lightweight adapters for well-recognized foundation models (such as CLIP), achieving excellent performance for downstream tasks.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7492