SpeedLimit: Neural Architecture Search for Quantized Transformer Models

Yuji Chai, Luke Bailey, Yunho Jin, Glenn Ko, Matthew Karle, David Brooks, Gu-Yeon Wei, H. Kung

Keywords: Quantization, Transformers, Neural Architecture Search

TL;DR: We combine quantization and neural architecture search to produce performant transformer models that adhere to upper-bound latency constraints.

Abstract:

While research in the field of transformer models has primarily focused on enhancing performance metrics such as accuracy
and perplexity, practical applications in industry often necessitate a rigorous consideration of inference latency constraints. Addressing this challenge, we introduce SpeedLimit, a novel Neural Architecture Search (NAS) technique that optimizes accuracy whilst adhering to an upper-bound latency constraint. Our method incorporates 8-bit integer quantization in the search process to outperform the current state-of-the-art technique. Our results underline the feasibility and efficacy of seeking an optimal balance between performance and latency, providing new avenues for deploying state-of-the-art transformer models in latency-sensitive environments.

Submission Number: 56