Binarized Neural Machine Translation

Yichi Zhang, Ankush Garg, Yuan Cao, Lukasz Lew, Behrooz Ghorbani, Zhiru Zhang, Orhan Firat

Keywords: neural network quantization, binarized transformer, machine translation, scaling law

TL;DR: We propose a novel binarization technique for Transformers applied to machine translation and show that one-bit Transformers scale and generalize well in both in-domain and out-of-domain setting.

Abstract:

The rapid scaling of language models is motivating research using low-bitwidth quantization.
In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind. We identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16$\times$ smaller in size. One-bit activations incur varying degrees of quality drop, but mitigated by the proposed architectural changes. We further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and out-of-domain settings. Implementation in JAX/Flax will be open sourced.

Submission Number: 9163