Accuracy Boosters: Epoch-Driven Mixed-Mantissa Block Floating-Point for DNN Training

Simla Burcu Harma, Canberk Sönmez, Babak Falsafi, Martin Jaggi, Yunho Oh

Keywords: DNN training, low-precision training, mixed-precision training, efficient training, number formats, numerical encodings, block floating-point, DNN accelerators

TL;DR: We propose an epoch-driven mixed-mantissa Hybrid Block Floating-Point training method converting 99.7% of arithmetic operations in training to 4-bit mantissas and using 6-bit mantissas in the last epoch, while preserving/outperforming FP32 accuracy.

Abstract: The unprecedented growth in DNN model complexity, size and the amount of training data have led to a commensurate increase in demand for computing and a search for minimal encoding. Recent research advocates Hybrid Block Floating-Point (HBFP) as a technique that minimizes silicon provisioning in accelerators by converting the majority of arithmetic operations in training to 8-bit fixed-point. In this paper, we perform a full-scale exploration of the HBFP design space including minimal mantissa encoding, varying block sizes, and mixed mantissa bit-width across layers and epochs. We propose Accuracy Boosters, an epoch-driven mixed-mantissa HBFP that uses 6-bit mantissa only in the last epoch and converts 99.7% of all arithmetic operations in training to 4-bit mantissas. Accuracy Boosters enable reducing silicon provisioning for an HBFP training accelerator by 16.98× as compared to FP32, while preserving or outperforming FP32 accuracy.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: General Machine Learning (ie none of the above)