Accuracy Boosters: Epoch-Driven Mixed-Mantissa Block Floating Point for DNN Training

Simla Burcu Harma, Ayan Chakraborty, Babak Falsafi, Martin Jaggi, Yunho Oh

Keywords: DNN training, low-precision training, mixed-precision training, efficient training, number formats, numerical encodings, block floating point, DNN accelerators

TL;DR: We propose an epoch-driven mixed-mantissa Hybrid Block Floating Point training method converting 99.7% of arithmetic operations in training to 4-bit mantissas and using 6-bit mantissas in the last epoch, while preserving/outperforming FP32 accuracy.

Abstract: The unprecedented growth in DNN model complexity, size, and amount of training data has led to a commensurate increase in demand for computing and a search for minimal encoding. Recent research advocates Hybrid Block Floating Point (HBFP) to minimize silicon provisioning in accelerators by converting the majority of arithmetic operations in training to 8-bit fixed point. In this paper, we perform a full-scale exploration of the HBFP design space using mathematical tools to study the interplay among various parameters and identify opportunities for even smaller encodings across layers and epochs. Based on our findings, we propose Accuracy Boosters, an epoch-driven mixed-mantissa HBFP technique that uses 6-bit mantissas only in the last epoch and first/last layers, and 4-bit mantissas for 99.7% of all other arithmetic operations in training. Using analytic models, we show Accuracy Boosters enable increasing arithmetic density for an HBFP training accelerator by up to 21.3x compared to FP32 and up to 4.4x compared to another SOTA format Bfloat16, while preserving or outperforming FP32 accuracy.

Workshop Track: MLArchSys

Presentation: In-Person

Presenter Full Name: Simla Burcu Harma

Presenter Email: [e-mail]

Presenter Bio:

I am a fourth-year Ph.D. student at EPFL advised by Prof. Babak Falsafi (EPFL) and Prof. Martin Jaggi (EPFL).
My research focuses on novel numerical formats for time- and energy-efficient Deep Neural Network (DNN) training and DNN accelerator design.