How Low Can We Go: Trading Memory for Error in Low-Precision TrainingDownload PDF

29 Sept 2021, 00:31 (edited 16 Mar 2022)ICLR 2022 PosterReaders: Everyone
  • Keywords: low-precision training, meta-learning, Pareto frontier, error-memory tradeoff, active learning, matrix factorization
  • Abstract: Low-precision arithmetic trains deep learning models using less energy, less memory and less time. However, we pay a price for the savings: lower precision may yield larger round-off error and hence larger prediction error. As applications proliferate, users must choose which precision to use to train a new model, and chip manufacturers must decide which precisions to manufacture. We view these precision choices as a hyperparameter tuning problem, and borrow ideas from meta-learning to learn the tradeoff between memory and error. In this paper, we introduce Pareto Estimation to Pick the Perfect Precision (PEPPP). We use matrix factorization to find non-dominated configurations (the Pareto frontier) with a limited number of network evaluations. For any given memory budget, the precision that minimizes error is a point on this frontier. Practitioners can use the frontier to trade memory for error and choose the best precision for their goals.
  • One-sentence Summary: Given a dataset and a memory budget, we use matrix factorization and active learning to efficiently pick the perfect low-precision configuration for a neural network.
12 Replies