Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems

Giung Nam, Juho Lee

Keywords: ensemble methods, low precision number systems

TL;DR: We present a novel insight on ensemble within low precision number systems, especially for large models.

Abstract: While ensembling deep neural networks has shown promise in improving generalization performance, scaling current ensemble methods for large models remains challenging. Given that recent progress in deep learning is largely driven by the scale, exemplified by the widespread adoption of large-scale neural network architectures, scalability emerges an increasingly critical issue for machine learning algorithms in the era of large-scale models. In this work, we first showcase the potential of low precision ensembling, where ensemble members are derived from a single model within low precision number systems in a training-free manner. Our empirical analysis demonstrates the effectiveness of our proposed low precision ensembling method compared to existing ensemble approaches.

Primary Area: Other (please use sparingly, only use the keyword field for more details)

Submission Number: 19991