Quantization and the Bottom of the Loss Landscape

Luca Di Carlo, Daniel T. Bernstein, David J. Schwab

Keywords: model quantization, compression, loss landscape geometry, deep learning

Abstract:

We introduce two physics-inspired methods for the compression of neural networks that encourage
weight clustering, in anticipation of model quantization, by adding attractive interactions between
parameters to the loss. Our two methods implement interactions either directly or via an intermediary
set of centroids. By applying these methods to pre-trained neural networks, we investigate the
existence of compressible configurations near the bottom of the loss landscape. The direct interaction
approach suggests the existence of multiple, qualitatively distinct compressed configurations close
to pre-trained models, and the centroid-mediated approach provides a pipeline for quantization that
is competitive with extant quantization methods.

Student Paper: Yes

Submission Number: 48