Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

TMLR Paper7572 Authors

Abstract: We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression.

Submission Type: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=IlPgaSwMJV&nesting=2&sort=date-desc

Changes Since Last Submission: A previous submission was desk rejected due to modified fonts. This has been resolved by removing a `times` import. The above link might not work since desk rejection doesn't generate a public shareable link.

Assigned Action Editor: ~Marco_Mondelli1

Submission Number: 7572