Finite-Sample Calibration for Low-Rank Adapter Pruning

Keywords: low-rank adaptation, adapter pruning, finite-sample calibration, spiked random matrices, BBP transition, Le Cam bounds, chi-square divergence, spectral thresholding, singular-value diagnostics, parameter-efficient fine-tuning

TL;DR: We derive exact finite-sample calibration tools for deciding which low-rank adapter components are distinguishable from noise, separating information-theoretic detectability from calibrated singular-value pruning rules.

Abstract: We derive the chi-square divergence between a rank-one Gaussian spike with spherical Haar prior and the noise-only law on $\mathbb{R}^{m\times n}$ exactly, as a convergent series in the even zonal moments of the uniform distribution on $S^{d-1}$. The formula is finite-dimensional and entire in the signal-to-noise parameter $\lambda=\theta^2/\sigma^2$; a geometric tail certificate makes it numerically computable at any $(m,n)$. The rectangular BBP subcritical limit $(1-c^4)^{-1/2}-1$ follows under joint dimension-signal scaling through the central-binomial generating function. Because the Haar prior averages over all directions, the chi-square expression yields a Le Cam minimax lower bound, obtained by applying Cauchy--Schwarz to the centered likelihood ratio, at the finite dimensions of a given adapter layer. A compact-manifold Laplace expansion on $S^{m-1}\times S^{n-1}$ shows that the full mixture likelihood ratio and $s_1$ agree only at leading exponential order: finite-$(m,n)$ thresholds also depend on the spectral-gap factor $s_1^{|m-n|}\prod_{i\geq2}(s_1^2-s_i^2)$. The resulting findings give practitioners explicit false-positive rates for pruning decisions and give researchers a finite-sample bridge between Le Cam certificates, BBP limits, and calibrated spectral diagnostics.

Submission Number: 61