Evidence Slopes and Effective Dimension in Singular Linear Models

Kalyaan Rao

Keywords: marginal likelihood evidence, singular learning theory, effective dimension, bayesian model selection, overparameterization

TL;DR: Laplace/BIC fails in singular models. In linear-Gaussian rank/subspace settings, the log-evidence bias scales as (d/2-λ)log n; using RLCT (λ=r/2) restores correct slopes and invariance.

Abstract: Bayesian model selection often uses Laplace’s approximation or BIC, which assume the complexity penalty in the log evidence is $(d/2)\log n$, where $d$ is the parameter dimension. Singular learning theory replaces $d/2$ with the real log canonical threshold (RLCT) $\lambda$, which can be strictly smaller in overparameterized low-rank models. We study linear–Gaussian rank and subspace/dictionary models where the marginal likelihood is available in closed form and $\lambda$ is analytically identifiable. We prove and empirically verify that Laplace/BIC incur a leading bias $((d/2)-\lambda)\log n$; in rank-$r$ regression, $\lambda=r/2$. In a dictionary model, the leading evidence term is invariant under overcomplete reparameterizations with the same span, while BIC is not. These results provide a clean finite-sample benchmark that makes “Laplace failure” in singular models explicit and motivates slope-based diagnostics for effective dimension.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Style Files: I have used the style files.

Submission Number: 30