Estimating Latent Regularization Parameters in Ill-Posed Problems with Semi-Definite Programming

Kritika Prakash, Sanjay Krishnan

Keywords: Inverse regularization, Tikhonov regularization, Semi-definite programming, Machine Learning, Hyperparameter estimation, precision matrix recovery

TL;DR: We propose an SDP-based solution for inverse regularization problem that recovers the prior mean and Tikhonov precision for a trained model from it’s observed optima, outperforming Bayesian & solver-agnostic baselines across diverse datasets.

Abstract: Tikhonov smoothing is often used in estimation problems in ill-posed settings. In a variety of applications ranging from human-computer interaction to model explainability, it is important to retroactively estimate smoothing parameters from an already trained model. We introduce an inverse regularization problem - one that infers latent smoothing hyperparameters obtained from a trained model and its dataset; and show a fast and effective solution using semi-definite programming. The algorithm directly exploits the stationarity conditions of Tikhonov models to jointly recover the model parameter's prior mean $\mu$ and Tikhonov precision matrix $T$ from observed optimum $\theta^*$. Our method formulates this as a multi-constraint least squares problem, providing a novel and interpretable approach. Empirically, our results show that our solution outperforms Bayesian approaches and solver-agnostic baselines on diverse benchmarks including diabetes, lung cancer, and California housing datasets.

Primary Area: optimization

Submission Number: 23250