Complexity-optimized sparse Bayesian learning for scalable classification tasks

Jiahua Luo, Junyang Chen, Junyi Xiang, Chi-Man Wong, Chi-Man Vong

Abstract: Sparse Bayesian Learning (SBL) is a powerful framework for constructing highly sparse probabilistic models with strong generalization performance. However, its practical application is hindered by the computational complexity associated with inverting a large covariance matrix, which scales as OM3 (M: feature size) during the update of regularization priors. This limitation poses significant challenges for high-dimensional feature spaces or large-scale datasets, often leading to memory overflow issues. To address this problem, this paper proposes a novel diagonal Quasi-Newton (DQN) method for SBL, termed DQN-SBL, which eliminates the need for covariance matrix inversion, thereby reducing the computational complexity to OM. The proposed DQN-SBL is extensively evaluated on both non-linear and linear classification tasks using a diverse set of benchmarks of varying scales. Experimental results demonstrate that DQN-SBL achieves competitive generalization performance while maintaining a highly sparse model and exhibits robust scalability to large-scale problems.