Sparse Covariance Supervised Principal Component Analysis

TMLR Paper9399 Authors

Abstract: Principal component analysis (PCA) is one of the most well-studied machine learning methods of the last century. However, the principal components derived by PCA are not guaranteed to be response-informative and are usually dense, meaning they are hard to interpret in high dimensional settings. The former has led to the development of supervised PCA techniques where the response is usually incorporated in an objective function to guide informative projections and enhance predictive accuracy, while the latter to sparse PCA methods that seek to induce sparsity by shrinking non-significant variables to zero, effectively improving interpretability. Sparse supervised PCA methods seek to combine the two concepts as a means of simultaneous supervised dimensionality reduction and variable selection, but they usually depend on iteratively biconvex solutions of auxiliary objective functions, with no robust convergence guarantees and are sensitive to initialisation. In this paper, we propose a novel sparse supervised PCA method, sparse covariance supervised PCA (SCS-PCA), that seeks to trade-off prediction accuracy and sparsity performance. We impose an $L_1$ penalty on a supervised objective function and we employ manifold proximal gradient descent to solve the derived optimization problem, which guarantees global convergence to a stationary point. Numerical results from simulations and real-world microarray data illustrate that SCS-PCA provides competitive performance in prediction tasks and is able to select more features compared to existing supervised sparse methods.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Pierre-Alexandre_Mattei3

Submission Number: 9399