Gaussian Processes with Bayesian Inference of Covariate Couplings

TMLR Paper4200 Authors

Abstract: Gaussian processes are powerful probabilistic models that are often coupled with Automatic Relevance Determination (ARD) capable of uncovering the importance of individual covariates. We develop covariances characterized by affine transformations of the inputs, formalized via a precision matrix between covariates, which can uncover covariate couplings for enhanced interpretability. We study a range of couplings priors from Wishart to Horseshoe and present fully Bayesian inference of such precision matrices within sparse Gaussian process. We demonstrate empirically the efficacy and interpretability of this approach.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=jbg6H7n6Um

Changes Since Last Submission:

Please see the responses to the reviews to the previous submission. We have highlighted in blue the changes with respect to the original submission. Here is a brief summary of the changes:

1. We provide a clarified description of the background and motivations in the introduction.
2. We emphasize in the introduction and in Section 6.5 the connection between the application context and the approaches adopted.
3. We better position our work in the literature adding a new "Related Works" section.
4. We clarify the construction of the model specifying both what the likelihood function is in Section 2.1 (for regression and classification) and the Gaussian Process construction in Section 2.2.
5. We created a new Section 4.4 to expand on the computational aspects of our proposal: we discuss the different costs of employing ACD covariances in contrast with ARD covariances.
6. In addition to the computational complexity analysis we have added one extra experiment showing the wall clock difference between running sparse GPs with the ARD and ACD covariance (Table 5 in the appendix).
6. We further motivate the importance of our contribution of inferring covariate couplings simultaneously with regression with respect to only applying PCA to reveal dependence/independence structure for the input covariates. We also added a new Figure 17 where we can observe richer insights stemming from the use of label information on the MoCap dataset to help better justify this point.
7. We further comment in Sections 4.3.2 and 6.4 on the added flexibility of an ACD parametrization after PCA. This effect is also illustrated in the updated version of Figure 6.
8. We clarify the training setup of the MoCap experiment in Table 2.
9. We fix a typo spotted by a reviewer in Section 2.

Assigned Action Editor: ~Arto_Klami1

Submission Number: 4200