Local Function Complexity for Active Learning via Mixture of Gaussian Processes
Abstract: Inhomogeneities in real-world data, e.g., due to changes in the observation noise level or variations in the structural complexity of the source function, pose a unique set of challenges for statistical inference. Accounting for them can greatly improve predictive power when physical resources or computation time is limited. In this paper, we draw on recent theoretical results on the estimation of local function complexity (LFC), derived from the domain of local polynomial smoothing (LPS), to establish a notion of local structural complexity, which is used to develop a model-agnostic active learning framework. Due to its reliance on pointwise estimates, the LPS model class is not robust and scalable with respect to large input space dimensions that typically come along with real-world problems. Here, we propose a GPR-based estimate of LFC, which is able to manage the curse of dimensionality. To this end, we train a mixture of experts (MoE) model where the experts are GPR models at different bandwidths. Being the key ingredient in the calculation of LFC, we then estimate locally optimal kernel bandwidths as the weighted average of these bandwidth candidates, where the weights are taken from the learned gate of the MoE model. We assess the effectiveness of our LFC estimate in an active learning application on a prototypical low-dimensional synthetic dataset, before taking on the challenging real-world task of reconstructing a quantum chemical force field for a small organic molecule and demonstrating state-of-the-art performance at a lower rate of sampling.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: In this revision of the original submission, we have highlighted the essential changes by blue text. We hope that this will be helpful for the reviewers to spot their requested changes and suggestions.
In a short second update, we have corrected for some further typos and some $p_{Opt}$ appearances that we have missed out to change to $p_{Sup}$
Assigned Action Editor: ~Sivan_Sabato1
Submission Number: 502
Loading