Score Matching with Missing Data

Josh Givens, Song Liu, Henry Reeve

TL;DR: We adapt score matching to missing data and demonstrate its general applicability by applying it to graphical model estimation.

Abstract: Score matching is a vital tool for learning the distribution of data with applications across many areas including diffusion processes, energy based modelling, and graphical model estimation. Despite all these applications, little work explores its use when data is incomplete. We address this by adapting score matching (and its major extensions) to work with missing data in a flexible setting where data can be partially missing over any subset of the coordinates. We provide two separate score matching variations for general use, an importance weighting (IW) approach, and a variational approach. We provide finite sample bounds for our IW approach in finite domain settings and show it to have especially strong performance in small sample lower dimensional cases. Complementing this, we show our variational approach to be strongest in more complex high-dimensional settings which we demonstrate on graphical model estimation tasks on both real and simulated data.

Lay Summary:

Score matching is an invaluable machine learning method which allows us to learn a distribution given data from it. This has many downstream application such as learning dependencies between different components of our data (graphical model edge detection), grouping data points (mode-seeking clustering), image generation (diffusion processes).

In this paper we propose two adaptations of score matching which still learn the distribution of the clean data even when the samples in our observed data have some chance of being partially missing or corrupted. This is a common real world scenario, encompassing cases such as missing measurements in medical data or corrupted pixels in image data.

Some work has looked to study adaptation of the aforementioned downstream processes to missing data but very little has focussed on adapting score matching itself. By constructing such an adaptation, we provide a tool that can tackle all of these downstream tasks as well as any other task which utilises score matching. We compare our new adaptations to existing methods for score matching and show our approach to improve upon them.

Link To Code: https://github.com/joshgivens/ScoreMatchingwithMissingData

Primary Area: General Machine Learning->Unsupervised and Semi-supervised Learning

Keywords: Score Matching, Missing Data, Variational Inference

Submission Number: 12176