Variational Inference for Discriminative Learning with Generative Modeling of Feature Incompletion

Kohei Miyaguchi, Takayuki Katsuki, Akira Koseki, Toshiya Iwamori

Keywords: Black-box variational inference, missing values, evidence upper bound

Abstract:

We are concerned with the problem of distributional prediction with incomplete features: The goal is to estimate the distribution of target variables given feature vectors with some of the elements missing. A typical approach to this problem is to perform missing-value imputation and regression, simultaneously or sequentially, which we call the generative approach. Another approach is to perform regression after appropriately encoding missing values into the feature, which we call the discriminative approach. In comparison, the generative approach is more robust to the feature corruption while the discriminative approach is more favorable to maximize the performance of prediction.
In this study, we propose a hybrid method to take the best of both worlds. Our method utilizes the black-box variational inference framework so that it can be applied to a wide variety of modern machine learning models, including the variational autoencoders. We also confirmed the effectiveness of the proposed method empirically.

One-sentence Summary: A new variational approximation and algorithm are proposed for discriminative inference with missing features.