Exploiting covariate embeddings for classification using Gaussian processes

Daniel Andrade, Akihiro Tamura, Masaaki Tsuchida

Abstract: Highlights • Introduces a new methodology to incorporate auxiliary covariate information. • Generalizes previous approaches to covariate smoothing. • Effectively learns all parameters without cross-validation. • Incorporates covariates at test time that did not occur in the training data. • Improves text classification performance for small training data sets. Abstract In many logistic regression tasks, auxiliary information about the covariates is available. For example, a user might be able to specify a similarity measure between the covariates, or an embedding (feature vector) for each covariate, which is created from unlabeled data. In particular for text classification, the covariates (words) can be described by word embeddings or similarity measures from lexical resources like WordNet. We propose a new method to use such embeddings of covariates for logistic regression. Our method consists of two main components. The first component is a Gaussian process (GP) with a covariance function that models the correlations between covariates, and returns a noise-free estimate of the covariates. The second component is a logistic regression model that uses these noise-free estimates. One advantage of our model is that the covariance function can be adjusted to the training data using maximum likelihood. Another advantage is that new covariates that never occurred in the training data can be incorporated at test time, while run-time increases only linearly in the number of new covariates. Our experiments demonstrate the usefulness of our method in situations when only small training data is available.