A representation-learning game for classes of prediction tasks

Neria Uzan, Nir Weinberger

Keywords: representation learning, dimensionality-reduction, regret, minimax solution, mixed strategies, multiplicative weights update

TL;DR: We derive optimal representations for classes of prediction tasks. We establish the theoretically optimal randomized representation in the linear-MSE setting, and propose an iterative algorithm for optimal representation in the general setting.

Abstract: We introduce a formulation for learning dimensionality-reducing representations of unlabeled feature vectors, when a prior knowledge on future prediction tasks is available. The formulation is based on a three-player game, in which the first player chooses a representation, the second player then adversarially chooses a prediction task, and the third player predicts the response based on the represented features. The first and third player aim is to minimize, and the second player to maximize, the regret: The minimal prediction loss using the representation compared to the same loss using the original features. Our first contribution is theoretical and addresses the mean squared error loss function, and the case in which the representation, the response to predict and the predictors are all linear functions. We establish the optimal representation in pure strategies, which shows the effectiveness of the prior knowledge, and the optimal regret in mixed strategies, which shows the usefulness of randomizing the representation. We prove that optimal randomization requires a precisely characterized finite number of representations, which is smaller than the dimension of the feature vector, and potentially much smaller. Our second contribution is an efficient gradient-based iterative algorithm that approximates the optimal mixed representation for a general loss function, and general classes of representations, response functions and predictors.

Submission Number: 2584