Evaluating representations by the complexity of learning low-loss predictors

William F Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, Kyunghyun Cho

Keywords: representation learning, representation evaluation, unsupervised learning, self-supervised learning

Abstract: We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest. To this end, we introduce two measures: surplus description length (SDL) and $\varepsilon$ sample complexity ($\varepsilon$SC). To compare our methods to prior work, we also present a framework based on plotting the validation loss versus dataset size (the "loss-data" curve). Existing measures, such as mutual information and minimum description length, correspond to slices and integrals along the data-axis of the loss-data curve, while ours correspond to slices and integrals along the loss-axis. This analysis shows that prior methods measure properties of an evaluation dataset of a specified size, whereas our methods measure properties of a predictor with a specified loss. We conclude with experiments on real data to compare the behavior of these methods over datasets of varying size.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Good representations allow simpler predictors to achieve low loss.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2009.07368/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=_DU2El79d