Revealing interpretable object representations from human behavior

Charles Y. Zheng, Francisco Pereira, Chris I. Baker, Martin N. Hebart

Abstract: To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories. These representations predicted a latent similarity structure between objects, which captured most of the explainable variance in human behavioral judgments. Individual dimensions in the low-dimensional embedding were found to be highly reproducible and interpretable as conveying degrees of taxonomic membership, functionality, and perceptual attributes. We further demonstrated the predictive power of the embeddings for explaining other forms of human behavior, including categorization, typicality judgments, and feature ratings, suggesting that the dimensions reflect human conceptual representations of objects beyond the specific task.

Keywords: category representation, sparse coding, representation learning, interpretable representations

TL;DR: Human behavioral judgments are used to obtain sparse and interpretable representations of objects that generalize to other tasks