Comparing supervised learning dynamics: Deep neural networks match human data efficiency but show a generalisation lag

Lukas S. Huber, Fred W. Mast, Felix A. Wichmann

Track: long paper (up to 9 pages)

Keywords: representation learning, object recognition, computer vision, deep neural networks, representational alignment, psychophysics, behavioral comparison, human-machine comparison, generalisation

TL;DR: An empirical side-by-side comparison of supervised representation learning in humans and deep neural networks revealed that, although DNNs are not inherently less data-efficient, they exhibit a pronounced generalisation lag.

Abstract:

Recent research has seen many behavioral comparisons between humans and deep neural networks (DNNs) in the domain of image classification. Often, comparison studies focus on the end-result of the learning process by measuring and comparing the similarities in the representations of object categories once they have been formed. However, the process of how these representations emerge—that is, the behavioral changes and intermediate stages observed during the acquisition—is less often directly and empirically compared.

Here we report a detailed investigation of the learning dynamics in human observers and various classic and state-of-the-art DNNs. We develop a constrained supervised learning environment to align learning-relevant conditions such as starting point, input modality, available input data and the feedback provided. Across the whole learning process we evaluate and compare how well learned representations can be generalized to previously unseen test data.

Comparisons across the entire learning process indicate that DNNs demonstrate a level of data efficiency comparable to human learners, challenging some prevailing assumptions in the field. However, our results also reveal representational differences: while DNNs' learning is characterized by a pronounced generalisation lag, humans appear to immediately acquire generalizable representations without a preliminary phase of learning training set-specific information that is only later transferred to novel data.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 41