How Classifiers Extract General Features for Downstream Tasks: An Asymptotic Analysis in Two-Layer Models

HEE BIN YOO, Sungyoon Lee, Cheongjae Jang, Dong-Sig Han, Jaein Kim, Seunghyeon Lim, Byoung-Tak Zhang

TL;DR: This paper investigates feature transfer in classifier-trained networks, analyzing the impact of similarity between training and unseen data on feature extraction and unseen data clustering performance.

Abstract:

Neural networks learn effective feature representations through intermediate layers, enabling feature transfer without additional training for new tasks. However, the conditions for successful feature transfer remain underexplored.
In this paper, we investigate feature transfer in classifier-trained networks, focusing on clustering in unseen distributions.
In binary classification, we find that higher similarity between training and unseen distributions improves Cohesion and Separability, while Separability further requires unseen data to be assigned to different training classes.
In multi-class classification, our analysis shows that the feature extractor maps input point based on their similarity to training classes, i.e. that unrelated training classes to input have negligible impact on feature extraction.
We validate our theoretical findings in synthetic dataset and demonstrate practical applicability utilizing ResNet and variations of CAR, CUB, SOP, ISC, and ImageNet datasets.

Primary Area: Deep Learning->Theory

Keywords: two-layer neural network, feature learning, transfer learning, metric learning, asymptotic analysis, random matrix theory

Submission Number: 10841