How Classifier Features Transfer to Downstream: An Asymptotic Analysis in a Two-Layer Model

HEE BIN YOO, Sungyoon Lee, Cheongjae Jang, Dong-Sig Han, Jaein Kim, Seunghyeon Lim, Byoung-Tak Zhang

Keywords: open-set clustering, feature transfer, two-layer neural network, feature learning, metric learning, retrieval

TL;DR: This paper investigates feature transfer in classifier-trained networks, analyzing the impact of similarity between training and unseen data on unseen data clustering performance and feature extraction.

Abstract:

Neural networks learn effective feature representations, which can be transferred to new tasks without additional training.
While larger datasets are known to improve feature transfer, the theoretical conditions for the success of such transfer remain unclear.
This work investigates feature transfer in networks trained for classification to identify the conditions that enable effective clustering in unseen classes.
We first reveal that higher similarity between training and unseen distributions leads to improved Cohesion and Separability.
We then show that feature expressiveness is enhanced when inputs are similar to the training classes, while the features of irrelevant inputs remain indistinguishable.
We validate our analysis on synthetic and benchmark datasets, including CAR, CUB, SOP, ISC, and ImageNet.
Our analysis highlights the importance of the similarity between training classes and the input distribution for successful feature transfer.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 15419