SynBench: Evaluating Pretrained Representations for Image Classification using Synthetic Data
Primary Area: visualization or interpretation of learned representations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Vision pretrained model; synthetic data; evaluation
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Fine-tuning large models pretrained at scale on broad data for solving downstream tasks has made considerable success in recent years. There seems to be indeed an ongoing paradigm shift in deep learning from task-centric model design to task-agnostic representation learning and task-specific fine-tuning. Specifically, the representations of pretrained models are used as a foundation for different downstream tasks. This paper proposes a new task-agnostic framework, \textit{SynBench}, to measure the quality of pretrained representations for image classification using synthetic data. To address the challenge of task-agnostic data-free evaluation, we design synthetic binary classification proxy tasks with class-conditional Gaussian mixtures. This way we probe and compare the robustness-accuracy performance on pretrained representations and input synthetic data. SynBench offers a holistic quantitative evaluation, informs the model designers of the intrinsic performance, and spares efforts on task-specific finetuning with real-life data. Evaluated with various pretrained vision models for different downstream image classification tasks, the experimental results show that our SynBench score matches well the actual linear probing performance of the pretrained model when fine-tuned on downstream tasks using real-life data. Finally, SynBench can also be used in robust linear probing to mitigate the robustness-accuracy tradeoff in downstream tasks.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6178
Loading