GRADSIMCORE: GRADIENT SIMILARITY BASED REPRESENTATIVE INSTANCES AS CORESET

Saumyaranjan Mohanty, Konda Reddy Mopuri

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Coreset selection, Data-efficient deep learning, gradient similarity

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Determination of Gradient similarity based representative instances as coreset selection.

Abstract: The rise in size and complexity of modern datasets and deep learning models have resulted in the usage of extensive computational resources and a rise in training time and effort. It also has increased the carbon footprint of training and fine-tuning models. One way to reduce the computational requirement is to extract the most representative subset (referred to as $\textit{coreset}$) that can substitute for the larger dataset. Coresets can thus replace huge datasets to train models and tune hyperparameters, especially in the early stages of training. This will result in a significant reduction of computational resource requirement and reduce carbon footprint. We propose a simple and novel framework based on the similarity of loss gradients for identifying the representative training instances as a coreset. Our method, dubbed as $\textit{GradSimCore}$, outperforms the state-of-the-art coreset selection algorithms on popular benchmark datasets ranging from MNIST to ImageNet. Because of its simplicity and effectiveness, our method is an important baseline for evaluating the effectiveness of the coreset selection algorithms. Anonymized codes for the proposed baseline are provided at https://anonymous.4open.science/r/GradSimCore-8884

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6690