Online importance sampling for stochastic gradient optimization

corentin salaun, Xingchang Huang, Iliyan Georgiev, Niloy Mitra, Gurprit Singh

Keywords: SGD, Importance sampling

TL;DR: This work introduce a novel importance sampling algorithm for machine learning frameworks improving convergence by prioritizing crucial data points through simplified importance functions based on loss derivative with minimal computational overhead.

Abstract:

Machine learning optimization commonly relies on stochastic gradient descent, where the accuracy of gradient estimation is crucial for model performance. Rather than relying on uniform sampling, importance sampling can improve accuracy by focusing on data points that have more significant impact on learning. However, existing methods for importance sampling face challenges with computational efficiency and integration into practical machine learning workflows.
In this work, we introduce a novel adaptive metric based on the loss derivative wrt the network output that can be used for both importance sampling and data pruning. Our metric not only enhances gradient accuracy by prioritizing influential data points but also enables effective pruning by identifying and removing data that contributes minimally to training. We propose an efficient adaptive algorithm that leverages this metric with minimal computational overhead. Our evaluations on classification and regression tasks demonstrate improved convergence and reduced training data requirements, validating the efficacy of our approach.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10076