Efficient Bayesian Updates for Deep Active Learning via Laplace Approximations
Keywords: Active Learning, Deep Learning, Batch Active Learning
TL;DR: We introduce an efficient Bayesian update using a last-layer Laplace approximation to replace costly retraining in deep active learning, enabling sequential batch selection and look-ahead strategies.
Abstract: Deep active learning (AL) involves selecting batches of instances for annotation since retraining large deep neural networks (DNNs) after each label acquisition is computationally impractical. Employing a naive top-$b$ selection can result in a batch of redundant (similar) instances. To address this issue, various batch AL strategies have been developed, many of which employ clustering for diversity as a heuristic. In contrast, we approach this issue by substituting the costly retraining with an efficient Bayesian update. Our proposed update represents a second-order optimization step using the Gaussian posterior from a last-layer Laplace approximation. Thereby, we achieve low computational complexity by computing the inverse Hessian in closed form. We demonstrate that in typical AL settings, our update closely approximates retraining while being considerably faster. Leveraging our update, we introduce a new framework for batch selection through sequential construction by updating the DNN after each label acquisition. Furthermore, we incorporate our update into a look-ahead selection strategy as a feasible upper baseline approximating optimal batch selection. Our results highlight the potential of efficient updates to advance deep AL research.
Primary Area: other topics in machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6156
Loading