An Efficient Plugin Method for Metric Optimization of Black-Box Models

Siddartha Devic, Nurendra Choudhary, Anirudh Srinivasan, Sahika Genc, Branislav Kveton, Gaurush Hiranandani

Keywords: optimization, black box systems, domain adaptation, distribution shift, classification

TL;DR: A simple coordinate-wise optimization method to adapt black-box models by post-processing their predictions with a scaling vector.

Abstract:

Many machine learning algorithms and classifiers are available only via API queries as a ``black-box'' --- that is, the downstream user has no ability to change, re-train, or fine-tune the model on a particular target distribution.
Indeed, a downstream user may not have any knowledge of the training distribution or performance metric used to construct and optimize the black-box model.
We propose a simple and efficient plugin method which takes as input arbitrary multiclass predictions and post-processes them in order to adapt them to a new target distribution, while simultaneously optimizing for a particular metric of the confusion matrix.
Importantly, the plugin method is \textit{post-hoc}, does not rely on feature information, and only requires a small number of probabilistic predictions along with their corresponding true label.
We empirically demonstrate that plugin has performance competitive with related methods on a variety of tabular and language tasks.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7768