Adapting in the Dark: Towards Stable and Efficient Black-Box Test-Time Adaptation

ICLR 2026 Conference Submission15947 Authors

Keywords: Test-Time Adaptation, Black-Box Test-Time Adaptation

Abstract:

Test-Time Adaptation (TTA) for black-box models accessible only via APIs presents a significant yet largely unexplored challenge.
Existing truly black-box methods are scarce; post-hoc output refinement shows minimal benefit, while naively introducing Zeroth-Order Optimization (ZOO) for prompt tuning at test time suffers from prohibitive query costs and catastrophic instability. To address these challenges, we introduce **BETA** (Black-box Efficient Test-time Adaptation), a novel framework that enables stable and efficient adaptation for both standard Vision Models and large Vision-Language Models. BETA uniquely employs a lightweight, local white-box steering model to create a tractable gradient pathway for optimization, circumventing the need for expensive ZOO methods. This is achieved through a prediction harmonization technique that creates a shared objective, stabilized by consistency regularization and a prompt learning-oriented filtering strategy. Requiring only *a single API call per test sample*, BETA achieves a +7.1\% gain on a ViT-B/16 model and a +3.4\% gain on powerful CLIP models; remarkably, its performance *surpasses* that of certain white-box and gray-box TTA methods (e.g., TENT and TPT). This practical effectiveness is further validated on a real-world commercial API, where BETA achieves a +5.2\% gain for just \$0.4—a 250x cost advantage over ZOO—establishing it as a robust and efficient solution for adapting models in the dark at test time.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 15947