Adapting in the Dark: Efficient and Stable Test-Time Adaptation for Black-Box Models

Yunbei Zhang, Shuaicheng Niu, Chengyi Cai, Feng Liu, Jihun Hamm

Abstract: Test-Time Adaptation (TTA) for black-box models accessible only via APIs remains largely unexplored. Existing approaches such as post-hoc output refinement offer limited adaptive capacity, while Zeroth-Order Optimization (ZOO) enables input-space adaptation but suffers from high query costs and instability in the unsupervised setting. We introduce BETA (Black-box Efficient Test-time Adaptation), a framework that uses a lightweight local steering model to create a tractable gradient pathway. Through prediction harmonization, consistency regularization, and prompt learning-oriented filtering, BETA enables stable adaptation with no additional API calls and negligible latency. On ImageNet-C, BETA achieves +7.1% accuracy gain on ViT-B/16 and +3.4% on CLIP, surpassing strong white-box methods. On a commercial API, BETA matches ZOO performance at 250x lower cost.

Submission Number: 26