Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation

Damian Sójka, Sebastian Cygert, Marc Masana

Keywords: test-time adaptation, backpropagation-free

TL;DR: We introduce PACE, a backpropagation-free continual test-time adaptation framework that achieves state-of-the-art accuracy and a 50% reduction in runtime while optimizing normalization layer parameters within a low-dimensional subspace.

Abstract:

We introduce PACE, a backpropagation-free continual test-time adaptation system that directly optimizes the affine parameters
of normalization layers. Existing derivative-free approaches struggle to balance runtime efficiency with learning capacity, as they either restrict updates to input prompts or require continuous, resource-intensive adaptation regardless of domain stability. To address these limitations, PACE leverages the Covariance Matrix Adaptation Evolution Strategy with the Fastfood projection to optimize high-dimensional affine parameters within a low-dimensional subspace, leading to superior adaptive performance. Furthermore, we enhance the runtime efficiency by incorporating an adaptation stopping criterion and a domain-specialized vector bank to eliminate redundant computation. Our framework achieves state-of-the-art accuracy across multiple benchmarks under continual distribution shifts, reducing runtime by over 50% compared to existing backpropagation-free methods.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 14