Ranked Entropy Minimization for Continual Test-Time Adaptation

Jisu Han, Jaemin Na, Wonjun Hwang

TL;DR: We propose an efficient solution for continual test-time adaptation based on ranked entropy minimization.

Abstract: Test-time adaptation aims to adapt to realistic environments in an online manner by learning during test time. Entropy minimization has emerged as a principal strategy for test-time adaptation due to its efficiency and adaptability. Nevertheless, it remains underexplored in continual test-time adaptation, where stability is more important. We observe that the entropy minimization method often suffers from model collapse, where the model converges to predicting a single class for all images due to a trivial solution. We propose ranked entropy minimization to mitigate the stability problem of the entropy minimization method and extend its applicability to continuous scenarios. Our approach explicitly structures the prediction difficulty through a progressive masking strategy. Specifically, it gradually aligns the model's probability distributions across different levels of prediction difficulty while preserving the rank order of entropy. The proposed method is extensively evaluated across various benchmarks, demonstrating its effectiveness through empirical results.

Lay Summary:

Artificial intelligence is getting better at classifying images, but it can still make mistakes in unfamiliar conditions, like bad weather or poor image quality. Letting it learn during prediction can help, but this often leads to unstable behavior, where AI gives the same answer for every image.
We developed a new approach where AI learns gradually by comparing easy and hard examples. For instance, it first sees a complete image and then learns to handle increasingly difficult versions by hiding important parts. The key is to maintain the correct order of prediction difficulty while learning in a stable and structured way.
This idea is inspired by the ancient Greek story of Achilles and the tortoise. Rather than rushing, AI takes deliberate steps to reduce errors. Our method helps AI adapt more reliably in changing environments, making it useful for real-world applications such as self-driving cars and robotics.

Link To Code: https://github.com/pilsHan/rem

Primary Area: Deep Learning->Robustness

Keywords: continual test time adaptation, entropy minimization, masked image modeling

Submission Number: 10166