Hyperspherical Simplex Representations from Softmax Outputs and Logits are Inherently Backward-Compatible

TMLR Paper6685 Authors

Abstract: Training modern AI models has become increasingly expensive, and updating deployed models can significantly alter the behavior of applications built upon them, due to changes in internal feature representations. In retrieval systems, such updates often require re-indexing the gallery-set by extracting feature vectors for all gallery data, a process that is computationally expensive and time-consuming, especially for large-scale datasets. Existing backward-compatibility methods allow direct comparison between the representations of updated and old models, avoiding the re-indexing of the gallery. However, they typically introduce a dependency on the old model by using auxiliary losses, mapping functions, or specific modifications to the model architecture. In this paper, we show that independently trained models are inherently backward-compatible when hyperspherical simplex representations derived from their softmax outputs or logits are used. Leveraging the geometric structure induced by the softmax function on these features, we define a deterministic projection that preserves their alignment across model updates. We demonstrate that these representations satisfy in expectation the formal definition of backward-compatibility. Without relying on regularization-based training, mapping functions, or modifications to the model architecture, we achieve competitive results on standard compatibility benchmarks involving model updates with new training classes and/or advanced model architectures.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Rahaf_Aljundi1

Submission Number: 6685