Non-Euclidean Harmonic Losses

ICLR 2026 Conference Submission21606 Authors

Keywords: Harmonic loss, Explainable AI, Green AI, Deep Learning

TL;DR: We extend harmonic loss beyond Euclidean distance by testing diverse metrics on vision and language models, showing that tailored variants can outperform cross-entropy and Euclidean, improving accuracy, interpretability, and sustainability.

Abstract: Cross-entropy loss has long been the standard choice for training deep neural networks, yet it suffers from interpretability limitations, unbounded weight growth, and inefficiencies that can contribute to costly training dynamics. Recent work introduced harmonic loss, a distance-based alternative grounded in Euclidean geometry, which improves interpretability and mitigates phenomena such as grokking, also known as delayed generalization on the test set. However, the study of harmonic loss remains narrow: only Euclidean distance is explored, and no systematic evaluation of computational efficiency or sustainability was conducted. In this paper, we extend harmonic loss by systematically investigating a broad spectrum of distance metrics as replacements for the Euclidean distance. We comprehensively evaluate distance-tailored harmonic losses on both vision backbones and large language models. Our analysis is framed around a three-way evaluation of model performance, interpretability, and sustainability. On vision tasks, cosine distances provide the most favorable trade-off, consistently improving accuracy while lowering carbon emissions, whereas Bray-Curtis and Mahalanobis further enhance interpretability at varying efficiency costs. On language models, cosine-based harmonic losses markedly improve gradient and learning stability, strengthen representation structure, and reduce emissions relative to cross-entropy and Euclidean heads. Our code is available at: https://anonymous.4open.science/r/rethinking-harmonic-loss-5BAB/

Primary Area: interpretability and explainable AI

Submission Number: 21606