Explaining to Learn: Regularization Using Contrastive Visual Explanation Pairs For Distribution Shifts

Paulo Mario P. Medina, Jose Marie Antonio Miñoza, Sebastian C. Ibañez

Keywords: distribution shifts, subpopulation shifts, domain generalization, spurious correlations, spatial confounders, GradCAM, explanation-based learning, regularization, fairness

TL;DR: The authors propose Explaining to Learn, an intersection between xAI and Distribution Shifts algorithm, which outperforms existing baselines regularization methods on datasets such as the Spawrious Hard Many-to-Many dataset.

Abstract: While a myriad of algorithms have been proposed to address distribution shifts, most algorithms are known to perform best only under specific conditions and fail to outperform the baseline empirical risk minimization (ERM) in other scenarios. Furthermore, the algorithmic complexity of some existing methods can render them less interpretable, and their approach to addressing spurious correlations, a hallmark of distribution shifts, is often indirect. To specifically address spatial confounders, we propose Explaining to Learn (ETL), an interpretable, explanation-based learning algorithm that removes spatial confounders from the primary classifier's latent representations during training. ETL achieves this by penalizing the similarity between GradCAM activation maps from a primary label classifier and a concurrently trained confounder classifier. On the more recent and difficult Spawrious Many-to-Many Hard Challenge benchmark, ETL achieves an average accuracy (AA) of 82.24% (±3.87) and a worst-group accuracy (WGA) of 66.31% (±8.73), outperforming leading state-of-the-art (SOTA) benchmarks by a significant 5% and 11%, respectively. This strong performance extends to other challenging benchmarks, where ETL also outperforms SOTA regularization methods on CMNIST (AA: 69.02% ±0.53; WGA: 67.63% ±1.39) and Waterbirds (AA: 92.12% ±0.67; WGA: 86.92% ±0.56). We complement these empirical results with theoretical analyses, demonstrating the viability of explanation-based learning for mitigating distribution shifts.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 24059