GIFT: A Framework for Global Interpretable Faithful Textual Explanations of Vision Classifiers

TMLR Paper6156 Authors

Abstract: Understanding the decision processes of deep vision models is essential for their safe and trustworthy deployment in real-world settings. Existing explainability approaches, such as saliency maps or concept-based analyses, often suffer from limited faithfulness, local scope, or ambiguous semantics. We introduce GIFT, a post-hoc framework that derives Global, Interpretable, Faithful, and Textual explanations for vision classifiers. GIFT begins by generating a large set of faithful, local visual counterfactuals, then employs vision–language models to translate these counterfactuals into natural-language descriptions of visual changes. These local explanations are aggregated by a large language model into concise, human-readable hypotheses about the model’s global decision rules. Crucially, GIFT includes a verification stage that quantitatively assesses the causal effect of each proposed explanation by performing image-based interventions, ensuring that the final textual explanations remain faithful to the model’s true reasoning process. Across diverse datasets, including the synthetic CLEVR benchmark, the real-world CelebA faces, and the complex BDD driving scenes, GIFT reveals not only meaningful classification rules but also unexpected biases and latent concepts driving model behavior. Altogether, GIFT bridges the gap between local counterfactual reasoning and global interpretability, offering a principled and extensible approach to causally grounded textual explanations for vision models.

Submission Length: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: 6146

Changes Since Last Submission: We got desk rejected because of latex template issues. We fixed the issues.

Assigned Action Editor: ~Krikamol_Muandet1

Submission Number: 6156