Decompose the model: mechanistic interpretability in image models with generalized integrated gradients (GIG)

Yearim Kim, Sangyu Han, Han Sangbum, Nojun Kwak

Keywords: explainable ai, interpretability, mechanistic interpretability

Abstract:

In the field of eXplainable AI (XAI) in language models, the progression from local explanations of individual decisions to global explanations with high-level concepts has laid the groundwork for mechanistic interpretability, which aims to decode the exact operations.
However, this paradigm has not been adequately explored in image models, where existing methods have primarily focused on class-specific interpretations.
This paper introduces a novel approach to systematically trace the entire pathway from input through all intermediate layers to the final output within the whole dataset.
We utilize Pointwise Feature Vectors (PFVs) and instance-specific Effective Receptive Fields (iERFs) to decompose model embeddings into interpretable Concept Vectors.
Then, we calculate the relevance between concept vectors with our Generalized Integrated Gradients (GIG), enabling a comprehensive, dataset-wide analysis of model behavior.
We validate our method of concept extraction and concept attribution in both qualitative and quantitative evaluations.
Our approach advances the understanding of semantic significance within image models, offering a holistic view of their operational mechanics. https://iclr2025gig.netlify.app/graph_visualization.html

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3513