XGRAG: A Graph-Native Framework for Explaining KG-based Retrieval-Augmented Generation

Zhuoling Li, Ha Linh Hong Tran Nguyen, Valeria Bladinieres, Maxim Romanovsky

Keywords: Explainable AI, Knowledge Graphs, Retrieval-Augmented Generation, Graph-based Perturbation

Abstract:

Graph-based Retrieval-Augmented Generation (GraphRAG) extends traditional RAG by using knowledge graphs (KGs) to give large language models (LLMs) a structured, semantically coherent context, yielding more grounded answers.
However, GraphRAG reasoning process remains a “black-box”, limiting our ability to understand how specific pieces of structured knowledge influence the final output.
Existing explainability (XAI) methods for RAG systems, designed for text-based retrieval, are limited to interpreting an LLM’s response through the relational structures among knowledge components, creating a critical gap in transparency and trustworthiness.
To address this, we introduce **XGRAG**, a novel framework that generates causally grounded explanations for GraphRAG systems by employing graph-based perturbation strategies, to quantify the contribution of individual graph components on the model’s answer.
We conduct extensive experiments comparing XGRAG against RAG-Ex, an XAI baseline for standard RAG, and evaluate its robustness across various question types, narrative structures and LLMs.
Our results demonstrate a 14.81\% improvement in explanation quality over the baseline RAG-Ex across NarrativeQA, FairyTaleQA, and TriviaQA, evaluated by F1-score measuring alignment between generated explanations and original answers.
Furthermore, XGRAG’s explanations exhibit a strong correlation with graph centrality measures, validating its ability to capture graph structure.
XGRAG provides a scalable and generalizable approach towards trustworthy AI through transparent, graph-based explanations that enhance the interpretability of RAG systems.

Primary Area: interpretability and explainable AI

Submission Number: 19848