DocImpact: Quantifying Document Impact in RAG-LLMs

ICLR 2025 Workshop BuildingTrust Submission84 Authors

Track: Tiny Paper Track (between 2 and 4 pages)

Keywords: RAG, Augmented Generation, Document Influence, LLM

TL;DR: We propose DocImpact, which is a novel methodology that quantifies individual document influence in RAG systems through counterfactual analysis

Abstract: We present DocImpact, a novel methodology for measuring the influence of individual documents in Retrieval-Augmented Generation (RAG) systems. While RAG architectures have become increasingly popular in modern language models, understanding the precise contribution of each retrieved document to model outputs remains challenging. Our algorithm employs a counterfactual analysis by systematically excluding individual documents and measuring the divergence in model outputs compared to the full-context baseline. We implement our RAG-LLM using Pinecone as the database and Llama-3.1-70b as the LLM.

Submission Number: 84