ChunkRAG: A Novel LLM-Chunk Filtering Method for RAG Systems

ICLR 2025 Workshop BuildingTrust Submission151 Authors

Track: Long Paper Track (up to 9 pages)

Keywords: LLM, RAG, Chunking, Fact Checking, Retrieval, Information Retrieval

TL;DR: ChunkRAG introduces a chunk-level retrieval strategy that segments documents into semantically coherent chunks, improving factual accuracy and reducing hallucinations by filtering out irrelevant information during retrieval-augmented generation.

Abstract: Retrieval-Augmented Generation (RAG) frameworks leveraging large language models (LLMs) frequently retrieve extraneous or weakly relevant information, leading to factual inaccuracies and hallucinations in generated responses. Existing document-level retrieval approaches lack sufficient granularity to effectively filter non-essential content. This paper introduces ChunkRAG, a retrieval framework that refines information selection through semantic chunking and chunk-level evaluation. ChunkRAG applies a dynamic greedy chunk aggregation strategy to segment documents into semantically coherent, variable-length sections based on cosine similarity. Empirical evaluations on the PopQA, PubHealth and Biography dataset indicate that ChunkRAG improves response accuracy over state-of-the-art RAG methods. The analysis further demonstrates that chunk-level filtering reduces redundant and weakly related information, enhancing the factual consistency of responses. By incorporating fine-grained retrieval mechanisms, ChunkRAG provides a scalable and domain-agnostic approach to mitigate hallucinations in knowledge-intensive tasks such as fact-checking and multi-hop reasoning.

Submission Number: 151