KDD Cup Meta CRAG 2024 Technical Report: Three-step Question-Answering Framework

Sungho Park, Jeongeum Seok, Jooyoung Lee, Joohyung Yun, Wonseok Lee

Keywords: Retrieval-Augmented Generation, Hallucination, External Knowledge Retrieval, Answer Confidence

Abstract: Large language models (LLMs) have shown significant capabilities in the question-answering task, but they often suffer from hallucination, where generated content deviates from real-world facts. Retrieval-augmented generation (RAG) has been proposed to address this issue, which enhances LLM performance by retrieving relevant information from external knowledge sources. KDD Cup Meta 2024 is a competition for advancing the practical application of RAG in real-world scenarios. Participants are asked to develop an innovative RAG system that can accurately and efficiently answer complex questions by integrating relevant external data while minimizing hallucination. Our team dRAGonRAnGers, composed of members from POSTECH Data Systems Lab, propose a methodology that addresses two primary challenges of RAG: reducing unnecessary retrievals and preventing the propagation of incorrect information. We enhance the standard RAG framework by incorporating the inherent knowledge of LLMs to avoid unnecessary retrievals and introducing a verification step to reassess generated answers. This approach optimizes the efficiency and reliability of QA systems, improving both response accuracy and computational efficiency. Our team is the first prize winner of the comparison question category for all three tasks and also the first prize winner of the post-processing category for task 1 in KDD Cup 2024.

Submission Number: 6