Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track

Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, Jimmy Lin

Abstract: Have you tried the new Bing Search? Or maybe you fiddled around with Google AI Overviews? These might sound familiar because the modern-day search stack has evolved to include retrieval-augmented generation (RAG) systems. They allow searching and incorporating real-time data into large language models (LLMs) to provide a well-informed, attributed, concise summary, in contrast to the traditional search paradigm that relies on displaying a ranked list of documents. Therefore, given these recent advancements, it is crucial to have an arena to build, test, visualize, and systematically evaluate RAG-based search systems. With this in mind, we propose TREC RAG to foster innovation in evaluating RAG systems. In our work, we lay out the steps we have taken towards making this track a reality—we describe the details of our reusable framework, Ragnarök, explain the curation of the new MS MARCO V2.1 collection, release the development topics, some relevance judgments and baselines for the track and standardize the I/O definitions which assist the end user. Next, using Ragnarök, we identify and provide key proprietary and open-source baselines such as OpenAI’s GPT-4o, Cohere’s Command R+, and Meta’s LLaMA3.1-70B. Further, we introduce a web-based user interface for an interactive arena allowing benchmarking pairwise RAG systems by crowdsourcing. We open-source Ragnarök and baselines to achieve a unified standard for future RAG systems.