FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents

Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, Andrew Drozdov

Keywords: Information Retrieval, Retrieval-Augmented Generation, Evaluation Benchmark, General Framework, Automated LLM Evaluation

TL;DR: FreshStack is a framework to build realistic IR & RAG evaluation benchmarks on niche and recent domains from community-asked questions and answers.

Abstract:

We introduce FreshStack, a holistic framework for automatically building information retrieval (IR) evaluation benchmarks by incorporating challenging questions and answers. FreshStack conducts the following steps:
(1) automatic corpus collection from code and technical documentation,
(2) nugget generation from community-asked questions and answers, and
(3) nugget-level support, retrieving documents using a fusion of retrieval techniques and hybrid architectures.
We use FreshStack to build five datasets on fast-growing, recent, and niche domains to ensure the tasks are sufficiently challenging. On FreshStack, existing retrieval models, when applied out-of-the-box, significantly underperform oracle approaches on all five domains, denoting plenty of headroom to improve IR quality. In addition, we identify cases where rerankers do not improve first-stage retrieval accuracy (two out of five domains) and oracle context helps an LLM generator generate a high-quality RAG answer.
We hope FreshStack will facilitate future work toward constructing realistic, scalable, and uncontaminated IR and RAG evaluation benchmarks.

Dataset URL: https://huggingface.co/freshstack

Code URL: https://github.com/fresh-stack/freshstack

Primary Area: Datasets & Benchmarks illustrating Different Deep learning Scenarios (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Flagged For Ethics Review: true

Submission Number: 1862