Learning from Synthetic Data Improves Multi-hop Reasoning

Anmol Kabra, Yilun Yin, Albert Gong, Kamilė Stankevičiūtė, Dongyoung Go, Johann Lee, Katie Z Luo, Carla P Gomes, Kilian Q Weinberger

Keywords: multi-hop reasoning, large language models, reinforcement learning, synthetic data

TL;DR: RL fine-tuning LLMs on synthetic data improves real-world multi-hop reasoning by teaching knowledge composition skills

Abstract:

Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks.
However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers.
All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow.
In this work, we investigate a cheaper alternative: RL fine-tuning on _rule-generated synthetic data_ for multi-hop reasoning tasks.
We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge.
On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to _compose knowledge_---a fundamental and generalizable reasoning skill.
Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 21366