Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

ICLR 2025 Conference Submission1169 Authors

Keywords: large language models, llms, synthetic-generation, dataset-generation, real-world data

TL;DR: We proposed a new method for generating and curating high-quality synthetic data grounded in real data sources that can be used to produce high-quality samples for finetuning LLMs on challenging tasks like reasoning and tool-use.

Abstract:

Large Language Models still struggle in challenging scenarios that leverage structured data, complex reasoning, or tool usage. In this paper, we propose Source2Synth: a new self-augmentation approach for teaching LLMs new skills that can be leveraged in low data regimes without relying on costly human annotations. Source2Synth takes as input a custom data source and produces synthetic data points with intermediate reasoning steps grounded in real-world sources.
Source2Synth improves the dataset quality by discarding low-quality generations based on their answerability.
We demonstrate the generality of this approach by applying it to two challenging domains: we test reasoning abilities in multi-hop question answering (MHQA), and tool usage in tabular question answering (TQA).
Our method improves performance by 25.51\% for TQA on WikiSQL and 22.57\% for MHQA on HotPotQA compared to the fine-tuned baselines.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1169