Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models

Gabriele Prato, Shagun Sodhani, Alessandro Sordoni, Sarath Chandar

Keywords: packing, latent, multi-hop, reasoning, large, language, model, models, llm, llms

TL;DR: We study the different aspects of packing and their effect on the downstream latent multi-hop reasoning capabilities of LLMs.

Abstract: The standard practice for training large language models involves packing multiple documents together to optimize computational efficiency. However, the impact of this process on the models' capabilities remains largely unexplored. To address this gap, we investigate how different document-packing strategies influence the latent multi-hop reasoning abilities of LLMs. Our findings indicate that packing can improve model performance compared to training on individual documents, at the expense of more compute. To further understand the underlying mechanisms, we conduct an ablation study, identifying key factors that explain the advantages of packing. Ultimately, our research deepens the understanding of LLM training dynamics and provides practical insights for optimizing model development.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 4692