Tune-n-Batch: Fine-Tuning LLMs for Batch Prompting
Abstract: The growing capabilities of Large Language Models (LLMs) have enabled batch prompting (BP), the technique of concatenating multiple questions into one prompt and answering all questions in one inference pass. However, current batch prompting techniques require lengthy prompts that need few-shot examples and formatting instructions, reporting decreased accuracy per question as the batch size grows. In this paper, we show that this accuracy loss can be mitigated by fine-tuning models for batch prompting. We aggregate training data for batch prompting by selecting batches from nine datasets with varying batch sizes. We demonstrate that after limited fine-tuning of LLaMA-3-8B-Instruct on our batch prompting dataset, BP can be performed effectively without in-prompt examples or repetitive formatting. Our fine-tuned LLaMA-3-8B-Instruct model exhibits consistent performance across various batch sizes on tasks seen and unseen during training.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: inference methods, prompting, fine-tuning
Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency
Languages Studied: English
Submission Number: 3956
Loading