Asking language models how to represent data for fine-tuning

ACL ARR 2025 February Submission4106 Authors

Abstract: Language models are often used for tasks involving structured data like tables and graphs, but there is no general approach for choosing the best format to represent such data across different tasks for fine-tuning. In this study, we show how the pre-trained model can suggest its own formats for representing structured data in a general task. We also compare the performance of different formats after fine-tuning the models to see how they relate to the pre-trained performance. Our results show that different formats perform best across different models after fine-tuning for the same task. Interestingly, the format that performs best before fine-tuning always remains one of the top choices afterward. This approach can help avoid the need for trial-and-error during fine-tuning, saving time, computational resources, and reducing the environmental impact of training large models.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: fine-tuning, generative models

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 4106