From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?

ACL ARR 2025 July Submission104 Authors

Abstract: Post-processing is crucial for the automatic evaluation of LLMs in fill-in-the-middle (FIM) code generation due to the frequent presence of extraneous code in raw outputs. This extraneous generation suggests a lack of awareness regarding output boundaries, requiring truncation for effective evaluation. The determination of an optimal truncation strategy, however, often proves intricate, particularly when the scope includes several programming languages. This study investigates the necessity of postprocessing instruction-tuned LLM outputs. Our findings reveal that supervised fine-tuning significantly enhances FIM code generation, enabling LLMs to generate code that seamlessly integrates with the surrounding context. Evaluating our fine-tuned Qwen2.5-Coder (base and instruct) models on HumanEval Infilling and SAFIM benchmarks demonstrates improved performances without post-processing, especially when the middle consists of complete lines. However, post-processing of the LLM outputs remains necessary when the middle is a random span of code.

Paper Type: Short

Research Area: Resources and Evaluation

Research Area Keywords: code generation, fill-in-the-middle, instruction-tuned LLMs

Contribution Types: Position papers

Languages Studied: English, Python

Submission Number: 104