On Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility

Zhitao He, Haolin Yang, Rui Min, Zeyu Qin, Yi R. Fung

Keywords: Long-Form Generation, Length Volatility

Abstract: Large Language Models (LLMs) excel at long-context understanding but exhibit significant limitations in long-form generation. Existing studies primarily focus on single-generation quality, generally overlooking the volatility of the output (i.e., the inconsistency in length and content across multiple generations). This volatility not only leads to significant computational costs but also severely impacts the models' reliable application. To address this gap, our work unfolds in three stages: *benchmarking, probing, and mitigation*. We first propose the **VO**latility in **L**ong-form **T**ext **Bench**mark (**VOLTBench**), a novel heterogeneous-task benchmark designed to systematically quantify the length volatility of long-form generation. Subsequently, by analyzing attention traces, we conduct an in-depth probe to identify several common internal patterns that cause this volatility. Finally, to mitigate long-form output volatility, we propose SELB (Structural Enforcement via Logits Boosting), a lightweight decoding-stage optimization strategy, designed to significantly enhance both the length accuracy and stability of long-form generation without additional training. Extensive experiments on VOLTBench provide the first systematic confirmation of severe long-form output instability in mainstream models and validate that our proposed method successfully improves the mean output length of the base model by 148\% and reduces the length volatility by 69\%, while maintaining high generation quality.

Primary Area: interpretability and explainable AI

Submission Number: 8616