SegTune: Structured and Fine-Grained Control for Song Generation

ACL ARR 2026 January Submission3101 Authors

Keywords: Song Generation, Non-autoregressive Generation, Fine-grained Controllability, Diffusion Transformer, Duration Prediction

Abstract: Recent advances in neural song generation have enabled high-quality synthesis from lyrics and global textual prompts. However, most systems fail to model temporally varying attributes of songs, severely limiting fine-grained control over musical structure and dynamics. To address this, we propose SegTune, a Diffusion Transformer-based framework enabling structured and fine-grained controllability by allowing users or large language models (LLMs) to specify local musical descriptions aligned to song segments. These segment prompts are temporally broadcast to corresponding time windows, while global prompts ensure stylistic coherence. To support precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamps in LyRiCs format. We further construct a large-scale data pipeline for high-quality song collection with aligned lyrics and prompts, and propose new metrics to evaluate segment alignment and vocal consistency. Experiments demonstrate that SegTune outperforms existing baselines in both musicality and controllability. Visit our demo page for more generated songs of SegTune.

Paper Type: Long

Research Area: Speech Processing and Spoken Language Understanding

Research Area Keywords: Generation

Languages Studied: English

Submission Number: 3101