Controllable Text Generation in the Instruction-Tuning Era
Abstract: While most research on controllable text generation has focused on steering base Language Models, the emerging instruction-tuning and prompting paradigm offers an alternate approach to controllability. We compile and release ConGenBench, a testbed of 17 different controllable generation tasks, using a subset of it to benchmark the performance of 9 different baselines and methods on Instruction-tuned Language Models. To our surprise, we find that prompting-based approaches outperform controllable text generation methods on most datasets and tasks, highlighting a need for research on controllable text generation with Instruction-tuned Language Models in specific. Prompt-based approaches match human performance on most stylistic tasks while lagging on structural tasks, foregrounding a need to study more varied constraints and more challenging stylistic tasks. To facilitate such research, we provide an algorithm that uses only a task dataset and a Large Language Model with in-context capabilities to automatically generate a constraint dataset. This method eliminates the fields dependence on pre-curated constraint datasets, hence vastly expanding the range of constraints that can be studied in the future.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Changes in new revision (same submission)
1. Experiment details made more clear (model sizes added to the main paper and Appendix, exact version specified in the appendix for reproducibility)
2. Standard deviations for individual surveys for each method and each metric added
3. Expanded discussion of contributions, emphasizing the importance of the results and the value of the ConGenBench testbed.
4. Miscellaneous formatting edits.
Assigned Action Editor: ~Edward_Grefenstette1
Submission Number: 2468
Loading