Increasing the Difficulty of Automatically Generated Questions via Reinforcement Learning with Synthetic Preference
Abstract: The demand for high-quality question-answering (QA) datasets has surged with the proliferation of language models and conversational agents in various emerging domains. As these models become ever more capable, the possibility of applying them to more challenging tasks is growing. Manual dataset annotation is costly and time-consuming, necessitating a more efficient approach. Automatically generated questions often suffer from a lack of quality or difficulty; hence, we propose a methodology to increase the difficulty of automatically generated questions using synthetic preference data, derived from SQuAD, to fine tune a question generation model using reinforcement learning. We empirically show an improvement in question difficulty over a supervised-finetuned model with minimal impact on question validity and perform an extensive error analysis. We believe our methodology provides a feasible approach to creating high quality synthetic datasets in emerging domains.
Paper Type: Long
Research Area: Question Answering
Research Area Keywords: Automatic Question Generation, Language Models, Reinforcement Learning with Human Feedback, Proximal Policy Optimisation, Synthetic Dataset Generation
Contribution Types: Approaches to low-resource settings, Theory
Languages Studied: English
Submission Number: 4912
Loading