Increasing the Difficulty of Automatically Generated Questions via Reinforcement Learning with Synthetic Preference

Anonymous

TL;DR: A methodology for increasing the difficulty of automatically generated questions using synthetic preference data for RLHF

Abstract: The demand for high-quality question-answering (QA) datasets has surged with the proliferation of language models and conversational agents in various emerging domains. As these models become ever more capable, the need for more challenging datasets for benchmarking and training is growing. Manual dataset annotation is costly and time-consuming, necessitating a more efficient approach. We propose a methodology to increase the difficulty of automatically generated questions using synthetic preference data, derived from SQuAD, to fine tune a question generation model using reinforcement learning. We empirically show an improvement in question difficulty and quality over a simple supervised-finetuned model and perform an extensive error analysis. We make all our code and results publicly available.

Paper Type: long

Research Area: Machine Learning for NLP

Contribution Types: Approaches to low-resource settings, Theory

Languages Studied: English