Exploring Domain Adaptation with LLMs for Real-World Augmented Question Answer Generation (RA-QAG) in Children Storytelling
Abstract: In the real world, external domain-specific knowledge is commonly required, for instance, teachers often apply their expertise to ask preschoolers educational-crafted, story-inspired questions beyond the story content during interactive storytelling; however, existing storytelling systems could not effectively support such activity as the generated questions are mostly text-based.
We formulate this type of common real-world application as a novel Real-World Augmented QAG (RA-QAG) task.
This work aims to explore how well LLMs, equipped with various domain adaptation strategies (e.g., few-shot In-Context Learning, Chain-of-Thoughts, Retrieval-Augmented Generation), perform on the RA-QAG task in the context of children storytelling.
We design and experiment with end-to-end and 2-Step QAG pipelines with different domain adaptation strategies to explore whether they can identify real-world knowledge and create QA pairs aligned with experts' annotation.
Our automatic evaluation and human evaluation show that 1) RAG is a promising direction to approach real-world domain-specific tasks; 2) human experts still have more nuanced knowledge from which generic LLMs need to learn.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: educational applications, domain adaptation, interactive storytelling, question generation
Contribution Types: NLP engineering experiment
Languages Studied: English
Submission Number: 5618
Loading