ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games
Submission Type: Regular Long Paper
Submission Track: NLP Applications
Submission Track 2: Theme Track: Large Language Models and the Future of NLP
Keywords: text games, code generation, simulation
TL;DR: We assess language models' ability to create world models via text-game generation, introducing ByteSized32, a corpus of Python text games for testing this. GPT-4 produced runnable games in 27% of cases, with potential for future improvement.
Abstract: In this work we investigate the capacity of language models to generate explicit, interpretable, and interactive world models of scientific and common-sense reasoning tasks. We operationalize this as a task of generating text games, expressed as hundreds of lines of Python code. To facilitate this task, we introduce ByteSized32, a corpus of 32 reasoning-focused text games totalling 20k lines of Python code. We empirically demonstrate that GPT-4 can use these games as templates for single-shot in-context learning, successfully producing runnable games on unseen topics in 28% of cases. When allowed to self-reflect on program errors, game runnability substantially increases to 58%. While evaluating simulation fidelity is labor intensive, we introduce a suite of automated metrics to assess game fidelity, technical validity, adherence to task specifications, and winnability, showing a high-degree of agreement with expert human ratings. We pose this as a challenge task to spur further development at the juncture of world modeling and code generation.
Submission Number: 3974
Loading