The First Challenge on Generating Instructions in Virtual Environments

Alexander Koller, Kristina Striegnitz, Donna Byron, Justine Cassell, Robert Dale, Johanna D. Moore, Jon Oberlander

Abstract: This paper describes the First Challenge on Generating Instructions in Virtual Environments (GIVE-1). GIVE is a shared task for generation systems which give real-time natural-language instructions to users in a virtual 3D world. These systems are evaluated by connecting users and NLG systems over the Internet. We describe the design and results of GIVE-1 as well as the participating NLG systems, and validate the experimental methodology by comparing the results collected over the Internet with results from a more traditional laboratory-based experiment.