ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality

Jaewook Lee, Filippo Aleotti, Diego Mazala, Guillermo Garcia-Hernando, Sara Vicente, Oliver James Johnston, Isabel Kraus-Liang, Jakub Powierza, Donghoon Shin, Jon E. Froehlich, Gabriel Brostow, Jessica Van Brummelen

Abstract: While augmented reality (AR) enables new ways to play, tell sto-ries, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, oftenrequiring professional tools and time. Prior systems have exploredAI-driven XR design but typically rely on manually defined VRenvironments and fixed asset libraries, limiting creative flexibil-ity and real-world relevance. We introduce ImaginateAR, the firstmobile tool for AI-assisted AR authoring to combine offline sceneunderstanding, fast 3D asset generation, and LLMs—enabling usersto create outdoor scenes through natural language interaction. Forexample, saying “a dragon enjoying a campfire” (P7) prompts thesystem to generate and arrange relevant assets, which can then berefined manually. Our technical evaluation shows that our custompipelines produce more accurate outdoor scene graphs and gener-ate 3D meshes faster than prior methods. A three-part user study(N=20) revealed preferred roles for AI, how users create in freeform use, and design implications for future AR authoring tools.ImaginateAR takes a step toward empowering anyone to create ARexperiences anywhere—simply by speaking their imagination.