3D-GPT: Procedural 3D Modeling with Large Language Models

Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, Stephen Gould

Abstract: In the pursuit of efficient automated content creation, procedural generation, leveraging modifiable parameters and rule-based systems, has emerged as a promising approach. Nonetheless, this can be a demanding endeavor, given its intricate nature necessitating a deep understanding of rules, algorithms, and parameters. To reduce workload, we introduce 3D-GPT, a framework utilizing large language models (LLMs) for instruction-driven 3D modeling. 3D-GPT positions LLMs as proficient problem solving agents, dissecting the procedural 3D modeling tasks into accessible segments and appointing the appropriate agent for each task. 3D-GPT integrates three core agents: the task dispatch agent, the conceptualization agent, and the modeling agent. They collaboratively achieve two objectives. First, they enhance the concise initial scene descriptions, elaborating details while dynamically adapting the text based on subsequent instructions. Second, they integrate procedural generation, extracting parameter values from enriched text to seamlessly interface with 3D software for asset creation. Our empirical investigations confirm that 3D-GPT not only correctly interprets instructions, delivering reliable results but also collaborates effectively with human designers. Furthermore, it integrates directly with Blender, unlocking expanded manipulation possibilities. Our work highlights the potential of LLMs in 3D modeling, offering a basic framework for future advancements in scene generation and animation.