LL3M: Large Language 3D Modelers

Summary

LL3M uses a team of large language models to write Python code that creates and edits 3D assets in Blender. Given user text instructions, the agents are capable of creating expressive shapes from scratch, and realizing complex, precise geometric manipulations in code. Whereas previous uses of code-writing LLMs for 3D creation have focused on specific subtasks or constrained procedural programs and primitives, our method is able to create unconstrained assets with geometry, layout, and appearance. With high-level code as a 3D representation, our pipeline is natively a loop of iterative refinement and co-creation: agents perform automatic code and visual self-critique, and users can provide continuous high-level feedback. Further editing avenues are enabled by the clear code and the parameters transparent in the generated Blender nodes and structures. Pipeline overview Our method includes three phases: initial creation, automatic refinement, and user-guided refinement. These are conceptual phases in the creation process; each phase involves different agent roles of its own. The first phase creates an initial shape, where implausible configurations, like a disconnected backrest, as well as simplistic geometry are automatically corrected and improved upon by the second phase. Afterwards, our system can accept additional edit instructions from the user, allowing for interactive and iterative 3D asset generation. Generating and refining iteratively is thus the native mode of operation for LL3M. More than just error correction, the pipeline realizes an iterative, coarse-to-fine creation process, involving both automatic and user-guided refinement. Gallery LL3M is capable of diverse shape generation. The results showcase detailed parts (e.g. the windmill architectural features) in intricate arrangements (e.g. the piano keys, the drum kit), and even a rich appearance (the skateboard) and material properties (the glossy lamp base). A notable feature of our approach is that ea...