SceneX: Procedural Controllable Large-scale Scene Generation via Large-language Models

Mengqi Zhou1, Jun Hou2, Chuanchen Luo2, Yuxi Wang2, Zhaoxiang Zhang2, Junran Peng1,2,
1Institute of Automation, Chinese Academy of Sciences, 2Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences
Paper arXiv Code

Abstract

Due to its great application potential, large-scale scene generation has drawn extensive attention in academia and industry. Recent research employs powerful generative models to create desired scenes and achieves promising results. However, most of these methods represent the scene using 3D primitives (e.g. point cloud or radiance field) incompatible with the industrial pipeline, which leads to a substantial gap between academic research and industrial deployment. Procedural Controllable Generation (PCG) is an efficient technique for creating scalable and high-quality assets, but it is unfriendly for ordinary users as it demands profound domain expertise. To address these issues, we resort to using the large language model (LLM) to drive the procedural modeling. In this paper, we introduce a large-scale scene generation framework, SceneX, which can automatically produce high-quality procedural models according to designers' textual descriptions.Specifically, the proposed method comprises two components, PCGBench and PCGPlanner. The former encompasses an extensive collection of accessible procedural assets and thousands of hand-craft API documents. The latter aims to generate executable actions for Blender to produce controllable and precise 3D assets guided by the user's instructions. Our SceneX can generate a city spanning 2.5 km * 2.5 km with delicate layout and geometric structures, drastically reducing the time cost from several weeks for professional PCG engineers to just a few hours for an ordinary user. Extensive experiments demonstrated the capability of our method in controllable large-scale scene generation and editing, including asset placement and season translation.

Tree generate demo

The proposed SceneX can create large-scale 3D natural scenes or unbounded cities automatically according to user instructions.

The proposed SceneX can create large-scale 3D natural scenes or unbounded cities automatically according to user instructions. The generated models are characterized by delicate geometric structures, realistic material textures, and natural lighting, allowing for seamless deployment in the industrial pipeline.

The generated models are characterized by delicate geometric structures, realistic material textures, and natural lighting, allowing for seamless deployment in the industrial pipeline.

The PCG Planner framework

The proposed SceneX can create large-scale 3D natural scenes or unbounded cities automatically according to user instructions. The generated models are characterized by delicate geometric structures, realistic material textures, and natural lighting, allowing for seamless deployment in the industrial pipeline.

The PCG Planner framework comprises three essential components: the task planner, asset retrieval, and action execution. This framework empowers LLMs with the capabilities for task planning in complex scenarios, utilizing multiple API actions, and facilitating large-scale scene generation.

Citation

@article{,
  author = {Mengqi Zhou and Jun Hou and Chuanchen Luo and Yuxi Wang and Zhaoxiang Zhang and Junran Peng},
  title  = {SceneX: Procedural Controllable Large-scale Scene Generation via Large-language Models},
  journal = {arXiv preprint arXiv:2403.15698},
  year = {2024},
}