PBR3DGen: A VLM-guided mesh generation with high-quality PBR textures


1The Hong Kong Polytechnic University, 2Xi'an Jiaotong University,
3Nanyang Technological University, 4Tencent Hunyuan
(*Equal Contribution)

Abstract

we present PBR3DGen, a two-stage mesh generation method with high-quality PBR materials that integrates the novel multi-view PBR material estimation model and a 3D PBR mesh reconstruction model. Specifically, PBR3DGen leverages vision language models (VLM) to guide multi-view diffusion, precisely capturing the spatial distribution and inherent attributes of reflective-metalness material. Additionally, we incorporate view-dependent illumination-aware conditions as pixel-aware priors to enhance spatially varying material properties. Furthermore, our reconstruction model reconstructs high-quality mesh with PBR material. Experimental results demonstrate that PBR3DGen significantly outperforms existing methods, achieving new state-of-the-art results for PBR estimation and mesh generation.

Method Overview

Overall pipeline. This pipeline of PBR3DGen is composed of Multi-View PBR generation module and PBR-LRM module.

Video

More Results

Image-condition PBR mesh generation

Text-condition PBR mesh generation

Applications for 3D generation assets

Relighting

Animation

-->