Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation

Yujin Chen1, Yinyu Nie1, Benjamin Ummenhofer2, Reiner Birkl2, Michael Paulitsch2, Matthias Müller2, Matthias Nießner1
1Technical University of Munich   2Intel Labs
Preprint
arXiv Video Code
MY ALT TEXT

We propose Mesh2NeRF, a novel method for extracting ground truth radiance fields directly from 3D textured meshes by incorporating mesh geometry, texture, and environment lighting information. Mesh2NeRF serves as direct 3D supervision for neural radiance fields, offering a comprehensive approach to leveraging mesh data for improving novel view synthesis performance. Mesh2NeRF can function as supervision for generative models during training on mesh collections, advancing various 3D generation tasks, including unconditional and conditional generation.

Abstract

We present Mesh2NeRF, an approach to derive ground-truth radiance fields from textured meshes for 3D generation tasks. Many 3D generative approaches represent 3D scenes as radiance fields for training. Their ground-truth radiance fields are usually fitted from multi-view renderings from a large-scale synthetic 3D dataset, which often results in artifacts due to occlusions or under-fitting issues. In Mesh2NeRF, we propose an analytic solution to directly obtain ground-truth radiance fields from 3D meshes, characterizing the density field with an occupancy function featuring a defined surface thickness, and determining view-dependent color through a reflection function considering both the mesh and environment lighting. Mesh2NeRF extracts accurate radiance fields which provides direct supervision for training generative NeRFs and single scene representation. We validate the effectiveness of Mesh2NeRF across various tasks, achieving a noteworthy 3.12dB improvement in PSNR for view synthesis in single scene representation on the ABO dataset, a 0.69 PSNR enhancement in the single-view conditional generation of ShapeNet Cars, and notably improved mesh extraction from NeRF in the unconditional generation of Objaverse Mugs.

Video

Single Scene Fitting

Mesh2NeRF can be applied to supervision of a single scene radiance field representation from a given mesh. We compare Mesh2NeRF NGP with Instant NGP.


Instant NGP Ours (Mesh2NeRF NGP) Ground Truth

NeRF Generation

Mesh2NeRF can be applied to supervise NeRF generation tasks. We compare Mesh2NeRF supervision to SSDNeRF with traditional NeRF supervision on conditional NeRF generation tasks.


BibTeX


      @article{chen2024mesh2nerf,
        title={Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation},
        author={Chen, Yujin and Nie, Yinyu and Ummenhofer, Benjamin and Birkl, Reiner and Paulitsch, Michael and Müller, Matthias and Nie{\ss}ner, Matthias},
        journal={arXiv preprint},
        year={2024}
      }