publications

2023

1. 

   NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds

   Jun-Kun Chen*, Jipeng Lyu*, and Yu-Xiong Wang

   In CVPR, 2023

   Abs arXiv Bib Website

   This paper proposes NeuralEditor that enables neural radiance fields (NeRFs) natively editable for general shape editing tasks. Despite their impressive results on novel-view synthesis, it remains a fundamental challenge for NeRFs to edit the shape of the scene. Our key insight is to exploit the explicit point cloud representation as the underlying structure to construct NeRFs, inspired by the intuitive interpretation of NeRF rendering as a process that projects or "plots" the associated 3D point cloud to a 2D image plane. To this end, NeuralEditor introduces a novel rendering scheme based on deterministic integration within K-D tree-guided density-adaptive voxels, which produces both high-quality rendering results and precise point clouds through optimization. NeuralEditor then performs shape editing via mapping associated points between point clouds. Extensive evaluation shows that NeuralEditor achieves state-of-the-art performance in both shape deformation and scene morphing tasks. Notably, NeuralEditor supports both zero-shot inference and further fine-tuning over the edited scene. Our code, benchmark, and demo video are available at this https URL.

   @inproceedings{chen2023neuraleditor,
     title = {NeuralEditor: Editing Neural Radiance Fields via Manipulating Point Clouds},
     author = {Chen*, Jun-Kun and Lyu*, Jipeng and Wang, Yu-Xiong},
     booktitle = {CVPR},
     year = {2023},
   }

2. 

   NeuralMPS: Non-Lambertian Multispectral Photometric Stereo via Spectral Reflectance Decomposition

   Jipeng Lv*, Heng Guo*, Guanying Chen, and 2 more authors

   In IJCAI, 2023

   Abs arXiv Bib Website

   Multispectral photometric stereo(MPS) aims at recovering the surface normal of a scene from a single-shot multispectral image captured under multispectral illuminations. Existing MPS methods adopt the Lambertian reflectance model to make the problem tractable, but it greatly limits their application to real-world surfaces. In this paper, we propose a deep neural network named NeuralMPS to solve the MPS problem under general non-Lambertian spectral reflectances. Specifically, we present a spectral reflectance decomposition(SRD) model to disentangle the spectral reflectance into geometric components and spectral components. With this decomposition, we show that the MPS problem for surfaces with a uniform material is equivalent to the conventional photometric stereo(CPS) with unknown light intensities. In this way, NeuralMPS reduces the difficulty of the non-Lambertian MPS problem by leveraging the well-studied non-Lambertian CPS methods. Experiments on both synthetic and real-world scenes demonstrate the effectiveness of our method.

   @inproceedings{lv2022neuralmps,
     title = {NeuralMPS: Non-Lambertian Multispectral Photometric Stereo via Spectral Reflectance Decomposition},
     author = {Lv*, Jipeng and Guo*, Heng and Chen, Guanying and Liang, Jinxiu and Shi, Boxin},
     booktitle = {IJCAI},
     year = {2023},
   }