From dark flash images to relightable 3D scenes with photometric stereo priors

High-quality 3D content reconstruction has always been the focus of computer vision and graphics research. Reconstructing scenes in dark environments is a challenging task, where flash photography acts as the primary light source but introduces complex photometric changes. The existing neural rendering methods generally have the problems of geometric noise and loss of details when processing such low-light data. To solve the problems, a research team from the National University of Defense Technology and Hunan Province’s Xiangjiang Laboratory (including Academician Chen Xiaohong, Prof. Xu Kai, Associate Prof. Yi Renjiao, and Prof. Zhu Chenyang, among others) published their new research in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature.

The team proposed a novel framework called Dark-2DGS, which combines photometric stereo priors with 2D Gaussian Splatting (2DGS) to reconstruct high-fidelity geometry. The method is verified and tested on both real-world dark scenes and synthetic datasets. Compared with the existing research results, the proposed method has higher geometric accuracy and visual quality.

In the research, they analyze the photometric cues in the flash images and use the physical lighting model to guide the geometry optimization of the neural field. In order to more accurately reconstruct the 3D scene from the dark data, they design a coarse-to-fine optimization strategy that integrates surface material information into the rendering pipeline to detect and recover intricate details.
Firstly, the standard 2DGS is used to establish the initial geometry of the scene. Then, a Photometric Stereo (PS) enhancement module is introduced to analyze the illumination variations in multi-view images to estimate feature values (material) of the surface, which are then integrated into the optimization process to refine the geometry and material. Finally, the method performs inverse rendering to decompose the scene attributes. The experimental data shows that compared with the existing view synthesis methods, the proposed method is more robust to dark and has the characteristics of high reconstruction accuracy and the ability to support relighting.