Physically-based neural rendering of multiplane images

Abstract

In this work, we explore an alternative method for view synthesis, drawing upon neural rendering and inverse rendering techniques applied to multiple images. Recent neural rendering approaches utilize inverse rendering to estimate parameters for physically-based rendering. However, due to its dependence on volume rendering, which accumulates color and density values from multiple samples to determine each pixel's radiance, neural rendering may not be the most preferred choice for real-time rendering applications yet. Multiplane image (MPI) rendering, in contrast, stores color and density information in multiple depth layers, which may provide a more efficient solution for real-time view synthesis. Nonetheless, the storage of image data in the normalized device coordinate system presents challenges when applying conventional inverse rendering methods directly to MPIs. Moreover, inverse rendering for physically-based rendering is seldom compatible with MPIs. To address these limitations, we propose a method for inverse rendering that aims to learn scene material information and light environment, enabling high-quality novel view synthesis, physics-based rendering, and scene editing. Our method represents geometric information in the form of an MPI and trains material data for each scene point. Furthermore, we differentiate the light environment into far-bound and near-bound regions to account for the global and local illumination of real scenes. Our rendering pipeline incorporates the spherical Gaussian approximation for reflectance and illumination, which may be more suited for real-time rendering applications and can integrate effectively with the traditional MPI architecture. The results of our study suggest the potential utility of our physically-based neural rendering approach, illustrating its possible applications in various scene editing tasks, such as relighting and seamlessly changing material appearance.