RGBD panorama synthesis using normal field-of-view cameras and mobile depth sensors in arbitrary configurations

Abstract

Omnidirectional images provide all-around visual information obtained from a single viewpoint, as opposed to normal field-of-view (NFoV) images having limited information of the scene. Meanwhile, having depth data along with RGB data enables us to generate colored 3D point cloud data, which is widely used for 3D reconstruction and representation. As such, generating omnidirectional RGBD data can provide much more immersive information than having a panorama image only or having NFoV RGBD data. However, this task is hard to achieve due to the highly correlated bi-modal information and has received less attention. This paper tackles a novel problem: ‘synthesizing’ 360º RGBD panorama from non-omnidirectional RGB and limited depth data. By training a deep convolutional neural network using input data obtained with arbitrary configurations of a camera and a depth sensor, the network can generate realistic 360º RGBD panorama correspondingly. As there is no performance evaluation metric for the generated RGBD panorama, I also propose a method to evaluate its perceptual quality. Experiments show that the proposed method generates visually pleasing RGBD panorama under arbitrary configurations of a camera and a depth sensor, and the proposed evaluation method matches well with human perception.