Time-consistent panoramic video inpainting with 3D convolutional neural networks

Abstract

With the rise of virtual reality, we can see an increase in the production of panoramic videos. Unlike perspective videos, panoramic videos have an unlimited field of view that captures the entire environment around the camera. The result is that everything, including film crew members and equipment, could end up in the final product because there is no place to hide them. To solve this issue, we could remove these unwanted objects from the video in post-production using inpainting. In this paper, a novel method of panoramic video inpainting is presented, one that is based on deep learning. This method makes use of two 3D convolutional neural networks to inpaint panoramic videos. The Frechet Video Distance is added to the loss function to encourage time consistency in the results. Furthermore, a border-matching term is used to make sure the inpainted results follow the border-matching constraint of panoramic imagery. Experiments show that this deep learning method is able to overcome the limitations of optical flow inpainting methods. Moreover, it also outperforms image inpainting methods that are based on deep learning, especially in terms of time consistency.