Multiview Stereoscopic Video Hole Filling Considering Spatiotemporal Consistency and Binocular Symmetry for Synthesized 3D Video

Abstract

This paper proposes a new hole-filling method with spatiotemporal consistency and binocular symmetry for synthesized 3D videos in view extrapolation. Disocclusion regions in the synthesized views at virtual viewpoints result in regions with missing content. These regions will be referred to as hole regions. To provide the high-quality synthesized 3D videos via 3D display, the hole regions need to be filled considering the characteristics of human visual perception. From the perceptual point of view, binocular asymmetry between synthesized left-and right-eye videos (i.e., stereo pair video) is one of the most important factors that induce visual discomfort in stereoscopic viewing. In addition, binocular symmetry without temporal consistency between temporally neighboring frames could cause visual discomfort by annoying flickering artifacts. In this paper, to maintain the spatiotemporal consistency and binocular symmetry in synthesized 3D videos at multiple virtual viewpoints, we propose a global optimization-based hole-filling method using the information from the already filled adjacent view and previous frame. Furthermore, to reduce the computational cost of the global optimization, we propose a label propagation method, which propagates reliable labels used in the adjacent view and previous frame to the target image to be filled. The performance of the proposed method has been evaluated by objective assessments of 3D image quality, temporal consistency, and computational efficiency. In addition, subjective assessment is conducted for measuring the visual comfort and overall quality. The experimental results proved that the proposed method provides hole filling results with spatiotemporal consistency and binocular symmetry.