Video outpainting with human-in-the-loop system and dual-domain propagation framework

Abstract

Video outpainting is a burgeoning research domain aimed at enriching the visual experience as well as the immersion of viewers by extending field-of-view (FOV) of videos. Its primary objective is to fill out the newly expanded regions beyond the original input video’s boundary. Our proposed method comprises two key components: (1) background estimation and (2) foreground estimation. Background estimation involves the segmentation of dynamic objects within the scene, facilitated by a human-in-the-loop system. Optical flow information is then utilized to ensure temporal consistency. On the other hand, foreground estimation employs a frame extrapolation technique to predict the motion of dynamic objects within the expanded areas. Compared to the baseline, our method excels in generating more realistic backgrounds with minimized artifacts. Additionally, it successfully maintains the contextual integrity of moving objects beyond the original video's borderline into the outpainted area. The effectiveness of our approach is evaluated on two datasets: DAVIS and a custom movie dataset comprising 300 clips spanning various movie genres. The results underscore the capability of our method to enhance video content and preserve visual coherence in the extended FOV.