Extreme video frame interpolation: extending VFI to video with extremely large motion

Abstract

Video Frame Interpolation (VFI) temporally interpolates one or more intermediate frames between every two consecutive frames with temporal coherence so that smoothly rendered fast motion is more visually pleasing. However, VFI is a long-standing and challenging task in computer vision, which is attributed to several factors such as non-linear motion, deformable object motion, large motions and motion blur in the video sequences. In this dissertation, we focus on the challenging issue of fast motions with extremely large pixel displacements which often occur in ultra-high-definition (UHD) video sequences and thus result in severe motion judder. To study VFI with large motion, we present an extreme VFI network, called XVFI-Net, that first handles the VFI for 4K videos with large motion. Even after training of the XVFI-Net, input frames can be flexibly down-scaled into any smaller size to cope with the spatial resolution and the degree of motion magnitudes of input frames during inference time, unlike the previous pyramid structures of a fixed number of scale levels. The recursive multi-scale shared structure of XVFI-Net allows for the large motion to be effectively captured in flexibly small-scale levels. In order to interpolate middle frames at any intermediate time instances, the bilateral optical flows are stably approximated by a novel complementary flow reversal (CFR) technique using the bidirectional motion. We also propose a coarse-direction-and-fine-attention (CDFA) module to extend the XVFI-Net, where multi-hypothesis flow vectors are utilized to deal with complex motion boundaries of fast-moving objects. The coarse direction vector is supervised by a self-guided training scheme. Then the fine attention vectors attend to the local details based on the coarse direction vector in a residual learning manner. Extensive experimental results show that our algorithms can successfully capture the essential information of objects with extremely large motions and complex textures while state-of-the-art methods exhibit poor performance. Furthermore, our XVFI-Net framework also performs comparably on the previous lower resolution benchmark dataset, which shows the robustness of our algorithm as well.