Deep joint deblurring and multi-frame interpolation with flow-guided attentive correlation and recursive boosting

Abstract

Video frame interpolation (VFI) synthesizes intermediate frames to temporally upscale a low frame rate (LFR) video to a high frame rate (HFR) one, which provides a visually pleasing experiences to users. On the other hand, motion blur is generally induced when capturing videos due to the accumulations of the light. Therefore, eliminating the motion blur, called deblurring, is also important to synthesize sharp intermediate frames while performing VFI. In this dissertation, we propose a novel joint deblurring and multi-frame interpolation (DeMFI) framework, called DeMFI-Net, which accurately converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate based on flow-guided attentive-correlation-based feature bolstering (FAC-FB) module and recursive boosting (RB), in terms of multi-frame interpolation (MFI). The DeMFI-Net jointly performs deblurring and MFI where its baseline version performs feature-flow-based warping with FAC-FB module to obtain a sharpinterpolated frame as well to deblur two center-input frames. Moreover, its extended version further improves the joint task performance based on pixel-flow-based warping with GRU-based RB. Our FAC-FB module effectively gathers the distributed blurry pixel information over blurry input frames in feature-domain to improve the overall joint performances, which is computationally efficient since its attentive correlation is only focused pointwise. Furthermore, we conduct diverse experiments on module locations, occlusion maps, efficient version, one-stage version and adaptivity on fps to deeply study our network for DeMFI. As a result, our DeMFI-Net achieves state-of-the-art (SOTA) performances for diverse datasets with significant margins compared to the recent SOTA methods, for both deblurring and MFI. All source codes including pretrained DeMFI-Net are publicly available at https://github.com/JihyongOh/DeMFI.