Frame rate up-conversion using convolutional neural networks

Abstract

Many motion-compensated frame interpolation (MCFI) methods use block matching algorithms (BMAs) for motion estimation (ME). However, the conventional BMAs that are originally developed by focusing on minimizing the prediction errors often fail to detect and project the object motion. In this paper, we present a new MCFI method that utilizes two convolutional neural networks (CNNs) to find the motion vector (MV) with greater reliability and to refine artifacts that are due to the incorrectly estimated MVs in the interpolated frame. The CNN model which is used to estimate MVs is trained to track the projected object motion as closely as possible. Furthermore, we also employ another CNN model to detect artifacts in the interpolated frame. As we apply a motion vector refinement (MVR) scheme only to the region that is needed to be refined, our proposed MVR method can refine the interpolated frame without producing ghost artifacts on the motion boundaries. Experimental results using the standard test video sequences show that our proposed ME method acquired MVs with greater reliability than conventional ME methods. Furthermore, our proposed MCFI method improves the average peak signal-to-noise ratio (PSNR) of interpolated frames.