Deep learning architecture for two-image analysis via reference based super resolution

Abstract

In computer vision, various two-image input based tasks including stereo vision, optical-flow estimation have been actively researched. As deep learning architectures are trained on a large-scale dataset to extract fundamental features, extracting the most general correspondence requires a specific dataset to learn. In this paper, in order to learn the most fundamental correspondences, we solve reference based super resolution (RefSR) selecting a dataset containing various correspondences. We propose a correspondence searching and extracting network (CSENet) and prove its utility solving RefSR, self-similarity SR, and sensor fusion. CSENet is able to handle small and large displacements with dynamic offset estimator for deformable convolution and robustly extract correspondences with relevancy-aware weight learning for cluttered or irrelevant input data. The proposed network is end-to-end trainable without any additional supervisions or heavy computations. Experimental results demonstrate a superior performance of the proposed method compared to previous works quantitatively and qualitatively.