Joint super resolution of color and depth images using a densely connected residual inception network

Abstract

Nowadays, there are many 3D applications which demand good 3D data including 2D high resolution (HR) image along with high accurate depth map at the same size. It is efficient to capture images and depth maps at low resolution (LR) then upscaling them to high resolution. However, when upsampling the LR images and depth maps especially with large scaling factors, the boundaries of them usually lose sharpness. A new method is designed to address the problem of single image and depth map super-resolution in which HR images and depth maps can infer deep features from LR version of each other. We propose a densely connected residual inception network (DRInet) for this joint problem. The DRInet aggregates an LR input image and its corresponding depth features together to guide each task allowing the network to do better for upsampling of both faraway and closer objects. In addition, when zooming in an image, there is huge difference in quality of further and closer structures. We present a depth-aware bicubic degradation model to formulate the effect of scene depth when downsampling an image and train our network on this degradation model. Our DRInet achieves the best performance comparing the more complex network in both tasks. Furthermore, the depth-aware degradation model improves the robustness of the network.