Context-guided depth completion via deep learning

Abstract

In this thesis, we present an end-to-end convolutional neural network (CNN) for depth completion. We aim to solve to major issues for effective depth completion. The first issue is to properly propagate the input depth samples. The second issue is to perform proper refinement of the initially propagated dense depth map. Our corresponding network consists of a geometry network, a convolutional spatial propagation network, and a context network. The geometry network, a single encoder-decoder network, learns to optimize a multi-task loss to generate an initial propagated depth map and a surface normal. The complementary outputs allow it to correctly propagate initial sparse depth points in slanted surfaces. The convolutional spatial propagation network learns the 8-way propagation affinity weights for better propagation from the input depth samples. The context network extracts a local and a global feature of an image to compute a bilateral weight, which enables it to preserve edges and fine details in the depth maps. We revisit and apply the traditional weighted median filter, with using the bilateral weight learnt from the context network. In order to validate the effectiveness and the robustness of our network, we performed extensive ablation studies and compared the results against state-of-the-art CNN-based depth completions, where we showed promising results on various scenes.