Robust 3D imaging using a single hand-held camera

Abstract

Recently, there has been increasing interest in the capture of depth information on mobile devices, to enable a variety of AR/VR and photographic applications. Conventional depth cameras require additional devices, such as the ToF sensor or structured-light type sensor, or Lightfield cameras, which increase the manufacturing costs and the size of devices. Due to the impracticality of the conventional depth sensor, the monocular vision-based approaches are emerging as a new alternative. However, vision-based methods are susceptible to the capturing environment. In this dissertation, we pursue to develop the robust 3D map generation method to the illumination changes and texture-less regions using hand-held cameras. We focus on vision-based technology that can be applied in the real world. We achieve the generality by using the data-driven approaches whose design is inspired by best practices of traditional geometry-based approaches.

First, we proposed a dense 3D reconstruction method for the rolling shutter cameras. Commercial hand-held cameras are mostly equipped with the rolling shutter cameras to reduce the manufacturing cost, so they typically cause the undesired rolling shutter artifact. In this study, we introduced a novel small motion bundle adjustment that effectively compensates for the rolling shutter effect. Moreover, we proposed a pipeline for a fine-scale dense 3D reconstruction that models the rolling shutter effect by utilizing both sparse 3D points and the camera trajectory from narrow-baseline images. In this reconstruction, the sparse 3D points are propagated to obtain an initial depth hypothesis using a geometry guidance term. Then, the depth information on each pixel is obtained by sweeping the plane around each depth search space near the hypothesis.

Second, we proposed a practical method that generates an all-around dense depth map using a narrow-baseline video clip captured by an SPC. While existing methods for depth from small motion rely on perspective cameras, we introduced a new bundle adjustment approach tailored for SPC that minimizes the re-projection error directly on the unit sphere. It enables to estimate approximate metric camera poses and 3D points. Additionally, we presented a novel dense matching method called sphere sweeping algorithm. This allows us to take advantage of the overlapping regions between the cameras.

Moreover, we presented a robust depth estimation method from a short burst shot with varied intensity (i.e., Auto-exposure bracketing) and strong noise (i.e., High ISO). Our key idea synergistically combines deep convolutional neural networks with a geometric understanding of the scene. In this study, we introduced a geometric transformation between optical flow and depth tailored for burst images, enabling our learning-based multi-view stereo matching to be performed effectively. we then described our depth estimation pipeline that incorporates this geometric transformation into our residual-flow network. It allows our framework to produce an accurate depth map even with a bracketed image sequence. We showed that the estimated depth is applicable for image quality enhancement and photographic editing.

Lastly, we presented a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches. Rather than directly estimating depth and/or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a context-aware cost aggregation, and regressing the depth map from the cost volume. The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network.