Integrated RGB-D imaging for efficient 3D scene understanding

Abstract

The objective of RGB-D imaging is to capture a color image associated with a depth or distance map. Capturing a scene's appearance and geometry information has become an essential imaging modality for applications in robotics, autonomous driving, and virtual and augmented reality. However, accurate RGB-D imaging systems often rely on multiple sensors, several shots, and/or high-power active illumination, resulting in increased hardware and computational costs.

This thesis presents imaging systems designed to recover the color and geometry of a scene under challenging constraints. The objective is to develop relevant RGB-D imaging approaches that enable higher-quality geometry and appearance acquisition at lower hardware and/or computational costs. Specifically, we attempt to minimize the number of sensors, reduce the use of high-power active illumination, and/or demonstrate real-time performance.

In this context, we develop three methods for RGB-D imaging in constrained scenarios. First, we present a monocular single-shot RGB-D imaging system based on uneven double refraction. Using a polarizer and a birefringent medium, we achieve real-time RGB-D imaging from a single RGB sensor. Second, we introduce an integrated omnidirectional system from fisheye images that provides RGB-D panoramas in real-time on an embedded board. Lastly, we propose a ToF/Stereo fusion method using mobile phone sensors. We accommodate the unknown lens motion and noisy low-power Time-of-Flight (ToF) module of the battery-powered device through per-snapshot calibration and effective inclusion of the ToF samples into the stereo correlation volume.

Overall, this thesis contributes to advancements in RGB-D imaging, offering practical solutions for efficient 3D scene understanding under various constraints.