Unsupervised synthetic-to-Real transfer learning of depth images in consideration of sensor characteristics

Abstract

In this work, we propose an unsupervised method to generate synthetic depth images with the noise characteristics of commercial depth cameras in table-top bin-picking scenarios. Due to the degradation and measurement uncertainties of depth cameras, depth images captured in the real world are vastly different from synthetically generated depth images. To model noise in Amplitude-Modulated Continous- Wave (AMCW) time-of-flight cameras, we integrate the time-of-flight depth measurement procedure of such cameras into a redefined Cycle-consistent Generative Adversarial Network (CycleGAN) framework to generate noise for synthetic depth images using a rendering approach. We show perceptually real- istic depth images generated based on the T-LESS dataset on cluttered table-top bin-picking scenarios collected from a time-of-flight camera (Kinect V2). Moreover, we compare perceptual similarity of our results with raw sensor measurements using structural similarity metrics and Fréchet distance in autoen- coder latent space.