(The) interplay between part segmentation and structure inference for 3D shape parsing

Abstract

We propose Seg&Struct, a supervised learning framework leveraging the interplay between part segmentation and structure inference and demonstrating their synergy in an integrated framework. Both 3D segmentation and structure inference have been extensively studied in the recent deep learning literature, while the supervisions used for each task have not been fully exploited to assist the other task. Namely, structure inference has been typically conducted with an autoencoder that does not leverage the point-to-part associations. Also, 3D segmentation has been mostly performed without structural priors that tell the plausibility of the structure created from the output segments. We present how these two tasks can be best combined while fully utilizing supervision to improve performance. Our framework first decomposes a raw input shape into segments using an off-the-shelf segmentation network, whose outputs are then mapped to nodes in a part hierarchy. Following this, a hierarchical message passing based on the structural priors regresses the parameters of part bounding boxes and classified the part relationships. Once the structure is predicted, the segmentation is rectified by examining the confusion of part boundaries using the structure-based part features. The refined segmentation can again be used to improve the predicted structure. Our experimental results based on the StructureNet dataset demonstrate that the interplay between the two tasks results in significant improvements in both tasks: 30.94% in structure inference and 0.6% in segmentation. We also showcase a structure-based shape retrieval that enables finding semantically similar shapes.