From unimodal to multimodal learning with adaptive alignment for 2D-3D visual recognition

Abstract

This dissertation considers unimodal and multimodal learning with adaptive alignment for 2D-3D visual recognition on images and point clouds. Regarding unimodality on 2D images, we investigate object detection and instance segmentation tasks, which are commonly formulated by a two-stage pipeline of RPN and R-CNN. We propose Cascade RPN with Adaptive Convolution to ensure alignment between features and reference boxes which is required for progressive refinement. For the R-CNN, we revisit Cascade Mask R-CNN and propose SCNet to align sample distribution between training and inference in existing cascade architectures. For unimodality on 3D point clouds, we propose SoftGroup to perform grouping on soft scores to avoid error propagation from hard semantic prediction into instance segmentation. SoftGroup is further extended to SoftGroup++ for scalable 3D instance segmentation with an adaptive strategy to reduce time complexity and search space. Finally, we propose Bird Eye View (BEV) fusion for multimodal object detection that aligns image and point features via BEV projection followed by weighted fusion to address the limitation of sparse points for far objects. Extensive experiments on various standard benchmarked datasets demonstrate the superiority and generality of the proposed methods.