Cross-modal alignment and translation for missing modality action recognition

Abstract

Multimodal data provides complementary information on the same context, leading to performance improvement in video action recognition. However, in reality, not all modalities are available at test time. To this end, we propose Cross-Modal Alignment and Translation (CMAT) framework for action recognition that is robust to missing modalities. Specifically, our framework first aligns representations of multiple modalities from the same video sample through contrastive learning, effectively alleviating the bias with respect to the type of missing modality. Then, CMAT learns to translate representations of one modality into that of another modality. This allows the representations of the missing modalities to be generated from the remaining modalities during the testing. Consequently, CMAT fully utilizes multimodal information obtained through abundant interactions across modalities. The proposed CMAT achieves the state-of-the-art performances in both complete and missing modality settings on NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA datasets. Moreover, extensive ablation studies demonstrate the effectiveness of our design.