Mitigating the modality bias in multispectral pedestrian detection using counterfactual strategies

Abstract

Multispectral pedestrian detection using RGB and thermal sensors (RGBT) has emerged as a promising solution for safety-critical vision applications that require non-stop operations all day/night. However, there are unsolved issues in multispectral pedestrian detection, including the modality bias problem. The imbalanced modality distribution in RGBT datasets provoke modality bias, where models tend to rely on one modality (thermal) over the other (RGB). Therefore, it is necessary to address the modality bias problem in order to learn multimodal relationships robustly in real-world environments. We deal with modality bias problems for multimodal representation through counterfactual approaches that can compensate for modality imbalance in datasets. First, we propose a novel model framework: Causal Mode Multiplexer (CMM) based on counterfactual intervention and guide the model to learn the causality between multimodal inputs and outputs. Different from the symmetrical fusion topology of existing methods, the proposed approach leverages two distinct causal graphs that are tailored to the multimodal data type. Second, we introduce a novel data augmentation framework: Prototypical Cross-modal Balancing (PCB) based on counterfactual image generation. Unlike existing augmentation methods, PCB generates multimodal data considering the modality balance of multimodal data. Each of the proposed methods from the model and data perspective are validated under extensive experiments including comparisons to the state-of-the-art methods, ablation studies, and further qualitative/quantitative results.