Probabilistic representation learning for improved cross-modal retrieval using density-wise similarity

Abstract

For cross-modal retrieval tasks, building a joint representation space for data samples from different modalities has been a common practice especially from the vision and language domains. The two characteristics of image and caption pairs that make this task especially challenging are the multiplicity of matches and partiality of matching pairs. Given an image or a caption, there are multiple positive captions or images and for each positive image-caption pair, the captions convey only the key concepts at interest while ignoring other components. Previous researches, which are based on learning pointwise embeddings in a deterministic way, fail to capture this one-to-many correspondences nor correctly calibrate the semantic intersection between arbitrary image-caption pairs. This paper proposes a generalized method of learning the representations of images and captions as probabilistic distributions in the joint representation space and explicitly model cross-modal uncertainty with differential entropy. The probabilistic embeddings are parametrically learned by fusing a visual, text head module to a pretrained visual text encoder and trained in a two-staged manner. Through extensive qualitative experiments on MS-COCO and Flickr30K datasets, the paper demonstrates the benefit of using probabilistic representations by showing how cross-modal uncertainty can measure the multiplicity within each sample and how density-wise similarity preserves the partial similarity of each image-caption pair.