Phrase-frames alignment network with contrastive attention loss for video description

Abstract

This paper considers a video caption generating network referred to as Phrase-Frames Alignment Network (PFAN) that solves the problem of information redundancy of successive sampled frame, prevalent in most video captioning algorithms. As consecutive sampled frames are less likely to provide unique information, prior methods have focused on encoding compact video representation from an input video through various methods such as using a hierarchical encoder or learning to sample informative frames. The PFAN attempts to compactly encode the input video by not only using the visual features of frames but also the semantics of a partially decoded caption. The PFAN (1) forms \textit{semantic groups} by aligning each video frame feature with discriminating word phrases of partially decoded caption and then (2) decodes the semantic groups to predict the next of the partially decoded caption. In contrast to the prior methods, the continuous feedback from decoded words enables the PFAN to dynamically update the video representation that adapts to the partially decoded caption. Furthermore, a contrastive attention loss is proposed to facilitate accurate alignment between word phrases and video frame features without requiring any manual annotations. The PFAN achieves state-of-the-art performances by outperforming runner-up methods by a margin of 2.1% and 2.4% in a CIDEr-D score on MSVD and MSR-VTT datasets, respectively. Extensive experiments are conducted to demonstrate the effectiveness and interpretability of the PFAN.