C2C

Abstract

Lip reading is to predict the spoken sentence from silent lip movement. However, due to the existence of homophenes that similar lip movement with different sound, lip reading is a challenging task and showing inferior performances than speech recognition. To mitigate the homophenes problem in lip reading, in this paper, we propose a novel Context to Context mapping (C2C) method which is mainly composed of two parts: 1) Audio Context Memory Network is designed to complement insufficient visual information by storing and providing both phoneme- and context-level audio knowledge without audio input during the inference phase, and 2) Visual Feature Decomposition Module (VFDM) is presented to figure out subtle differences in similar lip movements by decomposing visual features into multiple latent features in order to capture the different amounts of temporal information. And reconstructed visual feature from latent features can distinguish subtle difference of lip movement. which also be helpful to reconstruct audio knowledge in viseme to phoneme level due to discriminative visual feature. Through the extensive experiments, we validate the effectiveness of the proposed C2C method achieving state-of-the-art performances on two public word-level lip reading datasets.