FaceSyncNet: A deep learning-based approach for non-linear synchronization of facial performance videos

Abstract

Given a pair of facial performance videos, we present a deep learning-based approach that can automatically return a synchronized version of these videos. Traditional methods require precise facial landmark tracking and/or clean audio, and thus are sensitive to tracking inaccuracies and audio noise. To alleviate these issues, our approach leverages large-scale video datasets along with their associated audio tracks and trains a deep learning network to learn the audio descriptors of a given video frame. We then use these descriptors to compute the similarity between video frames in a cost matrix and compute a low-cost non-linear synchronization path. Both quantitative and qualitative evaluations have shown that our approach outperforms existing state-of-the-art methods.