Video-guided facial animation

Abstract

In this thesis, we present a novel method for synthesizing a realistic facial animation taking as input a single video which captures various facial expressions and the 3D template mesh for a head model. In the capture process, we do not employ any markers or structured light on the face as it obstructs the natural performance of the actor or actress. Our basic idea is to use the temporal coherency between consecutive frames and model the facial skin physically to drive the 3D facial animation. To do that, we first estimate the camera matrix using interactively specified corresponding point pairs between the given mesh and the image at the initial frame. At the same time, the temporal coherency between all pairs of consecutive images is estimated using optical flow algorithm. Then, the head motion is estimated to extract the local motion from the optical flow. We refine the given template mesh using the local motion with modeling a facial skin. After that, we apply the estimated head motion to the template mesh for a natural animation. To estimate the error of deformed mesh, we first synthesize an image by projecting the texture-mapped mesh onto the image plane. The error map is constructed by subtracting the synthesized image from the image of the next frame. Then, we locally compute the optical flow using the error map. Only the part of the mesh is refined again where the optical flow is updated. We iterate this process until the error reduces below the predefined threshold or with fixed number of times.