Design and fusion of pose-invariant face-identification experts

Abstract

We address the problem of pose-invariant face recognition based on a single model image. To cope with novel view face images, a model of the effect of pose changes on face appearance must be available. Face images at an arbitrary pose can be mapped to a reference pose by the model yielding view-invariant representation. Such a model typically relies on dense correspondences of different view face images, which are difficult to establish in practice. Errors in the correspondences seriously degrade the accuracy of any recognizer. Therefore, we assume only the minimal possible set of correspondences, given by the corresponding eye positions. We investigate a number of approaches to pose-invariant face recognition exploiting such a minimal set of facial features correspondences. Four different methods are proposed as pose-invariant face recognition "experts" and combined in a single framework of expert fusion. Each expert explicitly or implicitly realizes the three sequential functions jointly required to capture the non-linear manifolds of face pose changes: representation, view transformation, and class discriminative feature extraction. Within this structure, the experts are designed for diversity. We compare a design in which the three stages are sequentially optimized with two methods which employ an overall single nonlinear function learnt from different view face images. We also propose an approach exploiting a three-dimensional face data. A lookup table storing facial feature correspondences between different pose images, found by 3-D face models, is constructed. The designed experts are different in their nature owing to different sources of information and architectures used. The proposed fusion architecture of the pose-invariant face experts achieves an impressive accuracy gain by virtue of the individual experts diversity. It is experimentally shown that the individual experts outperform the classical linear discriminant analysis (LDA) method on the XM2VTS face data set consisting of about 300 face classes. Further impressive performance gains are obtained by combining the outputs of the experts using different fusion strategies.