Subtle changes discriminative latent feature representation for facial expression recognition

Abstract

Recently, spontaneous facial expression recognition (FER) has gained increasing attention for computing human affect. In addition, recognizing spontaneous and subtle facial expressions are becoming increasingly important because current and future face-related applications lie in a more natural context. However, the recognition of spontaneous and subtle facial expression (or micro-expression) which appear as a subtle move-ment of facial parts has been regarded as a very difficult problem, due to its low intensity and short duration. In this thesis, we propose subtle changes discriminative latent spatio-temporal feature representation in 3D convo-lutional neural network (CNN) for spontaneous FER. To effectively exploit spatial and temporal characteristics of subtle facial expressions in CNN, following methods have been devised

1) expression prior based filter initial-ization that generates prototypical patches which have meaningful spatio-temporal characteristics of facial parts to utilize the prior of subtle facial expression, 2) filter update that emphasizes temporal differences of fil-ters, to make the network more sensitive to temporal changes of input sequence, 3) feature fusion with motion parts based CNN to give complementary information with texture. By the extensive experiments using CASME II dataset (containing spontaneous and subtle facial expressions), the results show that every step of the pro-posed latent feature representation helps to achieve higher spontaneous FER accuracy than existing latent fea-ture representation as well as hand-crafted features. Also, when visualizing the latent features by the proposed methods compared to the conventional CNN, the significant increase of discriminability in the training and test set is observed.