Learning to embed, align, and augment

Abstract

In computer vision, images are transformed into various forms for various purposes. For example, images are transformed into embedding vectors lie on special space to improve performance and robustness. Face images are transformed for locating their eyes, nose, and mouth in the same position. Images are augmented through label preserving transformations. This dissertation studies models and learning methods to transform images---embed, align, augment---with application to face and object recognition.

First, this dissertation proposes a sparsity sharing embedding method that transforms face images into embedding vectors which are robust to variations in pose, illumination, and expression. The SSE is built on a generic identity dataset where each identity contains multiple faces under large intra-personal settings. An embedding space is learned to preserve inter-personal structures of intra-personal settings. Face images are transformed into embedding vectors, thus robust face verification under large variations in pose, illumination, and expression can be achieved.

Second, two face alignment methods that locating eyes, nose, and mouth at the same position are proposed. The first method, a parallel joint boosting, simultaneously estimates poses and face landmarks. It iteratively updates the poses and face landmarks in a stage-wise manner: pose probabilities are updated based on previous face landmark estimates and face landmark estimates are updated based on previous pose probabilities. The second method is cascade Gaussian process regression trees (cGPRT). Here, GPRT is a Gaussian process with a kernel defined by a set of trees. Without increasing prediction time, the prediction of cGPRT can be performed in the same framework as the cascade regression trees (CRT) but with better generalization.

Lastly, a data augmentation method is proposed to learn image transformations that improve generalization performance. Data augmentation has a large impact on the generalization performance of the image classification model. However, it is currently conducted on the basis of trial and error, and thus, the generalization performance cannot be predicted during training. This study considers an influence function that predicts how generalization performance is affected by a particular augmented training sample in terms of validation loss. The influence function provides an approximation of the change in validation loss without actually comparing the performances that include and exclude the sample in the training process. Based on this function, a differentiable augmentation network is learned to augment the input training sample to reduce validation loss.