Face matting in image and video

Abstract

Recently, there has been active research in matting, focusing on the precise separation of objects from the background, especially when dealing with complex boundaries. This task is challenging as it involves predicting the position and extent of objects and requires a fine-grained distinction of object boundaries. To address this challenge, prior research has often resorted to providing hints to models about the target's location and extent using auxiliary inputs such as binary masks or trimaps. However, generating auxiliary inputs for each input can be costly. Meanwhile, facial transformation technologies are gaining increased demand across various application domains. However, the quality of facial transformation may significantly deteriorate when there are occlusions such as hair, or motion blur of objects on the face. In light of this, this paper introduces a novel problem known as ``Facial Matting", which aims to overcome these limitations by leveraging matting technology. As creating datasets for face matting is time-consuming and labor-intensive, we construct an artificially generated image dataset for training purposes. We empirically validated the dataset's effectiveness across various matting networks in trimap-free way, and assessed their performance using synthetically generated test benchmarks. Furthermore, to imbue time consistency into facial transformation techniques primarily performed in videos, we propose a framework for ``Temporal Consistency in Video Facial Matting" that utilizes non-labeled video training datasets. This framework aims to ensure that video facial transformations maintain temporal consistency throughout the duration of the video. In the end, we confirmed that employing this method enhances the quality of facial transformation technologies.