RGB-IR cross-modality person re-identification with pose-transferred image generation

Abstract

Person re-identification (Re-ID) is a popular task in video surveillance and robotic applications, and targets the retrieval of an expected pedestrian image that has the same identity as the query image. The use of cameras at night or in dark environments is common in real-world scenarios. When lighting conditions are poor or unavailable, RGB cameras cannot capture sufficient information, and most camera system would turn to infrared camera mode. Thus, RGB-IR cross-modality Re-ID is introduced for robust person identity retrieval under flexible illumination conditions. However, RGB-IR Re-ID has an additional modality variance issue, originated from the different imaging processes of spectrum cameras. This called “cross-modality variance” and it makes RGB-IR Re-ID research very challenging. However, RGB-IR Re-ID suffers from not only a cross-modality variance, but also “human pose variation” which is a conventional challenging issue in Re-ID tasks. The pose information is an important factor with respect to improving the performance of RGB-IR Re-ID. Existing studies have focused mainly on addressing the cross-modality variance, and the pose information is rarely exploited. In this thesis, RGB-IR Re-ID systems are newly proposed, and they can cover not only cross-modality variation but also human pose variation through a generative adversarial network (GAN)-based approach. RGB-IR Re-ID systems are proposed based on two GAN-based approaches: The single-generator based approach and the two-separate generator-based approach. Both proposed systems deal with the pose variance issue by introducing a pose transfer module, which generates a new target pose-transferred image. The pose transfer module is first verified to have effective performance increments when applied to the RGB-IR Re-ID system. In addition, the effect of the pose transfer module depending on the intensity of the pose variance is confirmed. Pose transfer has a valid effect on all pose variance level cases. They show the greatest increment in the high pose variance gallery set. Finally, it was possible to verify which approaches show better performance in a real robotic application. The single-generator based approach exhibits a better performance with a Rank-1 accuracy set to 40.5%, and it is the final proposed model. The proposed model is expected to be robust in a real scenario where the pose variance is large, such as robotic applications.