(A) study on salient-object-aware image super-resolution using a generative adversarial network

Abstract

Single image super-resolution (SR), which aims to reconstruct a high-resolution image from a low-resolution image, is an essential task in the field of computer vision. Recently, various SR methods based on convolutional neural networks have been proposed, significantly enhancing the reconstruction quality of SR. Among them, GAN-based models are frequently employed due to their ability to generate realistic results. Although high-frequency components are well restored in these approaches, unnecessary structures tend to be excessively generated, causing unpleasant artifacts and lowering the fidelity of SR results. Inspired by the fact that humans generally focus on salient regions in an image, in this thesis, we propose a Salient-Object-Aware Super-Resolution Generative Adversarial Network (SOASRGAN) model that concentrates on salient objects in the given image. Unlike existing SR methods that restore all regions at the same degree, the proposed model performs SR by applying different attentions and loss functions depending on the saliency of the region. First, we propose a method for computing the object saliency score by utilizing instance segmentation and saliency detection. Then, we extract the saliency feature map while predicting the scored mask. By providing the extracted saliency feature map to the proposed Salient-Object-Guided Attention (SOGA) module, the network can attend to the salient object area. Also, three loss functions and a relativistic average discriminator are employed to assign weights according to the saliency of objects. Our proposed method allows the SR network to treat each region differently based on salient objects. As a result, images generated by the proposed SOASRGAN demonstrate better visual quality than conventional methods on various datasets, maximizing the image quality of salient object regions and suppressing excessive generation of unnecessary structures that causes artifacts in background areas. To the best of our knowledge, this is the first attempt to induce discriminatory results guided by salient objects in the field of single image SR.