Radar-to-photometric image translation using context-aware, misalignment-robust and confidence-guided generative models

Abstract

Satellite synthetic aperture radar (SAR) images are immensely valuable because they can be obtained regardless of weather and time conditions. However, SAR images have fatal noise and less contextual information, thus making them harder and less interpretable. Thus, translation of SAR to electro-optical (EO) images is highly required for easier interpretation. In this article, we propose a novel coarse-to-fine context-aware SAR-to-EO translation (CFCA-SET) framework and a misalignment-resistant (MR) loss for the misaligned pairs of SAR-EO images. With our auxiliary learning of SAR-to-near-infrared translation, CFCA-SET consists of a two-stage training: 1) the low-resolution SAR-to-EO translation is learned in the coarse stage via a local self-attention module that helps diminish the SAR noise and 2) the resulting output is used as guidance in the fine stage to generate the SAR colorization of high resolution. Our proposed auxiliary learning of SAR-to-NIR translation can successfully lead CFCA-SET to learn distinguishable characteristics of various SAR objects with less confusion in a context-aware manner. To handle the inevitable misalignment problem between SAR and EO images, we newly designed an MR loss function. We also use a pre-trained segmentation network to provide the segmentation regions with their labels into learning the SET. Our ECFCA-SET can be trained to effectively learn the translation for the regions of confusing contexts by utilizing the segmentation and ㅣlocally adaptive confidence mask loss function. Furthermore, due to the difficulty of SAR-EO pair image dataset collection, we propose a novel SAR-EO data augmentation strategy via diffusion process. Extensive experimental results show that our CFCA-SET can generate more recognizable and understandable EO-like images compared to other methods in terms of nine image quality metrics.