Deep neural network optimization methods for efficient training and inference on image synthesis

Abstract

Recently, deep neural networks have demonstrated impressive image synthesis and reconstruction performances, and have been extensively studied in various computer vision tasks. Unfortunately, training and evaluating recent deep learning models such as diffusion models or NeRF(Neural Radiance Field) consume a lot of time and computational resources. To address this problem and to allow training with even a single GPU, here we present novel methods for training diffusion model and a model for sparse view 3D object reconstruction. First of all, we present a novel pyramidal diffusion model that can generate high resolution images starting from much coarser resolution images using a single score function trained with a positional embedding. This enables a neural network to be much lighter and also enables time-efficient image generation without compromising its performances. Furthermore, we show that the proposed approach can be also efficiently used for multi-scale super-resolution problem using a single score function. Next, inspired by NeRF that have achieved state-of-the-art performance on novel view synthesis using implicit neural representation, we present a novel neural support optimization method. The method only estimates the support set and updates values in the support so that the speed of training and inference can be improved. Additionally, reconstructing a 3D object from two perpendicular views has been intriguing research topic in tomographic reconstruction with many applications such as X-ray baggage scanners, electron microscopy, etc. Unfortunately, the convention reconstruction algorithms such as filtered back-projection as well as modern deep learning methods completely fail to reconstruct in this extreme sparse view acquisition setting. To overcome the problem and further enhance the reconstruction quality, we propose a novel feature domain loss using the pre-trained VGGNet and DINO-ViT. We demonstrate successful 3D reconstruction using real two projection images in baggage scanning and STEM-EDX tomography dataset.