Diffusion models for few-shot image generation

Abstract

Recently, diffusion models have demonstrated excellent performance on a range of image generation tasks, outperforming Generative Adversarial Networks (GANs). However, diffusion models require large-scale training datasets for training, and if the number of training samples is insufficient, the model is unable to generate diverse samples and simply replicates the training data. In this work, I propose a novel diffusion model adaptation method that utilizes a pre-trained model to adapt to few-shot datasets, which provide 10 or fewer training samples. To achieve this, I fine-tune diffusion models with the diffusion loss and use a modified cross-domain distance consistency loss to prevent overfitting. In addition, I propose source model-guided sampling, which preserves the overall structure of the generated sample from the source model. To demonstrate the performance and adaptation capability in few-shot settings, I conducted quantitative and qualitative experiments in various data domains. Furthermore, I extend the proposed framework to various image translation tasks by using the pre-trained source model jointly in the sampling process. The experimental results show that the proposed method is effective at adapting diffusion models to few-shot datasets and can be applied to various image translation tasks.