Simple but effective attention calibration for CLIP-guided diffusion models

Abstract

While Contrastive Language-Image Pre-training (CLIP) model has significantly advanced text-to-image generation, we uncover two notable issues in its application to diffusion models, particularly with the implementation of local embeddings. First, the model disproportionately focuses on word embeddings with less information of the input prompt. Second, local embeddings disrupt the image geometry established by global embeddings at initial timesteps, risking misalignment with the original prompt. To mitigate the identified issues, we introduce two adjustments to cross-attention: sequence-dependent and time-dependent attention calibration. Our method employs simple numerical operations, for which we provide the values, ensuring easy implementation. In the sequence-dependent attention calibration, constants are added to the logits in the cross-attention layer to counterbalance the diminishing attention across the word sequence. The time-dependent attention adjustment enhances the attention towards global embeddings in the initial stages, facilitating better geometry formation. Our experiments on various datasets show that this simple method significantly improves the performance of Stable Diffusion, yielding images that more accurately depict the input prompts.