Object-Conditioned Energy-Based Model for Attention Map Alignment in Text-to-Image Diffusion Models

Yasi Zhang, Peiyu Yu, Ying Nian Wu

Keywords: Attention map alignment, Energy-based Models, Text-to-Image Diffusion Models

Abstract: Text-to-image diffusion models have shown great success in generating high-quality text-guided images. Yet, these models may still fail to semantically align generated images with the provided text prompts, leading to problems like incorrect attribute binding and/or catastrophic object neglect. Given the pervasive object-oriented structure underlying text prompts, we introduce a novel object-conditioned Energy-Based Attention Map Alignment (EBAMA) method to address the aforementioned problems. We show that an object-centric attribute binding loss naturally emerges by approximately maximizing the log-likelihood of a $z$-parameterized energy-based model with the help of the negative sampling technique. We further propose an object-centric intensity regularizer to prevent excessive shifts of objects attention towards their attributes. Extensive qualitative and quantitative experiments on the AnE benchmark demonstrate the superior performance of our method over previous strong counterparts.

Submission Number: 24