Object-Conditioned Energy-Based Model for Attention Map Alignment in Text-to-Image Diffusion Models

Published: 09 Apr 2024, Last Modified: 13 Apr 2024SynData4CVEveryoneRevisionsBibTeXCC BY 4.0
Keywords: Attention map alignment, Energy-based Models, Text-to-Image Diffusion Models
Abstract: Text-to-image diffusion models have shown great success in generating high-quality text-guided images. Yet, these models may still fail to semantically align generated images with the provided text prompts, leading to problems like incorrect attribute binding and/or catastrophic object neglect. Given the pervasive object-oriented structure underlying text prompts, we introduce a novel object-conditioned Energy-Based Attention Map Alignment (EBAMA) method to address the aforementioned problems. We show that an object-centric attribute binding loss naturally emerges by approximately maximizing the log-likelihood of a $z$-parameterized energy-based model with the help of the negative sampling technique. We further propose an object-centric intensity regularizer to prevent excessive shifts of objects attention towards their attributes. Extensive qualitative and quantitative experiments on the AnE benchmark demonstrate the superior performance of our method over previous strong counterparts.
Supplementary Material: pdf
Submission Number: 24