Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models

TMLR Paper5246 Authors

Abstract:

Despite the ability of text-to-image models to generate high-quality, realistic, and diverse images, they face challenges in compositional generation, often struggling to accurately represent details specified in the input prompt. A prevalent issue in compositional generation is the misalignment of spatial relationships, as models often fail to faithfully generate images that reflect the spatial configurations specified between objects in the input prompts.
To address this challenge, we propose a novel probabilistic framework for modeling the relative spatial positioning of objects in a scene, leveraging the concept of \textit{Probability of Superiority (PoS)}. Building on this insight, we make two key contributions. First, we introduce a novel evaluation metric, \textit{PoS-based Evaluation (PSE)}, designed to assess the alignment of 2D and 3D spatial relationships between text and image, with improved adherence to human judgment. Second, we propose \textit{PoS-based Generation (PSG)}, an inference-time method that improves the alignment of 2D and 3D spatial relationships in T2I models without requiring fine-tuning. \emph{PSG} employs a Part-of-Speech PoS-based reward function that can be utilized in two distinct ways: (1) as a \textit{gradient-based} guidance mechanism applied to the cross-attention maps during the denoising steps, or (2) as a \textit{search-based} strategy that evaluates a set of initial noise vectors to select the best one. Extensive experiments demonstrate that the \emph{PSE} metric exhibits stronger alignment with human judgment compared to traditional center-based metrics, providing a more nuanced and reliable measure of complex spatial relationship accuracy in text-image alignment. Furthermore, \emph{PSG} significantly enhances the ability of text-to-image models to generate images with specified spatial configurations, outperforming state-of-the-art methods across multiple evaluation metrics and benchmarks.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Shuangfei_Zhai3

Submission Number: 5246