Uncovering Limitations in Text-to-Image Generation: A Contrastive Approach with Structured Semantic Alignment
Submission Type: Regular Long Paper
Submission Track: NLP Applications
Submission Track 2: Syntax, Parsing and their Applications
Keywords: Multi-modal generation, Semantic consistency, Structure information learning
TL;DR: This work presents a contrastive approach that explores the limitations of text-to-image models in generating fine-grained semantics, ultimately revealing the challenges in real-world applications.
Abstract: Despite significant advancements in text-to-image generation models, they still face challenges when it comes to producing highly detailed or complex images based on textual descriptions. In order to explore these limitations, we propose a Structured Semantic Alignment (SSA) method for evaluating text-to-image generation models. SSA focuses on learning structured semantic embeddings across different modalities and aligning them in a joint space. The method employs the following steps to achieve its objective: (i) Generating mutated prompts by substituting words with semantically equivalent or nonequivalent alternatives while preserving the original syntax; (ii) Representing the sentence structure through parsing trees obtained via syntax parsing; (iii) Learning fine-grained structured embeddings that project semantic features from different modalities into a shared embedding space; (iv) Evaluating the semantic consistency between the structured text embeddings and the corresponding visual embeddings. Through experiments conducted on various benchmarks, we have demonstrated that SSA offers improved measurement of semantic consistency of text-to-image generation models. Additionally, it unveils a wide range of generation errors including under-generation, incorrect constituency, incorrect dependency, and semantic confusion. By uncovering these biases and limitations embedded within the models, our proposed method provides valuable insights into their shortcomings when applied to real-world scenarios.
Submission Number: 46
Loading