SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis

ICLR 2026 Conference Submission13891 Authors

Keywords: Benchmarking and Evaluation, Text-to-Image, Generation, Open-Vocabulary Object Detection, Model Interpretability, Alignment, Explanability

TL;DR: We introduce SANEval, an open-vocabulary T2I benchmark that not only scores compositional accuracy but also provides fine-grained diagnostic actionable feedback

Abstract: The rapid progress of text-to-image (T2I) models has unlocked unprecedented creative potential, yet their ability to faithfully render complex prompts involving multiple objects, attributes, and spatial relationships remains a significant bottleneck. Progress is hampered by a lack of adequate evaluation methods; current benchmarks are often restricted to closed-set vocabularies, lack fine-grained diagnostic capabilities, and fail to provide the interpretable feedback necessary to diagnose and remedy specific compositional failures. We solve these challenges by introducing **SANEval** (Spatial, Attribute, and Numeracy Evaluation), a comprehensive benchmark that establishes a scalable new pipeline for open-vocabulary compositional evaluation. SANEval combines a large language model (LLM) for deep prompt understanding with an LLM-enhanced, open-vocabulary object detector to robustly evaluate compositional adherence unconstrained by a fixed vocabulary. Through extensive experiments on six state-of-the-art T2I models, we demonstrate that SANEval's automated evaluations provide a more faithful proxy for human assessment; our metric achieves a Spearman's rank correlation with statistically different results than that of existing benchmarks across tasks of attribute binding, spatial relations, and numeracy. To facilitate future research in compositional T2I generation and evaluation, we will release the SANEval dataset and our open-source evaluation pipeline.

Primary Area: datasets and benchmarks

Submission Number: 13891