Go to ICLR 2025 Conference homepageCycleAug: Cycle-Consistent Visual Augmentation for Large Multimodal Models
Training multimodal large language models (MLLMs) requires high-quality image-question-answer (IQA) triplets, which are labour-intensive to curate and often lack diversity. We propose a novel data augmentation framework for visual instruction tuning that efficiently generates diverse synthetic images based on existing IQA anchor triplets. To ensure that the generated images align with their associated QA pairs, we propose CycleAug --- cycle-consistency visual augmentation which involves synthesizing images from text (text $\rightarrow$ image) and then performing a verification step to confirm that the answers derived from the synthetic images match the original answers (image $\rightarrow$ text), ensuring consistency across images, questions, and answers. By combining synthetic images with high-quality real data in the training phase, we demonstrate these synthetic triplets act as an implicit regularization, which improves the robustness of MLLMs and enables analogical reasoning. Extensive experiments show that our approach improves model performance on multiple visual question-answering benchmarks without additional real-world data. This work highlights the potential of leveraging visual foundational models to enhance visual instruction tuning in MLLMs.