CF-VLM：CounterFactual Vision-Language Fine-tuning

Jusheng Zhang, Kaitong Cai, Yijia Fan, Jian Wang, Keze Wang

Keywords: VLM, causal, clip

TL;DR: VLMs lack deep causal reasoning. CF-VLM, via counterfactuals & novel training, boosts causal logic. It surpasses SOTA in reasoning/generalization, cuts hallucinations, aids real-world VLM use.

Abstract: Recent advances in vision-language models (VLMs) have greatly improved cross-modal semantic understanding, yet significant limitations remain in fine-grained discrimination and deep causal reasoning tasks. Existing VLMs often rely on superficial statistical correlations, lacking the ability to capture the underlying causal logic between visual and textual content. To address this, we propose the **CounterFactual Vision-Language Fine-tuning Model (CF-VLM)**, a novel framework that enhances the causal reasoning capabilities of VLMs through the targeted use of counterfactual samples. CF-VLM introduces three complementary training objectives: maintaining foundational cross-modal alignment, reinforcing the uniqueness, and stability of factual scene representations against coherent counterfactuals, and sharpening the model’s sensitivity to minimal but critical causal edits. Extensive experiments demonstrate that CF-VLM consistently outperforms strong baselines and state-of-the-art methods on compositional reasoning and generalization benchmarks. Furthermore, it shows promise in mitigating visual hallucinations, indicating improved factual consistency. Our CF-VLM provides a robust foundation for deploying VLMs in high-stakes, real-world scenarios requiring reliable reasoning and interpretability.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 4499