CPCF: A Cross-Prompt Contrastive Framework for Referring Multimodal Large Language Models

Lanyun Zhu, Deyi Ji, Tianrun Chen, Haiyang Wu, De Wen Soh, Jun Liu

Abstract: Referring MLLMs extend conventional multimodal large language models by allowing them to receive referring visual prompts and generate responses tailored to the indicated regions. However, these models often suffer from suboptimal performance due to incorrect responses tailored to misleading areas adjacent to or similar to the target region. This work introduces CPCF, a novel framework to address this issue and achieve superior results. CPCF contrasts outputs generated from the indicated visual prompt with those from contrastive prompts sampled from misleading regions, effectively suppressing the influence of erroneous information outside the target region on response generation. To further enhance the effectiveness and efficiency of our framework, several novel designs are proposed, including a prompt extraction network to automatically identify suitable contrastive prompts, a self-training method that leverages unlabeled data to improve training quality, and a distillation approach to reduce the additional computational overhead associated with contrastive decoding. Incorporating these novel designs, CPCF achieves state-of-the-art performance, as demonstrated by extensive experiments across multiple benchmarks. Project page: https://lanyunzhu.site/CPCF/

Lay Summary: Conventional MLLMs are typically limited to answering abstract and general questions about the entire image, whereas referring MLLMs are designed to receive visual prompts that point to specific regions and generate responses tailored to those indicated areas. However, these models often struggle with suboptimal performance, frequently producing incorrect responses caused by confusion with misleading regions adjacent to or visually similar to the target. To address this issue, this work proposes a novel framework, CPCF, which incorporates an automatic contrastive decoding strategy to reduce such errors, a self-training method to enhance its effectiveness, and a distillation algorithm to mitigate the additional computational overhead introduced by contrastive decoding. Experimental results across multiple benchmarks demonstrate the superior performance of CPCF, paving the way for more reliable and fine-grained MLLMs with fewer mistakes and more accurate region-specific understanding.

Primary Area: Deep Learning->Large Language Models

Keywords: Referring multimodal large language models

Submission Number: 359