CoT-Seg: Rethinking Segmentation with Chain-of-Thought Reasoning and Self-Correction

Huaiqian Liu, Chak Ho Huang, Shiu-hong Kao, Yu-Wing Tai, Chi-Keung Tang

Keywords: segmentation, recursive self improvement, CoT, reasoning segmentation, vision language integration

TL;DR: a recursive self-correction framework using MLLM reasoner and verifier to enhance reasoning segmentation

Abstract:

Existing works of reasoning segmentation often fall short in complex cases, particularly when addressing complicated queries and out-of-domain images. Inspired by the chain-of-thought reasoning, where harder problems require longer thinking steps/time, this paper aims to explore a system that can think step-by-step, look up information if needed, generate results, self-evaluate its own results, and recursively refine the results, in the same way humans approach harder questions. We introduce CoT-Seg, a framework that rethinks reasoning segmentation by combining chain-of-thought reasoning with recursive self-correction. Instead of fine-tuning, CoT-Seg leverages the inherent reasoning ability of pre-trained MLLMs (e.g., GPT-4o) to decompose queries into meta-instructions, extract fine-grained
semantics from images, and identify target objects even under implicit or complex prompts. Moreover, CoT-Seg incorporates a self-correction stage: the model evaluates its own segmentation against the original query and reasoning trace, identifies mismatches, and iteratively refines the mask. This tight integration of reasoning and correction significantly improves reliability and robustness, especially in ambiguous or error-prone cases. Furthermore, our CoT-Seg framework allows easy incorporation of retrieval-augmented reasoning, enabling the system to access external knowledge when the input lacks sufficient information. Our results highlight that combining chain-of-thought reasoning, self-correction, offers a powerful paradigm for vision language integration driven segmentation.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 63