Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

Chengzhi Mao, Xudong Lin, Wen-Sheng Chu

Keywords: Vision Language Model

TL;DR: We revert the paradigm of feeding image into large language models, and show language can guide image encoder to learn more sophisticated features and combat hallucinations.

Abstract: Vision foundation models are typically trained as static feature extractors, forcing the burden of task adaptation onto large downstream models. We propose a different paradigm: instead of solely feeding visual features into language, we use language itself to dynamically guide the vision encoder. Our method, Language-Instructed Vision Embeddings (LIVE), leverages language as high-level guidance to produce task-centric embeddings at inference time—without requiring task-specific retraining. This enables the encoder to focus attention on contextually relevant aspects of the input, yielding more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations (+34 points on MMVP), outperforms vision–language models with orders of magnitude more parameters on visual question answering, and generalizes to unseen instructions and tasks---offering a direct path toward adaptive, instruction-driven visual intelligence.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 3452