Contrastive Conceptor Activation Steering (COAST): Unlocking Vision-Language-Action Models through Hidden States

Miranda Muqing Miao, Subin Kim, Brandon Y. Yang, Dinesh Jayaraman, Lyle Ungar

Keywords: Methods (probing, steering, causal interventions), Other, Applications of interpretability

TL;DR: We successfully steered robotics VLA using contrastive conceptors and found generalization across 3 models and 3 benchmarks and established cross-task transfer.

Abstract: Vision-Language-Action (VLA) models leverage powerful perceptual priors from web-scale vision-language model (VLM) pre-training, yet they remain surprisingly brittle in practice, frequently failing at simple robotic tasks. To mitigate this, we propose \underline{C}ontrastive C\underline{o}nceptor \underline{A}ctivation \underline{St}eering (COAST). COAST builds on the notion of a conceptor, a linear operator that soft-projects data into the principal components of a target distribution. COAST uses conceptors to identify success-critical subspaces for a target robotic task from a few examples of success and failure rollouts. At inference time, it steers VLA latents into these identified success subspaces to improve task outcomes. Across three architecturally distinct neural policies (flow-matching VLAs, autoregressive VLA, and Diffusion Policy), COAST improves mean simulation and real-robot task success rate by approximately 20 and 40 percentage points, respectively. The activation subspace geometry reveals that failure modes share substantial structure across tasks while success representations remain largely task-specific. When tasks share similar failure modes, this structure enables zero-shot transfer of previously fitted conceptors to new tasks. Ultimately, our results suggest that the bottleneck in current VLAs is not a lack of relevant knowledge in VLM latents, but an inability to retrieve it during action generation. COAST provides a lightweight, training-free path to unlocking these latent capabilities by steering the model towards its own ``success'' distributions.

Submission Number: 110