RoboAlign: Reinforcement Learning for Action-Aligned Multimodal Large Language Models

Dongyoung Kim, Sumin Park, Woomin Song, Seungku Kim, Taeyoung Kim, Huiwon Jang, Jaehyung Kim, Younggyo Seo, Jinwoo Shin

Keywords: MLLM, VLA, RL, Alignment, Embodied Reasoning

TL;DR: RoboAlign directly aligns MLLM's representation with low-level action while simultaneously enhancing embodied reasoning and preserving general-purpose knowledge, thereby producing models well-suited for VLA

Abstract: In recent years, state-of-the-art vision–language–action models (VLAs) have been built upon pre-trained multimodal large language models (MLLMs). However, how to systematically train MLLMs to improve VLA performance remains an open problem. While prior approaches primarily focus on strengthening embodied reasoning via linguistic actions, the modality gap limits the transferability of language-based knowledge to non-linguistic low-level actions produced by VLAs. To address this problem, we propose a novel framework RoboAlign that aligns MLLM representations with low-level actions, thereby producing MLLMs well-suited for VLA. Specifically, we achieve action alignment through reinforcement learning, where the model generates action tokens via zero-shot reasoning in natural language. To validate the effectiveness of RoboAlign, we train VLAs by adding a diffusion-based action head on top of an MLLM backbone and evaluate them on major robotics benchmarks. Specifically, training base MLLMs with RoboAlign improves the performance on robotic tasks by 17.5%, 18.9%, and 106.6% on LIBERO, CALVIN, and real-world robotic environments, respectively. Moreover, RoboAlign outperforms models aligned only with language-described actions or with supervised fine-tuning based approaches such as ECoT, demonstrating its effectiveness and broad applicability.

Primary Area: applications to robotics, autonomy, planning

Submission Number: 24158