Vid2Sid: Videos Can Help Close the Sim2Real Gap

Kevin Qiu, Yu Zhang, Marek Cygan, Josie Hughes

Track: long paper (up to 8 pages)

Keywords: system identification, sim2real, vision language models

TL;DR: We present a video-driven system identification pipeline that calibrates simulation parameters using only camera observations, leveraging foundation-model perception and VLM reasoning to diagnose physical mismatches and propose parameter updates.

Abstract: Calibrating a robot simulator’s physics parameters (friction, damping, material stiffness) to match real hardware is often done by hand or with black-box optimizers that reduce error but cannot explain which physical discrepancies drive the error. When sensing is limited to external cameras, the problem is further compounded by perception noise and the absence of direct force or state measurements. We present Vid2Sid, a video-driven system identification pipeline that couples foundation-model perception with a VLM-in-the-loop optimizer that analyzes paired sim-real videos, diagnoses concrete mismatches, and proposes physics parameter updates with natural language rationales. We evaluate our approach on a tendon-actuated finger (rigid-body dynamics in MuJoCo) and a deformable continuum tentacle (soft-body dynamics in PyElastica). On sim2real holdout controls unseen during training, Vid2Sid achieves the best average rank across all settings, matching or exceeding black-box optimizers while uniquely providing interpretable reasoning at each iteration. sim2sim validation confirms that Vid2Sid recovers ground-truth parameters most accurately (mean relative error under 13% vs. 28–98%), and ablation analysis reveals three calibration regimes. VLM-guided optimization excels when perception is clean and the simulator is expressive, while model-class limitations bound performance in more challenging settings.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 44