Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation

Rachel Luo, Heng Yang, Michael Watson, Apoorva Sharma, Sushant Veer, Edward Schmerling, Marco Pavone

Keywords: metric estimation, sample efficiency, control variates

Abstract: Learning-based robotic systems demand rigorous validation to assure reliable performance, but extensive real‐world testing is often prohibitively expensive and if conducted may still yield insufficient data for high-confidence guarantees. In this work, we introduce a general estimation framework that leverages *paired* data across test platforms, e.g., paired simulation and real‐world observations, to achieve better estimates of real-world metrics via the method of control variates. By incorporating cheap and abundant auxiliary measurements (for example, simulator outputs) as control variates for costly real‐world samples, our method provably reduces the variance of Monte Carlo estimates and thus requires significantly fewer real‐world samples to attain a specified confidence bound on the mean performance. We provide theoretical analysis characterizing the variance and sample-efficiency improvement, and demonstrate empirically in autonomous driving and quadruped robotics settings that our approach achieves high‐probability bounds with markedly reduced sample complexity. Our technique can lower the real‐world testing burden for validating the performance of the stack, thereby enabling more efficient and cost‐effective experimental evaluation of robotic systems.

Submission Number: 874