Real-Time Coaching of Human Physical Skills with Large Language Models

ICLR 2026 Conference Submission18119 Authors

Keywords: Concurrent AI Coaching

Abstract: Concurrent coaching of humans with language instruction has the potential to dramatically accelerate skill acquisition in high-stakes domains like driving and sports. However, effective concurrent coaching requires two key capabilities: determining when to intervene with fast, proactive timing decisions, and determining what to say through free-form instruction generation for diverse scenarios. Existing approaches struggle because they either sacrifice real-time responsiveness for content quality or sacrifice content flexibility for speed. Our key insight is to decompose concurrent coaching into two stages: deciding when to intervene and determining what to say, bridged by a shared representation. We introduce StreamCoach, a two-stage coaching framework that encodes learner state into lightweight language embeddings, enabling intervention decisions within 17 ms that trigger generation of contextually appropriate instructions. In the fast inference stage, StreamCoach compares current state embeddings against past coaching scenarios to trigger interventions. In the slow reasoning stage, the same embeddings retrieve relevant examples for Retrieval-Augmented Generation of adaptive instructions. By separating timing-critical decisions from content generation,StreamCoach achieves both key capabilities simultaneously. Evaluated in high-performance driving, StreamCoach significantly outperforms existing approaches in both intervention timing and instruction quality, demonstrating effective concurrent coaching of humans through language.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 18119