Imitation learning of dexterous hand control uncovers muscle-level representations in primate sensorimotor cortex
Keywords: Imitation learning, representation learning, sensorimotor representations, cortical dynamics, neural decoding, dexterous manipulation, motor control, embodied control
Abstract: Dexterous grasping requires the integration of proprioceptive feedback with predictive motor commands. Yet, how cortical circuits combine afferent feedback with efference signals during grasp remains poorly understood. Here we combine deep reinforcement learning, biomechanics, and neural recordings to build a closed-loop, muscle-level controller of primate grasping. A neural network policy, trained via imitation learning on a 39-muscle musculoskeletal hand, reproduces naturalistic pre-contact shaping and develops internal states that explain single-neuron activity in primary motor (M1) and somatosensory (S1) cortex. Three principles emerged. First, muscle-based control aligns more closely with cortical dynamics than joint-based control, even when the latter achieves higher tracking accuracy. Second, task optimization drives the emergence of internal representations that capture both trial-to-trial variability and object-specific tuning. Third, trajectory embeddings learned within the network can be decoded from M1 to directly drive the controller with only tens of neurons, outperforming joint-angle decoding. Together, these findings establish a stimulus-computable framework that reveals integrated, muscle-centric and goal-latent codes in S1/M1 while opening a novel route for creating brain-body models.
Submission Number: 36
Loading