From Movement Memories to Robotic Dialogue: A Language-Mediated Pipeline for Somatic Machine Translation

Pinyao Liu, Keon Ju Lee, Paul Pu Liang

Track: Paper

Keywords: robotic art, human movements, human experience, embodied AI, human-AI interaction

Abstract: Human movements recalled from memories are often fleeting, private, and difficult to externalize for reflection. Traditional body-centered therapeutic practices, such as Gestalt-based approach, rely on re-enacting a movement as remembered. We propose Somatic Machine Translation, a language-mediated pipeline behind a robotic artwork that captures human re-enacted movements from memories via an Inertial Measurement Unit (IMU), interprets them into natural-language descriptions using a large language model, and then generates new robotic movement sequences. This transformation reframes movement through the body of a non-human performer—a robotic sculpture. We situate this work at the intersection of therapeutic dreamwork, creative AI and embodied interaction, arguing that interpretive divergence can open novel perspectives on somatic memory. The contribution includes a language-mediated robotic movement generation pipeline embedded in the artwork, and a conceptual framework for physical transformation in creative embodied AI.

Submission Number: 220