Semantic Glitch: Agency and Artistry in an Autonomous Pixel Cloud

Qing Zhang, Jing Huang, Mingyang Xu, Jun Rekimoto

Track: Paper

Keywords: Creative AI, Multimodal Large Language Models (MLLMs), Autonomous Navigation, Human-Robot Interaction (HRI), Robotic Art, Speculative Design

TL;DR: A flying pixel robot navigates using a large language model instead of traditional sensors, prioritizing the creation of character over efficiency.

Abstract: While mainstream robotics pursues metric precision and flawless performance, this paper explores the creative potential of a deliberately "lo-fi" approach. We present the "Semantic Glitch," a soft flying robotic art installation whose physical form—a 3D pixel style cloud—is a "physical glitch" derived from digital archaeology. We detail a novel autonomous pipeline that rejects conventional sensors like LiDAR and SLAM, relying solely on the qualitative, semantic understanding of a Multimodal Large Language Model to navigate. By authoring a bio-inspired personality for the robot through a natural language prompt, we create a "narrative mind" that complements the "weak," historically-loaded body. Our analysis begins with a 13-minute autonomous flight log, and a follow-up study statistically validates the framework's robustness for authoring quantifiably distinct personas. The combined analysis reveals emergent behaviors—from landmark-based navigation to a compelling "plan-to-execution" gap—and a character whose unpredictable, plausible behavior stems from a lack of precise proprioception. This demonstrates a lo-fi framework for creating imperfect companions whose success is measured in character over efficiency.

Submission Number: 73