Instruction-Based Self-Supervised Online Training of the Perceptual Subsystem of a Cognitive Robotic Architecture

Sarah Schneider, Evan Krause, Daniel Soukup, Matthias Scheutz

Keywords: Adaptive Perception, Cognitive Architectures, Representation Learning

Abstract: Traditional AI systems often rely on the closed-world assumption, limiting their adaptability in dynamic environments. We present a cognitive architecture (CA) that expands its perceptual capabilities by leveraging object prototypes from user-provided natural language descriptions. These prototypes, composed of superellipsoid primitives, are used by the CA to train a convolutional semantic encoder with rendering parameterizations as automated ground-truth supervision. Once trained, the CA employs the encoder to infer superellipsoid-based representations from real-world object observations. The bidirectional mapping between superellipsoid parameters and natural language descriptions enables translation of inferred geometric information into human-understandable descriptions. We describe the design of these fundamental object prototypes, the synthetically supervised training pipeline, and the mapping process. Experimental results demonstrate that the CA enhances its perceptual repertoire through structured, interpretable object representations.

Paper Track: Technical paper

Submission Number: 34