Synapse: Learning Preferential Concepts from Visual Demonstrations

Sadanand Modak, Noah Tobias Patton, Isil Dillig, Joydeep Biswas

Keywords: Visual-Language Models, Concept learning, Neuro-symbolic programming, Program Synthesis, Visual Reasoning

TL;DR: Synapse is a data-efficient framework to evaluate and learn complex visual preferential concepts through natural language and physical demonstrations.

Abstract: We address the problem of preference learning, which aims to learn user-specific preferences (e.g., "good parking spot", "convenient drop-off location") from visual input. Despite its similarity to learning factual concepts (e.g., "red cube"), preference learning is a fundamentally harder problem due to its subjective nature and the paucity of person-specific training data. We address this problem using a new framework called Synapse, which is a neuro-symbolic approach designed to efficiently learn preferential concepts from limited demonstrations. Synapse represents preferences as neuro-symbolic programs in a domain-specific language (DSL) that operates over images, and leverages a novel combination of visual parsing, large language models, and program synthesis to learn programs representing individual preferences. We evaluate Synapse focusing on mobility-related concepts in mobile robotics and autonomous driving. Our evaluation demonstrates that Synapse significantly outperforms existing baselines. The code and other details can be found on the project website https://amrl.cs.utexas.edu/synapse.

Submission Number: 19