Synapse: Learning preferential concepts from visual demonstrations

S Modak, NT Patton, I Dillig, J Biswas - First Workshop on Vision …, 2024 - openreview.net
First Workshop on Vision-Language Models for Navigation and …, 2024openreview.net
We address the problem of preference learning, which aims to learn user-specific
preferences (eg," good parking spot"," convenient drop-off location") from visual input.
Despite its similarity to learning factual concepts (eg," red cube"), preference learning is a
fundamentally harder problem due to its subjective nature and the paucity of person-specific
training data. We address this problem using a new framework called Synapse, which is a
neuro-symbolic approach designed to efficiently learn preferential concepts from limited …
We address the problem of preference learning, which aims to learn user-specific preferences (e.g., "good parking spot", "convenient drop-off location") from visual input. Despite its similarity to learning factual concepts (e.g., "red cube"), preference learning is a fundamentally harder problem due to its subjective nature and the paucity of person-specific training data. We address this problem using a new framework called Synapse, which is a neuro-symbolic approach designed to efficiently learn preferential concepts from limited demonstrations. Synapse represents preferences as neuro-symbolic programs in a domain-specific language (DSL) that operates over images, and leverages a novel combination of visual parsing, large language models, and program synthesis to learn programs representing individual preferences. We evaluate Synapse focusing on mobility-related concepts in mobile robotics and autonomous driving. Our evaluation demonstrates that Synapse significantly outperforms existing baselines. The code and other details can be found on the project website https://amrl.cs.utexas.edu/synapse.
openreview.net
Showing the best result for this search. See all results