Human Feedback Guided Reinforcement Learning for Unknown Temporal Tasks via Weighted Finite Automata

Nathaniel Smith, Yu Wang

Keywords: Reinforcement learning, human feedback, temporal, weighted finite automata, bradley terry, minigrid

TL;DR: This work focuses on inferring the structure of a temporal task from pairwise human preferences and using it to guide reinforcement learning toward behavior aligned with human intent.

Abstract: Robots in real-world settings are commonly expected to complete complex temporal tasks—goals that have explicit order dependencies. Because such goals are difficult to specify mathematically, we explore how humans with intuitive task understanding can provide meaningful feedback on observed robot behavior. This work focuses on inferring the structure of a temporal task from pairwise human preferences and using it to guide reinforcement learning toward behavior aligned with human intent. Our method leverages limited feedback to construct a weighted finite automaton (WFA) that tracks task progress and shapes the learning process. We validate the approach in a Minigrid case study using real human feedback, showing that it enables agents to learn temporally structured tasks that would otherwise remain unlearned.

Submission Number: 79