Automaton Distillation: Neuro-Symbolic Transfer Learning for Deep Reinforcement Learning

Precious Nwaorgu, Suraj Singireddy, Andre Beckus, Aden McKinney, Mahyar Alinejad, Chinwendu Enyioha, Sumit Kumar Jha, Alvaro Velasquez, George K. Atia

Abstract:

Reinforcement learning (RL) agents often struggle to reuse knowledge when task dynamics change, even when the underlying objective remains the same. This sample inefficiency is compounded by poor generalization beyond the training distribution. We introduce automaton distillation—a neuro-symbolic transfer learning approach that addresses both challenges by distilling Q-value estimates from a teacher agent into a compact automaton representation of the shared task objective. Critically, our method requires no explicit alignment
between source and target state-action spaces: the automaton serves as a domain-agnostic intermediary through which value information is transferred. We propose two variants. Static transfer performs value iteration over the abstract MDP induced by the automaton, providing a lightweight initialization. Dynamic transfer distills empirical Q-values from a teacher’s replay buffer onto automaton transitions, grounding symbolic abstractions in actual environment dynamics and correcting for mismatches between automaton trace length and true trajectory cost. We evaluate both variants on discrete and continuous gridworld tasks with sparse, non-Markovian rewards, and on a continuous benchmark. These results demonstrate that a shared symbolic objective is a sufficient bridge for effective few-shot transfer, even when source and target environments differ substantially in dynamics.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/MaduabuchiNwaorgu/automaton_distillation

Assigned Action Editor: ~Chinmay_Hegde1

Submission Number: 4905