A Direct Approximation of AIXI Using Logical State Abstractions

Samuel Yang-Zhao, Tianyu Wang, Kee Siong Ng

Keywords: AIXI, artificial general intelligence, higher-order logic, reinforcement learning, state abstraction, feature selection, binary decision diagrams

Abstract: We propose a practical integration of logical state abstraction with AIXI, a Bayesian optimality notion for reinforcement learning agents, to significantly expand the model class that AIXI agents can be approximated over to complex history-dependent and structured environments. The state representation and reasoning framework is based on higher-order logic, which can be used to define and enumerate complex features on non-Markovian and structured environments. We address the problem of selecting the right subset of features to form state abstractions by adapting the $\Phi$-MDP optimisation criterion from state abstraction theory. Exact Bayesian model learning is then achieved using a suitable generalisation of Context Tree Weighting over abstract state sequences. The resultant architecture can be integrated with different planning algorithms. Experimental results on controlling epidemics on large-scale contact networks validates the agent's performance.

TL;DR: Incorporating state representations based in higher-order logic using state abstraction theory allows for a practical AIXI approximation on a large model class