The World Is Bigger: A Computationally-Embedded Perspective on the Big World Hypothesis

Alex Lewandowski, Aditya A. Ramesh, Edan Meyer, Dale Schuurmans, Marlos C. Machado

Keywords: resource constraints, algorithmic information theory, reinforcement learning, continual learning

Abstract: Continual learning is often motivated by the idea, known as the big world hypothesis, that the world is bigger than the agent. Recent problem formulations capture this idea by explicitly constraining an agent relative to the environment. These constraints lead to solutions in which the agent continually adapts to best use its limited capacity, rather than converging to a fixed solution. However, explicit constraints can be ad hoc, difficult to incorporate, and limiting to the effectiveness of scaling up the agent's capacity. In this paper, we characterize a problem setting in which an agent, regardless of its capacity, is implicitly constrained by being embedded in the environment. In particular, we introduce a computationally-embedded perspective that represents an embedded agent as an automaton simulated within a universal (formal) computer. We prove that such an automaton is implicitly constrained and that it is equivalent to an agent that interacts with a reward-free and partially-observable Markov decision process over a countably infinite state-space. We propose an objective for this setting, which we call interactivity, that measures an agent's ability to continually adapt its behaviour to learn new predictions. To support experimentation on continual adaptation, we develop a synthetic benchmark in which an interactivity-seeking agent constructs its own non-stationary stream of experience from which it must continually learn to predict.

Submission Number: 19