Probing the Inductive Bias of Neural Networks through Learning Random Cellular Automata

Jan Disselhoff, Michael Wand

Keywords: Cellular Automata, Generalization, Inductive Bias, Deep Learning, Coarse-Graining, Discrete Dynamical Systems, Physics for Deep Learning

Abstract:

In this paper, we empirically examine whether the inductive bias of deep networks can be linked to structural properties of dynamical systems inspired by physics, such as symmetry, locality, and coarse‑grained observation of outcomes.
To explore this question, we generate “toy universes” by sampling random cellular‑automaton rules that satisfy these constraints, and train convolutional neural networks (CNNs) to predict their evolution under three experimental factors: temporal coarse‑graining, spatial pooling, and a structured (low‑entropy) initial state. Throughout, we measure each network’s average generalization performance relative to a baseline.
While classical constraints such as symmetry and locality are necessary, they alone are not sufficient for learnability. However, when we account for the perturbation sensitivity of the target function, we observe a strong negative correlation with learnability. Further, using a structured (low‑entropy) initial state leads networks to favor coarser macroscopic patterns over details.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 9533