Measuring Memorization and Generalization in Forecasting Models via Structured Perturbations of Chaotic Systems

Max Kanwal, Caryn Tran

Keywords: dynamical systems, OOD genearalization

TL;DR: We quantify generalization vs. memorization in forecasting models using structured test-time perturbations of chaotic systems.

Abstract: We introduce a benchmarking method for evaluating generalization and memorization in time series forecasting models of chaotic dynamical systems. By generating two complementary types of test sets—by perturbating training trajectories to minimally/maximally diverge over a fixed time horizon—we quantify each model's sensitivity to distribution shift. Our results reveal consistent trade-offs between training accuracy and OOD generalization across neural architectures, offering a lightweight diagnostic tool for model evaluation in the small-data regime.

Submission Number: 34