The Brittleness of Priors: An Empirical Case for Adaptive Neural Architectures

Dawud Hage

Keywords: Brittleness, Adaptive Neural Architectures

Abstract: Deep and compositional architectures enjoy strong theoretical advantages, yet practical models still hinge on fixed architectural priors whose alignment with data is rarely scrutinized. We report a rigorous isoparametric study comparing fully-connected ReLU MLPs, residual MLPs, and periodic-activation networks (SIREN) on canonical 1D function approximation. To isolate architecture from capacity, we employ a strict isoparametric control, matching total parameter counts across all models; to avoid memorization, we inject Gaussian noise, hold out validation data, apply early stopping, and evaluate both interpolation and extrapolation error. Our results reveal a fundamental brittleness of priors: no single static architecture dominates. Shallow- and-wide MLPs excel on smooth, non-compositional targets; deeper MLPs win on compositional signals; periodic activations are decisively superior on oscillatory targets; and residual connections, despite optimization benefits, underperform when their identity-preserving bias misaligns with the target family. These empirical trends cohere with theory on depth-enabled expressivity and spectral bias. We conclude that maximal generalization arises only when architectural bias matches data structure, motivating a shift from static design towards heterogeneous modular systems and, in the longer term, Architecturally Plastic Networks (APNs).

Serve As Reviewer: ~Dawud_Hage1

Submission Number: 49