Self-Partitioning Adaptive Widening Networks

Keywords: Supervised Learning, Machine Learning, Growing Network, Adaptive Methods, Neural Architecture, MLP

Abstract:

The design of neural architectures is commonly treated as a process separate from parameter optimization, which necessitates expensive retraining whenever the architecture is modified and frequently results in over-parameterized, memory-intensive models.
To overcome these limitations, we introduce the Self-Partitioning Adaptive Widening Network (SPAWN), a novel framework that jointly learns both its architecture and its parameters by dynamically increasing its representational capacity during training. SPAWN starts from a single shallow predictor and incrementally builds a soft-gated mixture of experts by partitioning the input space where additional complexity is needed.
In contrast to prior approaches that expand networks by adding neurons or layers, SPAWN increases capacity by recursively subdividing the input space, yielding a differentiable tree of affine experts trained end-to-end. This expansion process is governed by data-driven criteria that determine when to grow, where to split, and how to preserve the model’s outputs, enabling seamless optimization without retraining from scratch.
Empirical evaluations on standard regression and classification benchmarks demonstrate that SPAWN automatically discovers compact architectures that match or surpass the predictive performance of substantially larger, fixed-capacity models, while using markedly fewer parameters.

Submission Number: 79