Layer-Wise Universal Approximation and Progressive Optimization for Residual Networks

Wei Wang, Xiaoyong Wei, Li Qing

Keywords: Interpretability and explainable AI

Abstract:

This paper provides a theoretical analysis to bridge the gap wherein the classical UAT, originally developed for feedforward networks (FNNs), fails to directly apply to modern residual networks (RNs) like ResNets and Transformers. Our key contributions are: First, we prove a layer-wise UAT for residual networks by formulating ResNet and Transformer blocks in a unified form compatible with FNNs,
ensuring that each layer satisfies the UAT conditions (compact inputs and continuity). Second, using this layer-wise formulation, we demonstrate that RN training can be effectively modeled as a compensatory additive model, enabling sequential optimization where layers collaboratively reduce input-output divergence. Unlike conventional end-to-end training (which suffers from instability risks), our layer-wise approach ensures bounded learning and superior convergence. Third, we propose Layer-wise Progressive Approximation (LPA), a training paradigm that operationalizes sequential optimization while increasing the probability of achieving UAT-compliant approximations at each layer. Experimental results across both synthetic datasets and standard benchmarks (CIFAR-10/100, Fashion-MNIST) demonstrate LPA’s significant advantages: up to 8.31% higher accuracy alongside early-layer convergence and improved training stability compared to conventional end-to-end approaches. We further show that adding an adaptive criterion to LPA
automatically discovers the effective layers and prunes the remainder, enabling aggressive compression, reducing model size by up to 79.17%. This suggests that simply scaling up the model is often unnecessary. The source code will be released
unpon acceptance at https://(open_upon_acceptance).

Primary Area: interpretability and explainable AI

Submission Number: 4455