Revisiting NeRN: Optimizing Training Strategies for Weight Reconstruction

Keywords: Neural Representation, Implicit Neural Networks, Weight Reconstruction, Knowledge Distillation

TL;DR: We enhance the accuracy and efficiency of neural representation that predict neural network weights

Abstract: The Neural Representation for Neural Network (NeRN) framework introduced a promising paradigm shift by using Implicit Neural Representations (INRs) to parameterize network weights as coordinate-based functions, rather than modeling data directly. Despite its conceptual novelty, NeRN remains impractical due to two main limitations: (1) the reconstructed weights fail to match the original model's performance, and (2) broader applicability such as compression, fine-tuning, and transfer learning remains underexplored. In this work, we revisit INR-based weight reconstruction by examining the limitations of NeRN's training strategy and propose improvements through two training strategies: (1) a reconstruction-only objective that, under overparameterization, enables the predictor to match and even exceed the performance of the original model, and (2) a decoupled training scheme that separates reconstruction and distillation phases, allowing each to specialize in its respective objective, thereby improving training stability and parameter efficiency. Our results advance INR-based parameterization toward practical use, demonstrating high-fidelity weight recovery and improved generalization.

Submission Number: 35