Meta-Learning Sparse Implicit Neural Representations

Jaeho Lee, Jihoon Tack, Namhoon Lee, Jinwoo Shin

Keywords: network pruning, implicit neural representation, coordinate-based representations, data compression

TL;DR: We develop a scalable method to learn sparse neural representations for a large set of signals.

Abstract: Implicit neural representations are a promising new avenue of representing general signals by learning a continuous function that, parameterized as a neural network, maps the domain of a signal to its codomain; the mapping from spatial coordinates of an image to its pixel values, for example. Being capable of conveying fine details in a high dimensional signal, unboundedly of its domain, implicit neural representations ensure many advantages over conventional discrete representations. However, the current approach is difficult to scale for a large number of signals or a data set, since learning a neural representation---which is parameter heavy by itself---for each signal individually requires a lot of memory and computations. To address this issue, we propose to leverage a meta-learning approach in combination with network compression under a sparsity constraint, such that it renders a well-initialized sparse parameterization that evolves quickly to represent a set of unseen signals in the subsequent training. We empirically demonstrate that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models with the same number of parameters, when trained to fit each signal using the same number of optimization steps.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

Code: https://github.com/jaeho-lee/MetaSparseINR

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2110.14678/code)