Branch-level Network Re-parameterization with Neural Substitution

Seungmin Oh, Jongbin Ryu

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Neural substitution, Re-parameterization, Branch-level connectivity

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We introduce the neural substitution method for the barnch-level network reparameterization.

Abstract:

In this paper, we propose the neural substitution method for network re-parameterization at branch-level connectivity. The proposed neural substitution method learns a variety of network topologies, allowing our re-parameterization method to exploit the ensemble effect fully. In addition, we introduce a guiding method for reducing the non-linear activation function in a linear transformation. Because branch-level connectivity necessitates multiple non-linear activation functions, they must be reduced to a single activation with our guided activation method during the re-parameterization process.
Being able to reduce the non-linear activation function in this manner is significant as it overcomes the limitation of the existing re-parameterization method, which works only at block-level connectivity.
If re-parameterization is applied only at the block-level connectivity, the network topology can only be exploited in a limited way, which makes it harder to learn diverse feature representations.
On the other hand, the proposed approach learns a considerably richer representation than existing re-parameterization methods due to the unlimited topology with branch-level connectivity, providing a generalized framework to be applied with other methods.
The proposed method improves the re-parameterization performance, but it is also a general framework that enables existing methods to benefit from branch-level connectivity.
In our experiments, we show that the proposed re-parameterization method works favorably against existing methods while significantly improving their performance when applied to the proposed branch-level connectivity.
Upon acceptance, we will make our implementation publicly available.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2449