Scalable Modular Network: A Framework for Adaptive Learning via Agreement Routing

Minyang Hu, Hong Chang, Bingpeng Ma, Shiguang Shan, Xilin CHEN

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: modular neural network; module selection process; adaptive learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: In this paper, we propose a novel modular network framework, called Scalable Modular Network (SMN), which enables adaptive learning capability and supports integration of new modules after pre-training for better adaptation.

Abstract:

In this paper, we propose a novel modular network framework, called Scalable Modular Network (SMN), which enables adaptive learning capability and supports integration of new modules after pre-training for better adaptation.
This adaptive capability comes from a novel design of router within SMN, named agreement router, which selects and composes different specialist modules through an iterative message passing process.
The agreement router iteratively computes the agreements among a set of input and outputs of all modules to allocate inputs to specific module.
During the iterative routing, messages of modules are passed to each other, which improves the module selection process with consideration of both local interactions (between a single module and input) and global interactions involving multiple other modules.
To validate our contributions, we conduct experiments on two problems: a toy min-max game and few-shot image classification task.
Our experimental results demonstrate that SMN can generalize to new distributions and exhibit sample-efficient adaptation to new tasks.
Furthermore, SMN can achieve a better adaptation capability when new modules are introduced after pre-training.
Our code is available at https://github.com/hu-my/ScalableModularNetwork.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: representation learning for computer vision, audio, language, and other modalities

Submission Number: 5102