Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal

Naresh Kumar Gurulingan, Elahe Arani, Bahram Zonooz

Keywords: Multi-task Learning, Structural Learning, Brain-inspired Neural Network, Neuron Creation, Neuron Removal

TL;DR: We propose a multi-task learning method inspired by structural learning in the brain that simultaneously learns the architecture and its parameters.

Abstract: Multi-task learning has the potential to improve generalization by maximizing positive transfer between tasks while reducing task interference. Fully achieving this potential is hindered by manually designed architectures that remain static throughout training. In contrast, learning in the brain occurs through structural changes that are in tandem with changes in synaptic strength. Therefore, we propose Multi-Task Structural Learning (MTSL) which simultaneously learns the multi-task architecture and its parameters. MTSL begins with an identical single task network for each task and alternates between a task learning phase and a structural learning phase. In the task learning phase, each network specializes in the corresponding task. In each of the structural learning phases, starting from the earliest layer, locally similar task layers first transfer their knowledge to a newly created group layer after which they become redundant and are removed. Our experimental results show that MTSL achieves competitive generalization with various baselines and improves robustness to out-of-distribution data.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/multi-task-structural-learning-using-local/code)