Parameter-Efficient Multi-Task and Multi-Domain Learning Using Factorized Tensor Networks

Yash Garg, Nebiyou Yismaw, Rakib Hyder, Ashley Prater-Bennette, Amit Roy-Chowdhury, Salman Asif

Abstract:

Multi-task and multi-domain learning methods seek to learn multiple tasks/domains, jointly or one after
another, using a single unified network. The primary challenge and opportunity lie in leveraging shared information
across these tasks and domains to enhance the efficiency of the unified network. The efficiency can be in terms of
accuracy, storage cost, computation, or sample complexity. In this paper, we introduce a factorized tensor network (FTN)
designed to achieve accuracy comparable to that of independent single-task or single-domain networks, while introducing
a minimal number of additional parameters. The FTN approach entails incorporating task- or domain-specific low-rank
tensor factors into a shared frozen network derived from a source model. This strategy allows for adaptation to numerous
target domains and tasks without encountering catastrophic forgetting. Furthermore, FTN requires a significantly smaller
number of task-specific parameters compared to existing methods. We performed experiments on widely used multidomain and multi-task datasets. We show the experiments on convolutional-based architecture with different backbones
and on transformer-based architecture. Our findings indicate that FTN attains similar accuracy as single-task or singledomain methods while using only a fraction of additional parameters per task.