Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer

Seungwon Lee, Sima Behpour, ERIC EATON

Keywords: lifelong learning, continual learning, architecture search

Abstract: Effective lifelong learning across diverse tasks requires diverse knowledge, yet transferring irrelevant knowledge may lead to interference and catastrophic forgetting. In deep networks, transferring the appropriate granularity of knowledge is as important as the transfer mechanism, and must be driven by the relationships among tasks. We first show that the lifelong learning performance of several current deep learning architectures can be significantly improved by transfer at the appropriate layers. We then develop an expectation-maximization (EM) method to automatically select the appropriate transfer configuration and optimize the task network weights. This EM-based selective transfer is highly effective, as demonstrated on three algorithms in several lifelong object classification scenarios.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Starting from the observation that performance of a lifelong learning architecture is significantly improved by transferring at appropriate layers, EM-based algorithm for selective transfer between tasks is proposed and evaluated in this paper.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=6Vci3yEdc2u