Incremental Learning through Deep Adaptation

Amir Rosenfeld, John K. Tsotsos

Abstract: Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added task, typically as many as the original network. We propose a method called Deep Adaptation Networks (DAN) that constrains newly learned filters to be linear combinations of existing ones. DANs preserve performance on the original task, require a fraction (typically 13%) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3% of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.

TL;DR: An alternative to transfer learning that learns faster, requires much less parameters (3-13 %), usually achieves better results and precisely preserves performance on old tasks.

Keywords: Transfer Learning, Learning without forgetting, Multitask Learning

Data: [DTD](https://paperswithcode.com/dataset/dtd), [Oxford 102 Flower](https://paperswithcode.com/dataset/oxford-102-flower), [SVHN](https://paperswithcode.com/dataset/svhn), [Sketch](https://paperswithcode.com/dataset/sketch), [Visual Domain Decathlon](https://paperswithcode.com/dataset/visual-domain-decathlon)