Accelerated Inference and Reduced Forgetting: The Dual Benefits of Early-Exit Networks in Continual Learning

Filip Szatkowski, Fei Yang, Bartłomiej Twardowski, Tomasz Trzcinski, Joost van de Weijer

Keywords: Continual learning, dynamic network

Abstract:

In the pursuit of a sustainable future for machine learning, energy-efficient neural network models are crucial. A practical approach to achieving this efficiency is through early-exit strategies. These strategies allow for swift predictions by making decisions early in the network, thereby conserving computation time and resources. However, so far the early-exit neural networks have only been developed
for stationary data distributions, which restricts their application in real-world scenarios where training data is derived from continuous non-stationary data. In this study, we aim to explore the continual training for early-exit networks. Specifically, we adapt the existing continual learning methods to fit early-exit architectures and introduce task-aware dynamic inference to improve the network accuracy for a given compute budgets. Finally, we evaluate continually those methods on the standard benchmarks to assess their accuracy and efficiency. Our work highlights the practical advantages of the early-exit networks in real-world continual learning scenarios.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 121