Characterizing Plastic Regions in Neural Networks

Gwyneth Liu, Nir N Shavit, Linghao Kong

Keywords: continual adaptation, neural network plasticity, localized adaptation, distribution shift, stability-plasticity tradeoff, interpretability

TL;DR: We define plastic regions, which are regions in the network that can be manipulated for better adaptation and are easily discoverable. We show that these regions are valuable and then study their presence in multiple architectures.

Abstract: Adapting a trained model to a new domain without overwriting prior knowledge is useful only when the model contains a region whose parameter state can support new learning. In vision classifiers, we study plastic regions: contiguous, easily-discoverable regions in which some manipulation of the region improves the target--source trade-off over size-matched control strips elsewhere in the same network. We first characterize a plastic region in ResNet-18 and show that it transfers across target domains, compounds under sequential adaptation, and can be manipulated to recover adaptation capacity at rigid checkpoints. We then analyze plastic-region existence across nine architectures and report observations about network properties that appear to enable or obstruct plastic-region formation.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 70