Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding

Depeng Li, Tianqi Wang, Junwei Chen, Qining Ren, Kenji Kawaguchi, Zhigang Zeng

Abstract:

Deep neural networks are susceptible to catastrophic forgetting
when trained on sequential tasks. Various continual
learning (CL) methods often rely on exemplar buffers or/and
network expansion for balancing model stability and plasticity,
which, however, compromises their practical value due to
privacy and memory concerns. Instead, this paper considers a
strict yet realistic setting, where the training data from previous
tasks is unavailable and the model size remains relatively
constant during sequential training. To achieve such desiderata,
we propose a conceptually simple yet effective method
that attributes forgetting to layer-wise parameter overwriting
and the resulting decision boundary distortion. This
is achieved by the synergy between two key components:
HSIC-Bottleneck Orthogonalization (HBO) implements nonoverwritten
parameter updates mediated by Hilbert-Schmidt
independence criterion in an orthogonal space and EquiAngular
Embedding (EAE) enhances decision boundary adaptation
between old and new tasks with predefined basis vectors.
Extensive experiments demonstrate that our method achieves
competitive accuracy performance, even with absolute superiority
of zero exemplar buffer and 1.02 the base model.