Using Multiple Heads to Subsize Meta-memorization Problem

Lu Wang, K. L. Eddie Law

Abstract: The memorization problem is a meta-level overfitting phenomenon in meta-learning. The trained model prefers to remember learned tasks instead of adapting to new tasks. This issue limits many meta-learning approaches to generalize. In this paper, we mitigate this limitation issue by proposing multiple supervisions through a multi-objective optimization process. The design leads to a Multi-Input Multi-Output (MIMO) configuration for meta-learning. The model has multiple outputs through different heads. Each head is supervised by a different order of labels for the same task. This leads to different memories, resulting in meta-level conflicts as regularization to avoid meta-overfitting. The resulting MIMO configuration is applicable to all MAML-like algorithms with minor increments in training computation, the inference calculation can be reduced through early-exit policy or better performance can be achieved through low cost ensemble. In experiments, identical model and training settings are used in all test cases, our proposed design is able to suppress the meta-overfitting issue, achieve smoother loss landscapes, and improve generalisation.