Cup Curriculum: Curriculum Learning on Model Capacity

Luca Scharr, Vanessa Toborek

Keywords: Curriculum Learning, Natural Language Processing, Iterative Magnitude Pruning

TL;DR: We apply curriculum learning for the model capacity in an NLP setting and show that it significantly increases the performance of early stopping.

Abstract:

Curriculum learning (CL) aims to increase the performance of a learner on a given task by applying a specialized learning strategy.
This strategy focuses on either the dataset, the task, or the model.
There is little to no work analysing the possibilities to apply CL on the model capacity in natural language processing.
To close this gap, we propose the cup curriculum.
In a first phase of training we use a variation of iterative magnitude pruning to reduce model capacity.
These weights are reintroduced in a second phase, resulting in the model capacity to show a cup-shaped curve over the training iterations.
We empirically evaluate different strategies of the cup curriculum and show that it outperforms early stopping reliably while exhibiting a high resilience to overfitting.

Submission Number: 47