Task-Adaptation Curriculum Learning

Anonymous

Abstract: Despite the prevailing applications of foundation models, when adapted to downstream tasks, their performance sensitively varies with the distribution shift/gap between the pertaining task and the target task. Moreover, direct fine-tuning might be overfitting to limited target task data. In the realm of NLP, tasks are semantically related with shared skills and those general purposed ones usually have more available data than the highly specific and user-defined ones. In this paper, we mitigate the distribution shift in task adaptation by developing a smooth transfer learning curriculum, which, by fine-tuning the model along a path of intermediate tasks on a graph, progressively bridges the gap between the pretrained model and a target task with limited data. To this end, we formulate the curriculum learning as a graph search problem and address its efficiency by a deep dive into accelerating the transferability estimation between tasks and two classical search algorithm applied to our problem, i.e., greedy best first search and Monte Carlo tree search. We evaluate our approach, i.e., ``task-adaptation curriculum learning (TaCL)'' on two benchmark settings with tasks drawn from GLUE. Extensive experiments on different target tasks demonstrate the effectiveness and advantages of \ours on more specific and data-deficient downstream tasks.

Paper Type: long

Research Area: Machine Learning for NLP

Contribution Types: Approaches to low-resource settings

Languages Studied: English

Preprint Status: There is no non-anonymous preprint and we do not intend to release one.

A1: n/a

A2: n/a

A3: yes

A3 Elaboration For Yes Or No: 1

B: yes

B1: yes

B1 Elaboration For Yes Or No: 5

B2: yes

B2 Elaboration For Yes Or No: 5

B3: yes

B3 Elaboration For Yes Or No: 5

B4: n/a

B5: n/a

B6: yes

B6 Elaboration For Yes Or No: 5

C: yes

C1: n/a

C2: yes

C2 Elaboration For Yes Or No: 5

C3: yes

C3 Elaboration For Yes Or No: 5

C4: yes

C4 Elaboration For Yes Or No: 5

D: no

D1: n/a

D2: n/a

D3: n/a

D4: n/a

D5: n/a

E: no

E1: n/a