Dynamics of Instruction Tuning: Each Ability of Large Language Models Has Its Own Growth Pace
Abstract: Instruction tuning is a burgeoning method to elicit the general intelligence of Large Language Models (LLMs). However, the understanding of its scaling properties remains underexplored. While some research advocates for expanding the number of instructions, others suggest that a small set of well-chosen examples is adequate. To understand such discrepancy, our work systematically studies the effectiveness of data volume, parameter size, and data construction methods on the development of each underlying ability of LLM, such as creative writing, code generation, and logical reasoning. Our study reveals three primary findings: (i) Despite these factors significantly influencing overall model performance, some abilities are more responsive to scaling, while others show high resistance. (ii) The sensitivity of different abilities to these factors can be explained by their Complexity and Transference, which indicate the relative importance of each factor in learning specific abilities. (iii) Tailoring data construction based on these sensitivities results in performance gains on two public benchmarks. Additionally, we curate a comprehensive dataset containing over 40k instances across ten abilities for our experiments.
Paper Type: Long
Research Area: Generation
Research Area Keywords: efficient models,data-efficient training
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Approaches to low-resource settings, Approaches low compute settings-efficiency, Data resources, Data analysis
Languages Studied: Chinese
Submission Number: 5880
Loading