Gauging Learnability in Supervised Fine-tuning Data

Haotian Zhou, Tingkai Liu, Qianli Ma, Jianbo Yuan, Pengfei Liu, Yang You, Hongxia Yang

Keywords: Large Language Models, Alignment, Supervised Fine-tune

Abstract: Supervised Fine-Tuning (SFT) serves as a crucial phase in aligning Large Language Models (LLMs) to specific task prerequisites. The selection of fine-tuning data profoundly influences the model’s performance, a choice traditionally grounded in data quality and distribution. However, this paper introduces an innovative dimension in data selection: learnability. SFT is regarded as a technique for unlocking the potential of pretrained models. However, given that different models have disparate capabilities, the data appropriate for one may not suit another. Thus, we introduce the term ``learnability" to define the suitability of data for effective learning by the model. We present the Loss Based SFT Data Selection (LoBaSS) method, utilizing data learnability as the principal criterion for the selection of secure, efficient, and high-quality data. This method provides a nuanced approach, allowing the alignment of data selection with inherent model capabilities, ensuring optimal compatibility and learning efficiency. In experimental comparisons involving 7B and 13B models, our LoBaSS method surpasses the full-data fine-tuning, requiring merely 6% of the data. When employing 16.7% of the data, LoBaSS harmonizes the model’s capabilities across conversational and mathematical domains, proving its efficacy and adaptability.

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3424