Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
Abstract: Self-Alignment serves as an efficient method to align pretrained large language models (LLMs) with human values while saving huge amount annotation costs. However, existing self-alignment methods utilize the pretrained LLM to generate alignment datasets in a few-shot manner, which gives rise to a question: Is the pretrained LLM the better few-shot generator rather than its aligned version? If not, to what extent could the aligned LLM continue providing benefits? In order to answer this question, we conduct a detailed study via bootstrapping self-alignment. We find the key role of in-context learning (ICL) examples, which serves as the only fresh data in this self-training loop and should be as much diverse and informative as possible. Ensuring this by putting forward an ICL example pool, we verify the ignored potential of multi-round(bootstrapping) self-alignment models. It regards the previous checkpoint as the generator, in contrast to the single round. This yields benefits in early stages(3～5 rounds) within both generation and classification tasks. To further improve the label quality in this bootstrapping pipeline, we focus on adjusting the training order from easy to hard and achieve a better performance. Moreover,
we discuss the collapse phenomenon in the later stage and offer two viewpoints: Data Processing Inequality and Sharper Output Distribution along with corresponding empirical study for explanation. Based on this, we give a validation dataset for early stop in case of further model collapse. Drawing on these findings, we propose Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced few-shot ability to boost zero or one-shot performance. Our experiments demonstrate the efficiency of SOFT across various classification and generation tasks, shedding light on the ignored potential of continually enhancing model self-alignment performance via inner capacity.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Brian_Kingsbury1
Submission Number: 3226
Loading