Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping

ACL ARR 2025 May Submission2488 Authors

Abstract: Self-alignment is an effective way to reduce the cost of human annotation while ensuring promising model capability. However, existing self-alignment methods utilize the pretrained LLM to generate alignment datasets in a few-shot manner, which gives rise to a question:Is the pretrained LLM the better few-shot generator rather than its aligned version? If not, to what leads to benefits? In this paper, our pioneering exploration delves into the impact of bootstrapping self-alignment on large language models. We find the key role of in-context learning (ICL) examples, which serves as the only fresh data in this self-training loop and should be as much diverse and informative as possible. Our findings reveal that bootstrapping self-alignment markedly surpasses the single-round approach. To further exploit the capabilities of bootstrapping, we investigate and adjust the training order of data, which yields improved performance of the model. We discuss the collapse phenomenon in the later stage and offer two viewpoints: Data Processing Inequality and Sharper Output Distribution along with ablation studies for explanation. Based on this, we give a validation dataset for early stop in case of further model collapse. We propose Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced few-shot ability to boost zero or one-shot performance, shedding light on the ignored potential of continually enhancing self-alignment performance.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Self-Alignment; Bootstrapping; Large Language Models; Bootstrapping

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 2488