Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates

Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, Sanjeev Arora

Keywords: safety, alignment, large language models, system prompt

TL;DR: To address the issue that fine-tuning an aligned model degrades safety, we introduce the “Free Tune, Safe Test” (FTST) Principle: fine-tune models without a system prompt that emphasizes safety, but include it at test time.

Abstract: Public LLMs such as the Llama 2-Chat have driven huge activity in LLM research. These models underwent alignment training and were considered safe, but it was recently documented by Qi et al. (2023) that even benign fine-tuning (e.g., on seemingly safe datasets) can give rise to unsafe behaviors in the models. The current paper is concerned with methods and best practices to mitigate such loss of alignment. Through extensive experiments on several chat models (Meta's Llama 2 Chat, Mistral AI's Mistral 7B Instruct v0.2, and OpenAI's GPT-3.5 Turbo), this paper uncovers that the prompt templates used during fine-tuning and inference play a crucial role in preserving safety alignment, and proposes the “*Free Tune, Safe Test*” (FTST) principle — fine-tune models without a system prompt that emphasizes safety, but include it at test time. Fine-tuning experiments on GSM8K, ChatDoctor, and OpenOrca show that FTST significantly reduces the rise of unsafe behaviors, and even almost eliminates them in some cases.

Submission Number: 80