Global Prefix-Tuning: Extremely Efficient Fine-Tuning for Shallow Alignment Using One Token
Abstract: We introduce \textit{Global-Token Tuner}, an extremely parameter-efficient fine-tuning (PEFT) method for adapting Large Language Models (LLMs) that uses only a few or just one learnable token, regardless of model size. Global-Token Tuner employs a unique design that constructs a globally shared set of tunable tokens that modify the attention of every layer. Therefore no matter how base model change the tunable parameter remains relatively constant. We showed that our method can attain comparable performance with LoRA across plenty of common tasks while reducing parameter requirements from multiple millions or more to as few as 5 thousand. We also believe the discovery that even one token can effectively finetune LLMs illuminates the inner workings of LLMs.
Paper Type: Short
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: finetuning llm, PEFT, alignment
Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency
Languages Studied: english
Submission Number: 4185
Loading