Aligner: One Global Token is Worth Millions of Parameters When Aligning LLMs

Zhou Ziheng, Ying Nian Wu, Song-Chun Zhu, Demetri Terzopoulos

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: LLM, Parameter-Efficient-Finetuning, Alignment, Human Preference, Value

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: we investigated in the notion of alignment as only a "form" learning, and proposed an extremely efficient LLM finetuning method, requiring as few as 1 vector while matching the performance of other SOTA methods like LLaMA-Adapter and LoRA

Abstract: We introduce Aligner, a novel Parameter-Efficient Fine-Tuning (PEFT) method for aligning multi-billion-sized Large Language Models (LLMs). Aligner employs a unique design that constructs a globally shared set of tunable tokens that will change the attention of every layer. Remarkably with this method, even when using one token accounting for a mere 5,000 parameters, Aligner can still perform comparably well to state-of-the-art methods like LoRA that require millions of parameters. This capacity is substantiated in both instruction following and value alignment tasks. Besides the multiple order-of-magnitude improvement in parameter efficiency, the insight Aligner provides into the internal mechanisms of LLMs is also valuable. The architectural features and efficacy of our method demonstrate that an LLM separates its handling of "form" and "knowledge" internally in some orthogonal manner. This finding should give impetus to new research into LLM mechanism understanding and value alignment.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 464