Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment

Max Wilcoxson, Morten Svendgård, Ria Doshi, Dylan Davis, Reya Vir, Anant Sahai

Track: short paper (up to 4 pages)

Keywords: Finetuning, alignment, in-context learning, soft prompting, LoRA, toy models

TL;DR: We demonstrate how polynomial regression presents a rich toy problem for exploring various phenomena of in-context learning in large language models, including parameter efficient finetuning and alignment.

Abstract: Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.

Submission Number: 50