Sample-Efficient Parametric Learning from Natural Language

Parth Asawa, Alex Dimakis, Matei Zaharia

Keywords: Sample-Efficient, Parametric Learning, NL Feedback, Context Distillation

TL;DR: Sample-Efficient Parametric Learning from Natural Language

Abstract: Large language models (LLMs) today rely heavily on in-context learning (ICL) to adapt at inference time, but such adaptations are transient. Prior work on context distillation and training with natural language feedback have shown that prompts or corrections can be internalized into weights, but largely focus on static instructions and assume large-scale access to data. In this work, we seek to study sample-efficient parametric learning from natural language feedback, sample efficiency being a key limiter of adoption of continual learning methods in the real world. Our method is simple: obtain feedback or an instruction for a model in natural language, sample a generation conditioned on the feedback, and fine-tune the model on the new generation with the feedback removed from the prompt. This procedure forces the model to incorporate feedback into its parameters rather than depend on it in-context. We evaluate on both factual and stylistic domains, and show that our approach can outperform ICL and standard SFT baselines under extremely limited data budgets. Preliminary robustness experiments further highlight the challenges of compounding feedback and regression on prior rules, motivating more algorithmic work for continual natural language feedback learning.

Serve As Reviewer: ~Parth_Asawa1

Submission Number: 53