Curvature Tuning: Provable Training-free Model Steering From a Single Parameter

Leyang Hu, Matteo Gamba, Randall Balestriero

Keywords: fine tuning, peft, spline theory, smoothing, adaptation, lora, robustness

TL;DR: A novel Parameter Efficient Fine-Tuning method which operates by smoothing model nonlinearities.

Abstract: The scaling of model and data sizes has reshaped the AI landscape, establishing finetuning pretrained models as the standard paradigm for solving downstream tasks. However, dominant finetuning methods typically rely on weight adaptation–thus often lacking interpretability– and depend on heuristically chosen hyperparameters. In this paper, we take a different perspective and shift the focus from weights to activation functions, viewing them through the lens of spline operators. We propose Curvature Tuning (CT), an interpretable and principled steering method that modulates a model’s decision boundary by injecting a single hyperparameter into its activation functions. Making this hyperparameter trainable gives rise to a novel and highly parameter-efficient finetuning method. This perspective complements current finetuning methods–whose effect lies primarily in feature adaptation–empirically improving both generalization and robustness.

Submission Number: 148