PeFoo-L: A General Framework for Preconditioned Enhanced Forward-Only Optimizer in LLM Fine-tuning on the Edge

ICLR 2026 Conference Submission17074 Authors

Keywords: Transformer, zeroth-order optimizer, fine-tuning

TL;DR: PeFoo is a Preconditioner-Enhanced Forward-Only Optimizer that integrates a preconditioning strategy to accelerate convergence for fine-tuning Large Language Models specifically on resource-constrained edge devices.

Abstract:

Fine-tuning Large Language Models (LLMs) on resource-constrained edge devices is a critical but challenging task, primarily due to the prohibitive memory and computational costs of backpropagation. While forward-only optimizers like MeZO mitigate these costs by eliminating the backward pass, they often suffer from slow and unstable convergence, particularly on loss landscapes with heterogeneous curvature. To address this limitation, we introduce PeFoo, a general framework for preconditioner enhanced forward only optimizer. PeFoo integrates a carefully designed preconditioning strategy into the forward-only paradigm, corrects a fundamental source of bias and instability present in prior work HiZOO. Furthermore, to counteract the memory overhead introduced by the preconditioner itself, we propose PeFoo-L, which employs a layer-wise update strategy. This approach constrains preconditioner storage and weight updates to a single layer per iteration, reducing the overall memory footprint and data traffic.
Experimental results validate the effectiveness of our framework. On the OPT-1.3B model, PeFoo surpasses the accuracy of leading zeroth-order methods MeZO and HiZOO by 2.7\% and 2.1\%, respectively. Furthermore, PeFoo-L achieves a memory footprint reduction of over 2.73$\times$ and 1.75$\times$ compared to Adam and HiZOO, while delivering faster convergence speed compared to MeZO and HiZOO.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 17074