HILDA: Hessian-Implicit Langevin with Damping and Adaptation for Diffusion Sampling

Jiale Chang, Can Jin, Ying Li, Siliang Tang, Yueting Zhuang

Keywords: Diffusion Models, Hessian-Vector Product, Conjugate Gradient, Training-free Sampling

Abstract: Diffusion models have achieved remarkable success in high-quality image generation through learning score functions of noise-corrupted data distributions. Contemporary sampling acceleration techniques predominantly focus on optimizing denoising trajectories along the temporal dimension, yet still rely on first-order Langevin dynamics for updates at individual noise levels. As the denoising process advances, curvature disparities along different principal directions of the target distribution become increasingly severe, resulting in pronounced anisotropic behavior. Methods that depend exclusively on first-order gradient information suffer from zigzag sampling trajectories in such regimes, thereby constraining effective step sizes and compromising sample quality. To address this limitation, we introduce HILDA—a training-free diffusion sampler that implicitly incorporates second-order geometric information at each noise level by employing Hessian-vector products combined with conjugate gradient methods to capture complete geometric information along coupled directions without explicitly constructing the Hessian matrix. To handle numerical ill-conditioning arising from strong anisotropy in later stages, we develop an adaptive damping coefficient λt based on condition number estimates and a spectral radius normalization factor ct, constructing a unified geometric operator Mt = ct(Ht + λtI)^(-1) that applies consistently to both drift and diffusion terms. HILDA functions as a plug-and-play geometric enhancement module that integrates seamlessly with existing ODE solvers, including DPM-Solver and UniPC. Experimental validation across multiple pre-trained diffusion models demonstrates that HILDA substantially mitigates zigzag artifacts and enhances both detail preservation and overall image quality under comparable or reduced sampling steps.

Primary Area: generative models

Submission Number: 6092