Go to ICML 2025 Conference homepagePropagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble
TL;DR: Propagation of chaos for mean-field Langevin dynamics
Abstract: Mean-field Langevin dynamics (MFLD) is an optimization method derived by taking the mean-field limit of noisy gradient descent for two-layer neural networks in the mean-field regime. Recently, the propagation of chaos (PoC) for MFLD has gained attention as it provides a quantitative characterization of the optimization complexity in terms of the number of particles and iterations. A remarkable progress by Chen et al. (2022) showed that the approximation error due to finite particles remains uniform in time and diminishes as the number of particles increases. In this paper, by refining the defective log-Sobolev inequality---a key result from that earlier work---under the neural network training setting, we establish an improved PoC result for MFLD, which removes the exponential dependence on the regularization coefficient from the particle approximation term of the optimization complexity. As an application, we propose a PoC-based model ensemble strategy with theoretical guarantees.
Lay Summary: Modern AI models adjust millions of internal “knobs” during training. A fresh viewpoint treats these adjustments as many particles moving together, which helps predict the time and compute needed. We improve the math behind this idea, eliminating a technical penalty that used to over-estimate training difficulty when the network includes regularization. The result: the wider the network, the closer real training stays to the ideal infinite-particle case. Our theory also supports a practical tip—combining several smaller models trained this way can reliably boost performance.
Primary Area: Theory->Deep Learning
Keywords: mean-field neural network, mean-field Langevin dynamics, propagation of chaos, finite-particle approximation, defective log-Sobolev inequality
Submission Number: 11521
Loading