KITINet: Kinetics Theory Inspired Network Architectures with PDE Simulation Approaches

Mingquan Feng, Yifan Fu, Tongcheng Zhang, Yu Jiang, Yixin Huang, Junchi Yan

Keywords: Physics Inspired Neural Network, Kinetic Theory

Abstract: Despite the widely recognized success of residual connections in modern neural networks, their design principles remain largely heuristic. This paper introduces KITINet (KInetics Theory Inspired Network), a way that reinterprets feature propagation through the lens of non-equilibrium particle dynamics and partial differential equation (PDE) simulation. We propose a new residual module that models feature updates as the stochastic evolution of a particle system, numerically simulated via a discretized solver for the Boltzmann transport equation (BTE). This formulation mimics particle collisions, enabling additional neuron-wise information propagation via physical interactions. Additionally, we reveal that this mechanism is an implicit regularization approach that induces network parameter condensation during training, where parameters progressively concentrate into a sparse subset of dominant channels. Experiments on large language modeling, image classification, scientific computation, and text classification show consistent improvements over classic network baselines, without additional inference cost.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 5656