TEL: A Thermodynamics-Inspired Layer for Adaptive, and Efficient Neural Learning

ICLR 2026 Conference Submission19065 Authors

Keywords: Iterative learning, Phsics based architecture, Gibbs free energy, adaptive nonlinearity, Non linear layer

Abstract: We introduce the Thermodynamic Equilibrium Layer (TEL), a neural building block that replaces fixed activations with a short, $K$-step energy-guided refinement. TEL performs $K$ discrete gradient steps on a Gibbs-inspired free energy with a learnable step size and an entropy-driven, adaptable temperature estimated from intermediate activations. This yields nonlinearities that are dynamic yet stable, expose useful per-layer diagnostics (temperature and energy trajectories), and run with a fixed, predictable compute budget. Across a broad suite of classification, regression, and reconstruction tasks, TEL matches or exceeds strong baselines, including MLPs, modern implicit/energy-based layers under compute matched dimensionality, FLOPs, and parameters. Swapping TEL in place of MLP feed forwards in standard CNN, LSTM, and Transformer blocks incurs minimal overhead while consistently improving accuracy. Together, these results position TEL as a scalable, drop-in alternative for constructing adaptable nonlinearities in deep networks.

Primary Area: learning theory

Submission Number: 19065