Diffusion-supplemented Implicit Layers: Operator Smoothing for better Implicit Solvers

Dinislam Gabitov, Bader Rasheed, Anastasia Antsiferova, Dmitriy S. Vatolin

Keywords: implicit NN, adversarial ml, diffusion models, neural ODE

TL;DR: This paper introduces Diffusion-Supplemented Implicit Layers (DSIL), a method to improve the convergence and robustness of implicit neural networks using diffusion layer.

Abstract: Implicit networks compute hidden states as fixed points. When the implicit map is poorly conditioned, solvers slow or fail. We propose Diffusion-Supplemented Implicit Layers (DSIL): insert a few denoising steps on the latent before each evaluation of the map. Under standard Lipschitz assumptions in a common metric, this preconditioning reduces the effective Lipschitz constant of the composed map, yielding stronger contraction; with a true proximal denoiser the contraction factor is explicitly tunable by the step size. On CIFAR-10 with a SODEF head, DSIL provides modest robustness gains without adversarial training. DSIL is architecture-agnostic and complements existing stabilization methods.

Submission Number: 124