Generalization Bounds for Neural Ordinary Differential Equations and Residual Neural Networks

Madhusudan Verma, Manoj Kumar

Keywords: NEURAL ORDINARY DIFFERENTIAL EQUATIONS, GENERALIZATION BOUNDS, BOUNDED VARIATION FUNCTIONS, MACHINE LEARNING

Abstract:

Neural ordinary differential equations (neural ODEs) represent a widely-used
class of deep learning models characterized by continuous depth. Understand-
ing the generalization error bound is important to evaluate how well a model is
expected to perform on new, unseen data. Earlier works in this direction involved
considering the linear case on the dynamics function (a function that models the
evolution of state variables) of Neural ODE Marion (2024). Other related work
is on bound for Neural Controlled ODE Bleistein & Guilloux (2023) that de-
pends on the sampling gap. We consider a class of neural ordinary differential
equations (ODEs) with a general nonlinear function for time-dependent and time-
independent cases which is Lipschitz with respect to state variables. We observed
that the solution of the neural ODEs would be of bound variations if we assume
that the dynamics function of Neural ODEs is Lipschitz continuous with respect
to the hidden state. We derive a generalization bound for the time-dependent
and time-independent Neural ODEs.Using the fact that Neural ODEs are limiting
cases of time-dependent Neural ODEs we obtained a bound for the residual neural
networks. We showed the effect of overparameterization and domain bound in the
generalization error bound. This is the first time, the generalization bound for the
Neural ODE with a more general non-linear function has been found.

Primary Area: learning on time series and dynamical systems

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 299