MOCK: an Algorithm for Learning Nonparametric Differential Equations via Multivariate Occupation Kernel Functions

TMLR Paper3885 Authors

Abstract: Learning a nonparametric system of ordinary differential equations from trajectories in a $d$-dimensional state space requires learning $d$ functions of $d$ variables. Explicit formulations often scale quadratically in $d$ unless additional knowledge about system properties, such as sparsity and symmetries, is available. In this work, we propose a linear approach, the multivariate occupation kernel method (MOCK), using the implicit formulation provided by vector-valued reproducing kernel Hilbert spaces. The solution for the vector field relies on multivariate occupation kernel functions associated with the trajectories and scales linearly with the dimension of the state space. We validate through experiments on a variety of simulated and real datasets ranging from 2 to 1024 dimensions. MOCK outperforms all other comparators on 3 of the 9 datasets on full trajectory prediction and 4 out of the 9 datasets on next-point prediction.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=zduPsND4Sy&noteId=Ni43hzewI5

Changes Since Last Submission:

Please find enclosed the research paper entitled ``MOCK: an Algorithm for Learning Nonparametric Differential Equations via Multivariate Occupation Kernel Functions". This is a major revision of the paper entitled ``Learning Nonparametric Differential Equations via Multivariate Occupation Kernel Functions", denoted Paper2929, originally submitted June 24th 2024 and rejected November 24 2024. We detail below the recommendations of the action editor and how we answered these recommendations.
\begin{enumerate}
\item {\em Revise the manuscript to enhance the clarity of the exposition. Ensure that all definitions, notations, and theorem statements are clearly presented. Incorporate the explanations and clarifications provided in the authors' responses into the main text. }

We modified the notations for the occupation kernel functions to improve clarity. See sections 2.1, 2.2, and 2.3., as pointed out by reviewer MBZQ. We included the paragraph regarding the well-posedness of the main optimization problem presented in Theorem 1 as was presented in the response to reviewer 8gNz.
\item {\em Provide detailed and consistent information about hyperparameter choices, including the relationship between key parameters and their impact on the algorithm's accuracy.}

We added section 4.3 Kernels, regularization, and hyperparameter tuning for the MOCK algorithm where we described the hyperparameter tuning that we used, specifically the gp\_minimize function from the Scikit-Optimize (skopt) package, version 0.10.2, which implements Gaussian Process optimization. In the same paragraph, we explained the respective roles of the regularization parameter $\lambda$ and the scale parameter $\sigma$.
\item {\em Include essential details about the computational methods used, such as how the integral quadrature is computed and specifics about the solver for the linear system.}
We included the details of the quadrature in equations (26) and (27). We also provided a reference for the linear solver, specifically numpy.linalg.solve from NumPy v1.26.4.
\item {\em Strengthen the experimental section by providing computational complexity comparisons (a major claim) with other frameworks to substantiate claims about efficiency.}
We added Table 3, providing the computational complexities of training for each method, including ours, and discussed this table in section 4.7. as follows: ``The MOCK algorithm with implicit and explicit kernel is linear in $d$, the dimension of the state space. Resnet and Lode are quadratic in $d$. In principle, SINDy-Poly, eDMD-Poly, eDMD-RFF, and eDMD-Deep are linear in $d$. However, $q$, the number of parameters in the learned model, which multiplies $d$, needs to increase with $d$ to obtain acceptable performances. This means the complexity is, in practice, more than linear in $d$. In the case of SINDy-Poly with quadratic basis functions, $q$ is $\mathcal{O}(d^2)$. On the contrary, in the case of MOCK explicit, $q$ is the number of features per dimension, and thus, it does not increase with $d$. In appendix F and G, we provide the rationale for the calculated complexities. In Section G, we also offer empirical runtimes for 3 datasets with respective state space dimensions 16, 32, and 128.

\end{enumerate}

We confirm that all authors have approved the manuscript for submission. We also confirm that the content of the manuscript has not been published or submitted for publication elsewhere.

Assigned Action Editor: ~Tongliang_Liu1

Submission Number: 3885