Ensemble Systems Representation for Function Learning over Manifolds

Wei Zhang, Vignesh Narayanan, Jr-Shin Li

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Function learning, dynamical systems, control theory

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We provide rigorous theoretical underpinnings of ensmeble system representations for function learning problems.

Abstract: Function learning concerns with the search for functional relationships among datasets. It coincides with the formulations of various learning problems, particularly supervised learning problems, and serves as the prototype for many learning models, e.g., neural networks and kernel machines. In this paper, we propose a novel framework to tackle function learning tasks from the perspective of ensemble systems theory. Our central idea is to generate function learning algorithms by using flows of continuous-time ensemble systems defined on infinite-dimensional Riemannian manifolds. This immediately gives rise to the notion of natural gradient flow that enables the generated algorithms to tackle function learning tasks over manifolds. Moreover, we rigorously investigate the relationship between the convergence of the generated algorithms and the dynamics of the ensemble systems with and without an external forcing or control input. We show that by turning the penalty strengths into control inputs, the algorithms are able to converge to any function over the manifold, regardless of the initial guesses, providing {\em ensemble controllability} of the systems. In addition to the theoretical investigation, concrete examples are also provided to demonstrate the high efficiency and excellent generalizability of these "continuous-time" algorithms compared with classical "discrete-time" algorithms.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8662