RNNS with gracefully degrading continuous attractors

Ábel Ságodi, Piotr A Sokol, Il Memming Park

Primary Area: applications to neuroscience & cognitive science

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: exploding gradient problem, gradient descent, bifurcation analysis

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We apply Persistence Theorem, which determines the persistence of invariant manifolds under perturbations, to derive RNN implementations of continuous attractors which do not bifurcate into divergent systems.

Abstract:

Attractor networks are essential theoretical components in recurrent networks for memory, learning, and computation.
However, the continuous attractors that are essential for continuous-valued memory suffer from structural instability---infinitesimal changes in the parameters can destroy the continuous attractor.
Moreover, the perturbed system's dynamics can exhibit divergent behavior with associated exploding gradients.
This poses a question about the utility of continuous attractors for systems that learn using gradient signals.
To address this issue, we use Fenichel's persistence theorem from dynamical systems theory to show that bounded attractors are stable in the sense that all perturbations maintain the stability.
This ensures that if there is a restorative learning signal, there will be no exploding gradients for any length of time for backpropagation.
In contrast, unbounded attractors may devolve into divergent systems under certain perturbations, leading to exploding gradients.
This insight also suggests that there can exist homeostatic mechanisms for certain implementations of continuous attractors that maintain the structure of the attractor sufficiently for the neural computation it is used in.
Finally, we verify in a simple continuous attractor that all perturbations preserve the invariant manifold and demonstrate the principle numerically in ring attractor systems.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3487