Parallelizing non-linear sequential models over the sequence length

Yi Heng Lim, Qi Zhu, Joshua Selfridge, Muhammad Firmansyah Kasim

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: parallel algorithm, recurrent neural networks, neural ordinary differential equations, sequential models

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We present a parallel algorithm to evaluate and training sequential models (e.g., RNN & NeuralODE) despite their inherent sequential nature

Abstract:

Sequential models, such as Recurrent Neural Networks and Neural Ordinary Differential Equations, have long suffered from slow training due to their inherent sequential nature.
For many years this bottleneck has persisted, as many thought sequential models could not be parallelized.
We challenge this long-held belief with our parallel algorithm that accelerates GPU evaluation of sequential models by up to 3 orders of magnitude faster without compromising output accuracy.
The algorithm does not need any special structure in the sequential models' architecture, making it applicable to a wide range of architectures.
Using our method, training sequential models can be more than 10 times faster than the common sequential method without any meaningful difference in the training results.
Leveraging this accelerated training, we discovered the efficacy of the Gated Recurrent Unit in a long time series classification problem with 17k time samples.
By overcoming the training bottleneck, our work serves as the first step to unlock the potential of non-linear sequential models for long sequence problems.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: general machine learning (i.e., none of the above)

Submission Number: 1788