Generalized Orders of Magnitude for Scalable, Parallel, High-Dynamic-Range Computation

TMLR Paper5259 Authors

Abstract: Many domains, from deep learning to finance, require compounding real numbers over long sequences, often leading to catastrophic numerical underflow or overflow. We introduce generalized orders of magnitude (GOOMs), a principled extension of traditional orders of magnitude that incorporates floating-point numbers as a special case, and which in practice enables stable computation over significantly larger dynamic ranges of real numbers than previously possible. We implement GOOMs, along with an efficient custom parallel prefix scan, to support native execution on parallel hardware such as GPUs. We demonstrate that our implementation of GOOMs outperforms traditional approaches with three representative experiments, all of which were previously considered impractical or impossible, and now become possible and practical: (1) compounding real matrix products {\em far} beyond standard floating-point limits; (2) estimating spectra of Lyapunov exponents in parallel, {\em orders of magnitude faster} than with previous methods, applying a novel selective-resetting method to prevent state colinearity; and (3) capturing long-range dependencies in deep recurrent neural networks with {\em non-diagonal recurrent states, computed in parallel via a prefix scan, without requiring any form of stabilization}. Our results show that our implementation of GOOMs, combined with efficient parallel scanning, offers a scalable and numerically robust alternative to conventional floating-point numbers for high-dynamic-range applications.

Submission Length: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~William_T_Redman1

Submission Number: 5259