Fuzzy PyTorch: Rapid Numerical Variability Evaluation for Deep Learning Models

TMLR Paper6227 Authors

Abstract: We introduce Fuzzy PyTorch, a framework for rapid evaluation of numerical variability in deep learning (DL) models. As DL is increasingly applied to diverse tasks, understanding variability from floating-point arithmetic is essential to ensure robust and reliable performance. Tools assessing such variability must be scalable, efficient, and integrate seamlessly with existing frameworks while minimizing code modifications. Fuzzy PyTorch enables this by integrating Stochastic Arithmetic into PyTorch through Probabilistic Rounding with Instruction Set Management, a novel library interfacing with Verificarlo, a numerical analysis compiler. The library offers two modes: stochastic rounding, preserving exact floating-point operations, and up-down rounding, a faster alternative. Comparative evaluations show Fuzzy PyTorch maintains model performance while up-down rounding achieves runtime reductions of $5\times$ to $60\times$ versus Verrou, a state-of-the-art tool. We further demonstrate scalability by running models from 1 to 341 million parameters, confirming applicability across small and large DL architectures. Overall, Fuzzy PyTorch provides an efficient, scalable, and practical solution for assessing numerical variability in deep learning, enabling researchers and practitioners to quantify and manage floating-point uncertainty without compromising performance or computational efficiency.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Jasper_Snoek1

Submission Number: 6227