A Fast Convoluted Story: Scaling Probabilistic Inference for Integer Arithmetics

Lennert De Smet, Pedro Zuidberg Dos Martires

Keywords: Probability theory, neurosymbolic AI, neuro-symbolic AI, neural-symbolic AI, integer arithmetic, linear integer arithmetic, integer programming, discrete random variables

TL;DR: We propose a differentiable tensorisation of linear arithmetics over integer-valued random variables that extends the horizon of exact probabilistic inference by exploiting the fast Fourier transform.

Abstract: As illustrated by the success of integer linear programming, linear integer arithmetics is a powerful tool for modelling combinatorial problems. Furthermore, the probabilistic extension of linear programming has been used to formulate problems in neurosymbolic AI. However, two key problems persist that prevent the adoption of neurosymbolic techniques beyond toy problems. First, probabilistic inference is inherently hard, #P-hard to be precise. Second, the discrete nature of integers renders the construction of meaningful gradients challenging, which is problematic for learning. In order to mitigate these issues, we formulate linear arithmetics over integer-valued random variables as tensor manipulations that can be implemented in a straightforward fashion using modern deep learning libraries. At the core of our formulation lies the observation that the addition of two integer-valued random variables can be performed by adapting the fast Fourier transform to probabilities in the log-domain. By relying on tensor operations we obtain a differentiable data structure, which unlocks, virtually for free, gradient-based learning. In our experimental validation we show that tensorising probabilistic integer linear arithmetics and leveraging the fast Fourier transform allows us to push the state of the art by several orders of magnitude in terms of inference and learning times.

Primary Area: Probabilistic methods (for example: variational inference, Gaussian processes)

Submission Number: 11332