Mind the Gap: Removing the Discretization Gap in Differentiable Logic Gate Networks

Shakir Yousefi, Andreas Plesner, Till Aczel, Roger Wattenhofer

Keywords: Logic gate networks, Gumbel noise, Faster training, Smoother minima

Abstract:

Modern neural networks exhibit state-of-the-art performance on many benchmarks, but their high computational requirements and energy usage have researchers exploring more efficient solutions for real-world deployment.
Logic gate networks (LGNs) learns a large network of logic gates for efficient image classification. However, learning a network that can solve a simple problem like CIFAR-10 can take days to weeks to train. Even then, almost half of the network remains unused, causing a \emph{discretization gap}. This discretization gap hinders real-world deployment of LGNs, as the performance drop between training and inference negatively impacts accuracy.
We inject Gumbel noise with a straight-through estimator during training to significantly speed up training, improve neuron utilization, and decrease the discretization gap.
We theoretically show that this results from implicit Hessian regularization, which improves the convergence properties of LGNs. We train networks $4.5 \times$ faster in wall-clock time, reduce the discretization gap by 98\%, and reduce the number of unused gates by 100\%.

Submission Number: 18