Lottery Ticket Structured Node Pruning for Tabular Datasets

Ryan Bluteau, Robin Gras, Mitchel Paulin, Zachary Innes

Keywords: Lottery Ticket Hypothesis, Tabular, Pruning

Abstract: In this paper we presented two pruning approaches on tabular neural networks based on the lottery ticket hypothesis that went beyond masking nodes by resizing the models accordingly. We showed top performing models in 6 of 8 datasets tested in terms of F1/RMSE. We also showed in 6 of 8 datasets a total reduction of over 85% of nodes and many over 98% reduced with minimal affect to accuracy. In one dataset the model reached a total size of one node per layer while still improving RMSE compared to the larger model used for pruning. We presented results for two approaches, iterative pruning using two styles, and oneshot pruning. Iterative pruning gradually reduces nodes in each layers based on norm pruning until we reach the smallest state, while oneshot will prune the model directly to the smallest state. We showed that the iterative approach will obtain the best result more consistently than oneshot.

One-sentence Summary: We prune tabular models to find lottery ticket weights that can generate a pruned network that outperforms the original in terms of training time and inference time while maintaining accuracy.