The Fragility of Polarity: A Perturbative Analysis of the sign Hypothesis in Sparse Networks

Arindam Biswas

Track: Extended Abstract Track

Keywords: polarity, perturbation analysis, lottery ticket hypothesis

TL;DR: We do a perturbative analysis on polarity experiments for the robustness of LTH

Abstract: The "sign Hypothesis" posits that preserving the initial signs of weights, rather than their exact magnitudes, is critical for successfully training sparse subnetworks found via the Lottery Ticket Hypothesis. While well-supported, the robustness of this sign structure across different task complexities remains unexplored. This paper introduces a framework of systematic, perturbative reinitialization strategies to probe the breaking points of sign-based training. We apply controlled perturbations—including random sign-flipping (Epsilon), targeted randomization (Delta), and threshold shifting (Phi)—to various convolutional architectures across MNIST, CIFAR-10, and CIFAR-100. Our findings reveal a clear "complexity gradient": on MNIST, the sign structure is extremely robust, with networks maintaining >97\% accuracy even with 90\% of signs flipped. On the more complex CIFAR-10, the classic lottery ticket rewinding strategy (`mask1`) becomes fragile, while strategies incorporating learned polarity (`gradual`, `mask0`) prove far more resilient. On CIFAR-100, this gap widens, with the choice of reinitialization strategy becoming a critical performance differentiator. A consistent finding across all datasets is a resilience to sign randomization of the lowest-magnitude weights, suggesting that magnitude pruning is effective precisely because it preserves the weights whose polarity is most critical for guiding optimization.

Submission Number: 149