Transferable Audio Lottery Tickets: Gradient Accumulation for Extreme Sparsity

Hyunjae Kim, Juhan Nam, Kyung Myun Lee

Submission Type: Global Research Track / 글로벌 연구 소개 트랙

Keywords: Lottery Ticket Hypothesis, Gradient Accumulation, Audio Classification

Abstract: Large-scale neural networks have achieved impressive performance across diverse audio domains, but their growing size raises the need for lightweight alternatives. The Lottery Ticket Hypothesis (LTH) offers a compelling direction by revealing sparse subnetworks that can match full network performance, yet its potential across various audio subdomains remains underexplored. In this work, we examine this potential by evaluating sparse subnetworks on diverse audio classification tasks, spanning speech, music, and environmental sound. In particular, we propose a simple modification to the training process that incorporates momentum-like gradient accumulation during subnetwork search. We show that this strategy enables finding extremely sparse subnetworks with less than 1.0 % of the initial parameters remaining, while still retaining up to 90 % of dense model performance without layer collapse even under severe unstructured pruning. Furthermore, these subnetworks were effectively transferred across different audio subdomains while sustaining their sparsity-robust characteristics.

Email Sharing: We authorize sharing author emails with Program Chairs. / 저자 이메일 공유에 동의합니다.

Data Release: We understand and agree to the OpenReview metadata visibility policy. / OpenReview 메타데이터 공개 정책을 이해하고 동의합니다.

Submission Number: 1