Resource Efficient Self-Supervised Learning for Speech Embeddings

Abhinav Mehrotra; Alberto Gil Couto Pimentel Ramos; Shucong Zhang; Salah Zaiem; Titouan Parcollet; Sourav Bhattacharya

Resource Efficient Self-Supervised Learning for Speech Embeddings

Abhinav Mehrotra, Alberto Gil Couto Pimentel Ramos, Shucong Zhang, Salah Zaiem, Titouan Parcollet, Sourav Bhattacharya

23 Sept 2023 (modified: 11 Feb 2024)Submitted to ICLR 2024EveryoneRevisionsBibTeX

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: SSL, Speech Processing

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Representation learning from sequential data using self-supervised learning (SSL) has proven to be a powerful technique and improved state-of-the-art (SOTA) results when fine-tuned for various downstream tasks. So far the success of SSL frameworks, e.g., Wav2Vec2 and Data2Vec2, for learning audio embeddings is primarily carried out by masking intermediate features and then solving a contrastive or non-contrastive task in an end-to-end manner, respectively. In comparison to contrastive SSL methods such as Wav2Vec2, non-contrastive techniques such as Data2Vec2 have emerged having better model quality and training time. However, Data2Vec2 is still quite demanding in terms of resources, namely infrastructure (more and better GPUs), which remains a significant barrier to further improving models for downstream tasks. In this work we show that non-contrastive learning, such as an extension of the Barlow--Twins methodology, when applied to a range of downstream tasks simultaneously decreases training time and resource requirements while maintaining or improving SOTA results in key benchmark datasets. From a computional point of view, our approach decreases Data2Vec2 training time by $2\times$ and permits effective training with smaller sequence lengths and batch sizes without requiring gradient accumulation reducing GPU VRAM requirements from NVIDIA A100's to V100's.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7951

Loading