Language Model Knowledge Distillation for Efficient Question Answering in Spanish

Adrián Bazaga, Pietro Lio, Gos Micklem

Keywords: Knowledge Distillation, Model Compression, Efficient Machine Learning, Spanish Language Models, Language Models, NLP, TinyRoBERTa

TL;DR: Efficient distilled RoBERTa model for question answering in Spanish

Abstract: Recent advances in the development of pre-trained Spanish language models has led to significant progress in many Natural Language Processing (NLP) tasks, such as question answering. However, the lack of efficient models imposes a barrier for the adoption of such models in resource-constrained environments. Therefore, smaller distilled models for the Spanish language could be proven to be highly scalable and facilitate their further adoption on a variety of tasks and scenarios. In this work, we take one step in this direction by developing SpanishTinyRoBERTa, a compressed language model based on RoBERTa for efficient question answering in Spanish. To achieve this, we employ knowledge distillation from a large model onto a lighter model that allows for a wider implementation, even in areas with limited computational resources, whilst attaining negligible performance sacrifice. Our experiments show that the dense distilled model can still preserve the performance of its larger counterpart, while significantly increasing inference speedup. This work serves as a starting point for further research and investigation of model compression efforts for Spanish language models across various NLP tasks.

Submission Number: 65