Parameter-Efficient Multilingual Summarization: An Empirical Study

Anonymous

TL;DR: We study Parameter-Efficient Fine-tuning, specifically LoRA, for Multilingual Summarization across diverse scenarios based on data availability.

Abstract: Although the emergence of pre-trained Large Language Models has significantly accelerated recent progress in NLP, their ever-increasing size poses significant challenges for conventional fine-tuning, especially in memory-intensive tasks. We investigate the potential of Parameter-Efficient Fine-Tuning, focusing on Low-Rank Adaptation (LoRA), in the domain of multilingual summarization, a task that is both challenging (due to typically long inputs), and relatively unexplored. We conduct an extensive study across different data availability scenarios, including high- and low-data settings, and cross-lingual transfer, leveraging models of different sizes. Our findings reveal that LoRA is competitive with full fine-tuning when trained with high quantities of data, and excels in low-data scenarios and cross-lingual transfer. We also study different strategies for few-shot cross-lingual transfer, finding that continued LoRA tuning outperforms full fine-tuning and the dynamic composition of language-specific LoRA modules.

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: The two datasets studied are multilingual hence consists of many languages (45 for XLSum, 5 for XWikis). The languages we evaluated with a focus include: Azerbaijani (AZ), Bengali (BN), Japanese (JA), Kirundi (RN), Korean (KO), Nepali (NE), Scottish Gaelic (GD), Somali (SO), Thai (TH), Yoruba (YO), English (EN), German (DE), French (FR), Czech (CS), Chinese (ZH).