DParT: Transferring knowledge between languages changing a few weights

Anonymous

Abstract: Large Language Models (LLMs) have significantly impacted both research and business domains, automating tasks previously unattainable by artificial intelligence. However, the primary focus on English and European languages presents a barrier in adapting and applying LLMs to other languages due to the challenges involved in data collection, pre-processing, and model training. To overcome this issue, we propose a Double Partial Tuning DParT strategy. It involves modifying the structure of the training data in the first stage and employing low rank adapters LoRA in the second stage, leading to knowledge transfer between languages and low computational efforts in terms of trainable parameters and data quantity. Tests on Arabic and Russian languages demonstrate the superiority of DParT over other training methods, potentially expanding the application of LLMs in various languages and further revolutionizing research and business fields.We selected Arabic and Russian languages, as they originate from distinct language families and utilize two different non-Latin scripts, in order to demonstrate the effectiveness of the proposed approach. Code and datasets will be made publicly available.

Paper Type: short

Research Area: Multilinguality and Language Diversity

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English, Arabic, Russian