Training-free LLM Merging for Multilingual Multi-task Learning

ACL ARR 2024 December Submission441 Authors

Abstract:

Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing (NLP) tasks. The release of open-source LLMs like LLaMA and Qwen has triggered the development of numerous fine-tuned models tailored for various tasks and languages. In this paper, we explore an important question: is it possible to combine these specialized models to create a unified model with multi-task and multi-lingual capabilities. We introduces \textbf{H}ierarchical \textbf{I}terative \textbf{Merging} (Hi-Merging), a training-free method for unifying multiple specialized LLMs into a single model. Extensive experiments on English and Chinese datasets, covering multiple-choice and question-answering tasks, validate Hi-Merging across three paradigms: multilingual merging, multi-task merging, and multilingual multi-task merging. The results demonstrate that Hi-Merging consistently outperforms existing merging techniques and surpasses the performance of models fine-tuned on combined datasets in most scenarios.
Code is available at this anonymous link: https://anonymous.4open.science/r/hi-merging.

Paper Type: Long

Research Area: Multilingualism and Cross-Lingual NLP

Research Area Keywords: model merging, large language models, multilingualism, multi-task learning

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English, Chinese

Submission Number: 441