JIONE: an Approach for Merging Large Language Models via Teacher–Student Prediction Refinement

ICLR 2026 Conference Submission25005 Authors

Keywords: Large Language Model, Large Languag Model Merging, Teacher-Student Approach, Prompt Engineering

TL;DR: This paper proposes an unconstrained model merging approach that accommodates both homogeneous and heterogeneous multiple LLMs. This is a teacher-student approach in which each query is processed by the student first and refined by the teacher.

Abstract: Large Language Models (LLMs) demonstrated remarkable capabilities across reasoning, problem-solving, and natural language understanding tasks such as Text classification, Multiple-choice question answering. However, relying on a single LLM faces limitations, as models are typically specialized to particular domains or objectives. For example, code-oriented models (e.g., Phi-3-mini) excel on programming benchmarks such as Mostly Basic Programming Problems (MBPP), conversational models (e.g., Qwen1.5) perform better on factual Q\&A tasks like TruthfulQA yet underperform in mathematical reasoning benchmarks such as Grade School Math 8K (GSM8K). This specialization highlights the need for \textbf{merging multiple LLMs} to leverage their complementary strengths. Therefore, a promising direction is to merge multiple LLMs, leveraging their complementary strengths while mitigating individual weaknesses. Existing approaches to model merging, such as SLERP and Task Arithmetic, primarily assume that the models share the same architecture. When different architectures are involved (e.g. FuseChat, ProDistill), prior work shows that existing approaches rely on training-heavy steps that incur computational and data costs. Consequently, an efficient and general method for merging heterogeneous LLMs remains an open challenge. In this paper, we introduce \textbf{JIONE}, a teacher-student prediction refinement approach designed to merge LLMs-agnostic architecture, without additional training/fine-tuning. It operates directly at the output level, where a teacher-student mechanism refines predictions and resolves inconsistencies before producing a merged answer. JIONE was evaluated on four benchmark datasets: TruthfulQA, GSM8K, MBPP, and SST-2 using Phi-3-mini-128k-instruct, Phi-3-mini-4k-instruct, Qwen1.5-1.8B-Chat and Distilbert-base-uncased-finetuned-sst-2-english models. Evaluation across Accuracy, ROUGE-N, and Exact Match Accuracy (EMA) shows that JIONE consistently outperforms SLERP and Task Arithmetic, achieving up to \textbf{+5.99\% improvement} for models of the same architecture and up to \textbf{+3.2\% improvement} when merging models of different architectures. These results demonstrate that JIONE enables effective and scalable merging of diverse LLMs, unlocking a path toward more general and versatile model integration. Experiments show that the teacher-student refinement process induces additional computational costs compared to baselines. However, the observed gain in performance and generalization justify this cost, particularly in applications such as medical diagnostics where prediction quality and robustness is critical. The code used in this work is released at \url{https://gitlab.com/tsotsa/jione}.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 25005