Improving In-Context Learning with Small Language Model Ensembles

M. Mehdi Mojarradi, Lingyi Yang, Robert McCraith, Adam Mahdi

Keywords: Large language model, ensemble methods, in-context learning, few-shot prompting

TL;DR: Ensemble SuperICL offers cheap and effective domain-specialisation in LLMs by combining the predictions and confidence scores of several small language models, outperforming baselines on a medical labelling task and NLP benchmarks.

Abstract: Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.

Submission Number: 35