"Call My Big Sibling (CMBS)" – A Confidence-Based Strategy Leveraging Instance Selection to Combine Small and Large Language Models for Cost-Effective Text Classification

ACL ARR 2025 February Submission4285 Authors

Abstract: Transformers have achieved cutting-edge results, with Large Language Models (LLMs) considered SOTA in many NLP tasks. However, the literature has not yet fully demonstrated that LLMs are always superior to first-generation Transformers (a.k.a. Small Language Models (SLMs)) in all NLP tasks and scenarios. This study compares four SLMs (BERT, RoBERTa, Qwen, BART) with four open LLMs (LLaMA 3.1, Mistral, Falcon, DeepSeek) across 9 sentiment analysis and 4 topic classification datasets, totaling over 1000 results. Findings indicate that open LLMs can moderately outperform or tie with SLMs in all tested datasets, though only when fine-tuned, at a very high computational cost. To address the cost-effectiveness trade-off, we propose "Call My Big Sibling" (CMBS), a novel confidence-based framework that efficiently integrates calibrated SLMs with open LLMs using advanced instance selection techniques. CMBS assigns high-confidence predictions to the cheaper SLM, while low-confidence cases are directed to LLMs in zero-shot, in-context, or partially-tuned (leveraging state-of-the-art instance selection approaches) modes, optimizing cost-effectiveness. Experiments show that CMBS significantly outperforms SLMs and achieves LLM-level effectiveness at a fraction of the cost, offering a practical alternative for cost-sensitive NLP applications.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: Machine Learning for NLP, Large Language Model

Contribution Types: Model analysis & interpretability, Data analysis

Languages Studied: English

Submission Number: 4285