LitePruner: A Lightweight Realtime Token Pruner before Large Language Models

ICLR 2026 Conference Submission16269 Authors

Keywords: Efficient NLP, Multilingual NLP, Realtime Method

TL;DR: In this paper, we study the realtime computing problem in the common in-context learning paradigm, attempting to reduce the total number of tokens per query but maintain decent performance in multilingual settings.

Abstract: Tokenization is one of the core steps of the language model pipeline. However, the tokenizer yields more tokens for the same context in non-English languages, especially in low-resource languages due to the shared multilingual settings, which results in unexpected fairness problems in terms of token fees, response latency, and long context processing. In this paper, we study the real-time computing problem, attempting to reduce the total number of tokens per query but maintain decent performance in multilingual settings. We present a simple, training-free, CPU-based pruner model to reuse pre-trained weights from the first attention layer of small models to rank token importance, only delivering important tokens to the target larger models. This method is motivated by the fact that early layers in both small and large models latch onto the similar shallow local signals due to similar tokenization algorithms (e.g., BPE) producing identical local signals. Massive in-context learning experiments on MGSM, Global-MMLU-Lite and ARC and RAG-based experiments on PubMedQA and MEMERAG show that our method can preserve decent performance for languages while reducing up to $30\%$ of the total number of tokens in both in-family and across-family model settings, where the pruner model and the target large model are in or not in the same model family. Our method is compatible with commercial LLM APIs and CPU-based, contributing to real-life applications.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 16269