TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination

ACL ARR 2026 January Submission4783 Authors

Keywords: Improve performance, Task-aware, Inference-time, Layer

Abstract:

Large Language Models (LLMs) are typically deployed using a fixed architecture, despite growing evidence that not all layers contribute equally to every downstream task. In this work, we introduce TALE (Task-Aware Layer Elimination), an inference-time method that improves task performance by selectively removing layers that are irrelevant or detrimental for a given task. TALE optimizes task-specific validation performance, yielding a task-adapted architecture without retraining or modifying model weights.
Across 9 tasks and 5 model families, under both zero-shot and few-shot settings, we show that TALE consistently matches or surpasses baseline performance while simultaneously reducing computational cost, outperforming general and layer-wise pruning approaches such as SLEB. Beyond inference-time gains, TALE synergizes with fine-tuning and few-shot learning, where task-adapted architectures lead to additional performance improvements. Computing TALE for a new task requires modest resources (1–2 GPU hours on an A100), making it a practical and deployable solution for task-specialized LLM inference.

Paper Type: Long

Research Area: LLM Efficiency

Research Area Keywords: LLM Efficiency, efficient models, analysis, model editing, NLP in resource-constrained settings

Contribution Types: Model analysis & interpretability, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 4783