Naive Bayes-based Context Extension for Large Language Models

Anonymous

Abstract: Large Language Models (LLMs) have shown promising in-context learning abilities. However, conventional in-context learning approaches are often impeded by length limitations of transformer architecture, which pose challenges when attempting to effectively integrate supervision from a substantial number of demonstration examples. To address this constraint, we introduce a novel framework called Naive Bayes-based Context Extension (NBCE) for large language models. NBCE enables existing LLMs to significantly expand their context size without the necessity for fine-tuning or reliance on specific model architectures, while maintaining linear efficiency. Across a wide range of tasks, experimental results demonstrate that NBCE substantially enhances performance, particularly as the number of demonstration examples increases, consistently outperforming alternative methods. The NBCE code will be made publicly accessible.

Paper Type: long

Research Area: Generation

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency, Theory

Languages Studied: English