ConvX: A Lightweight Converter to Bridge Indexed Dense Representations and Large Language Models for Retrieval-Augmented Generation

ACL ARR 2026 January Submission9285 Authors

Keywords: Retrieval-Augmented Generation, Context Compression, Compression-based RAG, Representation Learning

Abstract: Retrieval-Augmented Generation (RAG) has significantly advanced open-domain question answering and dialogue systems by incorporating external knowledge into large language models. Despite its effectiveness, existing RAG pipelines suffer from critical efficiency limitations. In particular, modern transformer-based generators exhibit quadratic or higher computational complexity with respect to input sequence length and hidden dimensionality, leading to substantial inference latency as model scales and contextual inputs increase. This issue is exacerbated in RAG settings, where retrieved contexts substantially expand the input prompt. To alleviate this challenge, we propose an effective compression-based RAG framework, ConvX, that directly leverages indexed dense representations produced by a retriever, entirely substituting to long text contexts. Our approach expands a single dense representation into a fixed number of memory slots using a lightweight converter to provide rich lexical information. This design enables efficient knowledge integration while significantly reducing input length and computational overhead. Empirical evaluations demonstrate that the proposed model achieves outstanding performances compared to existing ad-hoc context compression methods in RAG setting, while offering substantially improved inference efficiency.

Paper Type: Long

Research Area: Retrieval-Augmented Language Models

Research Area Keywords: Machine Learning for NLP

Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 9285